Method and system for learning intent classification model using intent description

Abstract

The system for learning an intention classification model using an intention description according to the present invention can generate a data set including the intention description of the intention classification model using a language model, use at least a portion of the data set as an input cue for the intention classification model, and perform intention classification by the intention classification model. Using at least one part of the data set as an input prompt word of an intention classification model, and determining the performance of the intention classification model, the dataset includes first data including independent intent descriptions created in a manner independent of the other intent descriptions and second data including dependent intent descriptions created in a manner dependent to the other intent descriptions.

Description

TECHNICAL FIELD

[0001] The present application relates to a learning method and system of an intent classification model, and more particularly, to a learning method and system of an intent classification model using intent specification, which learns an intent classification model based on input data including intent specification and is capable of improving accuracy of intent classification. BACKGROUND

[0002] Recently, artificial intelligence (AI) technology is showing a cutting-edge development situation, attracting the attention of the entire society. Artificial intelligence refers to a phenomenon in which a computer performs human-specific intelligence at a high level of ability, such as "a computer brain that performs the field of human intelligence," "an engineering technology and science that creates an intelligent machine," "a series of algorithm systems designed to think, perceive, and act like a human being," and the like.

[0003] It is introduced that artificial intelligence, when used in combination with technologies such as augmented reality, the Internet of Things, edge computing, and digital twinning, can provide a highly integrated intelligent space, and is emphasized as a core new technology that will lead the fourth industrial revolution era. And, artificial intelligence is not only capable of solving standardization problems, but also is attracting attention as the next-generation growth engine that can drive the evolution of the industrial ecosystem, and is actively applied to knowledge service industries such as IT, medical care, agriculture, energy, automobiles, robots, distribution, finance, law, education, real estate, advertising, and communication, in addition to IT, medical care, agriculture, energy, automobiles, and robots. That is, artificial intelligence is not only committed to improving convenience or level in real life, but also is combined with all existing systems including the entire culture and art of our society, and is preparing to usher in a new era.

[0004] Recently, as a task-oriented conversational system that has a conversation with a user by using voice or text input of the user and performs a specific task, a chatbot has been introduced and applied in various fields. The chatbot uses a conversational artificial intelligence technology such as natural language processing (NLP) to understand a user's question (Query) and automatically display a response thereto.

[0005] On the other hand, in such a task-oriented conversational system, in order to accurately understand a user's question and provide appropriate services accordingly, an intent classification technology that accurately grasps a user's intent from a user's question can be essential. Research is being conducted on such an intent classification that accurately grasps a user's intent from a user's Query for mapping a user's expression to one of a pre-defined set of intents.

[0006] As related art, there is disclosed Effectiveness of pre-training for few-shot intent classification. (https: / / aclanthology.org / 2021.findings-emnlp.96 / ) (Zhang et al., Findings 2021). SUMMARY

[0007] TECHNICAL PROBLEM TO BE SOLVED

[0008] An embodiment of the present disclosure is to provide a learning method and system of an intent classification model capable of improving accuracy of intent classification and using intent descriptions, which learns an intent classification model of a large-scale language model including intent descriptions in input data, thereby mapping a user expression to one of a predefined set of intents.

[0009] An embodiment of the present disclosure is to provide a method of generating and providing a high-accuracy answer to a user question through phased processing for advanced reasoning and inference functions, and a system thereof.

[0010] MEANS FOR SOLVING TECHNICAL PROBLEM

[0011] The system of an embodiment of the present disclosure can include at least one or more processors, and at least one or more memories storing instructions or information executed in the at least one or more processors, wherein the actions performed by the instructions or information executed in the at least one or more processors include an action of receiving a user question including a plurality of user intents through a user input / output unit, an action of outputting a plurality of intent descriptions corresponding to the plurality of user intents from a language model stored in the memory, an action of providing the plurality of intent descriptions stored in the memory to the user through the user input / output unit and receiving a user input, and an action of using at least one intent description determined through the user input as an input prompt of an intent classification model executed by the at least one or more processors, and performing intent classification by the intent classification model, the plurality of intent descriptions including at least one or more of an independent intent description created in a manner independent of other intent descriptions, a dependent intent description created in a manner subordinate to other intent descriptions, or a cleaned intent description input through the user input unit after being cleaned by the user based on the independent intent description or the dependent intent description.

[0012] Also, the instruction prompt word input to the language model to output the plurality of intent descriptions from the language model can include an independent prompt word that acquires first data including a plurality of intent descriptions for one intent, a subordinate prompt word that acquires second data including a plurality of intent descriptions for a plurality of intents and analyzes a context of the plurality of intents and each of the plurality of intent descriptions, and a cleaned prompt word that acquires third data based on the independent intent description or the subordinate intent description by user input.

[0013] Also, the input prompt word can be composed of an instruction and a user query section, an intent option section including an intent name and a plurality of intent descriptions for the intent name.

[0014] Also, further including a ranker that adjusts the number of the intent options, wherein the ranker can calculate a similarity between the user query and the intent description.

[0015] Also, the intent classification model can re-learn using at least one of the independent intent description or the subordinate intent description or the cleaned intent description.

[0016] Also, at least a part of the first data can be provided as an input prompt word in determining the performance of the intent classification model, or at least a part of the second data can be provided as an input prompt word in determining the performance of the intent classification model.

[0017] The method performed by the at least one processor of an embodiment of the present application can include the steps of receiving a user query including a plurality of user intents through a user input / output unit, outputting a plurality of intent descriptions corresponding to the plurality of user intents using a language model stored in a memory, providing the plurality of intent descriptions stored in the memory to the user through the user input / output unit and receiving a user input through the user input unit, and using at least one intent description determined through the user input as an input prompt word of an intent classification model performed by the at least one processor, and classifying an intent by the intent classification model, wherein the plurality of intent descriptions include at least one of an independent intent description created independently of other intent descriptions, a subordinate intent description created in a manner subordinate to other intent descriptions, or a cleaned intent description input through the user input unit based on the independent intent description or the subordinate intent description and cleaned by the user.

[0018] Furthermore, in the step of outputting the aforementioned multiple intent descriptions, instruction prompts are constructed by inputting them into the language model to generate a dataset including the aforementioned multiple intent descriptions from the language model. The instruction prompts include: independent prompts, which acquire first data including multiple intent descriptions for a single intent; dependent prompts, which acquire second data including multiple intent descriptions for multiple intents and analyze the context of each of the multiple intents and the multiple intent descriptions; and cleaned prompts, which acquire third data based on the aforementioned independent intent descriptions or the aforementioned dependent intent descriptions, obtained from user input.

[0019] Furthermore, in the step of performing intent classification by the intent classification model described above, the input prompt can consist of an instruction and user query section, an intent option section including the intent name and multiple intent descriptions for the intent name.

[0020] Furthermore, the sorter that adjusts the number of the aforementioned intent options can calculate the similarity between the aforementioned user query and the aforementioned intent description, and arrange the aforementioned multiple intent descriptions in descending order based on the aforementioned similarity.

[0021] Furthermore, in the step of classifying intent by the intent classification model described above, the intent classification model can be relearned using at least a portion of the independent intent description, the subordinate intent description, or the cleaned intent description.

[0022] Furthermore, it may include: a step of determining the performance of the intent classification model, wherein, in determining the performance of the intent classification model, at least a portion of the first data is provided as input prompt words, or, in determining the performance of the intent classification model, at least a portion of the second data is provided as input prompt words.

[0023] A system according to an embodiment of the present invention may include at least one processor; and at least one memory storing instructions or information executed by the at least one processor, wherein the actions performed by the instructions or information executed by the at least one processor include: inputting a user intent from question data obtained by the user through prompts from a user input / output unit into a language model stored in the memory, thereby generating multiple intent descriptions for the user intent and specifying the question intent; retrieving information for generating an answer corresponding to the question intent; detecting response data corresponding to the question intent from the retrieved information; and generating an answer to the question intent based on the detected response data.

[0024] Also, the action of specifying the question intent can be performed by outputting the independent intent specification and the subordinate intent specification from the user input / output unit, selecting at least one of the independent intent specification or the subordinate intent specification from the user input / output unit, or inputting a cleaned intent specification cleaned by the user based on the independent intent specification or the subordinate intent specification from the user input / output unit, and classifying the intent based on the selected or input intent specification by the intent classification model.

[0025] Also, the action of specifying the question intent can be performed by outputting the independent intent specification and the subordinate intent specification from the user input / output unit, selecting at least one of the independent intent specification or the subordinate intent specification from the user input / output unit, or inputting a cleaned intent specification cleaned by the user based on the independent intent specification or the subordinate intent specification from the user input / output unit, and classifying the intent based on the selected or input intent specification by the intent classification model.

[0026] Also, the action of retrieving the information can be performed by retrieving information associated with the user intent and the plurality of intent specifications from a database or the Internet.

[0027] Also, the action of detecting the response data can include an action of determining an evidence paragraph associated with the question data from the intent specification, an action of acquiring basis data for the question data based on the determined evidence paragraph, and an action of generating response data for the question data based on the acquired basis data.

[0028] Also, the basis data can be a file or a website collected from a database or the Internet.

[0029] The method performed by the at least one processor of the embodiment of the present application can include the steps of inputting a user intent of question data acquired from a prompt word of a user input / output unit of an interactive system into a language model stored in a memory, generating a plurality of intent specifications for the user intent, and specifying a question intent; retrieving information for generating an answer corresponding to the question intent from a database or the Internet; detecting response data corresponding to the question intent from the retrieved information; and generating an answer for the question intent based on the detected response data.

[0030] Also, in the step of specifying the question intent, the language model can generate an independent intent specification created independently of other intent specifications or a plurality of intent names and a subordinate intent specification created subordinate to other intent specifications.

[0031] Also, the step of specifying the above question intent can include a step of outputting the above independent intent specification and the above dependent intent specification through the above user input / output unit, a step of selecting at least one of the independent intent specification or the dependent intent specification through the above user input / output unit or inputting a cleaned intent specification cleaned by the user based on the independent intent specification or the dependent intent specification through the above user input / output unit, and a step of classifying the intent by the intent classification model according to the selected or inputted intent specification.

[0032] Also, the step of detecting the above response data can include a step of determining an evidence paragraph associated with the above question data in the above intent specification, a step of acquiring basis data for the above question data based on the determined above evidence paragraph, and a step of generating response data for the above question data based on the acquired above basis data, the above basis data being a file and a website collected through a database or the Internet.

[0033] Inventive Effects

[0034] According to an embodiment of the present invention, an intent classification model learning method and system using intent specification, which can provide mapping of a user's expression to one of a predefined set of intents and can improve the accuracy of intent classification by learning of the intent classification model including intent specification in input data.

[0035] According to an embodiment of the present invention, an intent of a user's question and an intent specification for the intent are generated, and based on high-level reasoning (Chain-of-thought reasoning) and inference functions implemented through a process similar to human cognitive reasoning, based on the intent specification, an effect of providing a faithful, fact-based, and clearly based answer to a deep question to the user can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 FIG. 1 is a diagram of an electronic device according to an embodiment of the present invention.

[0037] Figure 2 FIG. 2 is a diagram illustrating an example of a command prompt word in an input language model.

[0038] Figure 3 FIG. 3 is a diagram illustrating an example of an input prompt word in an input intent classification model.

[0039] Figure 4 FIG. 4 is a diagram of an intent classification model learning method using intent specification according to an embodiment of the present invention.

[0040] Figure 5 FIG. 5 is a diagram illustrating accuracy performance of an intent classification model according to the number of intent options.

[0041] Figure 6 is a learning method using intention according to other embodiments of the present invention.

[0042] Figure 7 is a method of generating an answer to a question according to other embodiments of the present invention. DETAILED DESCRIPTION

[0043] In order to clarify the technical idea of the present disclosure, a plurality of embodiments of the present invention are explained in detail below with reference to the accompanying drawings. In explaining the present disclosure, when it is judged that a specific explanation for a related well-known function or configuration element is likely to obscure the gist of the present disclosure, the detailed explanation thereof is omitted. In the drawings, the same reference numerals and symbols are attached to configuration elements having substantially the same function configuration even if they are shown in different drawings. For the convenience of explanation, the device and the method are explained at the same time if necessary. The actions of the present disclosure are not necessarily performed in the order as recorded, and can be performed in parallel, selectively, or individually.

[0044] The terms used in the plurality of embodiments of the present disclosure are selected as the general terms that are currently used as widely as possible in consideration of the functions in the present disclosure, but are different according to the intention or precedents of the skilled in the art, the appearance of new technology, etc. Also, there are terms arbitrarily selected by the applicant in a specific case, and in this case, the meaning thereof is recorded in detail in the explanation part of the corresponding embodiment. Therefore, the terms used in the present specification should be defined based on the meaning inherent to the terms and the overall content of the present disclosure, and should not be defined as simple names of the terms.

[0045] Throughout the present disclosure, the expression of the singular includes the plural unless it is explicitly stated that the context clearly distinguishes the meaning. The terms such as "include" or "have" are used to specify the presence of features, numbers, steps, actions, configuration elements, components, or combinations thereof, and should not be understood as excluding the possibility of adding one or more other features, numbers, steps, actions, configuration elements, components, or combinations thereof in advance. That is, throughout the present disclosure, when it is recorded that a certain part "includes" certain configuration elements, it means that other configuration elements can also be included unless it is specifically recorded to the contrary, rather than excluding other configuration elements.

[0046] The expression "at least one" or the like modifies the entire list of configuration elements and does not individually modify the configuration elements in the list. For example, "at least one of A, B, and C" and "at least one of A, B, or C" mean only A, only B, only C, all of A and B, all of B and C, all of A and C, all of A and B and C, or a combination thereof.

[0047] Also, the terms "…unit", "…module", and the like, as recited in the present disclosure, mean a unit that processes at least one function or action, which can be realized by hardware or software or a combination of hardware and software.

[0048] Throughout the present disclosure, when recited that one part is "connected" with other parts, it includes not only the case of "directly connected" but also the case of "electrically connected" with other elements interposed therebetween. Also, when recited that one part "includes" one constituent element, it means that other constituent elements can also be included, not excluding other constituent elements, unless specifically recited to the contrary.

[0049] "Configured to", as used throughout the present disclosure, can be replaced with, for example, "Suitable for", "having the capacity to", "designed to", "adapted to", "made to", or "capable of", depending on the situation. The term "configured to" does not necessarily mean a case of "Specifically designed to" in hardware. However, in any case, the expression of "configured to" can mean that the system "is capable of" performing with other devices or parts. For example, the sentence "a processor configured to perform processes of A, B, and C" can mean a customized processor (e.g., an embedded processor) for performing corresponding actions or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing the corresponding actions by executing one or more software programs stored in a memory.

[0050] Artificial intelligence (AI) is a field of computer engineering and information technology that studies how to enable a computer to implement thinking, learning, self-development, and the like, which can be achieved by human intelligence, aiming to enable a computer to simulate human intelligent behavior. Also, artificial intelligence does not exist independently, but is directly or indirectly associated with other fields of computer science. In particular, currently, in many fields of information technology, attempts are being actively made to introduce artificial intelligence elements to solve problems in the field.

[0051] Machine learning is a field of artificial intelligence, and is a field of research that gives a computer the ability to learn without explicit programming. Specifically, machine learning can be said to be a technology that researches and constructs a system that learns based on experience data, performs prediction, and improves its own performance, and an algorithm for the system. The algorithm of machine learning adopts a way of constructing a specific model in order to derive a prediction or a decision based on input data, rather than strictly executing a static program instruction set. The term "machine learning" can be used interchangeably with "mechanical learning".

[0052] In the field of machine learning, numerous machine learning algorithms have been developed around the issue of how to classify data. Representative ones are decision tree or Bayesian network, support vector machine (SVM), and artificial neural network (ANN). The decision tree is an analysis method that charts a decision rule as a tree structure, thereby performing classification and prediction. The Bayesian network is a model that expresses the probabilistic relationship (conditional independence) between a plurality of variables in a chart structure. The Bayesian network is suitable for data mining by unsupervised learning. The support vector machine is a supervised learning model for pattern recognition and data analysis, and is mainly used for classification and regression analysis. The artificial neural network is an information processing system that connects a plurality of neurons called nodes or processing elements in a layer structure, modeling the working principle of biological neurons and the connection relationship between neurons.

[0053] The artificial neural network is a model for machine learning, and is a statistical learning algorithm that draws inspiration from biological neural networks (especially the brain in the central nervous system of animals) in the fields of machine learning and cognitive science. Specifically, the artificial neural network can represent the model as a whole that artificial neurons (nodes) that form a network by the combination of synapses change the synaptic connection strength by learning and thereby acquire the ability to solve problems. The artificial neural network can be used interchangeably with the term neural network.

[0054] An artificial neural network can include a plurality of layers, each of which can include a plurality of neurons. Also, the artificial neural network can include synapses connecting the neurons with each other. The artificial neural network can be generally defined by the following three elements, i.e., (1) a connection pattern between the plurality of neurons of other layers, (2) a learning process of updating weights of the connections, and (3) an activation function of generating an output value based on a weighted sum of inputs received from a previous layer.

[0055] The artificial neural network can include, but is not limited to, a network model of a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), a BRDNN (Bidirectional Recurrent Deep Neural Network), an MLP (Multilayer Perceptron), a CNN (Convolutional Neural Network), etc. In the present specification, the term "layer" can be used interchangeably with "level".

[0056] The artificial neural network can be classified into a single-layer neural network and a multi-layer neural network according to the number of levels. A general single-layer neural network is configured with an input layer and an output layer. Also, a general multi-layer neural network is configured with an input layer, one or more hidden layers, and an output layer.

[0057] The input layer is a layer that receives external data, and the number of neurons of the input layer is the same as the number of input variables. The hidden layer is located between the input layer and the output layer, and receives signals from the input layer and extracts features, and then transmits the extracted features to the output layer. The output layer receives signals from the hidden layer, and outputs an output value based on the received signals. Input signals between the neurons are multiplied by each connection strength (weight), and then summed. If the sum is greater than a threshold value of the neuron, the neuron is activated, and an output value obtained through an activation function is output.

[0058] On the other hand, a deep neural network including a plurality of hidden layers between the input layer and the output layer can be a representative artificial neural network that implements deep learning as one of machine learning techniques. On the other hand, the term "deep learning" can be used interchangeably with "deep-level learning", and the term "learning" can be used interchangeably with "training".

[0059] The workflow of machine learning consists of a series of processes of collecting data for learning and verification and modeling, and training a model after modeling, and can include collection of training data, inspection and exploration of data, data preprocessing and cleaning, modeling, and training processes.

[0060] 1. Collection of training data

[0061] Training data suitable for learning of a learning model in the present specification can be generated using data collected from a plurality of samples. In the present specification, at least one or more different types of training data sets can be used for training of a learning model, and each training data can further include one or more experiment-based results used as a function label. At least a part of the training data sets is used for training of a learning model, and another part can be used for verifying the learned learning model.

[0062] 2. Inspection and exploration of data

[0063] When training data for learning of a learning model is collected, inspection and exploration of the structure of the collected training data, noise data, and data cleaning methods suitable for machine learning can be performed.

[0064] Such a data inspection and exploration step is referred to as an Exploratory Data Analysis (EDA) step, and can be said to be a process of observing and understanding collected data from various angles. Before learning data, inspection can be performed on independent variables, dependent variables, variable types, data types of variables, etc. by means such as visualization such as a graph, statistical tests, etc., and the characteristics and internal structural relationships of the data can be confirmed in advance. Through such EDA, the distribution and values of the data are reviewed, and the phenomenon expressed by the data is better understood, and potential problems with the data can be discovered. Also, through the process of inspecting the data from various angles, various patterns that were not apparent in the problem definition stage can be discovered, and the original hypothesis can be modified or a new hypothesis can be proposed based thereon. Exploratory data analysis can generally include a process of exploring outliers of data and a process of analyzing relationships between attributes of data.

[0065] The process of exploring outliers is a process of confirming whether there are outliers in the data, and can include specimen extraction methods, statistical methods, and visual methods, etc. The specimen extraction method is to randomly extract specimens from the data and confirm the overall trend and special matters of the data values. The statistical method can use summary statistics such as mean, median, mode for confirming the center of data, or range, variance for confirming the dispersion of data, etc. The visual method can determine which statistical index is suitable for the respective properties of the collected data using probability density function, histogram, dot plot, word cloud, time series chart, map, etc. It should be noted that when using statistical indexes, the mean reflects all data values within the set, so if there are outliers, the mean will be affected, and the median only uses one value located in the middle position, so even if there are outliers, a representative result can be obtained.

[0066] The process of analyzing the relationship between data properties is a process of finding combinations of properties that have a meaningful correlation with each other within the data. According to the property combinations between qualitative properties (Categorical Variable; Qualitative) that cannot be represented by numbers but can be digitized arbitrarily and quantitative properties (Numeric Variable; Quantitative) that can be numerically, relationship analysis can be performed differently. Categorical-Categorical relationship can represent the number of values corresponding to each attribute value pair using a cross table or a mosaic chart, Numeric-Categorical relationship can be expressed in a visual manner by observing each category statistical value (mean, median, etc.) or by a box plot, and Numeric-Numeric relationship can analyze the correlation between two properties by correlation coefficient. The correlation coefficient -1 can represent a negative correlation relationship in which two properties change in opposite directions, 0 represents no correlation, and 1 represents a positive correlation relationship in which two properties always change in the same direction. The relationship of two properties with a correlation coefficient can also appear in many ways, which can be visually represented using a scatter plot.

[0067] 3. Data preprocessing and cleaning

[0068] Data preprocessing is a process of cleaning data to convert it into a form that the model can understand, and data preprocessing can generally include Handling Missing Data, Outlier Removal, Scaling, Categorical Data Encoding, Feature Selection and Extraction, and Data Transformation. The detailed process of data preprocessing can be implemented in whole or selectively implemented in part, and other machine learning models can also be used for data preprocessing.

[0069] Handling Missing Data is a process of handling missing values when there are missing values in data, and missing values can be represented as NaN (Not a Number) or null or can be deleted. Missing values can be filled in data or deleted to improve data integrity, and when missing values are filled, mean or median or mode, etc. can be used.

[0070] Outlier Removal is a process of removing values that deviate from the usual data pattern, i.e. outliers, from data. Outliers can reduce the performance of the model, so they need to be removed or replaced, and after identifying outliers, the corresponding row or column can be deleted or replaced with other values.

[0071] Scaling is a process of adjusting the size of data, and through data scaling, the data range can be adjusted to improve model performance or improve convergence speed. Through data scaling, the characteristics of data can be adjusted to a similar range, and typical data scaling can use Standardization and Normalization. Standardization is a method of converting data to a distribution with a mean of 0 and a standard deviation of 1, and mainly uses mean and standard deviation for conversion, and the standardized value z can be expressed as (where x is the original value, μ is the mean, and σ is the standard deviation). Normalization is a method of converting the data range to [0, 1] or [-1, 1], and mainly uses the minimum and maximum values to convert data, and the normalized value x norm can be expressed as (where x is the original value, x min is the minimum value, and x max is the maximum value).

[0072] Categorical Data Encoding is a method of converting a string that cannot be directly input to a model or a categorical variable represented in an integer value into a numeric type that can be input to a model. In general, a categorical variable can be converted into a numeric type using one-hot encoding or label encoding.

[0073] Feature Selection and Extraction is a method of selecting the most useful features in model learning or extracting new features to improve the performance of a model, through which the complexity of a model can be reduced and overfitting can be prevented.

[0074] Data Transformation is a method of extracting new information or making a model more understandable by transforming data, which can include tokenization of text data or preprocessing of image data, etc. Through data transformation, useful features can be extracted from raw data or data can be converted into an appropriate form, thereby improving the performance of a model.

[0075] Through the data preprocessing as described above, it is possible to improve the performance of a learning model of machine learning and to ensure stability.

[0076] On the other hand, in training a learning model according to an embodiment of the disclosure, a process of preprocessing information tagged in a natural language and a process of learning a language model based on preprocessed data can be implemented.

[0077] 3-1. Text Preprocessing for Large-Scale Language Model

[0078] If the collected data is not converted into a desired state through preprocessing, tokenization & cleaning & normalization can be performed according to the purpose of using the corresponding data.

[0079] Tokenization refers to an operation of dividing data provided in a unit called a token, and the unit of a token can be defined as a meaningful unit in general. Tokenization can include word tokenization and sentence tokenization in general.

[0080] Tokenization refers to an operation of dividing data provided in a unit called a token, and the unit of a token can be defined as a meaningful unit in general. Tokenization can include word tokenization and sentence tokenization in general.

[0081] Word tokenization refers to a case where a criterion for a word unit is set as a word, and a word includes not only a word unit but also a sentence, a character string having a meaning. Word tokenization is performed based on a space or a punctuation mark such as a period, a comma, a question mark, a semicolon, an exclamation mark, and the like. Note that, when tokenization is performed, there is a case where a token loses a meaning when a punctuation mark or a special character is removed, and thus an accurate algorithm for tokenization can be required. For example, in a case where a word itself includes a punctuation mark or a special character is used, it is not possible to solve the problem by simply removing the punctuation mark or the special character. Thus, when tokenization is performed, a tokenization rule such as a Penn Treebank Tokenization rule can be applied.

[0082] Article tokenization refers to a case where a text is divided into articles. If data is not cleaned, a corpus is not divided into articles, and thus article tokenization can be performed according to a purpose of use. Article tokenization can define various rules according to a language used, how a special character is used in a corresponding corpus, and the like.

[0083] An operation of classifying tokens according to a purpose is referred to as tokenization. Cleaning and normalization of text data according to a purpose can be performed before and after tokenization. Cleaning is a process of removing noise data, and normalization is a process of integrating and converting a plurality of words having different expression methods into the same word.

[0084] In order to remove a portion that hinders tokenization, cleaning can be performed before tokenization. However, in order to remove noise remaining after tokenization, cleaning can be repeatedly performed. Noise data removed by cleaning is a character having no meaning, and a method of removing unnecessary words includes a method of removing a stop word and a method of removing a word having a low frequency of occurrence and a short length.

[0085] Normalization includes a process of integrating words having different marks and a process of integrating upper and lower case characters. The process of integrating upper and lower case characters is a normalization method capable of reducing a number of words in an English language. In the English language, a capital letter is used only in a specific case such as the beginning of an article, and most characters are written in a lower case. Thus, most of the process of integrating upper and lower case characters can be implemented by a process of converting a capital letter into a lower case.

[0086] In a computing system, in order to process a natural language, a preprocessing operation of numerically converting a text is required, and for this, an operation of mapping each word of the text to a unique integer is performed. This mapping process can utilize techniques such as Integer Encoding, Padding, and One-Hot Encoding.

[0087] Integer Encoding is a method of assigning an integer to a word, and a word set (vocabulary) is made in order of frequency of occurrence, and an integer is sequentially assigned from a small number in order of frequency of occurrence. Integer Encoding starts with text data including a plurality of articles, and at the same time, a cleaning operation and a normalization operation are performed, thereby performing word tokenization. At this time, a plurality of words have been lowercased and thus the number of words has been unified, and deletion of words can be performed according to non-use words and word length. Thereby, the frequency of occurrence of each word can be recorded as a value (value) using the word as a key. After arranging in order of frequency of occurrence within the text, an integer is assigned to a word having a high frequency of occurrence, and thus Integer Encoding can be performed.

[0088] Padding is an operation for arbitrarily aligning the lengths of articles having different lengths within a text. A computing system can bundle articles having the same length into one matrix and perform parallel operation. That is, in order to perform parallel operation of a computing system, "0" can be arbitrarily added to the integer encoding result of an article having a different length within a text, thereby unifying the lengths of a plurality of articles. That is, the longest article can be found from a word set in which Integer Encoding is ended, and "0" is added to the integer matrix so as to correspond to the length of the longest article. A computing system can recognize a plurality of articles having the same length as one matrix, and thus parallel processing can be performed, and at this time, the computing system can ignore "0" words recognized as having no meaning. This operation of filling a specific value in data to adjust the size (shape) of data is called padding, and a case where a number "0" is added in order to adjust the length is called zero padding.

[0089] One-hot encoding is a way of expressing a word as a vector, and the size of the word set is used as the dimension of the vector, and the index of the word to be expressed is given a value of 1, and other indexes are given a value of 0. The vector thus expressed is called a one-hot vector. One-hot encoding is constructed by the integer encoding and index assignment process. After performing integer encoding to assign a unique integer to each word, the unique integer of the word to be expressed is regarded as an index, and "1" is assigned to the corresponding position, and "0" is assigned to the index positions of other words. It should be noted that one-hot encoding has a problem of increasing the necessary space for storing the vector (increasing the dimension of the vector) as the number of words increases, and a problem of being unable to confirm the similarity between words. In order to solve such problems, as a technique for vectorizing in a multi-dimensional space reflecting the latent meaning of a word, LSA (Latent Semantic Analysis) as a count-based vectorization method, NNLM, RNNLM, Word2Vec, and FastText, which vectorize based on prediction, GloVe, which uses both a count-based and a prediction-based method, and the like can be cited.

[0090] On the other hand, in order for a computer to understand and process text, it is necessary to appropriately convert the text into numbers. Depending on the method of expressing a word, the natural language processing performance varies greatly, and therefore many techniques for digitizing a word have been proposed. So far, the most widely used method is a word embedding method that vectorizes each word through artificial intelligence neural network learning.

[0091] Word embedding is a method of expressing a word as a vector, and converts a word into a dense expression. The result derived through the word embedding process is called a dense vector or an embedding vector. As a word embedding methodology, LSA, Word2Vec, FastText, Glove, and the like are proposed.

[0092] 4. Modeling and training

[0093] An artificial neural network can be learned (trained) using training data. Here, learning can refer to a process of determining the parameters of an artificial neural network using learning data in order to achieve the purpose of classifying (classification) or regression analysis (regression) or clustering (clustering) input data. As representative examples of the parameters of an artificial neural network, weights assigned to synapses or biases applied to neurons can be cited.

[0094] An artificial neural network that learns using training data can classify or cluster input data according to a pattern that the input data has. On the other hand, an artificial neural network that learns using training data can be referred to as a trained model in this specification.

[0095] Second, a learning method of an artificial neural network is described. A learning method of an artificial neural network can be broadly classified into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0096] Supervised learning is a kind of machine learning method for generalizing a function from training data. In addition, a function that outputs a continuous value in such a generalized function can be referred to as regression, and a function that outputs a class of an input vector can be referred to as classification.

[0097] In supervised learning, learning of an artificial neural network is performed in a state in which a label for training data is provided. Here, the label can indicate a true value (or a result value) that an artificial neural network needs to infer when the training data is input to the artificial neural network. In this specification, the true value (or the result value) that an artificial neural network needs to infer when training data is input is referred to as a label or labeling data. Also, in this specification, an operation of setting a label for training data for learning of an artificial neural network is referred to as labeling labeling data for training data. In this case, training data and a label corresponding to the training data constitute a training set, and an artificial neural network can be input in the form of a training set.

[0098] On the other hand, training data indicates a plurality of features, and labeling a label for training data can mean labeling a label for features indicated by training data. In this case, training data can indicate features of an input object in the form of a vector. An artificial neural network can generalize a function regarding a correlation between training data and labeling data using training data and labeling data. In addition, by evaluating the function generalized by the artificial neural network, a parameter of the artificial neural network can be determined (optimized).

[0099] Unsupervised learning is a kind of mechanical learning that does not provide labels for training data. Specifically, unsupervised learning is a learning method that classifies after learning a pattern in training data itself by a neural network, rather than finding a correlation between training data and labels corresponding to the training data. As examples of unsupervised learning, clustering or independent component analysis (ICA) can be cited. In this specification, the term "clustering" can be used interchangeably with "aggregation".

[0100] As an example of a neural network using unsupervised learning, a generative adversarial network (GAN) or an autoencoder (AE) can be cited.

[0101] A generative adversarial network is a machine learning method in which two different artificial intelligences, a generator and a discriminator, compete with each other to improve performance. In this case, the generator is a model that creates new data, and can generate new data based on original data. Also, the discriminator is a model that recognizes a pattern of data, and can function to recognize whether input data is original data or new data generated by the generator. In addition, the generator can learn from data that fails to deceive the discriminator, and the discriminator can learn from data that is deceived by the generator. As a result, the generator can evolve to the extent of deceiving the discriminator to the maximum, and the discriminator can evolve to the extent of being able to well distinguish between original data and data generated by the generator.

[0102] An autoencoder is a neural network that aims to reproduce the input itself by output. The autoencoder includes an input layer, at least one hidden layer, and an output layer. In this case, the number of nodes of the hidden layer is less than the number of nodes of the input layer, so the dimension of data is reduced, thereby achieving compression or encoding. Also, data output from the hidden layer enters the output layer. In this case, the number of nodes of the output layer is more than the number of nodes of the hidden layer, so the dimension of data is increased, thereby achieving decompression or decoding.

[0103] On the other hand, the autoencoder expresses input data in the form of hidden layer data by adjusting the connection strength of neurons through learning. In the hidden layer, information is expressed by a smaller number of neurons than the input layer, and the input data can be reproduced by the output, which can indicate that the hidden layer has discovered and expressed a hidden pattern from the input data.

[0104] Semi-supervised learning is a kind of machine learning, and can be expressed as a learning method using all of training data for which a label is provided and training data for which a label is not provided. As a technique of semi-supervised learning, there is a technique of learning using a label that is inferred after inferring a label of training data for which a label is not provided. This technique is advantageous in a case where a cost required for labeling is large.

[0105] The theory of reinforcement learning is that if an environment in which an agent can judge what action should be taken at each moment is provided, an optimal path can be found only through experience without relying on data. Reinforcement learning can be mainly executed through a Markov Decision Process (MDP). The Markov Decision Process is explained as follows. First, an environment in which a plurality of information required for an agent to perform a next action is provided. Second, how the agent acts in the environment is defined. Third, what is handled well by the agent is defined to be rewarded and what is handled not well is defined to be penalized. Fourth, experience is repeated until a future reward reaches a maximum point, thereby deriving an optimal policy.

[0106] The structure of an artificial neural network is specified through a model configuration, an activation function, a loss function or a cost function, a learning algorithm, an optimization algorithm, and the like, and a hyperparameter is set in advance before learning. After that, a model parameter is set through learning, and thus the content can be specified.

[0107] For example, elements for determining the structure of an artificial neural network can include the number of hidden layers, the number of hidden nodes included in each hidden layer, an input feature vector, a target feature vector, and the like.

[0108] The hyperparameter includes a plurality of parameters that need to be set in the early stage in order to perform learning, such as an initial value of a model parameter. In addition, the model parameter includes a plurality of parameters that need to be determined through learning. For example, the hyperparameter can include an initial value of a weight between nodes, an initial value of a bias between nodes, a mini-batch size, a number of learning repetitions, a learning rate, and the like. In addition, the model parameter can include a weight between nodes, a bias between nodes, and the like.

[0109] The loss function can be used as an index (criterion) for determining optimal model parameters in the learning process of the artificial neural network. Learning in the artificial neural network refers to a process of manipulating model parameters in order to reduce the loss function, and the purpose of learning is to determine model parameters that minimize the loss function. The loss function can mainly use a mean squared error (MSE) or a cross entropy error (CEE), but the present invention is not limited thereto. The cross entropy error can be used when true value labels are one-hot encoded. One-hot encoding is an encoding method in which a true value label value is set to 1 only for a neuron corresponding to a true value and is set to 0 for a neuron that is not a true value.

[0110] In machine learning or deep learning, in order to minimize the loss function, a learning optimization algorithm can be used, and the learning optimization algorithm includes a gradient descent (GD), a stochastic gradient descent (SGD), a momentum, a NAG (Nesterov Accelerate Gradient), Adagrad, AdaDelta, RMSProp, Adam, Nadam, etc.

[0111] The gradient descent is a technique of adjusting model parameters in a direction in which the loss function value is reduced from a current state, in consideration of the gradient of the loss function. The direction in which the model parameters are adjusted is referred to as a step direction, and the size of the adjustment is referred to as a step size. At this time, the step size can represent a learning rate. The gradient descent can perform partial differentiation of the loss function with respect to each model parameter, thereby obtaining a gradient, and change the model parameters by a size corresponding to the learning rate in the direction of the obtained gradient, thereby implementing update.

[0112] The stochastic gradient descent is a technique of dividing learning data into small batches, performing the gradient descent with respect to each small batch, and thereby increasing the frequency of the gradient descent.

[0113] Adagrad, AdaDelta, and RMSProp are techniques of adjusting the step size in the SGD, thereby improving optimization accuracy. In the SGD, the momentum and the NAG are techniques of adjusting the step direction, thereby improving optimization accuracy. Adam is a technique of combining the momentum and the RMSProp, thereby adjusting the step size and the step direction to improve optimization accuracy. Nadam is a technique of combining the NAG and the RMSProp, thereby adjusting the step size and the step direction to improve optimization accuracy.

[0114] The learning speed and accuracy of an artificial neural network have characteristics that are affected by hyperparameters in addition to the structure of the artificial neural network and the type of learning optimization algorithm. Therefore, in order to obtain a good learning model, in addition to determining the appropriate structure of the artificial neural network and learning algorithm, it is also important to set appropriate hyperparameters.

[0115] Generally, for hyperparameters, learning of an artificial neural network is performed while experimentally setting various values, and then the optimal value that provides stable learning speed and accuracy of the learning result is set.

[0116] The plurality of embodiments of the present application apply to a chatbot that has a conversation with a user by using a voice or a text-based input of the user and performs a specific operation, and functions to understand a question of a speaker and output an appropriate answer. According to the core function, the structure of the chatbot can include a question intent classification, an entity name recognition, a core keyword extraction, an answer retrieval, etc. The question intent classification function is a function of grasping the intent of the question of the speaker, and predicts the intent category of the corresponding question using an intent classification model. The entity name recognition function is a function of recognizing the entity name of each word token in the question of the speaker, and the core keyword extraction function is to extract a noun or a verb called a core in the meaning of the question of the speaker using a morpheme analyzer, etc. The answer retrieval function is to retrieve a corresponding appropriate answer in a learning database based on the intent of the question, the recognized entity name, the extracted core keyword, etc., and to display it. That is, when there is an article of a question (query) of a user is input, the chatbot performs preprocessing for the corresponding article, extracts keywords (word tokens) through a morpheme analyzer, extracts only necessary keywords such as nouns or verbs, and removes non-words. After that, the chatbot performs intent analysis and entity name recognition on the extracted keywords, and derives a corresponding answer, and for this, in order to natural language processing, the chatbot generally uses a deep learning model such as an intent analysis (classification) model and an entity name recognition model.

[0117] The present embodiment can involve fine tuning or prompt engineering for improving the accuracy of the intent classification model in the main configuration of such a chatbot. Fine tuning or prompt engineering is a technique for enhancing the accuracy and usability of a deep learning model. Fine tuning is a method of relearning a pre-learned model to conform to a specific operation or data set, and requires domain-specific data. Prompt engineering is a technique of optimizing input prompts for a model to obtain a desired output result, and can be applied to various operations in a relatively short time using a smaller amount of data than fine tuning. The learning method and system of the intent classification model using intent specification according to an embodiment of the present application can include an intent classification model that relearns a model pre-learned based on learning data including intent specification, and an intent classification model that infers from the intent classification model based on input data (input prompts) including intent specification.

[0118] The learning method and system of the intent classification model using intent specification according to an embodiment of the present application can be applied to various fields using a chatbot based on a large-scale language model. The intent classification model optimized according to the present embodiment can be applied to a customer service chatbot based on enterprise customer reception data, a domain model optimized for a specific domain (work field) such as medical care, law, finance, various natural language processing operations such as text generation, summarization, translation, market trend analysis reflecting the latest information and trends, a personalized recommendation system based on user service usage data, etc.

[0119] Figure 1 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present application.

[0120] As Figure 1As shown, the electronic device 100 (hereinafter, referred to as an electronic device) according to an embodiment of the present application can include at least one or more processors 110, a memory 120, and a communication unit 130. The electronic device 100 is a basic configuration for executing a computing environment, and in other embodiments, the electronic device 100 can be configured to further include a part of other constituent elements or to be replaced with a part of other constituent elements, or to be configured as one or more entities, or to be configured only by a part of the disclosed configuration. A plurality of constituent elements or at least a part of the constituent elements inside or outside the electronic device 100 can be connected to each other through a BUS, a GPIO (General Purpose Input / Output), an SPI (Serial Peripheral Interface), an MIPI (Mobile industry Processor Interface), or the like, so that data or signals can be transmitted and received.

[0121] The processor 110 can represent a combination of one or more processors capable of driving at least software (e.g., instructions, programs, etc.) stored in the memory 120 to control the processor 110 and constituent elements of the electronic device 100, without a clear expression of different meanings in the context. Also, the processor 110 can perform various operations, processing, data generation or processing, etc., can read data, etc. from the memory 120 or store data, etc. in the memory 120. The processor 110 can be constituted of at least one or more cores, can include a central processing device (CPU), a general-purpose image processing device (GPGPU), a tensor processor (TPU), etc. for performing data analysis, machine learning (ML), or deep learning (DL). The processor 110 can perform data processing for machine learning (or deep learning) of the present disclosure after reading software stored in the memory 120. According to an embodiment of the disclosure, the processor 110 can perform an operation for learning of a neural network. The processor 110 can perform processing of input data for learning in deep learning, feature extraction in input data, error calculation, weight update of a neural network using backpropagation, etc. for learning of a neural network. At least one of the CPU, the GPGPU, and the TPU of the processor 110 can process learning of a neural network model. For example, the CPU and the GPGPU can together process learning of a neural network model, data classification using a neural network model. Also, in an embodiment of the disclosure, at least one or more processors 110 of the electronic device 100 can be used at the same time to process learning of a neural network model, data classification using a neural network model.

[0122] The memory 120 is used to store various data, which is data acquired or processed or used by at least one constituent element of the electronic device 100, and can include software (e.g., instructions, programs, etc.). The memory 120 can represent a set of one or more memories without a clear expression of different meanings in the context, and can include at least one type of storage medium of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a web storage that performs a storage function in the Internet, etc. The instructions or programs or software stored in the memory 120 can refer to an operating system for controlling the operations of the constituent elements of the electronic device 100, an application program, or a middleware that provides various functions to the application program to enable the application program to utilize the constituent elements of the electronic device 100, etc. In an embodiment, the memory 120 can store instructions corresponding to a specific operation performed by the processor 110 when the processor 110 performs the specific operation.

[0123] The communication unit 130 performs wireless or wired communication between the electronic device 100 and other devices (e.g., a user terminal or other server), and can employ a wireless communication system according to eMBB, URLLC, MMTC, LTE, LTE-A, NR, UMTS, GSM, CDMA, WCDMA, TDMA, FDMA, OFDMA, SCFDMA, WiBro, WiFi, Bluetooth, NFC, GPS, or GNSS, etc. Also, the communication unit 130 can employ various wired communication systems such as USB, HDMI, RS-232 (Recommended Standard-232), POTS (Plain Old Telephone Service), a public switched telephone network (PSTN: Public Switched Telephone Network), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN), etc. In an embodiment of the present application, the communication unit 130 can be configured not to be limited to wired and wireless communication methods, and can employ various communication networks such as a personal area network (PAN), a wide area network (WAN), etc. Also, the above network can be a publicly known World Wide Web (WWW), and can employ a wireless transmission technology for short-range communication such as infrared (IrDA: Infrared Data Association) or Bluetooth (Bluetooth). The technology described in an embodiment of the present application can be applied to other networks mentioned above.

[0124] The electronic device 100 according to an embodiment of the present application can constitute a learning system using an intent classification model described with reference to the drawings or can execute software constituting a learning method using an intent classification model described with reference to the drawings.

[0125] The learning system of the intent classification model according to the embodiment of the present application can be applied to a task-oriented dialogue system such as a chatbot, and can be used to improve the accuracy of the intent classification model that understands the intent of the user's question. In general, the intent classification model can predict a domain (or category) for learning. Therefore, even when unknown data (unseen data) is provided as input to the intent classification model that has been pre-learned in this way, it is necessary to smoothly perform prediction and classification for the unknown data, and in this case, it can be necessary to further operate the intent classification model (fine-tuning or prompt engineering). That is, the intent classification model can be modified so that even when unknown data is input to the intent classification model, it is predicted or classified as an unknown domain (category; unseen domain or unseen class) that is not included in the learning data set, rather than being classified into one of the domains (categories; seen domain or seen class) learned by the internal mechanism. In modifying the intent classification model in this way, the learning system and method of the intent classification model according to the embodiment of the present application can be used.

[0126] In an embodiment, an independent intent explanation, a dependent (dependent) intent explanation, and a cleaned intent explanation can be generated. The independent intent explanation includes only an explanation for one intent, and can be obtained by simultaneously providing three examples of user questions for the corresponding intent. At this time, information for other intents is not included, so the corresponding intent can be explained alone without a comparison object. Thus, the independent intent explanation cannot sufficiently reflect the relative difference with other intents, and therefore the quality can be relatively low. The dependent intent explanation provides a prompt word including a list of all intents, and is generated so as to be able to distinguish the explanation for each intent from other intents. That is, the generated dependent intent explanation can clearly distinguish a specific intent from other intents, and therefore can have a relatively high quality than the independent intent explanation. In view of the fact that the automatically generated intent explanations such as the independent intent explanation and the dependent intent explanation can not clearly reflect the difference between multiple intents, a cleaned intent explanation that is directly cleaned by a person can also be generated. The cleaned intent explanation is an intent explanation in which an explanation for all intents has been manually created by a person, and is created so as to be able to be clearly distinguished from other intents, and therefore can have a higher quality than the automatically generated explanation.

[0127] According to the learning system of the intent classification model using the intent description according to an embodiment of the present application, a dataset including intent descriptions of the intent classification model can be output from a language model, at least a part of the dataset can be used as an input prompt of the intent classification model, intent classification can be performed by the intent classification model, at least a part of the dataset can be used as an input prompt of the intent classification model, and performance of the intent classification model can be determined. Here, the dataset can include first data including independent intent descriptions and second data including dependent intent descriptions. That is, the intent classification model can be re-learned or tested using the first data or the second data.

[0128] The learning system according to an embodiment can acquire or generate a dataset for re-learning of the intent classification model after pre-learning from other language models. The language model according to an embodiment of the present application can use an AI chat robot that generates an output corresponding to an input prompt, for example, ChatGPT. The learning system according to an embodiment can acquire two datasets from the language model, the first data can be data including independent intent descriptions, and the second data can be data including dependent intent descriptions.

[0129] That is, as Figure 2 As shown, the learning system according to an embodiment inputs instruction prompts including independent intent descriptions (independent prompts) and instruction prompts including dependent intent descriptions (dependent prompts) to a language model such as ChatCPT, respectively, so that the first data and the second data can be acquired.

[0130] Referring to Figure 2The first data for a manner of including an intent description for one intent can be acquired through an independent prompt word. That is, the first data acquired through the independent prompt word can include an intent name and an intent description for the corresponding intent name. The first data acquired through such an independent prompt word includes a plurality of intent names and intent descriptions for the corresponding intent names, and it is difficult for the intent classification model to determine or judge the differences between the plurality of intent names included in the first data. That is, the independent prompt word can include one intent and three user queries for the corresponding intent, and other intents are excluded from the prompt word. Also, the intent description for the intent included in the independent prompt word is created in an independent manner without analyzing the context of the intent descriptions for other intents, and thus is called an independent intent description. As such, the independent intent description is not analyzed or considered or referenced or referred to in the context of other intent descriptions, and thus is created or set in an independent manner regardless of other intent descriptions, and thus lacks a comparative context in the intent descriptions collected, reducing the quality of the intent descriptions. In an embodiment, in order to acquire a plurality of independent intent descriptions for one intent name, the independent prompt word can be called multiple times.

[0131] Referring to Figure 2, the second data can be acquired by the sub-prompt word. That is, the second data acquired by the sub-prompt word can include a plurality of intent names and a plurality of intent descriptions for the corresponding intent names. The plurality of intent names and the intent descriptions for the corresponding intent names included in the second data can be distinguished from each other by the intent classification model. That is, the sub-prompt word can include a plurality of intents and three user queries for each intent in one prompt word. The plurality of intents are included in one sub-prompt word, and the indication sentence of the prompt word can be set to acquire intent descriptions for the plurality of intents and each intent description does not contain the upper or lower concept of the other intent description. That is, each intent description can be set not to overlap or repeat the content of the other intent description. The sub-intent description can thus be created or set in a subordinate manner after analyzing the context of the other intent description. That is, the sub-intent description analyzes or considers or refers to or refers to the context of the other intent description, and thus can depend on and be influenced by the other intent description. The sub-intent description is thus created or set in relation to the content of the other intent description, and thus the prompt word includes as many intents as possible and the plurality of intent descriptions generated thereby have unique differences from each other, and thus the quality of the intent descriptions of the second data can be higher than that of the first data. In an embodiment, in order to acquire a plurality of sub-intent descriptions for one intent name, the sub-prompt word can be called multiple times.

[0132] The data set acquired from the language model in this way can be used as data for relearning of fine-tuning of the intent classification model and test data for testing or determining the performance of the intent classification model that has completed fine-tuning.

[0133] On the other hand, in an embodiment, in addition to the first data or the second data, third data in which an operator manually reviews and adjusts the intent and the intent description for the corresponding intent can be included. The third data can include cleaned intent descriptions. That is, the third data is data having the highest quality for accurate judgment of the intent classification model, and the third data can be used only for verification of the intent classification model, and can also be selectively included in the data set in order to improve the accuracy of the intent classification model.

[0134] And, the data set secured as described above can be set as an input prompt word for fine-tuning of the intent classification model, Figure 3 An example of an input prompt word based on a data set is shown.

[0135] Referring to Figure 3The input prompt word inputted to the intent classification model can be divided into an instruction for classifying the provided intent, an instruction and a user query, and an instruction and an intent option. The intent option can include an intent name and an intent description for the corresponding intent name. That is, the input prompt word for fine-tuning the intent classification model is generated based on the first data, the second data, and the third data, and the form of the input prompt word based on the first data to the third data can be generated to be the same.

[0136] To improve the performance of the intent classification model, the learning system of an embodiment can perform intent classification of the intent classification model using the input prompt word of the data set based on the language model as described above as input. At this time, the input prompt word inputted to the intent classification model can be set based on the first data, the second data, or the third data or can further include the data.

[0137] Also, the learning system of an embodiment inputs a test data set to the intent classification model that has been re-learned based on the re-learning data set, and thus can determine the performance of the intent classification model.

[0138] In this way, the classification accuracy of the intent classification model that has been re-learned based on the data set including the intent description can be higher than that of the model that has not been re-learned, and evaluation of the model will be described below.

[0139] On the other hand, the learning system of an embodiment can further include a ranker for adjusting the number of intent options included in the re-learning input prompt word. If the intent description is included in the model re-learning input prompt word, the length of the input prompt word is proportionally increased according to the number of intent options.

[0140] For example, assuming that the model can accept a maximum of 1024 tokens as input, if an explanation of 10 words is provided for each of 100 intents, the likelihood of exceeding the input length limit increases. To solve this problem, the ranker of an embodiment calculates the similarity between the user query and the intent description, and after arranging all the intent options in order of low similarity, the top k intents can be delivered to the intent classification model. According to an embodiment, the k intent options with the highest similarity are selected, and thus the entire intent is included in the model input but the input length can be optimized. Evaluation of the performance of the intent classification model by increasing or decreasing the number of intent options will be described below.

[0141] As Figure 4It is shown that the learning method of the intent classification model using a language model implemented in the learning system of the intent classification model using a language model according to an embodiment of the present application can include the step of generating a data set including intent descriptions of the intent classification model using a language model (S110), the step of classifying the intent by the intent classification model using at least a part of the data set as an input prompt word of the intent classification model (S120), and the step of determining the performance of the intent classification model using at least a part of the data set as an input prompt word of the intent classification model (S130). Among them, the data set can include first data containing independent intent descriptions and second data containing dependent intent descriptions.

[0142] In the step of generating a data set including intent descriptions of the intent classification model using a language model (S110), the learning system can acquire a data set for the pre-learned intent classification model from other language models. That is, the learning system of the present embodiment can acquire a data set for fine-tuning of the intent classification model from other language models, such as ChatGPT. In order to acquire a data set from a language model, the learning system inputs a predetermined instruction prompt word into the language model to acquire a data set. At this time, in order to determine the data quality, the instruction prompt word input to the language model can include two instruction prompt words. The independent prompt word can be an instruction prompt word for acquiring first data including independent intent descriptions from the language model, and the dependent prompt word can be an instruction prompt word for acquiring second data including dependent intent descriptions.

[0143] The first data acquired through the independent prompt word includes multiple intent names and intent descriptions for the corresponding intent names, but the intent classification model has difficulty in determining or cannot determine the differences between the multiple intent names included in the first data. That is, the intent descriptions of the first data lack a comparative context, thereby reducing the quality of the intent descriptions.

[0144] The second data acquired through the dependent prompt word includes multiple intent names and multiple intent descriptions for the corresponding intent names. Such second data includes multiple intent names and intent descriptions for the corresponding intent names, and the intent classification model can determine the differences between the multiple intent names included in the second data. That is, the intent descriptions of the second data can be uniquely distinguished, and thus the quality of the intent descriptions of the second data can be higher than that of the first data.

[0145] The data set acquired from the language model in this way can be used as relearning data for fine-tuning of the intent classification model and test data for testing or determining the performance of the intent classification model that has completed fine-tuning.

[0146] In the step (S120) of classifying the intent by the intent classification model using at least a part of the data set as an input prompt of the intent classification model, the input prompt of the intent classification model can have a manner of including an instruction for classifying the provided intent and a part of the user query and a part of the intent option. The data set obtained from the language model is thus reconfigured in the form of the input prompt of the intent classification model. The first input prompt based on the first data including the independent intent specification or the second input prompt based on the second data including the dependent intent specification is input to the intent classification model, thereby implementing fine-tuning of the intent classification model. The intent classification model re-learned by the first input prompt and the intent classification model re-learned by the second input prompt can have a difference in classification accuracy and performance. The performance comparison between the models is described below.

[0147] In the step (S130) of determining the performance of the intent classification model using at least a part of the data set as an input prompt of the intent classification model, the learning system of an embodiment can perform a test using the first input prompt based on the first data including the independent intent specification or the second input prompt based on the second data including the dependent intent specification. At this time, the input prompts for the test are different from the input prompts for the learning of the intent classification model, and the input prompts for the test are input to the baseline model, i.e., the original intent classification model, the intent classification model re-learned by the first input prompt, and the intent classification model re-learned by the second input prompt, respectively, and the models can be tested and the performance of the models can be evaluated.

[0148] On the other hand, the learning method of the intent classification model using the language model of an embodiment can further include a step of adjusting the number of intent options included in the first input prompt based on the first data and the second input prompt based on the second data and the third input prompt based on the third data, i.e., adjusting the number of intent options of the input prompt of the intent classification model.

[0149] To this end, the learning system of an embodiment can arrange all the intent options in order of similarity between the user query and the intent specification included in the input prompt in a ranker that calculates the similarity and then select only the top k intents.

[0150] According to the learning method and system of the intent classification model using the intent specification of an embodiment of the present application having the above-described configuration, the performance of the intent classification model can be evaluated.

[0151] The first intent classification model re-learned with the first input cues including independent intent descriptions derived from the language model, the second intent classification model re-learned with the second input cues including dependent intent descriptions, and the baseline model trained without intent descriptions were tested, so that the intent classification performance of each model can be evaluated.

[0152] [Table 1]

[0153]

[0154] As described in [Table 1], based on the results of testing the first intent classification model (independent descriptions), the second intent classification model (dependent descriptions), and the baseline model (without descriptions) on the test data set not including intent descriptions, the baseline model showed the highest performance of 84.28%, the first intent classification model showed lower performance of 81.93%, and the second intent classification model showed performance of 82.1%. However, when the models were tested with the first input cues including independent intent descriptions (independent descriptions), the first intent classification model and the second intent classification model showed performance of 85.64% and 86.99%, respectively, showing results of improving the performance of the baseline model by 1.49% and 2.84%, respectively. When the models were tested with the second input cues including dependent intent descriptions (dependent descriptions), the first intent classification model and the second intent classification model showed performance of 90.97% and 91.75%, respectively, showing results of improving the performance of the baseline model by 0.42% and 1.2%, respectively. It can be seen that when tested based on input cues including intent descriptions, intent descriptions are more effective in explaining the detailed meaning of the user's expression of intent, and the performance of the test results is improved when the model is adjusted based on intent descriptions.

[0155] Also, for the second intent classification model re-learned with dependent intent descriptions, when tested with the first input cues including independent intent descriptions, it showed performance of 86.99%, and in contrast, when tested with the second input cues including dependent intent descriptions, it showed performance of 91.75%, so it can be seen that when tested, improving the quality of input cues including intent descriptions has a greater impact on improving the performance of the model than the training of the model.

[0156] Also, the second intent classification model that is re-learned based on the second input prompt word including the dependent intent description shows higher performance compared to the first intent classification model at the time of testing, and thus it can be seen that the performance of the model can be improved by improving the quality of the input prompt word used for re-learning.

[0157] [Table 2]

[0158]

[0159] On the other hand, in an embodiment, the performance can be evaluated according to the number of intent descriptions (the number of intent options) within the input prompt word used at the time of re-learning and testing of the intent classification model. As shown in [Table 2], testing was performed using a model that was learned after changing the number of intent descriptions in the second input prompt word including the dependent intent description. At this time, performance evaluation according to the quality of the intent description was performed using a third input prompt word based on third data, which is data in which the data including the dependent intent description was qualitatively adjusted by an operator.

[0160] An intent classification model that was re-learned using an input prompt word including five dependent intent descriptions for each intent (5 dependent descriptions), an intent classification model that was re-learned using an input prompt word including one dependent intent description for each intent (1 dependent description), and an intent classification model that was re-learned using an input prompt word including one dependent intent description for each intent, which was manually adjusted (cleansed) by an operator, were prepared, and then each model was tested based on an input prompt word including one dependent intent description for each intent and an input prompt word including one dependent intent description for each intent, which was manually adjusted by an operator. At this time, the input prompt word manually adjusted by the operator can have the highest quality intent description. As shown in [Table 2], the performance of the model learned with one intent description and the model learned with multiple intent descriptions showed little difference, but it can be seen that the model learned with the intent description having higher quality (manually adjusted by the operator) showed the highest performance. Thus, it can be seen that the intent classification model is more affected by the quality of the intent description than the number of intent descriptions.

[0161] On the other hand, with reference to Figure 5The number of intent options (number of intent descriptions) included in the input prompt for relearning of the intent classification model can be optimized. The learning system of one embodiment includes a ranker for adjusting the number of intent options included in the input prompt for relearning. The ranker can calculate the similarity between the user query of the input prompt and the intent description and rank all the intent options in order of low similarity. That is, the ranker performs a function of passing only the top k intents among the intent options ranked in order of low similarity to the intent classification model. Referring to Figure 5 It can be seen that as the number of top intent options k increases, the performance of the intent classification model improves. For example, it can be seen that in the CLINC dataset, the intent classification model exhibits an accuracy (performance) of about 44.21% when k is 1, and as k increases, the performance improves, so that when k reaches 13, an accuracy of about 90% is exhibited. From this result, it can be seen that when relearning or testing of the intent classification model is performed using an input prompt having the top 10 intent options, the best performance is exhibited. For example, in the CLINC dataset including 75 intent options, the length of each intent description is composed of 10 wordpieces, and about 1,200 to 1,300 wordpieces are required, but in the case of using the top 10 intent options using the ranker, the length of the input prompt can be reduced to 300 to 400 wordpieces. That is, the learning system of the present embodiment sets the number of intent descriptions of the input prompt to 10 using the ranker, thereby having the effect of reducing the input size by about 75%, and the intent classification model can obtain the same similar performance.

[0162] As described above, from the learning method and system of the intent classification model using intent descriptions according to an embodiment of the present invention and the evaluation results of the learning system of one embodiment, it can be seen that when testing, the intent classification model relearned based on the intent descriptions of the intent exhibits higher performance when using the prompt based on the intent descriptions, and the higher quality of the intent descriptions in the training and testing of the model has an effect on the improvement of the performance of the model, and the quality of the intent descriptions is more effective than the number of intent descriptions in improving the performance of the model.

[0163] In one embodiment, the processor receives a user expression, and with reference to a database including a plurality of intent candidates and natural language-based descriptions for corresponding intents, evaluates the similarity between the user expression and the natural language-based descriptions, determines the top k intent candidates having high relevance to the user expression based on the evaluation of the similarity, and inputs the determined k intent candidates and the natural language-based descriptions, so that the intent suitable for the user expression can be determined.

[0164] In one embodiment of the present invention, independent intent descriptions, dependent intent descriptions, and cleaned intent descriptions are generated, and these multiple intent descriptions can be provided to the user through an interactive system. In the prompts displayed to the user via the interactive system's display, the independent intent description includes a description of one intent and can simultaneously provide three examples of user questions related to the corresponding intent. In this case, the independent intent description does not include information about other intents, thus explaining the corresponding intent alone without a comparison object. Therefore, the independent intent description cannot adequately reflect the relative differences with other intents, and its quality may be relatively low. The dependent intent description displayed to the user can provide prompts including a list of all intents, generating descriptions of each intent in a way that distinguishes it from other intents before providing them to the user. That is, the generated dependent intent description can clearly distinguish a specific intent from other intents, and therefore may have a relatively higher quality than the independent intent description. Since multiple automatically generated intent descriptions, such as independent intent descriptions and dependent intent descriptions, may not clearly reflect the differences between multiple intents, cleaned intent descriptions can also be generated by the user directly cleaning them.

[0165] In other words, based on the prompts displayed to the user through the interactive system's display, the user can create cleaned intent descriptions. These user-created cleaned intent descriptions are manually created for all intents, making them clearly distinguishable from other intents, and thus achieving a higher quality than automatically generated descriptions.

[0166] That is, in one embodiment of the present invention, independent / dependent intent descriptions generated from the language model can be provided to the user through prompts in the interactive system, and user input can be obtained from the user based on the provided intent descriptions. The user input can be an independent intent description or at least one or more dependent intent descriptions selected for learning or relearning the intent classification model learning system, and can also be a cleaned intent description directly created by the user.

[0167] An interactive system according to an embodiment of the present invention includes a user computing device and a server computing system, and the devices and systems can be connected in a communicable manner through a communication unit.

[0168] In one embodiment, a user computing device can utilize an artificial intelligence model built into the device or an artificial intelligence model provided by a server computing system to perform a process of learning or relearning the artificial intelligence model, or predicting or inferring the characteristics of a target. Furthermore, the server computing device can provide a service predicting target characteristics to the user computing device via an application or webpage, based on requests made by the user through the user computing device.

[0169] The user computing device can include a smartphone, a tablet PC, a wearable device such as a smartwatch, and other all types of computing devices such as a desktop computer. Such a user computing device includes at least one processor and a memory. The memory can include one or more non-transitory / transitory computer readable storage media and combinations thereof, and can include web page saving of a server performing a storage function of the memory on the Internet. Such a memory can store data and instructions required when the at least one processor performs an action of an application for learning / relearning of an artificial intelligence model or prediction of a target characteristic.

[0170] Also, the user computing device can store at least one artificial intelligence model. For example, the user computing device can store a plurality of machine learning models such as neural networks, other types of machine learning models including linear / non-linear models. Also, the user computing device can store a prompt template as an input means utilized in a process of learning / relearning of an artificial intelligence model or prediction of a target characteristic. That is, in an embodiment, the user computing device can perform a target prediction based on data received after a prompt word request execution in a process of learning / relearning of a model or prediction of a target characteristic. Also, for a task requested by the user computing device, a process corresponding to the requested task can be performed by a server computing system through an embedded artificial intelligence model, and a result of the execution can be delivered to the user computing device. Such a user computing device can include at least one user input unit for sensing a user input. For example, the user input unit can include a touch screen sensing a touch of a user input medium such as a finger or a stylus, an image sensor sensing a gesture input of a user, a microphone sensing a voice input of a user, a button, a mouse, a keyboard, and the like.

[0171] The server computing system includes at least one or more processors and a memory, which can be at least one of a central processing device (CPU), a graphic processing device (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, micro-controllers, microprocessors, and / or electrical units for performing other functions, or a plurality of processors electrically connected.

[0172] The memory can include one or more non-transitory / transitory computer-readable storage media and combinations thereof. Such a memory can store at least one or more processors for learning / relearning of an artificial intelligence model or an artificial intelligence model for prediction of a target characteristic, data, and instructions. The memory can store a neural network or a linear / non-linear model, which can include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network, etc. In an embodiment, the server computing system can further include a database, which is a storage for persistently storing and managing original data that becomes a basis for learning of an artificial intelligence model, enhancement data (oversampling data) for improving a prediction performance of an artificial intelligence model, etc. Such a database can include a storage in various manners including a file system and cloud storage. For example, the database can include at least one of a relational database using a structured query language (SQL) for defining and operating data, a NoSQL database designed for flexibility and expansibility and processing non-fixed format and semi-fixed format data, a data warehouse that centrally processes multi-source large-capacity data most suitable for query and analysis as a system for reporting and data analysis, a data warehouse that stores a large amount of original data as a basic form of structured data, semi-structured data, unstructured data, a local storage or a NAS (Network Attached Storage) that generally stores data in a file accessible in a computer operating system, etc. Also, the server computing system can further include other input / output devices for input / output of data, etc.

[0173] The communication section performs wireless or wired communication between the user computing device and the server computing system, and can use a wireless communication system such as eMBB, URLLC, MMTC, LTE, LTE-A, NR, UMTS, GSM, CDMA, WCDMA, TDMA, FDMA, OFDMA, SCFDMA, WiBro, WiFi, Bluetooth, NFC, GPS, or GNSS. Also, the communication section can use various wired communication systems such as USB, HDMI, RS-232 (Recommended Standard-232), POTS (Plain Old Telephone Service), PSTN (Public Switched Telephone Network), xDSL (x Digital Subscriber Line technology), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and LAN (Local Area Network). In an embodiment, the communication section can be configured to not be limited to wired and wireless communication methods, and can include various communication networks such as PAN (Personal Area Network) and WAN (Wide Area Network). Also, the communication network can be a publicly known WWW (World Wide Web), and can use a wireless transmission technology for short-range communication such as IrDA (Infrared Data Association) or Bluetooth.

[0174] The interactive system of an embodiment of the present application can perform an action of specifying a question intention in question data obtained from a user (S310), an action of retrieving information for generating an answer corresponding to the question intention (S320), an action of detecting response data corresponding to the question intention from the retrieved information (S330), and an action of generating an answer to the question intention based on the detected response data (S340).

[0175] The interactive system of an embodiment can receive a user question including a plurality of user intents from a prompt word of a user input / output unit (S210), output intent descriptions corresponding to the plurality of user intents from a language model stored in a memory (S220), provide the output intent descriptions to the user through the prompt word, and thus receive a user input (S230), use a data set including an intent description determined through the user input as an input prompt of an intent classification model, and perform intent classification by the intent classification model (S240). Here, the data set can include first data including independent intent descriptions, second data including dependent intent descriptions, or third data including intent descriptions cleaned by the user. That is, the intent classification model can be re-learned or tested with at least one or more of the first data, the second data, or the third data.

[0176] The interactive system of an embodiment inputs a prompt word provided to the user, acquires a plurality of intent descriptions corresponding to the user intents included in the user question from a language model such as ChatGPT, and can receive a selection or input of an intent description for re-learning of the pre-learned intent classification model from among the plurality of intent descriptions thus acquired from the user. That is, based on a user input including at least one of the plurality of intent descriptions of independent / dependent or cleaned, a data set for re-learning of the intent classification model can be generated.

[0177] To this end, the interactive system of an embodiment can generate instruction prompts for acquiring intent descriptions from a language model using the user intents included in the user question. The instruction prompts can include a prompt for acquiring independent intent descriptions (independent prompt), a prompt for acquiring dependent intent descriptions (dependent prompt), and a prompt for acquiring intent descriptions cleaned by the user (cleaned prompt). The interactive system can display the plurality of generated prompts to the user and can receive a user input from the user.

[0178] Reference Figure 2, the independent prompt word for acquiring the independent intent description provided to the user can acquire first data for one intent including the way of intent description from the language model. That is, the first data acquired by the independent prompt word can include an intent name and an intent description for the corresponding intent name. By including multiple intent names and intent descriptions for the corresponding intent names in the first data acquired by the independent prompt word, it is difficult for the intent classification model to determine or determine the difference between the multiple intent names included in the first data. And the intent description for the intent contained in the independent prompt word does not analyze or compare or determine the context of the intent description for other intents, so the independent intent description can be created independently of other intent descriptions. That is, the independent intent description collected by the independent prompt word lacks a comparison context, so the quality of the intent description is reduced. In an embodiment, in order to acquire multiple independent intent descriptions for one intent name, the independent prompt word can be called multiple times. Referring to Figure 2 , the dependent prompt word for acquiring the dependent intent description provided to the user can acquire second data for multiple intents including the way of intent description from the language model. That is, the second data acquired by the dependent prompt word can include multiple intent names and intent descriptions for the corresponding intent names. By including multiple intent names and intent descriptions for the corresponding intent names in the second data acquired by the dependent prompt word, the intent classification model can determine the difference between the multiple intent names included in the second data. That is, it can be set that the dependent intent description collected by the dependent prompt word includes multiple intents and intent descriptions for multiple intents and each intent description does not contain the upper or lower concept of other intent descriptions. Each intent description generated by the dependent prompt word can be set to not overlap or repeat the content of other intent descriptions. That is, the dependent intent description analyzes or considers or refers to or refers to the content of other intent descriptions, thereby depending on other intent descriptions and can be affected by them. Thus, the multiple intent descriptions generated within the prompt word as much as possible include all intents, and each of the multiple intent descriptions has a unique distinction from each other, so the quality of the intent description of the second data can be higher than that of the first data. In an embodiment, in order to acquire multiple dependent intent descriptions for one intent name, the dependent prompt word can be called multiple times.

[0179] The multiple intent descriptions acquired from the language model in this way can be provided to the user through the prompt word of the interactive system, and the user can select at least one of the independent intent description or the dependent multiple intent descriptions (first data or second data or all).

[0180] On the other hand, in an embodiment of the present application, based on the independent intention specification or the subordinate intention specification, the intention specification after cleaning (third data) of the prompt word after cleaning by the user manually reviewing and adjusting the intention and the intention specification for the corresponding intention can be acquired. The intention specification after cleaning is the intention specification received directly from the user, and the intention specification after cleaning can be received from the user through the prompt word of the interactive system. That is, the third data is the highest quality data for judging the accuracy of the intention classification model, and in order to verify the intention classification model or improve the accuracy of the intention classification model, the third data can also be included in the data set.

[0181] That is, in an embodiment, the data set for relearning of the intention classification model can include at least one or more of the independent intention specification or the subordinate intention specification selected by the user based on the user input or the cleaned multiple intention specifications, such as Figure 3 As shown, the data set thus ensured is set as an input prompt word mode input intention classification model for relearning of the intention classification model.

[0182] The intention classification model that relearns based on the newly input data set can further accurately grasp the intention contained in the user's question and thus more accurately grasp or specify the user's question intention.

[0183] As described above, the action of retrieving information for generating an answer according to the grasped or specified user's question intention (S320) can include retrieval of professional files or documents included in the database of the interactive system, or external databases (for example, a DB storing academic papers or report files) connected through the communication unit, web browsing performed through the communication unit in the interactive system, etc. The answer response model executed by the interactive system of an embodiment can retrieve professional files, etc. including contents corresponding to the user's question intention specified by the intention classification model through online / offline retrieval through the database or the web, etc. At this time, the retrieved files or contents can be retrieved based on the question intention of the user's question data, the intention name, the independent / subordinate / cleaning intention specification selected by the user after being output from the language model. For example, the intention name included in the user's question data, the keyword extracted from the selected intention specification, the question intention determined from the intention classification model, or the professional file corresponding to the intention category can be ensured through database retrieval and web browsing.

[0184] The action of detecting response data corresponding to the question intention from the retrieved information (S330) can be performed by an answer response model, which is an artificial intelligence model that takes at least one or more of the retrieved information (for example, a paper, a report, or a website, etc.), the question data, etc. as input and outputs at least one response data based on the input intention specification and the question data.

[0185] In an embodiment, such an answer response model implements reasoning for a phased process (in an embodiment, a multi-stage reasoning process) for advanced reasoning and inference functions when input is an intent specification as well as a question, thereby being able to provide high-quality response data for the input question. In an embodiment, the answer response model can be a large language model (LLM) or the like that is pre-trained based on a training data set including a plurality of files in order to perform the above-described functional actions.

[0186] In an embodiment, the answer response model can generate response data for a predetermined user question based on a multi-stage reasoning process. The multi-stage reasoning process can represent a structured process of generating response data based on question data. Such a multi-stage reasoning process can include a relevance selection process, a context generation process, and a system configuration process.

[0187] The relevance selection process can determine the evidence passage from the intent specification related to the question data (S331). The answer response model can extract at least one or more passages, i.e., evidence passages, including the answer to the question data and / or the basis from the intent specification used in the intent classification model, i.e., the passage selected by the user or included in the cleaned intent specification. At this time, in an embodiment, the answer response model can perform the above-described relevance selection process based on a prompt engineering algorithm. For reference, the prompt engineering can mean a process of making and optimizing a prompt word in order to obtain a desired result from an artificial intelligence model. Among them, the prompt word can mean a structured text input in order to generate an output from an artificial intelligence model. Such a prompt word can guide the artificial intelligence model to retrieve specific content from the learned data so as to output a result that meets the target. Illustratively, the constituent elements of the prompt word can include an "instruction" that specifies an operation or a guideline desired to be performed by a deep learning model, a "context information" that specifies external information or additional context capable of adjusting the deep learning model, an "input data" that specifies an input or a question for finding an answer, and / or an "output data" that specifies an output type or form, etc. That is, in an embodiment, the answer response model can perform a relevance selection process of generating a structured prompt word (hereinafter referred to as a professional professional question prompt word) according to a predetermined intent specification and question data based on a prompt engineering algorithm, and extracting at least one evidence passage within the corresponding intent specification based on the generated question prompt word. Accordingly, the answer response model can extract at least one evidence passage from the intent specification based on a more explicit explanation of the question data, such as a further accurate understanding of the question intent within the question data and the premises contained therein, etc. In this way, the answer response model significantly improves the explanation performance for the question data, and thus can also directly improve the quality of the response data generated thereafter.

[0188] In an embodiment, the answer response model can acquire basis data for the question data based on the determined evidence passage (S332). The basis data can represent data that serves as a basis for generating response data for the question data. That is, in an embodiment, the basis data can be a collection of various data utilized in generating the response data. In an embodiment of the present disclosure, the basis data can be information collected through retrieval of files or documents included in the database of the interactive system retrieved in the above-described action of retrieving information for generating an answer corresponding to the question intent (S320), or through retrieval of files or documents of an external database (e.g., a DB storing academic papers or report files) connected through the communication unit, web browsing performed in the interactive system through the communication unit, etc. In an embodiment, the answer response model can detect main answer data including a direct answer to the question data, explanatory article data explaining the direct answer, and / or supplementary information data including related background knowledge from the retrieved information based on at least one evidence passage. In addition, the answer response model can acquire basis data as described above based on the detected plurality of data.

[0189] Also, in an embodiment, the answer response model can generate response data based on the acquired basis data (S333). That is, in an embodiment, the answer response model performs data processing based on the acquired basis data, and thus can generate response data for the user question. In an embodiment, the answer response model can perform predetermined data processing (e.g., removal of duplicate text, etc.) for enhancing the conciseness and readability of the answer based on the basis data, and thus generate an answer (S340).

[0190] As described above, in an embodiment of the present disclosure, the interactive system explicitly identifies the user question intent based on the intent explanation for the user question intent, extracts a basis passage that can be an answer to the question data from the intent explanation selected by the user or cleaned by the user, generates evidence basis based on the content of the extracted basis passage, and based on the generated evidence basis, can provide response data that can provide a more in-depth answer. That is, in an embodiment, the interactive system can resolve the limitations, i.e., hallucination and answer ambiguity, that the existing interactive model (e.g., OpenAI GPT, etc.) has when performing a deep question based on the intent explanation and a response task therefor. Also, according to the interactive system of an embodiment of the present disclosure, based on the advanced reasoning (Chain-of-thought reasoning) and inference function implemented through a process similar to human cognitive reasoning, for a deep question related to the question intent and the intent explanation, it is possible to provide answer data that is faithful to the question, based on facts, and gives clear evidence.

[0191] The interactive system of an embodiment of the present application can provide the generated response data to the user through an output device of the user computing device or the server computing system. Specifically, an application program executed in the user computing device or the server computing system can provide the user with at least one response data acquired based on the user's query data.

[0192] In an embodiment of the present application, the application program can provide an answer output interface based on the intelligent interactive system. In this regard, the answer output interface according to the embodiment can represent a user interface in which predetermined response data and various data and information associated therewith are visually displayed in a predetermined manner. Also, in the embodiment, the application program can display the generated response data to the user based on the provided answer output interface.

[0193] In this regard, in an embodiment, the application program can match and display at least one basis information corresponding to each of the provided response data in association with the corresponding response data. The basis information can represent data and / or information used as a basis for generating the predetermined response data. For example, the basis information can include text data of an intention explanation corresponding to the predetermined response data, retrieved professional document identification information (e.g., paper number and / or paper title, etc.), evidence paragraph data, and / or basis data, etc. Also, in an embodiment, the basis information can further include query data corresponding to the predetermined response data. In an embodiment, the application program can display at least one basis information corresponding to each of the response data. As an embodiment, the application program can detect at least one professional document data related to the generation of each of the response data, professional document identification information for the professional document data, evidence paragraph data, evidence paragraph identification information for the evidence paragraph data, and / or basis data. Also, in an embodiment, the application program can match the detected basis information to each of the response data, and display the same in association with each of the response data in a predetermined manner (e.g., list, image, and / or text, etc.) through the answer output interface.

[0194] On the other hand, an embodiment of the present application can be implemented as an application specific integrated circuit (ASIC; Application Specific integrated circuit) customized for a specific application field and a special function of a device.

[0195] The application-specific integrated circuit can also be referred to as a custom semiconductor, which is different from a standard semiconductor that has specifications that can be applied to any electronic product or application as long as certain conditions are met, and is an integrated circuit manufactured by a semiconductor manufacturer according to a specific order. That is, a custom semiconductor is designed and manufactured to perform only the functions required for a specific device or a specific function, and is generally classified into a full custom IC designed and manufactured from the beginning according to user requirements and a semi-custom IC designed and manufactured using a part of the standardized design, according to the design method.

[0196] The custom semiconductor is mainly used in communication systems, high-performance computing systems, consumer electronics, automobiles, industrial automation, medical devices, military, aerospace industries, etc., and recently, in AI semiconductors required for large-scale operations to implement AI with high performance and high power efficiency.

[0197] In communication systems, custom semiconductors are used as core constituent elements of network routers, switches, modems, etc., and perform data packet processing, protocol conversion, signal processing, etc., thereby providing high throughput and low latency. In high-performance computing systems, ASICs are used as core constituent elements to implement high-speed processing and parallel processing, and in consumer electronics such as digital cameras, smartphones, tablet PCs, game consoles, etc., ASICs provide solutions for high performance and low power consumption required to perform specific functions. In the automotive industry, ASICs are used to control various electronic systems inside the car, and in industrial automation systems, ASICs provide solutions for high-precision control and high-performance processing.

[0198] The application-specific integrated circuit to which the embodiment of the present application is applied includes a memory in which an independent memory interface (I / F) is implemented, and can include a plurality of functional blocks that request memory access. Each functional block can be a direct memory access (DMA) functional block, a processor, a video processor, a cache controller, a decompression block, a data path block. The basic constituents of the application-specific integrated circuit can include transistors that amplify or switch electrical signals, circuits that combine transistors to perform logic functions, i.e., logic gates, memory units that store data, analog circuits that combine transistors to process continuous voltage or current, microprocessors, DSPs, graphic cores, etc., IP cores (Intellectual Property Core) that are designed in advance to perform specific functions, etc.

[0199] The ASIC can also include a separate memory I / F that interfaces with the separate memory and a built-in memory I / F that interfaces with the built-in memory, wherein the separate memory I / F is connected to each of the functional blocks so as to be able to receive memory access signals (e.g., control signals, address signals, and data signals) and generate signals for controlling the separate memory based on these input signals. The built-in memory I / F is connected to each of the functional blocks, receives memory access signals (e.g., control signals, address signals, data signals), and based on these input signals, can generate modified memory access signals for controlling the built-in memory. The separate memory I / F and the built-in memory I / F are designed inside the memory control block of the ASIC, whereby a memory control structure that can be flexibly applied to both the separate memory and the built-in memory can be provided.

[0200] Also, an application specific integrated circuit (ASIC) for an artificial neural network (ANN) is composed of a plurality of neurons and a plurality of synapse circuits configured in an array, each neuron is composed of a register, a microprocessor, and at least one input, and each synapse circuit can be configured to include a memory for storing a synaptic weight. Wherein each neuron of the ASIC can be connected to at least one other neuron through one of the plurality of synapse circuits.

[0201] While it is generally possible to implement the present disclosure by a computing device as explained above, it should be understood by those skilled in the art that the present disclosure can be implemented by computer executable instructions executable on more than one computer and / or in combination with other program modules and / or by combining hardware and software.

[0202] Those skilled in the art of the technical field to which this disclosure pertains will understand that information and signals can be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0203] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps associated with the embodiments disclosed herein can be implemented by electronic hardware, various forms of program or design code (referred to herein as software for convenience), or combinations thereof. To clearly illustrate this hardware and software interoperability, the functionality of various exemplary components, blocks, modules, circuits, and steps has been generally described above. Whether such functionality is implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art can implement the functionality described in various ways for each specific application, but such implementation decisions should not be construed as exceeding the scope of this disclosure.

[0204] The various embodiments described herein can be implemented by methods, apparatus, or articles employing standard programming and / or engineering techniques. The term "article" includes computer programs, carriers, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, memory cards, flash drives, key drives, etc.). Furthermore, the various storage media disclosed herein include one or more devices for storing information and / or other mechanically readable media.

[0205] It should be understood that the specific order or hierarchical structure of the steps in the proposed process is merely one example of an exemplary method. It should be understood that, based on design priorities, the specific order or hierarchical structure of the steps in the process can be rearranged within the scope of this disclosure. The appended method claims provide the elements of each step in a sample order, but this does not imply limitation by the proposed specific order or hierarchical structure.

[0206] The description of the disclosed embodiments is provided to enable those skilled in the art to utilize or implement this disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the various embodiments disclosed herein, and should be interpreted within the broadest scope of the principles and novel features disclosed herein.

Claims

1. A system comprising: At least one processor; as well as At least one memory that stores instructions or information executed in the aforementioned at least one processor. The actions performed by the above instructions or information executed by at least one of the above processors include: The user input / output unit receives user queries that include multiple user intentions. The language model stored in the above memory outputs actions corresponding to multiple user intents; The action of providing the aforementioned multiple intent descriptions stored in the aforementioned memory to the aforementioned user through the aforementioned user input / output unit and receiving user input; and The intent description determined through the user input is used as an input prompt word for an intent classification model executed by at least one processor, and the intent classification model performs the intent classification action. The aforementioned multiple intent descriptions include at least one of the following: an independent intent description created independently of other intent descriptions, a subordinate intent description created in a manner dependent on other intent descriptions, or a cleaned intent description entered by the user through the aforementioned user input section after being cleaned by the user based on an independent intent description or a subordinate intent description.

2. The system according to claim 1, wherein, The instruction prompts input into the language model above in order to output the above multiple intent descriptions include: Independent prompt words, which acquire first data including multiple intent descriptions for an intent; Subordinate prompt words, which acquire second data including multiple intent descriptions for multiple intents and analyze the context of each of the multiple intents and the multiple intent descriptions; and The cleaned prompts are based on the aforementioned independent intent statement or the aforementioned subordinate intent statement, and are obtained by user input to acquire third-party data.

3. The system according to claim 1, wherein, The input prompt consists of an instruction and user query section, an intent option section including the intent name and multiple intent descriptions for the intent name.

4. The system according to claim 3, wherein, Also includes: The sorter adjusts the number of the aforementioned intent options. The sorter calculates the similarity between the user query and the intent description.

5. The system according to claim 1, wherein, The above intent classification model is relearned using at least a portion of the above independent intent description, the above subordinate intent description, or the above cleaned intent description.

6. The system according to claim 2, wherein, When determining the performance of the above intent classification model, at least a portion of the first data is provided as input prompt words, or when determining the performance of the above intent classification model, at least a portion of the second data is provided as input prompt words.

7. A method executed by at least one processor, the method comprising: The steps involve receiving user questions, including multiple user intentions, through the user input / output unit; The steps involve using a language model stored in memory to output multiple intent descriptions corresponding to the aforementioned multiple user intents. The steps of providing the user with the multiple intent descriptions stored in the memory and receiving user input through the user input unit; as well as The step of classifying intent by using at least one intent description determined through the aforementioned user input as an input prompt word for an intent classification model executed by at least one or more processors. The aforementioned multiple intent descriptions include at least one of the following: an independent intent description created independently of other intent descriptions, a subordinate intent description created in a manner dependent on other intent descriptions, or a cleaned intent description entered by the user through the aforementioned user input section after being cleaned by the user based on an independent intent description or a subordinate intent description.

8. The method according to claim 1, wherein, In the steps of outputting the above-mentioned multiple intent descriptions The instruction prompts are input into the language model to generate a dataset containing the multiple intent descriptions. The instruction prompts include: independent prompts, which acquire first data including multiple intent descriptions for a single intent; dependent prompts, which acquire second data including multiple intent descriptions for multiple intents and analyze the context of each of the multiple intents and the multiple intent descriptions; and cleaned prompts, which acquire third data based on user input of either the independent intent description or the dependent intent description.

9. The method according to claim 7, wherein, In the steps of performing intent classification by the intent classification model described above The input prompt consists of an instruction and user query section, an intent option section including the intent name and multiple intent descriptions for the intent name.

10. The method according to claim 9, wherein, A sorter that adjusts the number of intent options calculates the similarity between the user query and the intent description, and sorts the multiple intent descriptions in descending order based on the similarity.

11. The method according to claim 7, wherein, In the step of classifying intent by the intent classification model described above, the intent classification model is relearned using at least a portion of the independent intent description, the subordinate intent description, or the cleaned intent description.

12. The method according to claim 8, wherein, Also includes: The steps to determine the performance of the above intention classification model are as follows: In the step of determining the performance of the above-mentioned intent classification model, at least a portion of the first data is provided as input prompt words when determining the performance of the above-mentioned intent classification model, or at least a portion of the second data is provided as input prompt words when determining the performance of the above-mentioned intent classification model.

13. A system comprising: At least one processor; as well as At least one memory that stores instructions or information executed by the aforementioned at least one processor. The actions performed by the above instructions or information executed by at least one of the above processors include: The user intent, obtained from the prompts of the user input / output unit, is input into the language model stored in the aforementioned memory, thereby generating multiple intent descriptions for the aforementioned user intent and specifying the action of the question intent; The action of retrieving information to generate an answer corresponding to the aforementioned question intent; Actions that detect response data corresponding to the aforementioned question intent from retrieved information; and Based on the detected response data, an action is generated to answer the aforementioned question intent.

14. The system according to claim 13, wherein, The action specifying the aforementioned question intent is generated by the aforementioned language model as either an independent intent description created independently of other intent descriptions or a subordinate intent description created in a manner dependent on other intent descriptions.

15. The system according to claim 14, wherein, The action specifying the above-mentioned question intent outputs the above-mentioned independent intent description and the above-mentioned subordinate intent description through the above-mentioned user input / output unit. The user input / output unit selects at least one of the above-mentioned independent intent description or the above-mentioned subordinate intent description, or the user input / output unit inputs a cleaned intent description cleaned by the user based on the independent intent description or the subordinate intent description. The intent is classified by the intent classification model according to the selected or input intent description.

16. The system according to claim 13, wherein, The act of retrieving the above information involves searching a database or the Internet for information related to the aforementioned user intent and the aforementioned multiple intent descriptions.

17. The system according to claim 13, wherein, The actions for detecting the above response data include: Identify the actions in the evidence paragraphs associated with the aforementioned questionable data from the above statement of intent; Based on the aforementioned evidentiary paragraphs, actions are taken to obtain the supporting data for the aforementioned questionable data; and Based on the aforementioned data, an action is taken to generate response data to the aforementioned question data.

18. The system according to claim 17, wherein, The data mentioned above is based on documents and websites collected from databases or the Internet.

19. A method executed by at least one processor, the method comprising: The step involves storing the user intent data obtained from prompts by the user input / output unit of the interactive system into the language model stored in the aforementioned memory, generating multiple intent descriptions for the aforementioned user intent, and then specifying the question intent. The step of retrieving information from a database or the internet to generate an answer corresponding to the aforementioned question intent; The steps involved in detecting response data that corresponds to the aforementioned question intent from the retrieved information; as well as The step of generating an answer to the above-mentioned question intent based on the detected response data.

20. The method according to claim 19, wherein, In the step of specifying the aforementioned question intent, the language model generates an independent intent description created in a manner consistent with other intent descriptions, or a subordinate intent description created in a manner consistent with other intent descriptions.

21. The method according to claim 20, wherein, The steps to specify the above question intent include: The steps of outputting the above independent intent description and the above subordinate intent description through the above user input / output unit; The user input / output unit selects at least one of the independent intent description or the subordinate intent description, or inputs a cleaned intent description based on the independent intent description or the subordinate intent description, through the user input / output unit; and The steps for classifying intents by an intent classification model, based on the selected or input intent description.

22. The method according to claim 19, wherein, The steps for detecting the above response data include: The steps for identifying the evidence paragraphs associated with the aforementioned questionable data in the above statement of intent; Based on the aforementioned evidentiary paragraphs, the steps for obtaining the supporting data for the aforementioned questionable data; and Based on the aforementioned data, the steps for generating response data to the aforementioned question data are as follows: The data mentioned above is based on documents and websites collected from databases or the Internet.

Images