Method and apparatus for automatically training speech recognition system

Abstract

The invention relates to a method and an apparatus for automatically training a speech recognition system. In a method and apparatus for automatically training a speech recognition system, the speech recognition system comprising one or more NLU engines, the method comprising: obtaining NLU results output by the one or more NLU engines; generating a hint for the large-scale language model based on comparing between the NLU results; determining whether the NLU result is appropriate by utilizing the generated hint for the large-scale language model; the speech recognition system is trained by using the determination result.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present application claims priority to Korean Patent Application Nos. 10-2023-0136176 and 10-2024-0101042 filed on Oct. 12, 2023 and Jul. 30, 2024, respectively, which are hereby incorporated by reference in their entirety for all purposes by this reference. Technical Field

[0003] The invention relates to a method and a device for automatically training a speech recognition system. Background Art

[0004] The descriptions in this section merely provide background information related to the present invention and do not constitute related art.

[0005] With the recent development of artificial intelligence technology, the scope of application of artificial intelligence is also expanding. Dialogue systems (such as chatbots or virtual assistants) that can talk with users using natural language are used in various fields. In order for the dialogue system to talk with the user, it is necessary to understand the user's words from the perspective of the dialogue system, in other words, to understand the input message. In order to achieve current natural language understanding (NLU), the dialogue system needs to derive the current context and the user's intended intention in the context from the dialogue between the dialogue system and the user, and analyze the input message based on the determined current context and / or intention.

[0006] The application range of these voice recognition services has expanded from homes to various fields such as vehicles. In addition, telematics technology includes various functions. Examples include real-time navigation functions, information search functions using the Internet, and functions such as optimizing the in-vehicle environment by utilizing vehicle location and weather information.

[0007] Speech recognition systems are widely used in daily life, for example in smartphones, smart speakers, and in-car infotainment systems. To build a natural language understanding system for analyzing the intent of utterances, it is necessary to pre-design a set of expected user intents, collect sentences, and train an intent classification model. However, pre-defined intents alone may not fully meet the changing needs of users. Accordingly, the intent classification model needs to be extended by iteratively incorporating new intents discovered in unlabeled user utterances.

[0008] To scale intent classification models, existing methods require human evaluators to select data and collect sentences. Current methods require a manual labeling process and are time-consuming and expensive. In addition, there is a problem of difficulty in obtaining consistent results when multiple people evaluate. In addition, speech recognition systems themselves contain errors, making it difficult to find misclassifications or detect new intents from speech recognition logs. In addition, even when training speech recognition and natural language processing models, users' speech patterns change over time, degrading model performance. Accordingly, methods and systems are needed to detect misclassifications and new intents and improve the system.

[0009] The information included in this background of the invention is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgment or any form of suggestion that this information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0010] Various aspects of the present invention are directed to providing a method and apparatus for automatically training a speech recognition system, which automatically trains the speech recognition system by discovering misclassifications and deriving new intents based on the results of the speech recognition system classifying intents.

[0011] Aspects of the present invention are not limited to those mentioned above, and other aspects not mentioned herein will be clearly understood by those skilled in the art from the following description.

[0012] According to one aspect of the present invention, there is provided a method for automatically training a speech recognition system, the speech recognition system comprising one or more NLU engines, the method comprising: obtaining NLU results output by the one or more NLU engines; generating prompts for a large-scale language model based on comparison between the NLU results; determining whether the NLU results are appropriate by utilizing the generated prompts for the large-scale language model, and training the speech recognition system by utilizing the determination results.

[0013] According to another aspect of the present invention, there is provided an apparatus for automatically training a speech recognition system, the speech recognition system comprising one or more NLU engines, the apparatus comprising a memory and one or more processors, the memory being configured to store one or more instructions, the one or more processors being operably connected to the memory and configured to execute the one or more instructions stored in the memory, wherein the one or more processors perform steps by executing the one or more instructions, the steps comprising: obtaining NLU results output by the one or more NLU engines; generating prompts for a large-scale language model based on comparison between the NLU results; determining whether the NLU results are appropriate by utilizing the generated prompts for the large-scale language model; and training the speech recognition system by utilizing the determination results.

[0014] According to an exemplary embodiment of the present invention, the speech recognition system can be automatically trained by discovering misclassifications and deriving new intents based on the results of intent classification.

[0015] According to an exemplary embodiment of the present invention, a user's continuously changing intentions may be detected by automatically training a speech recognition system.

[0016] The methods and apparatus of the present invention have other features and advantages that will be apparent from, or will be described in detail in, the accompanying drawings and subsequent detailed descriptions incorporated herein, which together serve to explain the specific principles of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram illustrating a speech recognition system.

[0018] Figure 2 is a schematic block diagram schematically illustrating an automatic learning apparatus based on a large-scale language model according to an exemplary embodiment of the present invention.

[0019] Figure 3 is a flow chart illustrating a process for detecting a new intent according to an exemplary embodiment of the present invention.

[0020] Figure 4 is a flow chart illustrating a process of determining an error classification according to an exemplary embodiment of the present invention.

[0021] Figure 5 is a flowchart of an intention analysis method according to an exemplary embodiment of the present invention.

[0022] Fig. 6A and Figure 6B is a schematic diagram illustrating a method for performing effective hinting within a limited input size for a large-scale language model.

[0023] Figure 7 is a schematic block diagram schematically illustrating an example of an exemplary computing device that can be used to implement a method or apparatus according to an exemplary embodiment of the present invention.

[0024] It should be understood that the accompanying drawings are not drawn to scale, but rather present a rather simplified representation of various features to illustrate the basic principles of the present invention. The specific design features of the present invention as included herein (including, for example, specific dimensions, orientations, locations, and shapes) will be determined in part by the specific intended application and use environment.

[0025] In the figures, reference numbers refer to the same or equivalent parts of the present invention throughout the several figures of the drawing. DETAILED DESCRIPTION

[0026] Reference will now be made in detail to various embodiments of the present invention, examples of which are shown in the accompanying drawings and described below. Although the present invention will be described in conjunction with exemplary embodiments of the present invention, it should be understood that this description is not intended to limit the present invention to those exemplary embodiments of the present invention. On the other hand, the present invention is intended to cover not only the exemplary embodiments of the present invention, but also various alternatives, modifications, equivalents and other embodiments that may be included within the spirit and scope of the present invention as defined by the appended claims.

[0027] In addition, unless otherwise specifically and clearly defined and explained, the terms (including technical terms and scientific terms) used in the exemplary embodiments of the present invention can be interpreted as the meanings that can be generally understood by those of ordinary skill in the art to which the present invention belongs. The meanings of commonly used terms (such as terms defined in dictionaries) can be interpreted based on the contextual meanings of the relevant technology.

[0028] Furthermore, the terms used in the exemplary embodiments of the present invention are for explaining the embodiments rather than for limiting the present invention.

[0029] Hereinafter, some exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals represent the same elements, although these elements are shown in different drawings. In addition, in the following description of various exemplary embodiments of the present invention, for the sake of clarity and brevity, the detailed description of the known functions and configurations incorporated therein will be omitted.

[0030] In addition, various terms such as first, second, A, B, (a), (b), etc. are used only to distinguish one component from another component, rather than to teach or imply the substance, order, or sequence of the components. Throughout this specification, when a component "includes" or "contains" a component, unless otherwise specifically stated, the component is intended to further include other components, rather than exclude other components. Terms such as "unit", "module", etc. refer to one or more units for processing at least one function or operation, and one or more units can be implemented by hardware, software, or a combination thereof.

[0031] The following detailed description in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the exemplary embodiments of the present invention may be practiced.

[0032] Figure 1 is a schematic diagram illustrating a speech recognition system.

[0033] Reference Figure 1 , the speech recognition system 100 recognizes the user's speech, understands the recognized speech, and provides a service corresponding to the user's speech.

[0034] To this end, the speech recognition system 100 includes a speech recognizer 110, a natural language understander 120, and a result processor 130, wherein the speech recognizer 110 converts a user's speech utterance into text, the natural language understander 120 is configured to determine the intent and entities included in the user's speech utterance, and the result processor 130 is configured to perform processing to provide a result corresponding to the user's intent and a result corresponding to the entity.

[0035] The speech recognizer 110 may utilize at least one speech recognition engine to convert the user's speech into an input sentence. In this document, the speech recognition engine may refer to a speech to text (Speech to Text, STT) engine, and by applying a speech recognition algorithm or a neural network model to a speech signal representing the user's speech, the speech signal may be converted into text.

[0036] For example, the speech recognizer 110 may extract a feature vector from the user utterance by applying a feature vector extraction technique such as cepstrum, linear predictive coefficient (LPC), Mel Frequency Cepstral Coefficient (MFCC), or filter bank energy.

[0037] The speech recognizer 110 can obtain the recognition result by comparing the extracted feature vector with the trained reference pattern. To this end, an acoustic model that models and compares the signal characteristics of the speech, or a language model that models the word order relationship of the words or syllables corresponding to the recognition vocabulary can be used.

[0038] The speech recognizer 110 may convert the user utterance into text based on a model to which machine learning or deep learning is applied.

[0039] In an exemplary embodiment of the present invention, the speech recognition result represents text obtained using a speech recognition engine.

[0040] The natural language understander 120 is configured to use at least one natural language understanding (NLU) engine to determine at least one of the user intent or entity included in the input sentence. In an exemplary embodiment of the present invention, the intent classifier is a language model that uses the natural language understander 120 to understand the user intent from the user's input.

[0041] The natural language understander 120 can extract information such as domain, entity name, and language behavior from the input sentence using an NLU engine, and recognize intent and entities according to the intent based on the extraction result. An entity can be called a word slot.

[0042] The NLU engine may segment the input sentence into morphemes, project the morphemes into a vector space, group the projected vectors, thereby classifying the intent according to the input sentence, and the NLU engine extracts word components as entities according to the intent within the input sentence.

[0043] In an exemplary embodiment of the present invention, the NLU result refers to at least one of the intents or entities obtained using the NLU engine.

[0044] The result processor 130 may output a result signal to the vehicle, the user device, or an external server to perform processing to provide a service corresponding to the user's intention.

[0045] For example, when the service corresponding to the user's intention is vehicle-related control, the result processor 130 can send a result signal for executing vehicle-related control to the vehicle. As an exemplary embodiment of the present invention, when the service corresponding to the user's intention is to provide specific information, the result processor 130 can search for the specific information and provide the searched information to the user terminal. If necessary, information retrieval can also be performed on an external server. As an exemplary embodiment of the present invention, when the service corresponding to the user's intention is to provide specific content, the result processor 130 can request to send the target content to an external server that provides the content. In another example, when the service corresponding to the user's intention is a continuation of a simple conversation, the result processor 130 can be configured to generate a response to the user's speech and output the response in a visual or auditory manner.

[0046] The speech recognition system 100 may be provided in an external server or a user terminal, and some of its components may be provided in an external server, while other components may be provided in a user terminal. The user terminal may be a mobile device such as a smart phone, a tablet PC or a wearable device, a home appliance or a vehicle having a user interface.

[0047] The speech recognition system 100 may further include a dialog manager that manages all dialogs between the speech recognition system 100 and a user.

[0048] The components of the speech recognition system 100 are classified based on their operation or function, and all or part of them can share memory or processor. The speech recognition system 100 can be implemented in a vehicle or a server. In addition, some of the components of the speech recognition system 100 can be included in the vehicle, while other components can be included in the server. For example, the vehicle sends a passenger's voice signal to the server, the server processes the passenger's voice signal, generates information or control instructions required by the passenger, and sends the information or control instructions to the vehicle.

[0049] Figure 2 is a schematic block diagram schematically illustrating an automatic learning apparatus based on a large-scale language model (LLM) according to an exemplary embodiment of the present invention.

[0050] The automatic learning device 200 includes a memory and a processor.

[0051] The automatic learning device 200 according to an exemplary embodiment of the present invention may include all or part of a misclassification candidate group selection unit 220 , a new intention estimation unit 230 , a prompt generation unit 240 , an automatic determination unit 250 , and a re-learning unit 260 . Figure 2 Not all of the blocks shown in FIG. 2 are essential components, and some of the blocks included in the automatic learning device 200 may be added, changed, or deleted. Figure 2 The constituent components shown in the figure represent functionally different elements, and at least one constituent component may be implemented in an integrated form in an actual physical environment.

[0052] The memory stores data and instructions required for the operation of the automatic learning device 200 .

[0053] The processor is configured to control all operations of the automatic learning device 200. The processor may be implemented as one or more processors. The processor may execute instructions stored in the memory.

[0054] The processor may include an intention determination unit 210 , an erroneous classification candidate group selection unit 220 , a new intention estimation unit 230 , a hint generation unit 240 , an automatic determination unit 250 , and a re-learning unit 260 .

[0055] In an exemplary embodiment of the present invention, the large-scale language model may be included in and utilized within the automatic learning apparatus 200, or the large-scale language model may be provided outside the automatic learning apparatus 200 and used to exchange data through communication with a server.

[0056] In an exemplary embodiment of the present invention, a domain may receive user utterances, and may include "utterances that can be processed, i.e., In-Domain (IND)" and "utterances that are difficult to process, i.e., Out-of-Domain (OOD)". IND refers to data or input that the system can process within the scope of training. This is data that the system can accurately understand and process based on trained data within a predefined domain. OOD refers to data or input that is beyond the scope of system training. OOD refers to data that includes new domains, patterns, or intents that the system is not currently trained for. The system may have difficulty responding appropriately to inputs that include OOD, and it may be difficult to predict accurately.

[0057] The intent determination unit 210 may include an intent classifier or a large-scale language model. The intent determination unit 210 may receive the user's log data and distinguish IND from ODD. The task of distinguishing IND from ODD may be performed using an intent classifier or a large-scale language model. In order to form the difference between IND and ODD, the intent determination unit 210 may train the intent classifier or the large-scale language model using labeled data.

[0058] The intent determination unit 210 may assign temporary labels to unlabeled data (pseudo labels) using a trained intent classifier or a trained large-scale language model.

[0059] Figure 1 The NLU engine may include an intent classifier or a large-scale language model of the intent determination unit 210. The NLU result may include IND and OOD datasets classified by the intent determination unit 210.

[0060] The error classification candidate group selection unit 220 can select the error classification candidate group using the intent classifier set. The intent classifier set refers to a method of predicting intent by connecting one or more intent classifiers in parallel. For example, for one utterance, different versions of intent classifiers can be executed in parallel to predict various intents.

[0061] The misclassification candidate group selection unit 220 may include one or more intent classifiers, wherein these intent classifiers are intent classifiers pre-trained using different methods.

[0062] In an exemplary embodiment of the present invention, a method for training an intent classifier may include: a first training method, which trains one or more models by changing the discard rate of models of the same type; a second training method, which uses the same algorithm but sets learning parameters (initialization, number of training rounds, batch size, etc.) differently to train one or more models; a third training method, which uses different algorithms such as BERT, ALBERT, ELECTRA, and RoBERTa to train one or more models; and a fourth training method, which includes common basic intents within the same entire intent set dataset but configures other intents differently to train one or more models.

[0063] For example, the misclassification candidate group selection unit 220 can select important intents from the entire intent set. In this article, important intents refer to intents common to all data sets. The misclassification candidate group selection unit 220 can be configured to generate three intent sets n1, n2, and n3 based on the entire intent set N. The entire intent set N is a set of all intents that the system can understand and process. Intent sets n1, n2, and n3 contain important intents, but each of intent sets n1, n2, and n3 contains other less important intents. The misclassification candidate group selection unit 220 can use intent sets n1, n2, and n3 as data sets to train one or more models, respectively.

[0064] The misclassification candidate group selection unit 220 may connect one or more trained intent classifiers in parallel.

[0065] The error classification candidate group selection unit 220 may connect one or more intent classifiers trained using, for example, the first training method in parallel. The error classification candidate group selection unit 220 may connect one or more intent classifiers trained using, for example, the second training method in parallel. The error classification candidate group selection unit 220 may connect one or more intent classifiers trained using, for example, the third training method in parallel. The error classification candidate group selection unit 220 may connect one or more intent classifiers trained using, for example, the fourth training method in parallel.

[0066] The misclassification candidate group selection unit 220 may be configured to use the trained intention classifier to predict the intention of the utterance according to the same input sentence. In addition, the prediction results of each intention classifier may be collected.

[0067] The misclassification candidate group selection unit 220 may evaluate the diversity of prediction results and select a misclassification candidate group that may cause an error.

[0068] The error classification candidate group selection unit 220 may select an error-prone error classification candidate group from the results of different intention classifiers.

[0069] In an exemplary embodiment of the present invention, a method for selecting a candidate group for misclassification may include: a first selection method for finding sentences with a high possibility of misclassification by sorting the sentences in the order of the most different prediction results; a second selection method for finding sentences with a high possibility of misclassification by grouping and sorting the sentences with the most different prediction results in the order of the highest frequency of intentions; and a third selection method for finding sentences with a high possibility of misclassification by analyzing the similarity between sub-words of the prediction results and sorting the sentences in order of decreasing similarity.

[0070] For example, the misclassification candidate group selection unit 220 is configured to determine that the more different the intents predicted by each intent classifier connected in parallel are, the higher the diversity is. The misclassification candidate group selection unit 220 searches for sentences with the most diverse prediction results and selects the sentences with the most diverse results as misclassification candidate groups.

[0071] The new intention estimation unit 230 may perform contrastive learning and clustering according to the data set classified as OOD by the intention determination unit 210 .

[0072] Contrastive learning is a method of embedding data points in a vector space and training to place similar data closer and different data farther apart. The new intent estimation unit 230 can use semi-supervised contrastive learning to train a large-scale language model for estimating new intents. Semi-supervised contrastive learning refers to contrastive learning using labeled data and unlabeled data together.

[0073] The new intent estimation unit 230 may utilize a large-scale language model to discover new intents.

[0074] In an exemplary embodiment of the present invention, a greedy method based on a pool sampling method is used. Pool-based sampling is a method of training a model by selecting part of the data from a large pool of unlabeled data. The current data pool is a ready-made unlabeled data set. The model is allowed to select the most useful samples from the unlabeled data for further learning. The greedy method refers to a method of repeatedly making the best choice in the current situation in order to make the best choice.

[0075] The new intention estimation unit 230 may select a cluster including the largest number of samples among clusters not included in the entire intention set. In addition, the new intention estimation unit 230 may select a representative sentence of a cluster.

[0076] Methods for selecting representative sentences of a cluster may include a method of selecting a sentence closest to the centroid of the cluster, a method of selecting a sentence with the highest frequency within a cluster, and a selection method considering the center distance and frequency of the cluster.

[0077] The prompt generation unit 240 may be configured to generate prompts to improve the intent classification performance of distinguishing IND from OOD. As an exemplary embodiment of the present invention, the prompt generation unit 240 may be configured to generate prompts for intent classification by adjusting the intent usage rate and the speech example sampling rate of the candidate group data set.

[0078] The prompt generation unit 240 can adjust the intent usage rate of the entire data set and adjust the utterance example sampling rate to generate prompts for generating new intents. The prompt generation unit 240 can be configured to generate prompts using a small number of examples and reduce the input size.

[0079] The automatic determination unit 250 may be configured to use a large-scale language model to determine whether the misclassification prediction result is appropriate. The automatic determination unit 250 may receive sentence data with a high possibility of misclassification and the prediction result of the intent classifier, and determine the relationship between the actual meaning of the utterance and the prediction result. The automatic determination unit 250 may analyze the meaning of the utterance and use the result as a hint.

[0080] The automatic determination unit 250 can detect new intents using a large-scale language model. The automatic determination unit 250 can detect new intents using clustered representative sentences. In addition, the automatic determination unit 250 can find true negative IND utterances in OOD data. The automatic determination unit 250 can be configured to generate new intent labels based on the results of detecting new intents.

[0081] The re-learning unit 260 may continuously train the intention classifier or the large-scale language model included in the automatic learning device based on the result of the automatic determination unit 250 .

[0082] Previously, human evaluators were needed to take advantage of continual and active learning. In order to identify cases where the model misclassified and detect new intents, domain experts or human evaluators needed to review and annotate the data. Current methods are time-consuming and expensive.

[0083] The automatic learning device 200 of the exemplary embodiment of the present invention can automate the role of a human evaluator, analyze the meaning of the text without a human evaluator, and automatically mark new intentions. Accordingly, the automatic learning device 200 of the exemplary embodiment of the present invention can perform infinite repetition of learning until all data is processed. Accordingly, the automatic learning device 200 can be useful in an environment where new data is continuously updated.

[0084] The automatic learning device 200 has appropriate termination conditions to maintain performance and efficiency. The condition is: the training is repeated until the performance of the automatic learning device 200 is not lower than a preset standard. For example, when the IND classification performance drops below a certain level, the training stops.

[0085] Figure 3 is a flow chart illustrating a process for detecting a new intent according to an exemplary embodiment of the present invention.

[0086] The intention determination unit 210 receives the user's speech and distinguishes IND from OOD (S300). The task of distinguishing IND from OOD can be performed using an intention classifier or a large-scale language model.

[0087] The hint generation unit 240 may be configured to generate hints to improve the intent classification performance of distinguishing IND from OOD. The hint generation unit 240 may be configured to generate hints for intent classification by adjusting the intent usage rate of the candidate group dataset and adjusting the utterance example sampling rate.

[0088] Table 1 shows the accuracy of distinguishing IND from OOD based on a large-scale language model and an intent classifier in an exemplary embodiment of the present invention.

[0089] [Table 1]

[0090]

[0091] The intent classifier utilizes ELECTRA-base based on Korean. The intent classifier utilizes a model pre-trained on a Korean dataset and fine-tuned on an intent classification dataset. Parameter settings (such as the model's layers and embedding size) are the same as the publicly available ELECTRA-base model. The large-scale language model utilizes the gpt-3.5-turbo-16k-0613 model. Although the input size of the large-scale language model is 16k, the large-scale language model exceeds the input size limit when 446 intents and 1 utterance sentence are used as an example. Accordingly, the intent classification task is performed by dividing the training data into 100%, 75%, 50% and 25% of the entire intent used. The test set includes a highest utterance test set and a uniformly distributed test set. The highest utterance test set includes a test set with a domain rate similar to the entire utterance data. For each intent, the uniformly distributed test set includes the same number of test sets. Table 1 shows the accuracy performance of the task of distinguishing IND from OOD based on a large-scale language model and an intent classifier in an exemplary embodiment of the present invention. When using a large-scale language model, it is impossible to match all intents even with a small number of samples providing 100% of the intent label examples. However, Table 1 shows that the performance degradation is small even with a small number of intent label examples.

[0092] The new intention estimation unit 230 may sample representative sentences of each cluster according to the data set classified as OOD by the intention determination unit ( S310 ).

[0093] The new intent estimation unit 230 may perform contrastive learning and clustering between labels. The new intent estimation unit 230 may select a cluster including the most samples among the clusters not included in the entire intent set. The new intent estimation unit 230 may select a representative sentence of the cluster.

[0094] The automatic determination unit 250 may detect new intents using a large-scale language model (S320). The automatic determination unit 250 may detect new intents using clustered representative sentences. The automatic determination unit 250 may find true negative IND utterances in OOD data.

[0095] The prompt generation unit 240 may adjust the intent usage rate of the entire data set and adjust the utterance example sampling rate to generate prompts for generating new intents.

[0096] In an exemplary embodiment of the present invention, a representative utterance of each cluster is selected from 19461 OOD sentences. Accordingly, a prompt is set to simultaneously perform the discovery of existing intents or new intents from a total of 161 sentences. As an exemplary embodiment of the present invention, a large-scale language model is provided with an intent label, and the new utterance is classified by intent, and when it is determined that the new utterance is not included in the existing intent, a new intent label is generated. The large-scale language model is not significantly affected by the change in the number of intent label examples. On average, about 14% are classified as IND, and the remaining 86% are predicted as new intent labels. Since the OOD data set does not have the correct label, the prediction results are evaluated by two human evaluators. As a result, it is determined that a new intent label is available with an accuracy of 80%. In general, 67% of the utterances classified as IND are correctly classified. The Cohen's Kappa coefficient, which can be used to determine the reliability of two evaluators, is 0.741.

[0097] The re-learning unit 260 may train the intent classifier or the large-scale language model of the automatic learning device based on the newly generated intent label (S330). In addition, the re-learning unit 260 may train the intent classifier or the large-scale language model based on the reconnaissance IND utterance.

[0098] The re-learning unit 260 may evaluate the intent classification performance based on the evaluation data (S340). The re-learning unit 260 may repeat the learning within a range in which the intent classification performance of the predefined intent is not reduced, and may stop the learning when the classification performance is reduced. The re-learning unit 260 may gradually include important clusters as new intent labels one by one to reduce the number of unknown intents. The re-learning unit 260 may repeat the learning indefinitely until all the data is processed.

[0099] Figure 4is a flow chart illustrating a process of determining an error classification according to an exemplary embodiment of the present invention.

[0100] The intention determination unit 210 receives the user's speech and distinguishes IND from OOD (S400). The task of distinguishing IND from OOD can be performed using an intention classifier or a large-scale language model.

[0101] The hint generation unit 240 may be configured to generate hints to improve the intent classification performance of distinguishing IND from OOD. The hint generation unit 240 may adjust the intent usage rate of the candidate group dataset and adjust the utterance example sampling rate to generate hints for intent classification.

[0102] The misclassification candidate group selection unit 220 may estimate misclassified sentences from the data set classified as IND by the intention determination unit ( S410 ).

[0103] The error classification candidate group selection unit 220 may include one or more intent classifiers. In this document, these intent classifiers refer to intent classifiers pre-trained in different ways. The error classification candidate group selection unit 220 may perform intent prediction based on the same input sentence by connecting one or more trained intent classifiers in parallel. The error classification candidate group selection unit 220 may evaluate the diversity of prediction results based on the same input sentence, and sort the prediction results based on the diversity of intent distribution. The error classification candidate group selection unit 220 may compare the prediction results of different intent classifiers, and select an error classification candidate group that may cause an error.

[0104] In an exemplary embodiment of the present invention, three intent sets n1, n2 and n3 are generated based on the entire intent set N. The entire intent set N is a collection of all intents that the system can understand and process. Intent sets n1, n2 and n3 contain important intents, but each of intent sets n1, n2 and n3 includes other less important intents. In this article, important intents refer to intents common to all data sets. The misclassification candidate group selection unit 220 uses intent sets n1, n2 and n3 as data sets to train one or more intent classifiers. Accordingly, the results predicted using the intent classifier are sorted based on the diversity of the intent distribution. The sentences that obtain the most diverse results are selected as the misclassification candidate group.

[0105] In an exemplary embodiment of the present invention, as a result of performing predictions, in the entire data set containing 161,843 utterances, 5,746 (3.55%) utterances had mismatched intent classifier results. Among these, after excluding the differences caused by different training intents, data that the intent classifier was confused with was found in 2,320 (1.43%) utterances.

[0106] The automatic determination unit 250 may be configured to use a large-scale language model to determine whether the misclassification prediction result is appropriate (S420). The large-scale language model of the automatic determination unit 250 may receive sentence data with a high possibility of misclassification and the prediction result of the intent classifier, and determine the relationship between the actual meaning of the utterance and the prediction result. The automatic determination unit 250 may analyze the meaning of the utterance and use the result as a hint.

[0107] In an exemplary embodiment of the present invention, the selected 2320 misclassified candidate group utterances are divided into three groups, and a large-scale language model is used to evaluate the prediction results of the intent classifier. The misclassified candidate group utterances are divided based on the utterance frequency. The three groups include Group A, Group B and Group C. Group A (395 utterances) has utterances more than 100 times, Group B (1348 utterances) has utterances more than 30 times but less than 100 times, and Group C (577 utterances) has utterances less than 30 times. 100 samples are randomly selected from each group, and a large-scale language model is used to determine the results of the intent classifier and infer the correct intent.

[0108] Table 2 shows the prediction results of the intent classifier, the results correctly inferred by the large-scale language model, and the results evaluated by human raters.

[0109] [Table 2]

[0110]

[0111]

[0112] When the large-scale language model infers correctly, enter 1 in the human column, and when the large-scale language model infers incorrectly, enter 0.

[0113] Table 3 shows the matching rates between human raters of the evaluation results.

[0114] [Table 3]

[0115] A B C LLM_adjuster 0.81 0.83 0.91

[0116] As a result, the table shows a performance with an average matching rate of 85.46%. This table shows that the determination ability of a large-scale language model can mimic the determination ability of humans.

[0117] The re-learning unit 260 may train the intention classifier of the automatic learning device or the large-scale language model based on the prediction result correctly inferred by the large-scale language model ( S430 ).

[0118] The re-learning unit 260 may evaluate the intent classification performance based on the evaluation data (S440). The re-learning unit 260 may repeat the learning within a range in which the intent classification performance of the predefined intent does not decrease, and may stop the learning when the classification performance decreases.

[0119] Figure 5 is a flowchart of an intention analysis method according to an exemplary embodiment of the present invention.

[0120] The intention determination unit 210 receives the data log of the user and distinguishes IND from OOD ( S500 ).

[0121] The misclassified candidate group selection unit 220 may estimate misclassified sentences based on data classified as IND by the intent classifier or the large-scale language model. In addition, the new intent estimation unit 230 may sample representative sentences of each cluster based on data classified as OOD by the intent classifier or the large-scale language model (S510).

[0122] The automatic determination unit 250 may be configured to use a large-scale language model to determine whether the prediction result of the intent classifier is appropriate. In addition, the automatic determination unit 250 may use a large-scale language model to detect a new intent (S520).

[0123] The re-learning unit 260 may train an intention classifier or a large-scale language model of the automatic learning device based on the result of the automatic determination unit 250 ( S530 ).

[0124] The re-learning unit 260 may evaluate the intention classification performance based on the evaluation data ( S540 ).

[0125] Fig. 6A and Figure 6B is a schematic diagram illustrating a method for performing effective hinting within a limited input size for a large-scale language model.

[0126] In large-scale language models, the longer the input and output tokens are, the slower the generation speed is. Accordingly, methods that use short prompts can increase the speed of new intent discovery and reduce costs. In addition, the performance of intent classifiers can be continuously improved by automation using active learning.

[0127] Fig. 6A is a schematic diagram showing a hinting method for efficiently performing intent classification tasks within a limited input size.

[0128] In order for a large-scale language model to predict the intent of a sentence, it is necessary to provide examples using a small sample. A large-scale language model includes a limited input size (context size). The input size refers to the maximum number of tokens that the model can process at one time. Due to the limitation on the number of tokens, there is a limit on the length of the input text. The prompt generation unit 240 can be configured to generate prompts to improve the intent classification performance within the limited input size.

[0129] As an exemplary embodiment of the present invention, although the input size of the large-scale language model is 16k, when 446 intents and 1 utterance sentence are used as examples, the large-scale language model exceeds the limit of the input size. In order to solve this problem, the prompt generation unit 240 can be configured to generate prompts for intent classification by adjusting the intent usage rate of the candidate group data set and adjusting the utterance example sampling rate. The generated prompts may include examples of utterances and intents forming a pair, such as "utterance: intent". For example, the utterance "open the window" and the intent "open the window" can form a pair and include an example of "open the window: open the window".

[0130] The prompt generation unit 240 may utilize N utterance examples and generate a prompt of 100% input size while utilizing 1 / N of the entire intent.

[0131] The hint generation unit 240 may utilize 1 utterance example and generate a hint of 100% input size while utilizing 100% of the entire intent. The method utilizing 100% of the entire intent may evenly classify all intents.

[0132] The prompt generation unit 240 may utilize 2 utterance examples and generate a prompt of 100% input size while utilizing 50% of the entire intent. The method utilizing 50% of the entire intent may be advantageous in classifying various intents on average.

[0133] For example, the prompt generation unit 240 can be configured to generate a prompt of 100% input size using 4 utterance examples and 25% of the entire intent. The method using 25% of the entire intent can classify statistically frequently used intents well.

[0134] Figure 6B is a schematic diagram showing a hinting method for efficiently generating new intents within a limited input size.

[0135] The prompt generation unit 240 may be configured to generate a prompt for generating a new intent by adjusting the intent usage rate of the entire data set and adjusting the utterance example sampling rate.

[0136] The generated prompt may include an example of an utterance and an intent forming a pair, such as "utterance: intent." For example, the utterance "open the window" and the intent "openWindow" may form a pair and include an example of "open the window: openWindow."

[0137] In an exemplary embodiment of the present invention, for the task of classifying existing intents, a large number of examples are required, while for the task of discovering new intents, sufficiently high accuracy can be achieved even with a small number (such as 5%) of examples.

[0138] Figure 7 is an exemplary block diagram schematically illustrating an exemplary computing device that can be used to implement a method or apparatus according to an exemplary embodiment of the present invention.

[0139] The computing device 700 may include all or part of a memory 710, a processor 720, a storage device 730, an input / output interface 740, and a communication interface 750. The computing device 700 may structurally and / or functionally include at least a portion of the automatic learning device 200. The computing device 700 may be a fixed computing device such as a desktop computer, a server, etc., and a mobile computing device such as a laptop computer, a smart phone, a vehicle electrical device, etc. The computing device 700 may be implemented as any dedicated hardware accelerator configured to effectively process the calculation of an artificial intelligence model. For example, the computing device 700 may include a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0140] The memory 710 may store a program that enables the processor 720 to perform methods or operations according to various embodiments of the present invention. For example, the program may include multiple instructions that can be executed by the processor 720, and the above-mentioned methods or operations may be performed by executing multiple instructions by the processor 720. The memory 710 may be a single memory or multiple memories. In this regard, the information required to perform the methods or operations according to various embodiments of the present invention may be stored in a single memory, or may be divided and stored in multiple memories. When the memory 710 is configured as multiple memories, the multiple memories may be physically separated. The memory 710 may include at least one of a volatile memory and a non-volatile memory. The volatile memory may include a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the non-volatile memory may include a flash memory, etc.

[0141] The processor 720 may include at least one core configured to execute at least one instruction. The processor 720 may execute instructions stored in the memory 710. The processor 720 may be a single processor or a plurality of processors.

[0142] The storage device 730 retains the stored data even when the power supplied to the computing device 700 is cut off. For example, the storage device 730 may include a nonvolatile memory, and may include a storage medium such as a tape, an optical disk, or a magnetic disk. The program stored in the storage device 730 may be loaded into the memory 710 before being executed by the processor 720. The storage device 730 may store a file written in a program language, and may load a program generated from the file by a compiler or the like into the memory 710. The storage device 730 may store data to be processed by the processor 720 and / or data processed by the processor 720.

[0143] The input / output interface 740 may be provided with an interface with an input device (such as a keyboard, a mouse, etc.) and / or an output device (such as a display device, a printer, etc.). A user may trigger the processor 720 to execute a program through an input device and / or check the processing result of the processor 720 through an output device.

[0144] The communication interface 750 may provide access to an external network. The computing device 700 may communicate with other devices through the communication interface 750 .

[0145] Each component of the device or method according to the exemplary embodiment of the present invention can be implemented as hardware or software, or can be implemented as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can be implemented as the function of executing the software corresponding to each component.

[0146] Various embodiments of the systems and techniques described herein can be implemented with digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. Various embodiments may include implementations of one or more computer programs executable on a programmable system. A programmable system includes at least one programmable processor, at least one programmable processor (which may be a dedicated processor or a general purpose processor) connected to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to a storage system, at least one input device, and at least one output device. A computer program (also referred to as a program, software, software application, or code) includes instructions for a programmable processor and is stored in a "computer readable recording medium".

[0147] The computer-readable recording medium may include all types of storage devices on which computer-readable data can be stored. The computer-readable recording medium may be a non-volatile or non-temporary medium, such as a read-only memory (ROM), a random access memory (RAM), a compact disc ROM (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device. In addition, the computer-readable recording medium may further include a volatile medium such as a data transmission medium. In addition, the computer-readable recording medium may be distributed on a computer system connected via a network, and may store and execute a computer-readable program code in a distributed manner.

[0148] In various exemplary embodiments of the present invention, the memory and the processor may be provided as one chip, or provided as separate chips.

[0149] In various exemplary embodiments of the present invention, the scope of the present invention includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) for enabling operations of methods according to various embodiments to be performed on a device or computer, and non-transitory computer-readable media including such software or instructions stored thereon and executable on a device or computer.

[0150] In various exemplary embodiments of the present invention, the computing device may be implemented in the form of hardware or software, or may be implemented in a combination of hardware and software.

[0151] Software implementations may include: software components (or elements), object-oriented software components, class components, task components, procedures, functions, attributes, processes, subroutines, program code segments, drivers, firmware, microcodes, data, databases, data structures, tables, arrays, and variables. Software, data, etc. may be stored in a memory and executed by a processor. The memory or processor may employ various devices known to those skilled in the art.

[0152] Furthermore, terms such as “unit”, “module” and the like included in the specification refer to a unit for processing at least one function or operation, which may be implemented by hardware, software or a combination thereof.

[0153] Hereinafter, the fact that a plurality of hardwares are operably coupled may include the fact that a direct and / or indirect connection between the plurality of hardwares is established by a wired and / or wireless manner.

[0154] In an exemplary embodiment of the present invention, a vehicle may be referred to as a concept based on various vehicles. In some cases, a vehicle may be interpreted as a concept based on various land vehicles including not only cars, motorcycles, trucks, and buses traveling on roads, but also various vehicles including airplanes, drones, ships, and the like.

[0155] To facilitate explanation and accurately define the appended claims, the terms "above", "below", "inside", "outside", "up", "down", "upward", "downward", "front", "rear", "back", "inside", "outside", "inwardly", "outwardly", "inside", "outside", "inner", "external", "front" and "rear" are used to describe features of the exemplary embodiments with reference to the positions of such features as shown in the accompanying drawings. It will be further understood that the term "connect" or its derivatives refer to both direct and indirect connections.

[0156] The term "and / or" may include a combination of a plurality of related listed items or any one of a plurality of related listed items. For example, "A and / or B" includes all three cases, such as "A", "B" and "A and B".

[0157] In an exemplary embodiment of the present invention, “at least one of A and B” may refer to “at least one of A or B” or “at least one of a combination of at least one of A and B”. Furthermore, “one or more of A and B” may refer to “one or more of A or B” or “one or more of a combination of one or more of A and B”.

[0158] In this specification, unless otherwise stated, a singular expression includes a plural expression unless the context clearly indicates otherwise.

[0159] In the exemplary embodiments of the present invention, it should be understood that terms such as "including" or "having" are intended to indicate the presence of the features, quantities, steps, operations, elements, parts, or a combination thereof described in the specification, and do not exclude the possibility of adding or existing one or more other features, quantities, steps, operations, elements, parts, or a combination thereof.

[0160] According to the exemplary embodiments of the present invention, components may be combined with each other to be implemented as one, or some components may be omitted.

[0161] The description of specific exemplary embodiments of the present invention is presented above for the purpose of illustration and description. The above description is not intended to be exhaustive, or to limit the present invention to the precise form disclosed, and it is obvious that several modifications and variations are possible according to the above teachings. Exemplary embodiments are selected and described to explain the specific principles of the present invention and their practical applications, thereby enabling other technical personnel in the field to implement and utilize various exemplary embodiments of the present invention and various alternatives and modifications thereof. The scope of the present invention is intended to be limited by the appended claims and their equivalents.

Claims

1. A method for automatically training a speech recognition system, the speech recognition system comprising one or more natural language understanding engines, the method comprising: Obtaining natural language understanding results output by one or more natural language understanding engines; Generate hints for large-scale language models based on comparisons between natural language understanding results; Determine whether natural language understanding results are appropriate by leveraging generated hints for large-scale language models; The speech recognition system is trained by utilizing the determination results.

2. The method for automatically training a speech recognition system according to claim 1, wherein: The natural language understanding result includes: at least one of the intent or entity obtained by using one or more natural language understanding engines, The method further comprises: performing processing to provide a result corresponding to the user's intent and a result corresponding to the entity; The vehicle is controlled based on the result.

3. The method for automatically training a speech recognition system according to claim 1, wherein: Obtaining a natural language understanding result output by one or more natural language understanding engines includes: obtaining the natural language understanding result by connecting one or more natural language understanding engines trained in different ways in parallel.

4. The method for automatically training a speech recognition system according to claim 3, wherein: Comparing between natural language understanding results includes: comparing the natural language understanding results, and filtering sentences whose results of predicted intent are most different.

5. The method for automatically training a speech recognition system according to claim 4, wherein: Determining whether the natural language understanding result is appropriate includes determining whether the result of predicting the intent using a large-scale language model is appropriate and inferring the correct intent.

6. The method for automatically training a speech recognition system according to claim 5, wherein: Training the speech recognition system by utilizing the determination results includes training the speech recognition system based on the results correctly inferred by the large-scale language model.

7. The method for automatically training a speech recognition system according to claim 1, wherein: Obtaining natural language understanding results output by one or more natural language understanding engines includes: performing contrastive learning and clustering between labels on the natural language understanding results output by one or more trained natural language understanding engines.

8. The method for automatically training a speech recognition system according to claim 7, wherein: Comparing between natural language understanding results includes filtering a cluster including the most samples among clusters not included in the entire intent set, and selecting representative sentences.

9. The method for automatically training a speech recognition system according to claim 8, wherein: Determining whether the natural language understanding results are appropriate includes detecting new intents by utilizing clustered representative sentences and generating new intent labels.

10. The method for automatically training a speech recognition system according to claim 9, wherein: Training the speech recognition system by utilizing the determination result includes: training the speech recognition system based on the newly generated intent label.

11. An apparatus for automatically training a speech recognition system, the speech recognition system comprising one or more natural language understanding engines, the apparatus comprising: a memory configured to store one or more instructions; one or more processors operably connected to the memory and configured to execute one or more instructions stored in the memory, Wherein, one or more processors execute one or more instructions to perform: Obtaining natural language understanding results output by one or more natural language understanding engines; Generate hints for large-scale language models based on comparisons between natural language understanding results; Determine whether natural language understanding results are appropriate by leveraging generated hints for large-scale language models; The speech recognition system is trained by utilizing the determination results.

12. The apparatus for automatically training a speech recognition system according to claim 11, wherein: The natural language understanding result includes at least one of an intent or an entity obtained by using one or more natural language understanding engines, The one or more processors further execute: processing to provide a result corresponding to the user's intent and a result corresponding to the entity; The vehicle is controlled based on the result.

13. The device for automatically training a speech recognition system according to claim 11, wherein: Obtaining a natural language understanding result output by one or more natural language understanding engines includes: obtaining the natural language understanding result by connecting one or more natural language understanding engines trained in different ways in parallel.

14. The device for automatically training a speech recognition system according to claim 13, wherein: Comparing between natural language understanding results includes: comparing the natural language understanding results, and filtering sentences whose results of predicted intent are most different.

15. The device for automatically training a speech recognition system according to claim 14, wherein: Training the speech recognition system by utilizing the determination results includes training the speech recognition system based on the results correctly inferred by the large-scale language model.

16. The device for automatically training a speech recognition system according to claim 11, wherein: Obtaining natural language understanding results output by one or more natural language understanding engines includes: performing contrastive learning and clustering between labels on the natural language understanding results output by one or more trained natural language understanding engines.

17. The device for automatically training a speech recognition system according to claim 16, wherein: Comparing between natural language understanding results includes filtering a cluster including the most samples among clusters not included in the entire intent set, and selecting representative sentences.

18. The device for automatically training a speech recognition system according to claim 17, wherein: Determining whether the natural language understanding result is appropriate includes detecting new intents by utilizing clustered representative sentences and generating new intent labels.

19. The device for automatically training a speech recognition system according to claim 18, wherein: Training the speech recognition system by utilizing the determination result includes: training the speech recognition system based on the newly generated intent label.

Images