Apparatus and computer-implemented method for evaluating utterance recognition system

Abstract

An apparatus and computer-implemented method for evaluating an utterance recognition system is disclosed. An embodiment device for evaluating an utterance recognition system comprising a plurality of utterance recognition engines and a plurality of natural language understanding (NLU) engines includes one or more processors and at least a storage device storing a program to be executed by the one or more processors, the program comprising instructions to obtain an utterance recognition result, wherein the plurality of utterance recognition engines recognize input audio, evaluate the plurality of utterance recognition engines based on a comparison between the utterance recognition results, obtain NLU results wherein the plurality of NLU engines understand each utterance recognition result, and evaluate the plurality of NLU engines based on a comparison between the NLU results.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2023-0126991, filed on September 22, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to methods and apparatus for evaluating a speech recognition system. Background Art

[0004] The following description merely provides background information related to the present disclosure and does not constitute prior art.

[0005] As artificial intelligence technologies have been developed recently, their applications are also expanding. Specifically, conversation systems that can conduct conversations with users using natural language (such as chat systems or virtual assistants) are being used in various fields, and the technology is gradually developing. In order for a conversation system to conduct a conversation with a user, the user's expression (i.e., input message) must be understood from the perspective of the conversation system. In order to achieve natural language understanding (NLU), the conversation system needs to derive the current context and the user's intention expected in the context from the conversation between the conversation system and the user, and analyze the input message based on the derived context and / or intention.

[0006] The application scope of these speech recognition services has expanded from the home to various fields, such as automobiles. In other words, the speech recognition assistant service and the telematics service are linked, and the voice command generated by the user's voice is sent to the car to control the car. Through this, the user can lock / unlock the door, or turn on the air conditioner in advance to control the temperature inside the vehicle.

[0007] In this case, data processing based on accurate speech recognition is the most important factor in ensuring user safety.

[0008] Even if the utterance recognition system can classify the user's voice intent with 95% accuracy, because millions of utterances are processed every day in actual utterance recognition services, the intent of hundreds of thousands of utterances may be misclassified.

[0009] In addition, due to the difference between the dataset used to train the speech recognition system and the user data in the speech recognition service phase, the accuracy of the speech recognition system may be reduced.

[0010] Therefore, there is an urgent need for a method of processing more accurate speech recognition in a vehicle and a method of providing more accurate speech recognition results according to a user request.

[0011] To achieve this goal, a speech recognition system that can accurately recognize the user's expression and intention is needed. In order to build such a speech recognition system, it is necessary to evaluate the processing results of the speech recognition system for the user's speech.

[0012] However, there is a lack of research on methods for evaluating the performance of speech recognition systems. Specifically, no method for automatically evaluating the processing results of speech recognition systems for a large number of user utterances has been studied. Summary of the invention

[0013] At least one embodiment of the present disclosure provides a device for evaluating a speech recognition system including multiple speech recognition engines and multiple natural language understanding (NLU) engines. The device includes at least one memory storing computer executable instructions and at least one processor. The at least one processor is configured to execute the computer executable instructions to obtain speech recognition results in which multiple speech recognition engines recognize input audio, evaluate the multiple speech recognition engines based on a comparison between the speech recognition results, obtain NLU results in which multiple NLU engines understand each speech recognition result, and evaluate the multiple NLU engines based on a comparison between the NLU results.

[0014] Another embodiment of the present disclosure provides a computer-implemented method for evaluating a speech recognition system including multiple speech recognition engines and multiple natural language understanding (NLU) engines. The method includes: obtaining speech recognition results in which multiple speech recognition engines recognize input audio; obtaining NLU results in which multiple NLU engines understand each speech recognition result; evaluating multiple speech recognition engines based on a comparison between speech recognition results; and evaluating multiple NLU engines based on a comparison between NLU results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a block diagram of the speech recognition system.

[0016] Figure 2 is a block diagram of an evaluation system and a speech recognition system according to one embodiment of the present disclosure.

[0017] Figure 3 is a diagram for explaining evaluation of a speech recognition engine according to one embodiment of the present disclosure.

[0018] Figure 4 is a diagram for explaining evaluation of an NLU engine according to one embodiment of the present disclosure.

[0019] Figure 5 is a diagram for explaining evaluation of an NLU engine according to another embodiment of the present invention.

[0020] Figure 6is a diagram for explaining dangerous text recognition according to one embodiment of the present disclosure.

[0021] Figure 7 is a diagram for explaining recognition failure detection according to one embodiment of the present disclosure.

[0022] Figure 8 is a flow chart of an evaluation method according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present disclosure provide a method and apparatus for evaluating and updating a speech recognition system based on a result of processing a large amount of voice data in the speech recognition system.

[0024] The following describes the embodiments of the present disclosure in detail using various drawings. In addition, when the components are marked with reference numerals in each figure, the same reference numerals are used as much as possible even for components shown in different drawings. In addition, in the description of the embodiments of the present disclosure, if it has been determined that the specific description of the relevant known configuration or function may obscure the key points of the present disclosure, its detailed description is omitted.

[0025] When describing components according to embodiments of the present disclosure, symbols such as first, second, i), ii), a), and b) may be used. These symbols are used only to distinguish components from other components. The identity or sequence or order of the components is not limited by the symbols. In this specification, when a component "includes" or "is equipped with" an element, it means that the component may further include other elements, without excluding other elements unless explicitly stated to the contrary. In addition, when an element in the written description and claims is described as "for" performing or executing a stated function, step, instruction set, etc., the element may also be regarded as "configured to" do so.

[0026] Each component of the device or method according to the embodiment of the present disclosure can be implemented by hardware or software or a combination of hardware and software. In addition, the function of each component can be implemented in software. A microprocessor or processor can execute the function of the software corresponding to each component.

[0027] Figure 1 It is a block diagram of the speech recognition system.

[0028] refer to Figure 1 , the speech recognition system 100 recognizes and understands the user's voice and provides a service corresponding to the user's voice.

[0029] To this end, the speech recognition system 100 includes a speech recognizer 110 that converts a user's speech expression into text, a natural language understander 120 that determines intentions and entities included in the user's speech expression, and a result processor 130 that performs processing to provide results corresponding to the user's intentions and entities.

[0030] The speech recognizer 110 may convert the user's expression into an input sentence using at least one speech recognition engine. In this case, the speech recognition engine may refer to a speech to text (STT) engine, and may convert a speech signal representing the user's expression into text by applying a speech recognition algorithm or a neural network model to the speech signal.

[0031] For example, the speech recognizer 110 may apply a feature vector extraction technique such as cepstrum, linear prediction coefficient (LPC), Mel-frequency cepstral coefficient (MFCC), or filter bank energy to extract a feature vector from the user expression.

[0032] The speech recognizer 110 can obtain a recognition result by comparing the extracted feature vector with the trained reference pattern. For this purpose, an acoustic model for modeling and comparing the signal features of the speech or a language model for modeling the language order relationship of words or syllables corresponding to the recognition vocabulary can be used.

[0033] The speech recognizer 110 can also convert the user expression into text based on a model applying machine learning or deep learning.

[0034] In an embodiment of the present disclosure, the speech recognition result means a text obtained using a speech recognition engine.

[0035] The natural language understander 120 determines at least one of a user intention and or an entity included in an input sentence using at least one natural language understanding (NLU) engine.

[0036] The natural language understander 120 may extract information such as domain, entity name, and speech act from the input sentence using an NLU engine, and recognize intent and entity according to the intent based on the extraction result. The entity may be referred to as a slot.

[0037] The NLU engine may segment an input sentence into morphemes, project the morphemes into a vector space, group the projected vectors according to the input sentence to classify intent, and extract word components as entities according to the intent in the input sentence.

[0038] For example, when the input sentence is "When do I change the engine oil?" The NLU engine segments the input sentence into "engine", "oil", "When", "change", and "do", and converts each morpheme into a vector. The NLU engine then classifies the intent corresponding to the vector based on the similarity between the vectors and their positions in the vector space. In the above example, the classified intent is consumable replacement check. The NLU engine extracts the word components "engine" and "oil" as entities based on the intent of consumable replacement check. The dialog manager can then provide the sentence "Engine oil change interval is 15,000 Km" based on the consumable replacement check intent and the "engine" and "oil" entities.

[0039] As another example, when the input sentence is "Let's go home", the domain is [navigation], the intent is [route setting], and the entity required to perform the control corresponding to the intent is [starting point, destination]. For another example, when the input sentence is "Turn on the air conditioner", the domain is [vehicle control], the intent is [air conditioner power on], and the entity required to perform the control corresponding to the intent is [temperature, air volume].

[0040] In an embodiment of the present disclosure, the NLU result refers to at least one of an intent and an entity obtained using an NLU engine.

[0041] The result processor 130 may output a result processing signal to the vehicle, the user device, or an external server to perform processing to provide a service corresponding to the user's intention.

[0042] For example, when the service corresponding to the user's intention is vehicle-related control, the result processor 130 may transmit a result processing signal for executing the vehicle-related control to the vehicle. As another example, when the service corresponding to the user's intention is to provide specific information, the result processor 130 may search for the specific information and provide the searched information to the user terminal. If necessary, information retrieval may be performed on another external server. As another example, when the service corresponding to the user's intention is to provide specific content, the result processor 130 may request that the target content be sent to an external server that provides the content. In another example, when the service corresponding to the user's intention is a simple continuation of the conversation, the result processor 130 may generate a response to the user's utterance and output the response visually or audibly.

[0043] The speech recognition system 100 may be provided on an external server or a user terminal, and some of its components may be provided on an external server and other components may be provided on a user terminal. The user terminal may be a mobile device such as a smart phone, a tablet PC, or a wearable device, a home appliance with a user interface, or a vehicle.

[0044] The speech recognition system 100 may further include a dialog manager that manages the overall dialog between the speech recognition system 100 and the user.

[0045] The components of the speech recognition system 100 are classified based on their operations or functions, and all or some of the components of the speech recognition system 100 may share a memory or a processor.

[0046] The speech recognition system 100 may be implemented in a vehicle or a server. Alternatively, some components of the speech recognition system 100 may be included in the vehicle, and other components may be included in the server. For example, the vehicle sends a passenger's voice signal to the server, and the server processes the passenger's voice signal to generate information or control commands required by the passenger, and sends the information or control commands to the vehicle.

[0047] Figure 2 is a block diagram of an evaluation system and a speech recognition system according to one embodiment of the present disclosure.

[0048] Reference Figure 2 , an evaluation system 200 and a speech recognition system 300 are shown.

[0049] The speech recognition system 300 is capable of inferring speech recognition and NLU on a large scale using a parallel speech recognition engine and a parallel NLU engine. The speech recognition engine and the NLU engine can be parallelized on a graphics processing unit (GPU). Conventionally, speech recognition of 100 to 30,000 audio data per day can be tested, while the speech recognition system 300 and the evaluation system 200 according to an embodiment of the present disclosure can test speech recognition of millions of audio data per day. That is, the speech recognition system 300 can perform speech recognition on a large amount of audio data at high speed. To this end, the speech recognition system 300 includes a speech recognizer 310 and a natural language understander 320.

[0050] The speech recognizer 310 includes a plurality of speech recognition engines, which may generate speech recognition results of input audio using different speech recognition algorithms or neural network models for speech recognition.

[0051] The natural language understander 320 includes multiple NLU engines. The NLU engines can use different NLU processes or NLU models.

[0052] The speech recognizer 310 and the natural language understander 320 may respectively execute Figure 1 Meanwhile, the speech recognizer 310 and the natural language understander 320 may both be provided in the vehicle or the server, or one may be provided in the vehicle and the other may be provided in the server.

[0053] In addition, the speech recognition system 300 may evaluate its performance through the evaluation system 200. Specifically, a plurality of speech recognition engines in the speech recognizer 310 and a plurality of NLU engines in the natural language understander 320 may process data provided from the evaluation system 200 and be evaluated based on the processing results.

[0054] Furthermore, the utterance recognition system 300 may be updated using the tags generated by the tag generator 260 using the very large artificial intelligence model.

[0055] The speech recognition system 300 may include at least one processor and a memory having at least one instruction, and may execute the instructions by the at least one processor to perform functions of the speech recognizer 310 and the natural language understander 320. The speech recognition system 300 may further include a communication unit for communicating with an external device.

[0056] The evaluation system 200 is a system for evaluating the performance of the speech recognition system 300. Specifically, the evaluation system 200 provides a large amount of speech data to the speech recognition system 300 and evaluates the performance of the speech recognition system 300 based on the processing result of the speech data by the speech recognition system 300. The evaluation system 200 can evaluate the speech recognition system 300 in an end-to-end manner.

[0057] Since the speech recognition system 300 processes a large amount of voice data generated from the customer log, the evaluation system 200 can find voice data with many errors based on the processing result of the speech recognition system 300. In other words, the evaluation system 200 can find new needs of customers.

[0058] Specifically, the evaluation system 200 includes at least one of a first text database (DB) 211 , a second text DB 213 , an audio DB 215 , a preprocessor 220 , an audio converter 230 , a noise model 240 , an evaluation unit 250 , and a tokenizer 260 .

[0059] The first text DB 211 and the second text DB 213 store text sentences that may be used by a plurality of users to utilize the speech recognition system 300. The text sentences may be a result of collecting voices from vehicle occupants.

[0060] Unlike the first text DB 211, the second text DB 213 may store tagged or marked text. The NLU tag or NLU mark of the text indicates the NLU target of the text. In other words, the NLU mark of the text is the target output that the natural language understander 320 wants to output in response to the text. On the other hand, the first text DB 211 is a user log and stores text sentences without tags or marks.

[0061] The audio DB 215 stores audio data that may be used by the user to utilize the speech recognition system 300. For example, the audio DB 215 may store audio data representing the result of collecting the user's voice utterance.

[0062] In order to allow the audio converter 230 to naturally convert a text sentence in the first text DB 211 or the second text DB 213 into audio data, the pre-processor 220 may perform question marks, numbers, other language words, special characters, sentence separation, and the like.

[0063] The audio converter 230 converts the text sentence preprocessed by the preprocessor 220 into audio data. The audio converter 230 may convert the text sentence into audio data using a text-to-speech (TTS) engine. The audio converter 230 may convert a large number of text sentences into audio data using a parallel TTS engine. The TTS engine may be a multi-speaker TTS engine. The TTS engine may consider phonemes based on Korean features, Korean pronunciation, car-related terms, algorithms for converting and reading long sentences into short sentences, and the like.

[0064] Specifically, the audio converter 230 converts the preprocessed text sentence into a plurality of speech audios that differ in at least one aspect of speech speed, pitch, and additional noise. In this case, the additional noise represents vehicle driving noise, tire friction noise, wind noise, rain noise, in-vehicle conversation, etc. The additional noise may be generated by the noise model 240.

[0065] The audio converter 230 can generate speech expressions that differ in terms of a person's gender, age, region, language, pitch, speech speed, etc. For example, the audio converter 230 can generate audio data that is a synthesis of the voice of a person in his 20s who speaks a specific text sentence at a medium pitch and a fast speed and the noise of an electric vehicle. As another example, the audio converter 230 can generate audio data that is a synthesis of the voice of a woman who speaks another text sentence in her 60s at a low pitch and a low speed and rain noise.

[0066] Due to the audio converter 230 , the evaluation system 200 can evaluate the speech recognition system 300 based on the result that the speech recognition system 300 recognizes various voice data.

[0067] The speech recognizer 310 receives the audio converted from the text sentence by the audio converter 230 as input audio. The speech recognizer 310 inputs the input audio to each speech recognition engine of the plurality of speech recognition engines and obtains a speech recognition result in text form from the speech recognition engine. Each speech recognition engine can output one speech recognition result from one input audio.

[0068] As an example, a speech recognition engine may use a stream-based speech recognition model and a language model together. The speech recognition model may be trained based on speech recognition based on vehicle-related voice data. The language model is a large language model and may be a generative model. The speech recognition model has high recognition accuracy for vehicle-related voice data, and the language model has relatively high recognition accuracy for general voice data. Some speech recognition engines use speech recognition models, while other speech recognition engines use language models, which allows one audio data to be recognized in different ways.

[0069] The natural language understander 320 receives the speech recognition result of the speech recognizer 310, inputs the speech recognition result to each of the plurality of NLU engines, and obtains an NLU result from each NLU engine, the NLU result including at least one of the intent or entity for each speech recognition result. Each NLU engine may output one NLU result from one speech recognition result.

[0070] As an example, the NLU engine can use different NLU models. A dual-LSTM model with fast training and inference speed and easy tuning and an ELECTTRA-based model with high intent classification accuracy for conversational and ambiguous expressions can be used. By combining different models, the NLU engine can be viewed as a fault-tolerant classifier.

[0071] The evaluation unit 250 evaluates the speech recognition system 300 based on the speech recognition result of the speech recognition engine and the NLU result of the NLU engine. In addition, the evaluation unit 250 may specify dangerous text that the speech recognition system 300 cannot recognize well.

[0072] To this end, the evaluation unit 250 may include at least one of a speech recognition engine evaluation unit 251, an NLU engine evaluation unit 253, a dangerous text recognition unit 255, or a recognition failure detection unit 257. In this case, the evaluation unit 250 includes at least one processor and a memory including at least one instruction, and the functions of the four evaluation units may be performed by executing the instructions by the at least one processor.

[0073] The speech recognition engine evaluation unit 251 obtains speech recognition results in which a plurality of speech recognition engines recognize input audio and evaluates the plurality of speech recognition engines based on comparison between the speech recognition results. For example, the speech recognition engine evaluation unit 251 may evaluate a speech recognition engine that outputs a speech recognition result different from most speech recognition results as having low performance.

[0074] The NLU engine evaluation unit 253 obtains NLU results of multiple NLU engines understanding each speech recognition result, and evaluates the multiple NLU engines based on the comparison between the NLU results. The NLU engine evaluation unit 253 may evaluate an NLU engine that outputs an NLU result different from most NLU results as having low performance. In addition, the NLU engine evaluation unit 253 may use previously stored tags to evaluate whether the NLU result of each NLU engine understanding the speech recognition text contains an error.

[0075] The dangerous text recognition unit 255 recognizes a dangerous text sentence that is difficult to recognize based on the degree of difference between the voice audio recognition results indicating the results of recognizing a plurality of voice audios by at least one utterance recognition engine among a plurality of utterance recognition engines.

[0076] The recognition failure detection unit 257 detects recognition failure of the input audio by comparing the NLU result of at least one NLU engine among the plurality of NLU engines understanding the utterance recognition result with the NLU tag of the utterance recognition result. Through this, the utterance recognition engine or NLU engine causing the recognition failure can be evaluated.

[0077] Meanwhile, when there is no NLU tag for the speech recognition result, the tag generator 260 generates an NLU tag for the speech recognition result by applying the language model to the NLU result. In addition, the tag generator 260 may generate an NLU tag for the speech recognition result classified as a domain not related to the vehicle. The tag generator 260 generates a prompt for the speech recognition result, inputs the prompt as a query into the language model, and modifies the prompt based on the processing result of the language model. Through repetition, the tag generator 260 may generate an NLU tag, that is, an intent tag for the speech recognition result.

[0078] In this case, the language model may generate NLU tags based on what parts of the current utterance are omitted, what requirements the current utterance contains, how the current utterance would be classified as an intent, and how different the generated tags are from the results of existing NLUs.

[0079] For example, the NLU engine may misunderstand the sentence "Raise the sun shade of the back seat" as "vehicle_wind_direction_topfloor_back" and "vehicle_window_close_back". When these sentences are applied to the language model, the language model can understand the errors in the NLU results. First, the language model understands that the input sentence means a command to raise the sun shade of the back seat of the car, understands that the sun shade represents a sunshade function, and understands that raising the sun shade is different from closing the window. In other words, the language model understands the expression sentence. Then, it can be self-feedback: the sentence "vehicle_wind_direction_topfloor_back" and the sentence "vehicle_window_close_back" are misclassified intentions. The language model understands that "vehicle_wind_direction_topfloor_back" means changing the wind direction and is therefore not suitable for the intention of the expression sentence. The language model understands that "vehicle_window_close_back" means closing the rear window of the car, and is therefore not suitable for expressing the intent of the sentence. Based on the NLU results of the NLU engine and its own feedback results, the language model can output "vehicle_sunshade_up_back". The tag generator 260 can use the language model to generate tags for unlabeled utterance recognition results.

[0080] The language model is a large language model and may be a generative model. The language model is an autoregressive model and may refer to a language model that allows inference without fine tuning using methods such as few-shot learning, and has 10 times more parameters than existing general language models (e.g., more than 100 billion parameters). For example, large-scale language models such as the Generative Pretrained Transformer 4 (GPT-4) model are excellent few-shot learners that can be controlled by natural prompts, allowing contextual learning with the ability to understand patterns with only a small amount of data through prompts to solve NLP problems.

[0081] Figure 3 is a diagram for explaining evaluation of a speech recognition engine according to one embodiment of the present disclosure.

[0082] Reference Figure 3 , the speech recognition engines 311 , 313 , and 315 each recognize input audio in text form and output a speech recognition result.

[0083] The utterance recognition engine evaluation unit 251 obtains an utterance recognition result in which the plurality of utterance recognition engines 311 , 313 , and 315 recognize the input audio.

[0084] The utterance recognition engine evaluation unit 251 may evaluate the plurality of utterance recognition engines 311, 313, and 315 based on comparison between utterance recognition results. Specifically, the utterance recognition engine evaluation unit 251 may evaluate the utterance recognition engine that outputs a specific utterance recognition result based on the number of utterance recognition results that are identical or similar to the specific utterance recognition result.

[0085] For example, the first speech recognition result may be the same as the second speech recognition result, while the third speech recognition result may be different from the first speech recognition result and the second speech recognition result. In this case, the speech recognition engine evaluation unit 251 may evaluate the first speech recognition engine 311 and the second speech recognition engine 313 as having high performance and may evaluate the third speech recognition engine 315 as having low performance.

[0086] When the speech recognition results of the speech recognition engines 311, 313, and 315 are different, the speech recognition engine evaluation unit 251 may evaluate the performance of all the speech recognition engines 311, 313, and 315 as low. In this case, the dangerous text recognition unit 255 may recognize the text before the corresponding input audio conversion as dangerous text.

[0087] In addition, the speech recognition engine evaluation unit 251 may evaluate the speech recognition engines 311, 313, and 315 in detail with reference to at least one of the speech speed, pitch, and additional noise used to generate the input audio. For example, the speech recognition engine evaluation unit 251 may evaluate the performance of the speech recognition engines 311, 313, and 315 in recognizing male speech expressions and the performance of the speech recognition engines 311, 313, and 315 in recognizing female speech expressions. Through this, the speech recognition engine evaluation unit 251 can distinguish between a speech recognition engine that recognizes male speech expressions well and a speech recognition engine that recognizes female speech expressions well among the speech recognition engines 311, 313, and 315.

[0088] Figure 4 is a diagram for explaining evaluation of an NLU engine according to one embodiment of the present disclosure.

[0089] Reference Figure 4 , the NLU engines 321, 323, and 325 understand the speech recognition results of the speech recognition engine and output NLU results including at least one of the intent or entity. Figure 4 In the explanation, use Figure 3 An operation of evaluating the performance of the NLU engines 321 , 323 , and 325 based on the first utterance recognition result among the utterance recognition results.

[0090] The NLU engine evaluation unit 253 obtains NLU results of the first utterance recognition result understood by multiple NLU engines, including 1-1 NLU result, 2-1 NLU result and 3-1 NLU result outputted by NLU engines 321, 323 and 325 according to the first utterance recognition result.

[0091] The NLU engine evaluation unit 253 evaluates a plurality of NLU engines based on comparison between NLU results. The NLU engine evaluation unit 253 may evaluate an NLU engine that outputs a specific NLU result based on the number of NLU results that are the same or similar to the specific NLU result.

[0092] For example, although the 1-1 NLU result and the 2-1 NLU result are the same, the 3-1 NLU result may be different from the 1-1 NLU result and the 2-1 NLU result. In this case, the NLU engine evaluation unit 253 may evaluate the performance of the first NLU engine 321 and the second NLU engine 323 as high, and evaluate the performance of the third NLU engine 325 as low.

[0093] Figure 5 is a diagram for explaining evaluation of an NLU engine according to another embodiment of the present disclosure.

[0094] refer to Figure 5 First, for the evaluation of the first NLU engine 321, the NLU tags of the speech recognition results are required. In this case, the NLU tags are the same as the NLU tags of the text corresponding to the speech recognition results. Specifically, the NLU tags of the first speech recognition result, the second speech recognition result, and the third speech recognition result are the same as the NLU tags of the text corresponding to the speech recognition results. Figure 1 The NLU tokens used to generate the input audio are the same as in .

[0095] First NLU Engine 321 Understanding Figure 1 and outputs a 1-1 NLU result, a 1-2 NLU result, and a 1-3 NLU result.

[0096] Figure 5 Shown in Figure 4 Among the NLU engines 321, 323 and 325, the NLU result recognition based on the first NLU engine 321 Figure 2 The operation of evaluating the performance of the first NLU engine 321 is performed based on the speech recognition result.

[0097] The NLU engine evaluation unit 253 may evaluate the first NLU engine 321 by comparing the NLU result of the first NLU engine 321 with the NLU token of the utterance recognition result.

[0098] When the 1-1 NLU result is the same as the NLU tag, the NLU engine evaluation unit 253 may evaluate the performance of the first NLU engine 321 as high. Conversely, when the similarity between the 1-2 NLU result and the NLU tag is low, the NLU engine evaluation unit 253 may evaluate the performance of the first NLU engine 321 as low.

[0099] In this manner, the NLU engine evaluation unit 253 evaluates the performance of the first NLU engine 321 based on the extent to which the 1-1 NLU result, the 1-2 NLU result, and the 1-3 NLU result are identical or similar to the NLU token.

[0100] Figure 6 is a diagram for explaining dangerous text recognition according to one embodiment of the present disclosure.

[0101] refer to Figure 6 , the audio converter 230 obtains the additional noise using the noise model 240 and converts the input text sentence into a plurality of speech audios that are different in at least one aspect of speech speed, pitch, and additional noise.

[0102] The first utterance recognition engine 311 recognizes each of the first voice audio, the second voice audio, and the third voice audio, and outputs a first voice audio recognition result, a second voice audio recognition result, and a third voice audio recognition result.

[0103] In this case, because the first voice audio, the second voice audio, and the third voice audio have different acoustic characteristics, the first voice audio recognition result, the second voice audio recognition result, and the third voice audio recognition result may be different.

[0104] The dangerous text recognition unit 255 obtains the speech audio recognition results of the first utterance recognition engine 311 recognizing the plurality of speech audios, and recognizes the text sentence as a dangerous text based on the degree of difference between the speech audio recognition results.

[0105] As an example, when the first voice audio recognition result and the second voice audio recognition result are the same, but the third voice audio recognition result is different from the first voice audio recognition result and the second voice audio recognition result, the dangerous text recognition unit 255 can recognize the text sentence input to the audio converter 230 as dangerous text.

[0106] As another example, when the first voice audio recognition result, the second voice audio recognition result, and the third voice audio recognition result are different, the dangerous text recognition unit 255 may recognize the text sentence as a dangerous text.

[0107] In this way, the dangerous text identification unit 255 can identify dangerous text having a high probability of being misrecognized by the first utterance recognition engine 311. The dangerous text can be used to update the utterance recognition engine later.

[0108] Figure 7 is a diagram for explaining recognition failure detection according to one embodiment of the present disclosure.

[0109] refer to Figure 7 , describes the operation of detecting recognition failure based on NLU results, where Figure 4 The first NLU engine 321 among the NLU engines 321 , 323 , and 325 recognizes the utterance recognition result.

[0110] First, the utterance recognition engines 311 , 313 , and 315 recognize input audio and output a first utterance recognition result, a second utterance recognition result, and a third utterance recognition result.

[0111] The first NLU engine 321 outputs an NLU result as a result of understanding the utterance recognition result.

[0112] The recognition failure detection unit 257 detects recognition failure of the input audio by comparing the NLU result of the first NLU engine 321 that understands the utterance recognition result with the NLU token of the utterance recognition result.

[0113] When any one of the 1-1 NLU result, the 1-2 NLU result, and the 1-3 NLU result is different from the NLU tag of the utterance recognition result, the recognition failure detection unit 257 determines that the utterance recognition system cannot recognize the input audio.

[0114] In addition, when the first utterance recognition result, the second utterance recognition result, and the third utterance recognition result are the same, the NLU engine evaluation unit may determine that the recognition failure of the input audio is caused by the first NLU engine 321. The NLU engine evaluation unit may evaluate the performance of the first NLU engine 321 as low.

[0115] In this way, by checking whether the input audio as the input of the utterance recognition system and the NLU result as the output of the utterance recognition system correspond, the performance of successful utterance recognition from a customer service perspective can be evaluated.

[0116] Meanwhile, when the 1-1 NLU result, the 1-2 NLU result, and the 1-3 NLU result are identical to the NLU tag of the utterance recognition result, the recognition failure detection unit 257 determines that the utterance recognition system successfully recognizes the input audio.

[0117] However, even in the case of successful recognition, when some of the first utterance recognition result, the second utterance recognition result, and the third utterance recognition result are different, the recognition failure detection unit 257 determines that the NLU result contains defects. In other words, although the utterance recognition is successful from the perspective of customer service, defects that occur in the process of deriving the NLU result from the input audio can be detected.

[0118] When some of the first speech recognition result, the second speech recognition result, and the third speech recognition result are different, the speech recognition engine evaluation unit 251 may determine that the defect in the NLU result is caused by at least one of the plurality of speech recognition engines 311, 313, and 315. Specifically, when the first speech recognition result and the second speech recognition result are the same, and the third speech recognition result is different from the first speech recognition result and the second speech recognition result, the speech recognition engine evaluation unit 251 may determine that the defect in the NLU result is caused by the third speech recognition engine 315. The speech recognition engine evaluation unit 251 may evaluate the performance of the speech recognition engine causing the defect as low. In addition, the dangerous text recognition unit 255 may recognize the input audio causing the speech recognition defect as dangerous audio.

[0119] Figure 8 is a flow chart of an evaluation method according to one embodiment of the present disclosure.

[0120] refer to Figure 8 , an evaluation device for evaluating a speech recognition system obtains speech recognition results in which a plurality of speech recognition engines recognize audio (S810).

[0121] The evaluation apparatus obtains an NLU result, wherein a plurality of NLU engines understand each utterance recognition result ( S820 ).

[0122] The evaluation device evaluates the plurality of utterance recognition engines based on a comparison between the utterance recognition results ( S830 ).

[0123] The evaluation device evaluates a plurality of NLU engines based on comparison between the NLU results ( S840 ).

[0124] In addition, the evaluation device may further evaluate each NLU engine by comparing each NLU result with the NLU tag for the utterance recognition result.

[0125] At the same time, the evaluation device can convert the pre-stored text sentences into multiple voice audios that are different in at least one aspect of speaking speed, pitch, and additional noise, and can identify the text sentences as dangerous texts based on the degree of difference between the voice audio recognition results, wherein in the voice audio recognition results, at least one speech recognition engine recognizes multiple voice audios.

[0126] In addition, the evaluation device can detect the recognition failure of the input audio by comparing the NLU result of at least one NLU engine that understands the speech recognition result with the NLU tag of the speech recognition result. When the speech recognition results are the same, the evaluation device can determine that the recognition failure is caused by the NLU engine.

[0127] Even if the NLU result of the NLU engine and the NLU tag of the speech recognition result are the same, when at least two speech recognition results are different, the evaluation device can determine that the NLU result of one NLU engine contains a defect. The evaluation device can determine that the defect in the NLU result of the NLU engine is caused by the speech recognition engine.

[0128] Meanwhile, when there is no NLU tag of the speech recognition result, the evaluation device can generate the NLU tag of the speech recognition result by applying the language model to the NLU result. The generated NLU tag can be used to evaluate the NLU engine or detect recognition failure in the speech recognition system.

[0129] As described above, according to one embodiment of the present disclosure, the speech recognition system may be evaluated and updated based on the result of processing a large amount of voice data in the speech recognition system.

[0130] The different techniques described herein may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The implementation may be in the form of a computer program tangibly embodied in a computer program product (i.e., an information carrier, such as a machine-readable storage device (computer-readable medium) or a propagation signal) for processing or controlling the operation of a data processing device (e.g., a programmable processor, a computer, or multiple computers) by the operation of the data processing device. Computer programs such as the computer programs mentioned above may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. The computer program may be deployed to run on a single computer or multiple computers at a site or distributed on multiple sites and interconnected by a communication network.

[0131] In addition, the components of the embodiments of the present disclosure may use integrated circuit structures such as memories, processors, logic circuits, lookup tables, etc. These integrated circuit structures perform each function described herein by controlling one or more microprocessors or other control devices. In addition, the components of the embodiments of the present disclosure may be specifically implemented by a part of a program or code including one or more executable instructions for performing specific logical functions, and executed by one or more microprocessors or other control devices. In addition, the components of the embodiments of the present disclosure may include or be implemented as a central processing unit (CPU), a microprocessor, etc. that performs various functions. In addition, the components of the embodiments of the present disclosure may store instructions executed by one or more processors in one or more memories.

[0132] For example, processors suitable for processing computer programs include general and special purpose microprocessors, and one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. Typically, a computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or operably coupled to receive data from or transmit data to or both. For example, information carriers suitable for embodying computer program instructions and data include semiconductor memory devices, such as magnetic media (such as hard disks, floppy disks, and tapes), optical media (such as compact disk read-only memories (CD-ROMs) and digital video disks (DVDs)), magneto-optical media (such as floppy disks), read-only memories (ROMs), random access memories (RAMs), flash memory, erasable programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), etc. The processor and memory may be supplemented or incorporated by dedicated logic circuits.

[0133] The processor may execute an operating system and software applications executed on the operating system. In addition, the processor device may access, store, manipulate, process, and generate data in response to software execution. For convenience, there are cases where a single processor device is used, but those skilled in the art will understand that the processor device may include multiple processing elements and / or multiple types of processing elements. For example, the processor device may include multiple processors or a single processor and a single controller. Other processing configurations (such as parallel processors) are also possible.

[0134] Additionally, non-transitory computer-readable media may be any available media that can be accessed by a computer and may include both computer storage media and transmission media.

[0135] This specification includes details of various specific implementations, but they should not be construed as limiting the scope of any invention or the protection claimed, and they should be construed as descriptions of features that may be unique to a particular embodiment of a particular invention. In the context of each embodiment, the specific features described herein may also be implemented in conjunction with a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments independently or in any suitable subcombination. Further, although features may operate in a particular combination and may initially be described as being claimed as such, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be modified into a subcombination or a variant of the subcombination.

[0136] Likewise, although operations are described in a particular order in the accompanying drawings, it should not be understood that such operations must be performed in the particular order shown or in a sequential order to achieve the desired result, or that all described operations should be performed. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of the various device components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and devices can generally be integrated in a single software product or packaged in multiple software products.

[0137] The above description is only an illustration of the technical concept of the present embodiment. A person of ordinary skill in the art can make various modifications and changes without departing from the basic features of each embodiment. Therefore, the present embodiment is not intended to limit but to describe the technical concept of the present embodiment. The scope of the technical concept of the embodiment is not limited by these embodiments. The protection scope of various embodiments should be interpreted by the attached claims. All technical concepts falling within the scope of their equivalents should be interpreted as being included within the scope of the present embodiment.

Claims

1. A device for evaluating a speech recognition system, the speech recognition system comprising a plurality of speech recognition engines and a plurality of natural language understanding engines, the device comprising: one or more processors; as well as At least one storage device storing a program executed by the one or more processors, the program comprising instructions for: Obtaining speech recognition results, wherein the plurality of speech recognition engines recognize input audio; evaluating the plurality of utterance recognition engines based on a comparison between the utterance recognition results; Obtaining a natural language understanding result, wherein the plurality of natural language understanding engines understand each utterance recognition result; and The plurality of natural language understanding engines are evaluated based on a comparison between the natural language understanding results.

2. The device according to claim 1, wherein: The program further includes instructions for: Converting pre-stored text sentences into a plurality of speech audios that differ in at least one of speech speed, pitch, and additional noise; as well as The text sentence is identified as a dangerous text based on the degree of difference between the speech audio recognition results obtained by one of the multiple speech recognition engines recognizing the multiple speech audios.

3. The device according to claim 1, wherein: The program further includes instructions for evaluating each natural language understanding engine by comparing each natural language understanding result to the natural language understanding indicia of the utterance recognition result.

4. The device according to claim 1, wherein: The program further includes instructions for the following operation: detecting recognition failure of the input audio by comparing a natural language understanding result obtained by a natural language understanding engine understanding the utterance recognition result with a natural language understanding tag of the utterance recognition result.

5. The device according to claim 4, wherein: The program further includes instructions for the following operation: determining that the recognition failure is caused by the one natural language understanding engine when the utterance recognition results are the same.

6. The device according to claim 4, wherein: In a case where the natural language understanding result of the one natural language understanding engine is identical to the natural language understanding tag of the speech recognition result, but at least two of the speech recognition results are different, the program further includes instructions for determining that the natural language understanding result of the one natural language understanding engine contains a defect.

7. The device according to claim 6, wherein: The program further includes instructions for determining that a defect in the natural language understanding result of the one natural language understanding engine is caused by at least one utterance recognition engine among the plurality of utterance recognition engines.

8. The device according to claim 1, wherein: The program further includes instructions for generating a natural language understanding tag of the utterance recognition result by applying a language model to the natural language understanding result.

9. A computer-implemented method for evaluating a speech recognition system, the speech recognition system comprising a plurality of speech recognition engines and a plurality of natural language understanding engines, the method comprising: Obtaining speech recognition results, wherein the plurality of speech recognition engines recognize input audio; Obtaining a natural language understanding result, wherein the plurality of natural language understanding engines understand each speech recognition result; evaluating the plurality of utterance recognition engines based on a comparison between the utterance recognition results; and The plurality of natural language understanding engines are evaluated based on a comparison between the natural language understanding results.

10. The method according to claim 9, further comprising: Converting pre-stored text sentences into a plurality of speech audios that differ in at least one of speech speed, pitch, and additional noise; and The text sentence is identified as a dangerous text based on the degree of difference between the speech audio recognition results obtained by one of the multiple speech recognition engines recognizing the multiple speech audios.

11. The method according to claim 9, further comprising: Each natural language understanding engine is evaluated by comparing each natural language understanding result to the natural language understanding tag of the utterance recognition result.

12. The method according to claim 9, further comprising: Recognition failure of the input audio is detected by comparing a natural language understanding result obtained by a natural language understanding engine understanding the speech recognition result with a natural language understanding tag of the speech recognition result.

13. The method according to claim 12, further comprising: When the speech recognition results are the same, it is determined that the recognition failure is caused by the one natural language understanding engine.

14. The method according to claim 12, further comprising: When the natural language understanding result of the one natural language understanding engine is identical to the natural language understanding mark of the speech recognition result, but at least two of the speech recognition results are different, it is determined that the natural language understanding result of the one natural language understanding engine contains a defect.

15. The method according to claim 14, further comprising: It is determined that a defect in the natural language understanding result of the one natural language understanding engine is caused by at least one utterance recognition engine among the plurality of utterance recognition engines.

16. The method according to claim 9, further comprising: A natural language understanding tag of the utterance recognition result is generated by applying a language model to the natural language understanding result.

Images