METHOD AND DEVICE FOR PROCESSING INPUT OUTPUTS BY A SPEECH RECOGNITION SYSTEM

Abstract

A method and a device process input utterances through a speech recognition system. The method and the device are implemented by a computer of a speech recognition system to process an utterance received as input. The method includes processing the utterance by a rule-based natural language understanding engine. The method further includes, if the rule-based natural language understanding engine cannot process the utterance, converting a representation of the utterance and enabling a machine learning-based natural language understanding engine to process the utterance using a large language model agent (LLM agent). The method further includes processing the utterance with a converted representation by the machine learning-based natural language understanding engine.

Description

TECHNICAL AREA

[0001] The present disclosure relates to a method and a device that enable a speech recognition system to process input utterances. More precisely, the disclosure relates to a method and a device that enable a speech recognition system to process input utterances using artificial intelligence. BACKGROUND

[0002] The statements in this section merely provide background information in connection with the present disclosure and do not necessarily represent the state of the art.

[0003] Current speech recognition systems are designed according to the single-turn approach. An intent classifier and a slot extractor are used to process user commands, i.e., utterances. Because conventional speech recognition systems use predefined intent classes to recognize and execute commands, they have the advantage of quickly and accurately recognizing the functions supported by the system. However, there are also multi-turn methods, which are context-based, such as human-to-human conversations. Multi-turn methods have limitations. They struggle to process utterances that abbreviate the content of the preceding utterance or refer to objects through the use of pronouns. Another difficulty lies in processing ambiguous utterances that can be interpreted in more than one way.In the field of natural language processing, the first problem is defined as the co-reference resolution problem, and the second as the ambiguity problem. Conventional speech recognition systems were developed by identifying co-reference resolution and ambiguity problems using out-of-domain (OOD) algorithms. The identified OOD utterances are subject to three types of exception handling: misclassification and misuse, incomplete recognition, or recognition with indications of unsupported features.

[0004] As large language models (LLMs), a type of generative AI, become more popular and readily available, the importance of multi-turn dialogue processing increases. However, incorporating generative AI into speech recognition systems presents a challenge. Large language models perform poorly at classifying intent, a task traditionally handled by natural language understanding (NLU). Another issue is the cost of running large language models. Because they have large parameters, they require significant GPU resources for development and execution. Therefore, indiscriminately deploying large language models can ultimately cost the employer more and slow down their service. OVERVIEW

[0005] The following overview is a simplified summary of certain features. It is not exhaustive and is not intended to identify important or critical elements.

[0006] One aspect of the present disclosure is the provision of a method and a device for processing input utterances by a speech recognition system.

[0007] According to at least one embodiment, the present disclosure provides a computer-implemented method for a speech recognition system for processing an input utterance. The method comprises processing the utterance by a rule-based natural language understanding engine. The method further comprises, if the rule-based natural language understanding engine cannot process the utterance, converting a representation of the utterance and enabling a machine learning-based natural language understanding engine to process the utterance using a large language model agent (LLM agent). The method further comprises processing the utterance with a converted representation by the machine learning-based natural language understanding engine.

[0008] According to another embodiment, the present disclosure provides a device for processing an input utterance, i.e., an input utterance. The device comprises at least one memory configured for storing computer-executable instructions and at least one processor.The at least one processor is configured to execute the computer-executable instructions to cause the at least one processor to process the utterance through a rule-based natural language understanding engine; if the rule-based natural language understanding engine cannot process the utterance, to convert a representation of the utterance to enable a machine learning-based natural language understanding engine to process the utterance using an LLM agent; and to process the utterance with a converted representation through the machine learning-based natural language understanding engine.

[0009] The aspects of the present disclosure are not limited to those mentioned above, and other aspects not mentioned here will be clearly understood by experts in this field of technology from the following description.

[0010] These and other features and advantages are described in more detail below. BRIEF DESCRIPTION OF THE DRAWING FIGURES Fig. Figure 1 is a schematic block diagram of a configuration of a speech recognition system according to at least one embodiment of the present disclosure. Fig. Figure 2 is a view of an illustrative procedure performed by a large language model agent to process an input utterance. Fig. Figure 3 is a flowchart of a speech recognition method according to at least one embodiment of the present disclosure. Fig. Figure 4 is a schematic representation of an illustrative configuration of a computing device that can be used to implement the devices and procedures described here. DETAILED DESCRIPTION

[0011] The present disclosure aims to solve technical problems, including the resolution of co-references and ambiguity in the processing of input utterances by a speech recognition system.

[0012] The present disclosure further aims to address the cost-effectiveness and technical problems of implementing a large language model into a speech recognition system.

[0013] The technical problems that this disclosure is intended to solve are not limited to those mentioned above. Other technical problems not mentioned should be apparent to the person skilled in the art from the following description.

[0014] In the following, terms in the singular may also be used in the plural unless otherwise stated.

[0015] Various embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the following description, it should be noted that identical or equivalent elements or components are designated by identical reference numerals, even if they are shown in different drawings. Furthermore, for the sake of clarity and brevity, the following description of various embodiments omits a detailed description of related known components and functions that obscure the subject matter of the present disclosure.

[0016] Furthermore, various ordinal numbers or alphacodes, such as first, second, i), ii), a), b), etc., are prefixed to distinguish one component from another, but not to imply or suggest the substances, order, or sequence of the components. When, in this description, a part "includes" or "encompasses" a component, this means that the part also includes, and does not exclude, other components, unless expressly stated otherwise.

[0017] The following description of the present disclosure in conjunction with the accompanying drawing figures is intended to describe various embodiments of the present disclosure and is not intended to represent the only embodiments in which the technical idea of ​​the present disclosure can be realized.

[0018] If a component, device, element or the like is described in the present disclosure as being intended for a particular purpose or for carrying out an operation, function or the like, the component, device or element should here be considered as being “specifically designed” to fulfill that purpose or to carry out that operation or function.

[0019] The term "engine" as used here refers to a software-based system, subsystem, or process programmed to perform one or more specific functions. An engine can be implemented by at least one or more processors as one or more software modules or components installed on one or more computing devices (computers) at one or more locations. In some examples, one or more computers may be dedicated to a particular engine; in other examples, multiple engines may run on the same computer or computers.

[0020] Fig. Figure 1 is a schematic block diagram of a configuration of a speech recognition system according to at least one embodiment of the present disclosure.

[0021] Referring to Fig. 1 is a speech recognition system 10 a device comprising a rule-based natural language understanding engine 100, a large language model agent (LLM agent) 120 and a machine learning natural language understanding engine 140.

[0022] The speech recognition system 10, which includes the rule-based natural language understanding engine 100, the LLM agent 120, and the machine learning-based natural language understanding engine 140, can be implemented by a computer or engine, such as at least one processor. The speech recognition system 10 can provide the ability to recognize human speech and convert the recognized speech into text or understand it as commands. The speech recognition system 10 enables users to interact with various devices, such as computers, without needing to use an input device, such as a keyboard or mouse. The speech recognition system 10 can be integrated into systems and devices in a wide variety of fields. Examples of systems and devices include mobile devices such as smartphones, smart devices (appliances?), smart speakers, and infotainment systems in cars.

[0023] Speech Recognition System 10 recognizes a user's utterance after it has been input, understands the recognized utterance, and offers a service that responds to the user's utterance. Speech Recognition System 10 may include a speech recognizer, such as a speech recognition device, that converts the user's speech utterance into text. The speech recognition device may use at least one speech recognition engine to convert the user's utterance into input text or an input sentence. The speech recognition engine may be a speech-to-text (STT) engine that applies a speech recognition algorithm or a neural network model to a speech signal representing the user's utterance, thereby converting the speech signal into text. The speech recognizer may also convert the user's utterance into text using a model derived through machine learning or deep learning.

[0024] The transcribed utterance can be understood using natural language understanding (NLU) techniques. Natural language understanding is a subfield of natural language processing (NLP) that enables computers to understand human language and determine its meaning. Natural language understanding uses three main processes to comprehend the meaning of an utterance. The first is intent detection. In this step, the intention behind an utterance is determined. The user of the sentence "Tell me what the weather is like" typically intends to obtain weather information. The second step is entity detection. This involves extracting specific elements (entities) from a sentence. In the sentence "What will the weather be like tomorrow in New York?", for example, "New York" and "tomorrow" are the entities.The third point is contextual understanding. Natural language comprehension is the ability to understand the meaning of words within their context. This is because one and the same word can have different meanings in different contexts. In particular, in at least one embodiment of the present disclosure, "processing" an utterance or text means that, as a result of processing the utterance or text, one obtains at least one intention or entity.

[0025] Embodiments of the present disclosure include two types of natural language understanding engines, including the rule-based natural language understanding engine 100 and the machine learning-based natural language understanding engine 140.

[0026] The rule-based Natural Language Understanding (NLA) engine uses predefined rules and pattern matching to understand user utterances. It analyzes sentence structure and determines meaning according to human-defined rules and patterns. Because the rules are clearly defined, the NLA engine benefits from fast processing speed and easy error detection. However, if the input data contains undefined expressions, the engine's performance can decline rapidly.

[0027] The machine learning-based Natural Language Understanding (NLA) Engine 140 incorporates a self-training algorithm to learn patterns and meanings in text using large datasets. With a labeled dataset, the NLA Engine 140 automatically learns intents and entities from texts. The algorithm derives rules from the data and generalizes them to process new data. Depending on the complexity of the NLA Engine 140, processing speed can vary. When the trained NLA Engine 140 is optimized, it can operate very quickly.In general, well-trained instances of the machine learning-based Natural Language Understanding Engine 140, based on large datasets, have higher performance and accuracy than rule-based models. However, collecting large datasets and training the machine learning-based Natural Language Understanding Engine 140 is time-consuming and expensive.

[0028] The rule-based natural language understanding engine 100 and the machine learning-based natural language understanding engine 140 each have different advantages and disadvantages and can be used in combination. For example, regarding command recognition in smart home control, instructions or commands commonly used in smart home systems have relatively simple and fixed forms. In particular, commands such as "turn on the light," "increase the temperature," and "close the door" can be easily processed using predefined rules. In contrast, a machine learning base is advantageous when the user-spoken commands are complex. A sentence like "Turn the temperature up a little," for example, is beneficial for a machine learning base. A hybrid approach that combines a rule-based and a machine learning base has the advantage of offering both fast response times and accuracy.

[0029] The rule-based Natural Language Understanding (NLA) Engine 100 processes specification-defined, unambiguous utterances, utterances that must be processed, and utterances that are difficult for the machine-learning-based NLA Engine 140 to process. Commands and formal utterances supported by the device's conventional speech recognition features, such as "give directions," "dial," and "help," can be processed by the rule-based NLA Engine 100. The machine-learning-based NLA Engine 140 can handle all unstructured, free-form utterance patterns that cannot be defined by a specification.

[0030] In some implementations, natural language understanding (NLI) techniques are well-suited to processing utterances for performing functions on board a vehicle system. In particular, they can offer scalable performance for domains with large populations of proper names, such as millions of points of interest (POIs) or tens of millions of music tracks. On the other hand, NLI techniques have significant difficulties processing utterances for undefined functions, utterances that do not use specific terminology for vehicle functions, and utterances that do not perform an existing function but instead ask a variety of related questions.For example, if a system only defines the terms "open window" and "close window" and encounters the utterance "My window is broken," the system will most likely respond to one of the two functions of opening or closing the window. Otherwise, the system may respond that it does not understand the utterance, or it may handle the utterance as unsupported under exceptions.

[0031] The processing of the natural language understanding engines 100, 140 is defined as shown in Equation 1. N(U)={Nrules(U)if U is a predefined rule-based utterance nml(U)otherwise

[0032] In some cases, the input utterance U is processed by a predefined rule-based natural language understanding engine (N). rules) is processed. If processing fails, the input utterance U is processed by a machine learning-based natural language understanding engine (N). ml ) processed.

[0033] The LLM agent 120 can complement the performance of rule-based and machine learning-based hybrid models for natural language understanding.

[0034] A large language model (LLM) is trained on a large amount of data. A large language model is typically finalized after fine-tuning its instructions to accurately understand and respond to user queries, and after reinforcement learning to prevent human-favored, biased, or harmful responses. The completed large language model possesses the general ability to understand complex and diverse human queries and to perform new tasks. When applied to speech recognition, large language models have the advantage of understanding the context of a dialogue and generating natural responses. The ability of large language models to understand and generate dialogues has applications in a wide variety of fields and has the potential to replace or complement predefined systems.

[0035] An agent is a system within an AI system that acts autonomously within a specific environment. The agent observes the environment using sensors and selects the optimal behavior through a decision-making algorithm. Based on this optimal behavior, the agent influences the environment.

[0036] The LLM-Agent 120 is an artificial intelligence agent that operates using a large language model. It can leverage natural language processing capabilities to perform various tasks. The LLM-Agent 120 can be used in a wide range of applications, such as virtual assistants, content creation, coding assistants, and more, relying on its ability to understand and generate text, respond in context, and provide information on a variety of topics.

[0037] The LLM agent 120 is used to determine whether the existing natural language understanding engines 100 and 140 can process utterances outside their processing range based on the existing dialogue. The LLM agent 120 restores omitted content based on the context or transforms the representation of an utterance, even if it deviates slightly from the specification, so that it matches the representation of the utterance defined by the specification, provided the utterance is semantically equivalent. If the utterance is ambiguous, the LLM agent 120 can ask the user a response question and continue the dialogue to clarify the ambiguity. If the user's response resolves the ambiguity, the LLM agent 120 can transform the user's utterance into a specification-defined utterance that can be processed by the natural language understanding engine based on the context.Consequently, the LLM agent 120 solves the problem of resolving co-references and the problem of ambiguity that arise in multi-turn dialogues and represent the inadequacies of existing technologies for understanding natural language.

[0038] A problem in resolving co-references arises when a statement omits the content of a preceding utterance or refers to it through the use of pronouns. For example, in the sentence "Maria looked tired. She slept very little last night," "she" refers to "Maria." The problem in resolving the co-reference is recognizing that "she" is "Maria."

[0039] An ambiguity problem arises when something can be interpreted in two or more ways. The word "bank," for example, can mean "river bank" or "money bank." The meaning can change depending on the context.

[0040] To organically integrate the LLM agent 120 with the existing natural language understanding engines 100 and 140, embodiments of the present disclosure place the LLM agent 120 between the rule-based natural language understanding engine 100 and the machine learning-based natural language understanding engine 140. Utterances that can be processed by the rule-based natural language understanding engine are considered unambiguous. Utterances input to the machine learning-based natural language understanding engine 140 may contain ambiguous utterances that were not filtered out by the rule-based natural language understanding engine 100, which the LLM agent 120 determines based on the context of the dialogue.

[0041] Fig. Figure 2 is a view of an illustrative procedure performed by a large language model agent to process an input utterance (input utterance) based on the dialogue context.

[0042] The LLM agent 120 processes the input utterance based on the context of the dialogue. The utterance processed by the LLM agent 120 can be efficiently processed by the machine learning-based natural language understanding engine 140 because it is a sentence that has been freed from co-reference resolution problems or ambiguity.

[0043] If the LLM agent 120 determines that the utterance is one that the conventional natural language understanding engines 100, 140 cannot process, it can use other external systems 1500 to process the response or the response with a feature that it does not support.

[0044] The existing natural language understanding engines 100 and 140 are unable to determine whether an unspecified utterance can be processed by them. Therefore, the LLM agent 120, which possesses strong linguistic knowledge, can process the unspecified utterance based on the context. In particular, the multi-turn dialogue procedure can transform the representation of the utterance sentence so that it can be processed by the existing natural language understanding engines 100 and 140.

[0045] The LLM agent 120 of at least one embodiment of the present disclosure has five types of functions. The functions include the processing of specification-defined utterances (L spec (U)), the processing of specification-defined similar utterances (L similar (U)), the processing of ambiguous statements (L disambiguate(U)), the processing of utterances that require external knowledge (L extend (U)), and the processing of other statements (L other (U)). The functions of the LLM agent 120 of the embodiment of the present disclosure are not limited to these five.

[0046] The respective functions of the LLM agent 120 function as in equation 2 for unspecified utterances that the rule-based natural language understanding engine 100 cannot process. 1. Check whether the utterance U can be processed by a rule-based NLU: Lrules(U)→Processing complete 2. If utterance U is not processed by rule-based NLU, the LLM agent performs context-based processing: −Lrules(U)→L(U,C) 3. The LLM agent performs the following processing based on utterance U and context C: L(U,C){Lspec(U) if U is a specification utterance;Lsimilar(U) if U is similar to a specification utterance;Ldisambiguate(U) if U is ambiguous;Lextend(U) if U requires external knowledge;Lother(U) otherwise

[0047] The user's utterance, converted by the LLM agent 120, is ultimately used as input for the machine learning-based natural language understanding engine 140 (see equation 3 below). L(U,C)→Nml(L,(U,C))

[0048] The LLM agent 120 is organically coupled with the existing natural language understanding engines 100, 140 by using the processes of Equation 2 and Equation 3.

[0049] Fig. 2 shows in (a) a procedure for processing specification-defined utterances (L spec(U)). The current utterance is then checked to see if it is a complete specification-defined utterance. If the current utterance is specification-defined, the LLM agent 120 passes the utterance set to the subsequent machine learning-based natural language understanding engine 140 without any special processing.

[0050] Fig. 2 shows under (b) a method for processing specified similar utterances (L similar(U)). The current utterance is then checked to see if it has the same semantics as the complete specification-defined utterance, even if it has a different expression. If the current utterance has the same meaning as the specification-defined utterance but a different expression, the LLM agent 120 assumes that it is within the functional range of the existing natural language understanding engines 100, 140 and transforms the current utterance into the representation of the specification-defined utterance. Suppose "Let's go to the L-Tower" is not a specification-defined utterance but is semantically equivalent to the specification-defined representative command "Take me to the <ziel>", then the LLM agent 120 passes a converted sentence "Take me to the L-Tower" to the subsequent machine learning-based natural language understanding engine 140.

[0051] Fig. 2 presents a procedure for handling ambiguous statements in (c) (L disambiguate (U)). If the user utterance is ambiguous and allows for multiple interpretations, the LLM agent 120 poses a response question based on the user utterance to clarify it. For example, if the user says, "It's too loud," this utterance is open to multiple interpretations. In this case, the LLM agent 120 clarifies the user's intention by asking a specific question, such as, "Do you want to turn the volume down?" or "Is the noise outside the vehicle the problem?" Once the ambiguity is resolved, the LLM agent 120 transforms the utterance into a specification-defined utterance so that it can be processed by the existing natural language understanding engines 100, 140. This method of handling ambiguous utterances preserves the continuity of the dialogue and supports the precise functionality desired by the user.

[0052] Fig. 2 illustrates under (d) a procedure for handling utterances that require external knowledge (L extend (U)). The LLM agent 120 can generate a hallucination when it receives an utterance that requires specific external knowledge or real-time information. For example, if a user inputs an utterance requiring real-time data, such as "What is the current traffic situation?", the LLM agent 120 may need to query a relevant API or database to respond based on the information received, rather than responding directly, to be precise. In this case, the LLM agent 120 does not respond directly but categorizes it as an intention that requires interaction with an external system. The subsequent generative artificial intelligence system 1400 can then invoke appropriate external knowledge systems 1500 to provide the user with a suitable response based on the information received.

[0053] Fig. 2 shows in (e) a procedure for processing other utterances (L other (U)). The LLM agent 120 applies two types of exception handling to respond appropriately to utterances that do not match the procedure defined by the specification for handling similar utterances (L similar (U)) in Fig. 2(b) and the procedure for dealing with ambiguous statements (L disambiguate (U)) in Fig. 2(c) can be handled. If the utterance attempts to perform an unsupported function, the user is informed that the function is not supported. If the utterance is semantically unintelligible and cannot be processed by the existing natural language understanding engines (100, 140), the user is informed that the utterance is unintelligible and is asked to try again or provide further information. The procedure for handling other utterances (L other (U)) can increase the system's flexibility and provide richer responses to various user utterances, as in Fig. 2 shown under (e).

[0054] As in Fig. As shown in Figure 1, the LLM agent 120 can be designed and implemented to include at least one of the following elements: task prompts 1100, a speech recognition specification document 1200, a dialogue history 1300, and "few-shot learning".

[0055] The task prompts 1100 are input texts used by the LLM agent 120 to guide its behavior when performing a specific task or working in a particular context. These prompts serve to clarify a given situation for the LLM agent 120 and to guide it to respond or act accordingly.

[0056] The Speech Recognition Specification Document 1200 defines the design, implementation, functional requirements, performance criteria, and the like for the Speech Recognition System 10. The Speech Recognition Specification Document 1200 clearly describes the behavior and performance goals of the Speech Recognition System 10 and provides developers with guidance for the design and creation of the Speech Recognition System 10.

[0057] Dialogue history 1300 is a record of the previous dialogue between the user and the LLM agent 120. Dialogue history 1300 preserves the context of the dialogue and helps the LLM agent 120 to conduct a consistent dialogue.

[0058] To maximize the performance of the LLM agent 120, this disclosure outlines specification document 1200 for speech recognition, including a list of representative commands and proper names, as well as detailed task prompts. For example, the utterance "Find a gas station nearby" can be replaced for processing by the representative command "Show me a gas station." In this way, the LLM agent 120 is guided to process a wide variety of utterances.

[0059] Few-shot learning refers to a technique in natural language processing and artificial intelligence where a model is taught to perform a specific task based on a small number of examples. To accurately process user utterances, the LLM agent 120 can contain specific examples (few-shots) and processing methods. As a result, the system should exhibit more consistent behavior. For example, a continuous utterance such as "Give me the Coffee Bean menu," followed by "Then Starbucks?", can be processed more effectively if it is based solely on the context of similar examples. Examples and rules are dynamically selected based on the current input, helping the LLM agent 120 to operate consistently across different situations and accurately determine user intent.

[0060] The speech recognition system 10 according to an embodiment of the present disclosure can also include a Dialogue Manager (DM) 160. The Dialogue Manager 160 plays a key role in the AI ​​system for controlling the dialogue flow, understanding the user's intent, generating appropriate responses, and managing multi-round dialogues. The Dialogue Manager 160 is capable of handling complex tasks by continuously monitoring the dialogue state, handling errors, and cooperating with external systems 1500 as needed. In this way, the Dialogue Manager 160 enables natural interaction with the user. The Dialogue Manager 160 can output the resulting signals to the vehicle, the user device, or the external server to perform processing and provide services that respond to the intent behind the utterance or text entered by the user.If the service that responds to the user's intention is, for example, a vehicle-related control, the Dialog Manager 160 can transmit the resulting signal to the vehicle to perform the vehicle-related control.

[0061] Fig. Figure 3 is a flowchart of a speech recognition procedure implemented by a computer of a speech recognition system 10 according to at least one embodiment of the present disclosure.

[0062] The speech recognition process can include recognizing a user's utterance upon receiving utterance input from the user (S300), understanding the recognized utterance, and providing a service corresponding to the user's utterance. The speech recognition system can include a speech recognizer that converts the user's speech utterance into text.

[0063] The procedure further includes processing, i.e., attempting to process the utterance using a rule-based natural language understanding engine. The procedure further includes determining, i.e., checking whether the transcribed utterance can be processed by the rule-based natural language understanding engine (S302).

[0064] If the utterance can be processed by the rule-based engine 100 for natural language understanding, the procedure can further include processing the utterance (S304) and passing it on, i.e., inputting it, to the dialogue manager 160. The procedure further includes the dialogue manager 160 understanding the intent of the input utterance and generating an appropriate response (S310).

[0065] The procedure may further include the LLM agent 120 determining whether, based on the existing dialogue, utterances other than those processed by the existing natural language understanding engines 100, 140 can be processed. Based on the context, the LLM agent 120 restores the omitted content or converts the representation of an utterance, even if the representation deviates somewhat from the specification, so that it matches the representation of the utterance defined by the specification, provided they are semantically equivalent. If the utterance is ambiguous, the LLM agent 120 poses a response question to the user and continues the dialogue in a way that resolves the ambiguity.Once the user's response has resolved the ambiguity, the LLM agent 120 can reconstruct the user's utterance into a specification-defined utterance that the natural language understanding engine can process based on the context. This solves the technical problems of resolving co-references or ambiguities in multi-party dialogues, a drawback of existing natural language understanding technologies (S306).

[0066] If the utterance cannot be processed by the rule-based natural language understanding engine (100), the procedure may still include the LLM agent (120) transforming the representation of the utterance for the machine learning-based natural language understanding engine (140) in order to process the transformed utterance (S308). In other words, the procedure may still include the utterance being processed by the machine learning-based natural language understanding engine (140) with a transformed representation.

[0067] The processed utterance is transmitted by the machine learning-based natural language understanding engine 140 to the dialogue manager 160. The procedure can further include the dialogue manager 160 understanding the intent of the input utterance and generating a corresponding response (S310).

[0068] Fig. Figure 4 is a schematic diagram of an exemplary configuration of a computing device 40 that can be used to implement the devices and procedures described here.

[0069] The computing device 40 can include non-volatile memory 400, a processor 420, memory 440, an input / output interface 460, and a communication interface 480, or a part thereof. The computing device 40 can be a stationary computing device, such as a desktop computer, server, or the like, as well as a mobile computing device, such as a laptop computer, smartphone, or the like. The computing device 40 can include any specialized hardware accelerator capable of efficiently processing computations on AI models. For example, the computing device 40 can include a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0070] The memory 400 can store programs which, when executed by the processor 420, cause the processor 420 to perform procedures or operations (operations) in accordance with various embodiments of this disclosure. For example, the programs can comprise a plurality of computer-executable instructions that can be executed by the processor 420. The multiple computer-executable instructions can be executed by the processor 420 to cause the processor 420 to perform the procedures or operations described above. The memory 400 can be a single memory or a plurality of memories. In this case, the information required to perform the procedures or operations according to the various embodiments of the disclosure can be stored in a single memory or distributed among the multiple memories.If the Memory 400 consists of multiple memory locations, these may be spatially separated. The Memory 400 can include at least one volatile and one non-volatile memory location. The volatile memory can, for example, include static random access memory (SRAM) or dynamic random access memory (DRAM), and the non-volatile memory can, for example, include flash memory.

[0071] The 420 processor can contain at least one core capable of executing at least one set of computer-executable instructions. The 420 processor can execute computer-executable instructions stored in memory 400. The 420 processor can be a single processor or a plurality of processors.

[0072] Memory 440 retains the stored data even if the power supply to computer 40 is interrupted. Memory 440 can, for example, consist of non-volatile memory or a storage medium such as magnetic tape, an optical disc, or a magnetic disk. Programs stored in memory 440 can be loaded into memory 400 before execution by processor 420. Memory 440 can contain files written in a programming language, and programs generated by a compiler or similar program can be loaded from these files into memory 400. Memory 440 can store data to be processed by processor 420 and / or data that has been processed by processor 420.

[0073] The 460 input / output interface can provide an interface to an input device, such as a keyboard, mouse, etc., and / or to an output device, such as a display, printer, etc. A user can trigger the execution of a program by the 420 processor via the input device and / or display the results of the processing by the 420 processor via the output device.

[0074] The communication interface 480 can provide access to an external network. The computing device 40 can communicate with other devices via the communication interface 480.

[0075] In the device or method according to the present disclosure, the respective components can be arranged such that they are implemented as hardware or software, or as a combination of hardware and software. Furthermore, each component can be functionally implemented by software, and a microprocessor can perform the function by software for each component, if implemented.

[0076] Various illustrative implementations of the systems and methods described here can be realized by digital electronic circuits, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can also be realized in one or more computer programs that can be executed on a programmable system. The programmable system comprises at least one programmable processor coupled to receive and transmit data and instructions to a memory system, at least one input device, and at least one output device, where the programmable processor can be a specialized or a general-purpose processor.Computer programs (also referred to as programs, software, software applications or code) contain computer-executable instructions for a programmable processor and are stored on a "computer-readable recording medium".

[0077] Computer-readable recording media encompasses all types of recording devices capable of storing data readable by a computer system. Examples of computer-readable recording media include non-volatile or non-transient media such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, optical / magnetic disk, storage devices, and the like. Computer-readable recording media can also include transient media such as data transmission media. Furthermore, computer-readable recording media can be distributed across computer systems connected via a network, allowing the computer-readable code to be stored and executed in a distributed manner.

[0078] Although the steps in the respective flowcharts / time diagrams in this description are presented as being carried out sequentially, they merely illustrate the technical concept of some embodiments of the present disclosure. Therefore, a person familiar with the respective embodiments could perform the steps without deviating from the concept and scope of protection of the embodiments by modifying the sequences described in the respective flowcharts / time diagrams or by performing two or more of the steps in parallel. Therefore, the steps in the respective flowcharts / time diagrams are not limited to the chronological sequences shown.

[0079] Although various embodiments of the present disclosure have been described for illustrative purposes, those skilled in the art should be aware that various modifications, additions, and substitutions are possible without deviating from the idea and scope of protection of the claimed disclosure. Therefore, various embodiments of the present disclosure have been described for the sake of brevity and clarity. The scope of the technical idea of ​​the embodiments of the present disclosure is not limited by the illustrations. Accordingly, the scope of the claimed disclosure is not limited by the embodiments expressly described above, but by the claims and their equivalents.

[0080] According to at least one embodiment of the present disclosure, a large language model can be used to solve problems in resolving co-references and ambiguities that arise when processing multi-turn dialogues in a speech recognition system, thereby providing a response that corresponds to the intention of the utterance.

[0081] According to the embodiments, the present disclosure can solve the problem of cost-efficiency in introducing a large language model into a speech recognition system.

[0082] The effects of this disclosure are not limited to those mentioned above. Other effects not mentioned should be apparent to the person skilled in the art from the above description.

[0083] REFERENCE NUMBERS 10 Speech recognition system 40 Computing device (data processing device)< / ziel>

Claims

[1] A method implemented by a computer for a speech recognition system to process an input utterance, the method comprising the following operational steps: Processing the utterance using a rule-based engine to understand natural language; If the rule-based natural language understanding engine cannot process the utterance, convert a representation of the utterance and allow a machine learning-based natural language understanding engine to process the utterance using a large language model agent (LLM agent); and Processing the utterance with a converted representation by the machine learning-based natural language understanding engine. [2] Method according to claim 1, wherein the LLM agent comprises: an agent that is implemented by including at least one of the following elements: a document containing speech recognition specifications, task prompts, a dialogue history, or few-shot learning. [3] The method of claim 1, wherein the conversion of the representation of the utterance comprises: Omitting the conversion of the utterance representation if the utterance is defined in a speech recognition specification document. [4] Method according to claim 1, wherein the conversion of the representation of the utterance comprises: If the utterance is a specification-defined similar utterance, convert the utterance so that it corresponds to a command representation of the utterance as defined in a speech recognition specification document. [5] The method of claim 1, wherein the conversion of the representation of the utterance comprises: If the utterance is one that is not defined in a speech recognition specification document, and the LLM agent is unable to interpret a meaning of the utterance, prompt the LLM agent to use a response question to interpret the meaning of the utterance. [6] The method according to claim 5 further comprises: If the LLM agent is unable to identify the meaning of the utterance using the answer question, the LLM agent should notify the user that the meaning is an unintelligible utterance and ask the user to try again or provide additional information. [7] Method according to claim 1, wherein the conversion of the representation of the utterance comprises: If the utterance is one that can only be answered using information obtained by calling an external system, instruct the LLM agent to call the external system to obtain the information needed to answer the utterance. [8] Method according to claim 1, wherein the conversion of the representation of the utterance comprises: If the utterance is one that is not defined in a speech recognition specification document, and the utterance refers to a feature that is not supported by the speech recognition system, notify the user that the feature is not supported by the speech recognition system. [9] Method according to claim 1, further comprising: Providing the utterance processed by the machine learning-based natural language understanding engine to a dialogue manager; and To prompt the dialogue manager to generate a response that corresponds to the intention of the statement. [10] Non-transitory, computer-readable recording medium with computer-executable instructions recorded thereon for performing each of the operational steps included in the method according to claim 1. [11] Device for processing an input utterance, the device comprising: at least one memory configured to store computer-executable instructions; and at least one processor configured to execute computer-executable instructions to cause the at least one processor to process the utterance using a rule-based engine for natural language understanding, If the rule-based natural language understanding engine cannot process the utterance, a representation of the utterance should be converted to enable a machine learning-based natural language understanding engine to process the utterance by using a large language modeling (LLM) agent, and The utterance is processed with a converted representation by the machine learning-based natural language understanding engine. [12] Device according to claim 11, wherein the LLM agent comprises: an agent that is implemented by including at least one of the following: a speech recognition specification document, task prompts, a dialogue history, or few-shot learning. [13] Device according to claim 11, wherein the at least one processor is configured to execute the computer-executable instructions to cause the at least one processor to: Omitting the conversion of the utterance representation if the utterance is defined in a speech recognition specification document. [14] Device according to claim 11, wherein the conversion of the representation of the utterance comprises: If the utterance is a similar utterance defined in the specification, convert the utterance so that it corresponds to a command representation of the utterance as defined in a speech recognition specification document. [15] Device according to claim 11, wherein converting the representation of the utterance comprises: If the utterance is one that is not defined in a speech recognition specification document, and the LLM agent is unable to interpret a meaning of the utterance, induce the LLM agent to use a response question to interpret the meaning of the utterance. [16] Device according to claim 15, wherein the at least one processor is configured to execute the computer-executable instructions to cause the at least one processor to: to respond to the LLM agent who cannot identify the meaning of the utterance with the answer question; and to instruct the LLM agent to notify a user that the utterance is unintelligible and to ask the user to try again or provide additional information. [17] Device according to claim 11, wherein converting the representation of the utterance comprises: If the utterance is one that can only be answered using information obtained by calling an external system, then the LLM agent should be instructed to call the external system to obtain the information needed to answer the utterance. [18] Device according to claim 11, wherein converting the representation of the utterance comprises: If the utterance is one that is not defined in a speech recognition specification document, and the utterance refers to a feature that is not supported by the device, notify the user that the feature is not supported by the device. [19] Device according to claim 11, wherein the at least one processor is configured to execute the computer-executable instructions to cause the at least one processor to: to make the utterance processed by the machine learning-based natural language understanding engine available to a dialogue manager; and to cause the dialogue manager to generate a response that corresponds to the intention of the statement.

Images