Voice interaction method and device based on large model, related equipment and program product

Abstract

The invention discloses a voice interaction method and device based on a large model, related equipment and a program product, and relates to the technical field of artificial intelligence. For an input voice instruction of a user, a voice recognition engine is called for voice recognition, and after a recognition text is obtained, a current task type is predicted through a task classification model; and if it is determined that the current interaction needs reference knowledge information, asynchronously acquiring and outputting the configured pad call content while calling a knowledge retrieval tool to retrieve target knowledge. And when the target knowledge is retrieved, splicing the recognition text, calling the large model by using the spliced text to generate a reply text, and outputting the reply text. In order to reduce the time delay brought by the knowledge retrieval process, the configured pad call content is asynchronously called to be output in the knowledge retrieval process, the transition time is smoothed through the interactive pad call function, and the response time delay of voice interaction is reduced. It is ensured that the large model can be called in combination with the retrieved target knowledge to generate the final reply text, and the generation quality of the large model is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a large model-based voice interaction method, device, related equipment and program product. Background Art

[0002] Currently, voice interaction solutions based on large models can achieve complex semantic understanding and natural language generation, and are widely used in scenarios such as smart assistants, smart homes, and in-vehicle systems.

[0003] However, due to the large size and complex calculation of large models, existing voice interaction solutions are prone to high response delays, especially in scenarios where complex conversations require a long wait from the time the user inputs the voice to the time the system outputs the response. Therefore, how to reduce the delay in voice interaction while ensuring the understanding and generation capabilities of large models has become an important challenge for technological development. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a large model-based voice interaction method, device, related equipment and program product to reduce the delay in voice interaction while ensuring the large model's understanding and generation capabilities. The specific solution is as follows:

[0005] In a first aspect, a large model-based voice interaction method is provided, comprising:

[0006] Get the user's input voice command;

[0007] Calling a speech recognition engine to perform speech recognition on the input speech instruction, and after obtaining the recognition text, calling a configured task classification model to determine the current task type based on the recognition text, wherein the task classification model is trained using the instruction text labeled with the task type;

[0008] When the current task type indicates that the interaction needs to refer to knowledge information, a knowledge retrieval tool is called to retrieve target knowledge related to the recognized text, and configured padding content is asynchronously obtained, and the padding content is output;

[0009] After the target knowledge is retrieved, the target knowledge and the recognition text are spliced ​​together, and the spliced ​​text is used to call the configured large model to obtain the final reply text generated by the large model, and the final reply text is output.

[0010] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of outputting the final reply text includes:

[0011] Calling a speech synthesis engine to perform speech synthesis on the final reply text, and temporarily storing the synthesized reply speech in a sending queue;

[0012] When the speech synthesis engine is called to perform speech synthesis on the final reply text, the review engine is asynchronously called to review the final reply text. If the review is passed, the reply speech in the sending queue is output.

[0013] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0014] When the review fails, the sending queue is refreshed to stop outputting the reply voice.

[0015] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0016] After obtaining the recognized text, while calling the configured task classification model, asynchronously calling the large model to instruct the large model to generate an initial reply text corresponding to the recognized text, and temporarily storing the initial reply text after obtaining the initial reply text generated by the large model;

[0017] After obtaining the final reply text generated by the large model, the initial reply text is overwritten.

[0018] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0019] When the current task type indicates that the current interaction does not require reference to knowledge information, the initial reply text is output as the final reply text.

[0020] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0021] When the current task type indicates that this interaction does not require reference to knowledge information, the large model is called to instruct the large model to generate a final reply text corresponding to the recognized text, and output the final reply text.

[0022] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the method is applied to a scheduling module of the server, and the scheduling module includes a primary scheduling module and a secondary scheduling module;

[0023] The primary scheduling module is responsible for data interaction with the speech recognition engine, the large model, the secondary scheduling module, the audit engine and the speech synthesis engine;

[0024] The secondary scheduling module is responsible for data interaction with the task classification model and the knowledge retrieval tool.

[0025] In a second aspect, a large model-based voice interaction device is provided, comprising:

[0026] An input voice acquisition unit, used to acquire a user's input voice command;

[0027] A first data processing unit is used to call a speech recognition engine to perform speech recognition on the input speech instruction, and after obtaining the recognition text, call a configured task classification model to determine the current task type based on the recognition text, wherein the task classification model is trained using the instruction text marked with the task type;

[0028] A second data processing unit is used to call a knowledge retrieval tool to retrieve target knowledge related to the recognized text when the current task type indicates that the interaction needs to refer to knowledge information, and asynchronously obtain the configured padding content and output the padding content;

[0029] The third data processing unit is used to splice the target knowledge and the recognition text after retrieving the target knowledge, and use the spliced ​​text to call the configured large model to obtain the final reply text generated by the large model, and output the final reply text.

[0030] In a third aspect, a large model-based voice interaction system is provided, including: a primary scheduling module and a secondary scheduling module;

[0031] The first-level scheduling module is used to obtain the user's input voice command, call the voice recognition engine to perform voice recognition on the input voice command, and send the recognized text to the second-level scheduling module;

[0032] The secondary scheduling module is used to call the configured task classification model, determine the current task type based on the recognition text, the task classification model is trained using the instruction text annotated with the task type, and when the current task type indicates that the current interaction requires reference to knowledge information, call the knowledge retrieval tool to retrieve the target knowledge related to the recognition text, and asynchronously obtain the configured padding content, and output the padding content;

[0033] The first-level scheduling module is also used to receive the target knowledge sent by the second-level scheduling module, splice the target knowledge and the recognition text, and use the spliced ​​text to call the configured large model to obtain the final reply text generated by the large model, and output the final reply text.

[0034] In a fourth aspect, an electronic device is provided, comprising: a memory and a processor;

[0035] The memory is used to store programs;

[0036] The processor is used to execute the program to implement the various steps of the large model-based voice interaction method described in any one of the first aspects of the present application.

[0037] In a fifth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the large model-based voice interaction method described in any one of the aforementioned first aspects of the present application are implemented.

[0038] In a sixth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the various steps of the large model-based voice interaction method described in any one of the aforementioned first aspects of the present application.

[0039] By means of the above technical scheme, the present application first calls the speech recognition engine for speech recognition for the user's input voice command, and predicts the current task type through the configured task classification model after obtaining the recognition text. If it is determined that the current task type indicates that this interaction requires reference to knowledge information, the configured padding content is obtained asynchronously while calling the knowledge retrieval tool to retrieve the target knowledge related to the recognition text, and the padding content is output in time. After the target knowledge is retrieved, the target knowledge and the recognition text are spliced, and the large model is called using the spliced ​​text to generate the final reply text, and the final reply text is output. Obviously, the present application scheme can timely predict whether this interaction needs to refer to knowledge information through the task classification model. If necessary, the target knowledge needs to be retrieved. In order to reduce the delay caused by the knowledge retrieval process, the present application asynchronously calls the configured padding content for output during the knowledge retrieval process, and smoothes this transition time through the interactive padding function, thereby reducing the response delay of the voice interaction. At the same time, it is ensured that the large model can be called in combination with the retrieved target knowledge to generate the final reply text, and the generation quality of the large model is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0041] Figure 1 A schematic diagram of an implementation system architecture of a large model-based voice interaction method provided in an embodiment of the present application;

[0042] Figure 2 A flow chart of a voice interaction method based on a large model provided in an embodiment of the present application;

[0043] Figure 3 A signaling diagram of a voice interaction process provided in an embodiment of the present application;

[0044] Figure 4 A signaling diagram of a speech synthesis and review process provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a system architecture for multi-component collaboration in a voice interaction process provided by an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of a voice interaction device based on a large model provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] With the rapid development of deep learning, natural language processing and speech recognition technologies, voice interaction systems have gradually become an important way of human-computer interaction. Voice interaction solutions based on large models are derived from the rapid development of deep learning, natural language processing (NLP) and speech recognition technologies. Secondly, the growing demand of users for smart devices has driven the development of these technologies, and traditional rule-based voice interaction systems have gradually been replaced by solutions driven by large models. Its core advantage lies in its powerful natural language understanding and generation capabilities, which can cope with complex conversations, multi-round interactions and multi-tasking. This system breaks the past reliance on fixed instructions and enables users to interact with devices in a more natural way, thereby significantly improving the user experience.

[0050] Due to the large number of parameters in large models, the inference time is long. Especially when facing complex or professional question scenarios raised by users, the model processing takes a long time, affecting the real-time interactive experience. In addition, for conversations in some specific fields (such as medical and legal), general large models still have certain limitations in understanding professional terms.

[0051] In view of this, the present application provides a voice interaction method based on a large model. The method of the present application can be applied to a variety of scenarios that require voice interaction, including but not limited to: smart assistants, smart homes, in-vehicle systems and other scenarios.

[0052] This application provides a large model-based voice interaction method that can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0053] The terminal 100 and the server 200 can be used together to execute the large model-based voice interaction method provided in the embodiments of the present application.

[0054] Next describe Figure 1 The product form of the mid-terminal 100;

[0055] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a vehicle-mounted device, a teaching screen, a wearable device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0056] The server 200 can deploy algorithm engines such as large models. The terminal 100 can be a client, and the user can perform voice interaction through the client. The terminal 100 can send the voice command input by the user to the server 200, and the server 200 performs voice recognition and reply generation, and returns the generated reply text or the synthesized speech corresponding to the reply text to the terminal 100 for output.

[0057] In addition, some algorithm functions on the server 200 can also be migrated to the terminal 100, for example, the terminal 100 performs a speech recognition process and sends the recognized text to the server 200 for reply generation, etc.

[0058] The present application embodiment provides a large model-based voice interaction method, which is illustrated by applying the method to a computer device. The computer device may be Figure 1 The system is composed of a terminal 100 and a server 200. Figure 2 , the voice interaction method based on the large model specifically includes the following steps:

[0059] Step S100: Acquire the user's input voice command.

[0060] Specifically, in a voice interaction scenario, the voice command input by the user in this interaction can be obtained.

[0061] Step S110: calling the speech recognition engine to perform speech recognition on the input voice instruction, and calling the configured task classification model after obtaining the recognition text, and determining the current task type based on the recognition text.

[0062] After the input voice command is obtained, the configured voice recognition engine can be called to perform voice recognition on the input voice command to obtain the recognized text.

[0063] Considering that the complexity of user input voice commands may vary in different scenarios, for some simple voice interaction scenarios, the large model can be directly used to generate reply text using the recognized text. For some complex voice interaction scenarios or vertical voice interaction scenarios, the large model does not work well when generating reply text based solely on recognized text, and needs to rely on external knowledge information.

[0064] In order to distinguish different interaction scenarios, the embodiment of the present application designs a task classification model that can predict the current task type based on the recognition text. The task classification model is trained using instruction text labeled with the task type. The task classification model can use a small model (relative to a large model, the task classification model can use a model with a smaller number of parameters) and is obtained by pre-training with labeled data. The task type of the current voice interaction task can be quickly predicted.

[0065] There are many types of tasks. One possible way to set the task type is to set knowledge tasks and pure generation tasks. Among them, knowledge tasks mean that this interaction needs to rely on external knowledge information to assist the big model in generating reply text. Pure generation tasks mean that this interaction does not need to rely on external knowledge information, and the big model can directly generate reply text based on the recognized text.

[0066] Step S120: When the current task type indicates that this interaction requires reference to knowledge information, a knowledge retrieval tool is called to retrieve target knowledge related to the recognized text, and the configured padding content is asynchronously obtained and output.

[0067] Specifically, if the current task type indicates that this interaction requires reference to knowledge information, the knowledge retrieval tool can be called to retrieve target knowledge related to the recognized text. Furthermore, in order to reduce the latency of the knowledge retrieval process, the present application can also asynchronously obtain the configured padding content while performing knowledge retrieval, and output the padding content.

[0068] The content of the pre-chat can be a pre-configured speech to relieve the user's waiting anxiety and provide a better user experience. The content of the pre-chat can be some interesting, relevant or guiding content. You can select or design appropriate pre-chat content according to specific application scenarios and user needs. The following examples show several different styles of pre-chat content:

[0069] Humor style:

[0070] "Hey, don't worry, I'm using my 'super brain' to generate the best answer for you, it will be ready soon!"

[0071] Informative style:

[0072] “Content is being prepared for you and our models are processing large amounts of data to ensure the most accurate information is provided.”

[0073] Encouraging style:

[0074] "Your request is important, we are working hard to generate an answer for you, please believe us, it will be soon!"

[0075] Guiding style:

[0076] "While you wait, you can think about what key points you would like to see in the answer so that we can better serve you."

[0077] Interaction Style:

[0078] "Hey, while you're waiting for the answer, do you want to take a quiz or guess a riddle? By the time you finish the answer, the answer will be almost ready!"

[0079] Professional style:

[0080] "Our models are generating content for you based on the latest data and algorithms, ensuring accuracy and timeliness.".

[0081] The output of the padding content may be in the form of text, or in the form of padding voice synthesized from the padding content, or in the form of images, videos, etc.

[0082] In this step, while retrieving the target knowledge, the padding content is asynchronously acquired and output, which can reduce the response delay of voice interaction and alleviate the user's waiting anxiety.

[0083] Step S130: After the target knowledge is retrieved, the target knowledge and the recognition text are spliced ​​together, and the configured large model is called using the spliced ​​text to obtain the final reply text generated by the large model, and the final reply text is output.

[0084] Specifically, after the target knowledge related to the recognized text is retrieved, the target knowledge and the recognized text can be spliced ​​to obtain a spliced ​​text. A prompt instruction is generated based on the spliced ​​text to call the large model and instruct the large model to generate a reply text for the recognized text with reference to the target knowledge.

[0085] After obtaining the final reply text generated by the large model, the final reply text can be output, specifically, the final reply text can be output in the form of text, or a speech synthesis engine can be called to synthesize the final reply text into speech, and then the synthesized speech can be output.

[0086] The voice interaction method provided by the embodiment of the present application, for the user's input voice command, first calls the voice recognition engine for voice recognition, and after obtaining the recognition text, predicts the current task type through the configured task classification model. If it is determined that the current task type indicates that this interaction needs to refer to knowledge information, then while calling the knowledge retrieval tool to retrieve the target knowledge related to the recognition text, asynchronously obtains the configured padding content, and outputs the padding content in time. After the target knowledge is retrieved, the target knowledge and the recognition text are spliced, and the large model is called using the spliced ​​text to generate the final reply text, and the final reply text is output. Obviously, the present application scheme can timely predict whether this interaction needs to refer to knowledge information through the task classification model. If necessary, the target knowledge needs to be retrieved. In order to reduce the delay caused by the knowledge retrieval process, the present application asynchronously calls the configured padding content for output during the knowledge retrieval process, and smoothes this transition time through the interactive padding function, thereby reducing the response delay of the voice interaction. At the same time, it is ensured that the large model can be called in combination with the retrieved target knowledge to generate the final reply text, and the generation quality of the large model is guaranteed.

[0087] In a possible implementation, after obtaining the recognized text output by the speech recognition engine, while calling the configured task classification model for classification prediction, the present application can also asynchronously call the large model to instruct the large model to generate a reply text (defined as the initial reply text) corresponding to the recognized text. After obtaining the initial reply text generated by the large model, the initial reply text is temporarily stored.

[0088] If the current task type predicted by the task classification model indicates that this interaction requires reference to knowledge information, search for the target knowledge, splice the target knowledge with the recognition text, and use the spliced ​​text to call the large model to generate the final reply text. After obtaining the final reply text generated by the large model, overwrite the temporarily stored initial reply text.

[0089] If the current task type predicted by the task classification model indicates that this interaction does not require reference knowledge information, the initial reply text can be directly output as the final reply text. In this case, the output timeliness of the final reply text can be further improved, and the efficiency of voice interaction can be improved.

[0090] In another possible implementation, after obtaining the recognized text output by the speech recognition engine, the configured task classification model can be first called for classification prediction, and the large model is temporarily called for generating the reply text. When the current task type predicted by the task classification model indicates that this interaction does not require reference to knowledge information, the large model is called again to instruct the large model to generate the final reply text corresponding to the recognized text, and the final reply text is output.

[0091] In the solution of this embodiment, after obtaining the recognized text, calling the large model to generate the reply text is temporarily postponed. Instead, the current task type is predicted first. Only when the current task type indicates that this interaction does not require reference to knowledge information, the large model is called to generate the final reply text. This can reduce the number of calls to the large model.

[0092] Reference Figure 3 , which illustrates a signaling diagram of an optional voice interaction process.

[0093] The voice interaction process involves the client on the terminal 100 side, and the scheduling module, speech recognition engine ASR, large model LLM, classification model (task classification model), knowledge retrieval tool, quick reply engine, etc. on the server 200 side. Among them, the scheduling module, speech recognition engine, large model, classification model (task classification model), knowledge retrieval tool, and quick reply engine can be deployed on different servers respectively, or they can be deployed on the same or several servers in any combination.

[0094] A possible interaction flow includes:

[0095] Step 1: The user inputs voice through the client, and the input voice command is transmitted to the scheduling module on the server side. The scheduling module is responsible for data interaction with other functional modules.

[0096] Step 2.1 The scheduling module sends the input voice command to the voice recognition engine, requesting voice recognition. Step 2.2 After the voice recognition is completed, the voice recognition engine returns the recognition text to the scheduling module.

[0097] Step 3.1 The scheduling module calls the classification model based on the recognized text to request a classification result prediction, that is, the classification model processes the recognized text to obtain the current task type. Step 3.2 After obtaining the classification result, the classification model returns it to the scheduling module.

[0098] There are two types of classification results (current task type):

[0099] In the first case, the current task type is a knowledge task, which means that this interaction requires reference to knowledge information. In this case, execute steps 4.1-4.7.

[0100] Step 4.1 The scheduling module calls the knowledge retrieval tool to retrieve the target knowledge related to the recognized text. At the same time, asynchronously execute step 4.2. The scheduling module requests the quick reply engine for the quick reply content (filler content). Step 4.3 The quick reply engine returns the filler content to the scheduling module. Step 4.4 After receiving the filler content, the scheduling module outputs the filler content to the client. The output method can specifically be to output the filler content text directly to the client, or the scheduling module can further call the speech synthesis engine to perform speech synthesis on the filler content and output the synthesized speech to the client. Step 4.5 After retrieving the target knowledge, the knowledge retrieval tool returns the target knowledge to the scheduling module. Step 4.6 After the scheduling module splices the target knowledge and the recognized text, it initiates a request to the large model LLM based on the spliced ​​text to request the large model to generate a reply text based on the spliced ​​text. Step 4.7 After the large model generates the reply text, it returns the reply text to the scheduling module.

[0101] In the second case, the current task type is a pure generative task, which means that this interaction does not require reference to knowledge information. In this case, execute steps 5.1-5.2.

[0102] Step 5.1 The scheduling module initiates a request to the large model LLM based on the recognition text to request the large model to generate a reply text based on the recognition text. Step 5.2 After the large model generates the reply text, it returns the reply text to the scheduling module.

[0103] Step 6: After receiving the reply text returned by the large model, the scheduling module outputs the reply text. The output method may be to directly output the reply text to the client, or the scheduling module may further call the speech synthesis engine to perform speech synthesis on the reply text and output the synthesized speech to the client.

[0104] In this embodiment Figure 3 Only one possible voice interaction process is illustrated. Combined with the introduction of the previous embodiment, it can be seen that other optional voice interaction processes can also be adopted, which will not be described in detail here.

[0105] Reference Figure 4 , which illustrates a signaling diagram for an optional speech synthesis and review process.

[0106] After obtaining the reply text to be output, the scheduling module can synthesize the reply text into speech before outputting it. This process involves speech synthesis of the reply text and output review. In order to save the extra time required for review and improve timeliness, this embodiment provides a strategy for parallel output review and speech synthesis, which specifically includes the following steps:

[0107] Step 1.1 After the scheduling module obtains the reply text to be output, it initiates a speech synthesis request to the speech synthesis engine TTS. At the same time, step 2.1 is asynchronously executed to send the reply text to the audit engine for review. The audit engine can review the reply text according to established rules. The audit items include but are not limited to: content security review, content compliance review, content instruction and accuracy review, etc.

[0108] Step 1.2: The speech synthesis engine returns the synthesized speech to the scheduling module after the synthesis is completed. Step 1.3: The scheduling module temporarily stores the synthesized speech in the sending queue.

[0109] Step 2.2 After the audit is completed, the audit engine returns the audit result to the scheduling module. The audit result includes whether the audit is passed or not.

[0110] When the review is passed, step 3.1 is executed and the scheduling module sends the synthesized speech in the sending queue to the client for output.

[0111] If the review fails, execute step 3.2 to refresh the dispatch queue by the scheduling module to stop outputting the synthesized speech.

[0112] By adopting the method of this embodiment, the speech synthesis process of the reply text and the output review process are executed in parallel, which saves the extra time required for review and improves the output response efficiency. According to the test, the time consumption of the server can be reduced by about 100ms.

[0113] In some embodiments of the present application, the scheduling module of the server described in the above embodiments may adopt a primary scheduling strategy, that is, the voice interaction method of the present application is completed by a separate scheduling module. In addition, this embodiment also provides a secondary scheduling strategy, that is, the scheduling module may include a primary scheduling module and a secondary scheduling module, such as Figure 5 shown.

[0114] Among them, the first-level scheduling module is responsible for data interaction with the speech recognition engine ASR, the large model, the second-level scheduling module, the review engine and the speech synthesis engine TTS.

[0115] The secondary scheduling module is responsible for data interaction with the classification model (task classification model), knowledge retrieval tool, quick reply engine, etc.

[0116] In an exemplary solution, the first-level scheduling module obtains the user's input voice command, calls the voice recognition engine to perform voice recognition on the input voice command, and sends the recognized text to the second-level scheduling module after obtaining it.

[0117] The secondary scheduling module calls the configured task classification model, determines the current task type based on the recognized text, and when the current task type indicates that this interaction requires reference to knowledge information, calls the knowledge retrieval tool to retrieve the target knowledge related to the recognized text, and asynchronously obtains the configured padding content, and outputs the padding content. Exemplarily, the secondary scheduling module outputs the padding content to the speech synthesis engine TTS for speech synthesis, and outputs the synthesized speech to the client through the primary scheduling module.

[0118] After the primary scheduling module receives the target knowledge retrieved from the secondary scheduling module, it splices the target knowledge and the recognition text, and uses the spliced ​​text to call the configured large model to obtain the final reply text generated by the large model, and outputs the final reply text. Exemplarily, the primary scheduling module outputs the final reply text to the speech synthesis engine TTS for speech synthesis, and outputs the synthesized speech to the client.

[0119] Furthermore, when the first-level scheduling module determines that the current task type indicates that this interaction does not require reference to knowledge information, it can directly call the large model based on the recognized text, obtain the final reply text generated by the large model, and output it to the speech synthesis engine TTS for speech synthesis. At the same time, it asynchronously calls the review engine to review the final reply text, and outputs the synthesized speech to the client after the review is passed.

[0120] The following describes a large model-based voice interaction device provided in an embodiment of the present application. The large model-based voice interaction device described below and the large model-based voice interaction method described above can refer to each other.

[0121] See also Figure 6 , Figure 6 A schematic diagram of the structure of a large-model-based voice interaction device disclosed in an embodiment of the present application.

[0122] like Figure 6 As shown, the device may include:

[0123] An input voice acquisition unit 11 is used to acquire a user's input voice command;

[0124] A first data processing unit 12 is used to call a speech recognition engine to perform speech recognition on the input speech instruction, and after obtaining the recognition text, call a configured task classification model to determine the current task type based on the recognition text, wherein the task classification model is trained using the instruction text marked with the task type;

[0125] The second data processing unit 13 is used to call a knowledge retrieval tool to retrieve target knowledge related to the recognized text when the current task type indicates that the interaction needs to refer to knowledge information, and asynchronously obtain the configured padding content and output the padding content;

[0126] The third data processing unit 14 is used to splice the target knowledge and the recognition text after retrieving the target knowledge, and use the spliced ​​text to call the configured large model to obtain the final reply text generated by the large model, and output the final reply text.

[0127] In a possible implementation, the process of the third data processing unit outputting the final reply text includes:

[0128] Calling a speech synthesis engine to perform speech synthesis on the final reply text, and temporarily storing the synthesized reply speech in a sending queue;

[0129] When the speech synthesis engine is called to perform speech synthesis on the final reply text, the review engine is asynchronously called to review the final reply text. If the review is passed, the reply speech in the sending queue is output.

[0130] In a possible implementation, the third data processing unit is further configured to:

[0131] When it is determined that the review fails, the sending queue is refreshed to stop outputting the reply voice.

[0132] In a possible implementation, the first data processing unit is further configured to:

[0133] After obtaining the recognized text, while calling the configured task classification model, asynchronously calling the large model to instruct the large model to generate an initial reply text corresponding to the recognized text, and temporarily storing the initial reply text after obtaining the initial reply text generated by the large model;

[0134] After obtaining the final reply text generated by the large model, the initial reply text is overwritten. On this basis, the second data processing unit is also used to:

[0135] When the current task type indicates that the current interaction does not require reference to knowledge information, the initial reply text is output as the final reply text.

[0136] In another possible implementation, the second data processing unit is further configured to:

[0137] When the current task type indicates that this interaction does not require reference to knowledge information, the large model is called to instruct the large model to generate a final reply text corresponding to the recognized text, and output the final reply text.

[0138] In some embodiments of the present application, a large model-based voice interaction system is further provided, which includes a primary scheduling module and a secondary scheduling module;

[0139] The first-level scheduling module is used to obtain the user's input voice command, call the voice recognition engine to perform voice recognition on the input voice command, and send the recognized text to the second-level scheduling module;

[0140] The secondary scheduling module is used to call the configured task classification model, determine the current task type based on the recognition text, the task classification model is trained using the instruction text annotated with the task type, and when the current task type indicates that the current interaction requires reference to knowledge information, call the knowledge retrieval tool to retrieve the target knowledge related to the recognition text, and asynchronously obtain the configured padding content, and output the padding content;

[0141] The first-level scheduling module is also used to receive the target knowledge sent by the second-level scheduling module, splice the target knowledge and the recognition text, and use the spliced ​​text to call the configured large model to obtain the final reply text generated by the large model, and output the final reply text.

[0142] The system provided in this embodiment can distribute the data interaction involved in the voice interaction process to the primary scheduling module and the secondary scheduling module by designing a secondary scheduling strategy. The secondary interaction strategy can execute the voice interaction process more efficiently and improve the overall response time of the system.

[0143] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, vehicle-mounted terminals, teaching large screens, wearable devices, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Figure 7 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603, so as to implement the voice interaction method based on a large model of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0145] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0146] Also provided in an embodiment of the present application is a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any large model-based voice interaction method provided in an embodiment of the present application.

[0147] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any large model-based voice interaction method provided in an embodiment of the present application.

[0148] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0149] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0150] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0151] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0152] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A voice interaction method based on a large model, characterized in that: include: Get the user's input voice command; Calling a speech recognition engine to perform speech recognition on the input speech instruction, and after obtaining the recognition text, calling a configured task classification model to determine the current task type based on the recognition text, wherein the task classification model is trained using the instruction text labeled with the task type; When the current task type indicates that the interaction needs to refer to knowledge information, a knowledge retrieval tool is called to retrieve target knowledge related to the recognized text, and configured padding content is asynchronously obtained, and the padding content is output; After the target knowledge is retrieved, the target knowledge and the recognition text are spliced ​​together, and the spliced ​​text is used to call the configured large model to obtain the final reply text generated by the large model, and the final reply text is output.

2. The method according to claim 1, characterized in that The process of outputting the final reply text includes: Calling a speech synthesis engine to perform speech synthesis on the final reply text, and temporarily storing the synthesized reply speech in a sending queue; When the speech synthesis engine is called to perform speech synthesis on the final reply text, the review engine is asynchronously called to review the final reply text. If the review is passed, the reply speech in the sending queue is output.

3. The method according to claim 2, characterized in that Also includes: When the review fails, the sending queue is refreshed to stop outputting the reply voice.

4. The method according to claim 1, characterized in that: Also includes: After obtaining the recognized text, while calling the configured task classification model, asynchronously calling the large model to instruct the large model to generate an initial reply text corresponding to the recognized text, and temporarily storing the initial reply text after obtaining the initial reply text generated by the large model; After obtaining the final reply text generated by the large model, the initial reply text is overwritten.

5. The method according to claim 4, characterized in that Also includes: When the current task type indicates that the current interaction does not require reference to knowledge information, the initial reply text is output as the final reply text.

6. The method according to claim 1, characterized in that Also includes: When the current task type indicates that this interaction does not require reference to knowledge information, the large model is called to instruct the large model to generate a final reply text corresponding to the recognized text, and output the final reply text.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to a scheduling module on the server side, wherein the scheduling module includes a primary scheduling module and a secondary scheduling module; The primary scheduling module is responsible for data interaction with the speech recognition engine, the large model, the secondary scheduling module, the audit engine and the speech synthesis engine; The secondary scheduling module is responsible for data interaction with the task classification model and the knowledge retrieval tool.

8. A voice interaction device based on a large model, characterized in that: include: An input voice acquisition unit, used to acquire a user's input voice command; A first data processing unit is used to call a speech recognition engine to perform speech recognition on the input speech instruction, and after obtaining the recognition text, call a configured task classification model to determine the current task type based on the recognition text, wherein the task classification model is trained using the instruction text marked with the task type; A second data processing unit is used to call a knowledge retrieval tool to retrieve target knowledge related to the recognized text when the current task type indicates that the interaction needs to refer to knowledge information, and asynchronously obtain the configured padding content and output the padding content; The third data processing unit is used to splice the target knowledge and the recognition text after retrieving the target knowledge, and use the spliced ​​text to call the configured large model to obtain the final reply text generated by the large model, and output the final reply text.

9. A voice interaction system based on a large model, characterized in that: include: Primary scheduling module and secondary scheduling module; The first-level scheduling module is used to obtain the user's input voice command, call the voice recognition engine to perform voice recognition on the input voice command, and send the recognized text to the second-level scheduling module; The secondary scheduling module is used to call the configured task classification model, determine the current task type based on the recognition text, the task classification model is trained using the instruction text annotated with the task type, and when the current task type indicates that the current interaction requires reference to knowledge information, call the knowledge retrieval tool to retrieve the target knowledge related to the recognition text, and asynchronously obtain the configured padding content, and output the padding content; The first-level scheduling module is also used to receive the target knowledge sent by the second-level scheduling module, splice the target knowledge and the recognition text, and use the spliced ​​text to call the configured large model to obtain the final reply text generated by the large model, and output the final reply text.

10. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the large model-based voice interaction method as described in any one of claims 1 to 7.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the large model-based voice interaction method as described in any one of claims 1 to 7 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the large model-based voice interaction method as described in any one of claims 1 to 7 is implemented.

Images