Voice interaction method, server and computer readable storage medium

Abstract

The application discloses a voice interaction method, a server and a computer readable storage medium. The method comprises the following steps: receiving a current voice request forwarded by a vehicle, searching a knowledge database according to the current voice request to obtain a first search result, performing slot identification and application program interface prediction and application program interface parameter filling on the current voice request based on a large language model according to a prompt information template and the first search result to obtain an execution result of the application program interface parameter filling, and outputting the execution result to the vehicle to complete voice interaction. In this way, the knowledge database is searched with the current voice request in the model reasoning process, so that the model can complete the reasoning of the current voice request based on the search result and the prompt information template. The timeliness of the reasoning result can be related to the information in the knowledge database, avoiding the situation that the training data of the model is outdated and the model gives an outdated answer, and ensuring the reasoning accuracy of the model.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, and in particular to a voice interaction method, a server, and a computer-readable storage medium. Background Technology

[0002] In related technologies, in-vehicle voice dialogue systems typically utilize a natural language generation module to parse user statements into machine-understandable semantic tags, and a dialogue state tracking module to maintain an internal dialogue state as a compact representation of the entire dialogue history. Based on this state, a dialogue strategy module selects appropriate dialogue actions, and finally, the natural language generation module converts the dialogue actions into natural language responses. However, the training data used by developers during model training may be classic or currently popular corpora, but these corpora may become outdated after the model is deployed, causing the model to provide invalid answers due to outdated information. Summary of the Invention

[0003] This application provides a voice interaction method, a server, and a computer-readable storage medium.

[0004] This application provides a voice interaction method, including:

[0005] Receive the current voice request forwarded by the vehicle;

[0006] Based on the current voice request, a search is performed on the pre-configured knowledge database to obtain the first search result;

[0007] Based on the large language model, according to the pre-configured prompt information template and the first retrieval result, slot recognition, application interface prediction and application interface parameter filling are performed on the current voice request to obtain the execution result of application interface parameter filling. The large language model is pre-trained and can generate the execution result of application interface parameter filling according to the voice request.

[0008] The execution result is output and sent to the vehicle to complete the voice interaction.

[0009] In the voice interaction method provided in this application, the server can receive the current voice request forwarded by the vehicle, and search the pre-configured knowledge database according to the current voice request to obtain a first search result that is related to the current voice request. Then, based on the pre-trained large language model that can generate application interface parameter filling execution results according to the voice request, the server performs slot recognition, application interface prediction and application interface parameter filling for the current voice request according to the pre-configured prompt information template and the first search result to obtain the application interface parameter filling execution result, and outputs the execution result to the vehicle, thereby completing the voice interaction.

[0010] Thus, in this embodiment of the application, the server can retrieve the knowledge database based on the current voice request during the model reasoning process to obtain the first retrieval result, so that the model can complete the reasoning work of the current voice request based on the first retrieval result and the prompt information template. In this way, the timeliness of the model reasoning result can be related to the information in the knowledge database, thereby avoiding the situation where the model gives an invalid or expired answer due to the expiration of the model's training data to a certain extent, and ensuring the reasoning accuracy of the model.

[0011] In some embodiments of this application, the knowledge database includes an index and data blocks corresponding to the index. The step of retrieving the pre-configured knowledge database according to the current voice request to obtain a first retrieval result includes:

[0012] Based on the current voice request, the index in the knowledge database is retrieved to obtain the target index;

[0013] The first retrieval result is determined based on the data block in the knowledge database that corresponds to the target index.

[0014] The server can use the data block that the target index refers to or represents as the retrieval result of the current voice request.

[0015] Thus, in this embodiment of the application, the server can use the current voice request to retrieve the index corresponding to the data block in the knowledge database, thereby avoiding the situation where the data block is retrieved directly through the current voice request. This reduces the computing resources required to retrieve the knowledge database and ensures the efficiency of the retrieval operation.

[0016] In some embodiments of this application, the method further includes:

[0017] The data block is subjected to embedding encoding to obtain the index corresponding to the data block.

[0018] Thus, in this embodiment of the application, the index of the data block can be obtained by embedding the data block, so that the pointing relationship between the data block and the index corresponding to the data block is clear and unambiguous.

[0019] In some embodiments of this application, the step of retrieving the index in the knowledge database based on the current voice request to obtain the target index includes:

[0020] The current voice request is subjected to the embedding encoding process to obtain the encoding result;

[0021] The target index is obtained by retrieving the index in the knowledge database based on the encoding result.

[0022] Thus, in this embodiment of the application, the server can perform embedded encoding processing on the current voice request to obtain an encoding result that is in the same space or dimension as the index, so that the target index can be reliably obtained by retrieving the encoding result.

[0023] In some embodiments of this application, the knowledge database includes multiple indexes and a data block corresponding to each index. The step of retrieving the indexes in the knowledge database based on the encoding result to obtain the target index includes:

[0024] The target index is determined from the plurality of indexes based on the similarity between the encoding result and each of the indexes.

[0025] Thus, in this embodiment of the application, the server can determine the target index from the indexes of each data block based on the similarity between the encoding result and the indexes of each data block, after determining the encoding result of the current voice request.

[0026] In some embodiments of this application, the knowledge database includes a plurality of indexes and a data block corresponding to each index, and the method further includes:

[0027] The predetermined knowledge data is split into multiple data blocks.

[0028] Thus, in this embodiment of the application, the server can split the complete knowledge data into multiple data blocks after obtaining the knowledge data, thereby avoiding the situation where the complete knowledge data is input into the large language model to prompt the large language model to perform reasoning work.

[0029] In some embodiments of this application, the training steps of the large language model include:

[0030] Obtain the voice request sample and the sample label corresponding to the voice request sample;

[0031] Based on the voice request sample, the knowledge database is searched to obtain a second search result;

[0032] The reference model is trained based on the voice request sample, the sample label, the second retrieval result, and the pre-configured prompt information template to obtain the large language model.

[0033] Thus, in this embodiment of the application, the server can train the reference model based on the voice request sample, the sample tag corresponding to the voice request sample, the second retrieval result corresponding to the voice request sample, and the pre-configured prompt information template, so as to obtain a large language model that can reliably process the downstream tasks required.

[0034] In some embodiments of this application, the method further includes:

[0035] Inject pre-configured vehicle domain knowledge into the base model;

[0036] The base model is trained to obtain the reference model.

[0037] In this way, by injecting knowledge from the vehicle domain into the base model and training the base model, the model can gain a certain understanding of knowledge from the vehicle domain, thereby improving the model's ability to process downstream tasks in the vehicle domain.

[0038] This application provides a server including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the above-described voice interaction method.

[0039] This application provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the above-described voice interaction method.

[0040] The server and computer storage medium provided in this application embodiment can retrieve the knowledge database according to the current voice request during the model reasoning process to obtain the first retrieval result. This allows the model to complete the reasoning work of the current voice request based on the first retrieval result and the prompt information template. In this way, the timeliness of the model reasoning result can be related to the information in the knowledge database, thereby avoiding the situation where the model gives invalid or expired answers due to the expiration of the model's training data to a certain extent, and ensuring the reasoning accuracy of the model.

[0041] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0042] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0043] Figure 1 This is a schematic diagram of the architecture of a traditional dialogue system in related technologies;

[0044] Figure 2 This is a schematic diagram of an end-to-end dialogue system architecture.

[0045] Figure 3 This is a flowchart illustrating the voice interaction method in some embodiments of this application;

[0046] Figure 4 This is a schematic diagram of the architecture of an end-to-end dialogue system based on a large language model in some embodiments of this application.

[0047] Figure 5 This is a flowchart illustrating the voice interaction method in some embodiments of this application;

[0048] Figure 6 This is a flowchart illustrating the voice interaction method in some embodiments of this application;

[0049] Figure 7 This is a flowchart illustrating the voice interaction method in some embodiments of this application;

[0050] Figure 8 This is a flowchart illustrating the voice interaction method in some embodiments of this application;

[0051] Figure 9 This is a flowchart illustrating the voice interaction method in some embodiments of this application. Detailed Implementation

[0052] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0053] Please see Figure 1 Traditional in-vehicle voice system architecture is based on a modular strategy, which divides the work among components to realize the entire dialogue process. It typically includes a natural language understanding module, a dialogue state tracking module, a dialogue strategy module, and a natural language generation module. Among them, the dialogue state tracking module and the dialogue strategy module are collectively referred to as the dialogue management module.

[0054] From the perspective of the overall dialogue processing chain, traditional in-vehicle voice system architectures, after acquiring the user's voice input request, first need to use a natural language understanding module to parse the user's voice request into machine-understandable semantic tags, i.e., perform domain classification, intent recognition, and slot recognition. Then, the dialogue state tracking module maintains the dialogue state, and based on this state, the dialogue strategy module selects the appropriate dialogue action. That is, in the dialogue management module, the dialogue state and dialogue strategy are combined to select and execute the application programming interface (API) that meets the user's input requirements. Finally, the natural language generation module converts the dialogue action into a natural language response, which is the system output returned to the user.

[0055] In traditional in-vehicle voice system architectures, each module integrates models and processing logic from different domains. Each module requires training its own model and implementing its own processing logic, necessitating significant development effort and sufficient domain-specific labeled data. Furthermore, interaction with real users is costly, and data feedback cycles are slow, insufficient to support rapid model iteration, thus limiting the scalability of the dialogue system. Additionally, maintaining the rule system requires substantial manpower and time, further restricting its extensibility. Moreover, due to the high latency requirements of voice interaction in in-vehicle scenarios, the timeliness of the dialogue system executing user commands through four modules and multiple models presents a significant challenge.

[0056] Please see Figure 2 To address the shortcomings and deficiencies of traditional in-vehicle voice system architectures, a new end-to-end voice system architecture has been proposed, comprising three core algorithm modules: NER (Named Entity Recognition), AP (Action Prediction), and AF (Argument Filling). The NER module identifies entities in the user's voice request. The AP module predicts the application programming interface (API) corresponding to the voice request, which aims to achieve the user's current goal. The AF module selects the parameters from the identified entities in the voice request to execute the API obtained in the previous step.

[0057] In other words, the NER module is used to obtain the entities that need to be called in the application programming interface (API), the AP module determines whether the API called by the user voice input is correct, and the AF module selects which entities to use as parameters for the API.

[0058] Understandably, the end-to-end in-vehicle voice system architecture integrates some modules of the traditional in-vehicle voice system architecture, reducing the relatively independent data requirements of each module and the need for manual rule formulation. Due to its streamlined architecture and lack of domain-specific distinctions, it reduces subsequent module maintenance costs and saves on machine and manpower expenses. Meanwhile, vehicle control requirements in in-vehicle scenarios are highly time-sensitive, and the computing power of the vehicle's hardware differs from the computing power used by developers to train offline models, resulting in high latency requirements for voice interaction in in-vehicle environments. The end-to-end architecture, by streamlining intermediate modules, can effectively reduce the latency of the in-vehicle system.

[0059] From the perspective of the overall dialogue processing chain, the end-to-end in-vehicle voice system architecture, after obtaining the user's voice request, will directly generate system instructions or responses by combining the dialogue history. That is, the NER module obtains slot information, the AP module obtains the API to be executed, and the AF module selects the parameters corresponding to the API execution from the identified slots.

[0060] However, the initial development process may be based on popular or classic corpora at the time. In later practical applications, these corpora may become outdated or obsolete, causing the system or module to provide incorrect or invalid answers, resulting in inaccurate, overly general, and weakly relevant responses.

[0061] For example, in the early stages of development, "AAA" might be a word in a music dictionary. However, in later practical applications, due to the broadcast of a certain television program, "AAA" might also refer to the name of a TV series. Therefore, when a user wants to watch a TV series and gives the voice command "play 'AAA'", the system / module might play music named "AAA" instead of the TV series named "AAA," resulting in an incorrect response.

[0062] Furthermore, it is understandable that if the system / module needs to be rebuilt or retrained to adapt to currently popular words or sentences, the reconstruction cost will be high and the number of reconstructions will be large.

[0063] Based on the issues mentioned above, please refer to Figure 3 This application provides a voice interaction method, including:

[0064] 01: Receive the current voice request forwarded by the vehicle;

[0065] 02: Based on the current voice request, search the pre-configured knowledge database to obtain the first search result;

[0066] 03: Based on the large language model, according to the pre-configured prompt information template and the first search result, slot recognition, application interface prediction and application interface parameter filling are performed on the current voice request to obtain the execution result of application interface parameter filling. The large language model is pre-trained and can generate the execution result of application interface parameter filling according to the voice request.

[0067] 04: Output the execution result and send it to the vehicle to complete the voice interaction.

[0068] This application provides a voice interaction device. The vehicle location determination method of this application can be implemented by the voice interaction device of this application. Specifically, the voice interaction device includes a receiving module, a retrieval module, a recognition module, and an interaction module. The receiving module receives a current voice request forwarded by the vehicle. The retrieval module searches a pre-configured knowledge database based on the current voice request to obtain a first retrieval result. The recognition module, based on a large language model, performs slot recognition, application programming interface (API) prediction, and API parameter filling on the current voice request according to a pre-configured prompt information template and the first retrieval result to obtain the API parameter filling execution result. The large language model is pre-trained and can generate the API parameter filling execution result based on the voice request. The interaction module outputs the execution result to the vehicle to complete the voice interaction.

[0069] This application also provides a vehicle, which includes a memory and a processor. The vehicle location determination method of this application can be implemented by the vehicle of this application. Specifically, the memory stores a computer program, and the processor is used to receive a current voice request forwarded by the vehicle, and to retrieve a first search result from a pre-configured knowledge database based on the current voice request, and to perform slot recognition, application interface prediction, and application interface parameter filling on the current voice request based on a large language model, a pre-configured prompt information template, and the first search result to obtain the execution result of application interface parameter filling. The large language model is pre-trained and can generate the execution result of application interface parameter filling based on the voice request, and is used to output the execution result to the vehicle to complete the voice interaction.

[0070] Specifically, with the rapid development of Large Language Models (LLMs), related technologies have made significant progress and have been applied in various fields. Advances in deep learning and big data technologies have increased the scale and complexity of LLMs, thereby improving their performance and expressive capabilities. LLMs can possess hundreds of billions or even trillions of parameters, enabling them to better understand and generate human language, covering a wider range of semantics and contexts. Consequently, LLMs have achieved performance comparable to human capabilities in various natural language processing tasks such as natural language understanding, natural language generation, contextual understanding, text summarization, text translation, and dialogue systems.

[0071] Therefore, this application provides an in-vehicle speech recognition solution based on a large language model, which can be found in the following details. Figure 4 , Figure 4This diagram illustrates the architecture of an end-to-end dialogue system based on a large language model (LLM) in certain embodiments of this application. Specifically, this embodiment leverages the superior capabilities of the LLM model in natural language processing tasks, and trains it accordingly based on the task at hand. This embodiment eliminates three modules (NER module, AP module, and AF module) in the end-to-end speech system architecture, meaning independent modules are no longer required. The language understanding, generation, and contextual understanding capabilities of the LLM model are directly utilized to generate the API and AF tags corresponding to the current speech request. This reduces the burden on developers and saves time and costs.

[0072] Understandably, large language models, which can have hundreds of billions or even trillions of parameters, have high training costs. Furthermore, if the training data used to train a large language model becomes outdated, retraining it presents two main challenges: firstly, the cost of retraining is high; secondly, even if the model is retrained to process currently popular words or sentences, those words or sentences will become outdated in the future, requiring another round of retraining and updates.

[0073] Based on this, in this embodiment, if a user wants the vehicle to perform a specific action, such as playing music, they utter a specific phrase at that moment. The vehicle then receives the user's voice request and forwards it to the server. Upon receiving the forwarded voice request, the server retrieves a pre-configured knowledge database based on the request to obtain a first search result relevant to the voice request. Next, the server inputs the first search result, a pre-configured prompt or instruction, and the voice request into a large language model. This allows the large language model to perform slot identification and application programming interface (API) prediction for the voice request, and to fill in the API parameters to obtain the API parameter filling result. Finally, the execution result of the large language model is output and sent to the vehicle, causing the vehicle to perform the aforementioned specific action to complete the interaction with the user.

[0074] It is understood that the current voice request in the embodiments of this application can represent the action that the user wants the vehicle to perform. For example, in one example, the current voice request is "Play song B by author A (person's name)".

[0075] It is also understood that, in the embodiments of this application, the knowledge database can be used to maintain and / or store corpus information such as currently popular characters, words, and sentences; or, in other words, the knowledge database can be understood as a dictionary used to maintain and / or store corpus information such as currently popular characters, words, and sentences. For example, in one example, the knowledge database includes the following contents: {"token":"X","label":"SONG"}, {"token":"X","label":"ALBUM"}, and {"token":"A","label":"SINGER"}.

[0076] Therefore, in this embodiment of the application, the first search result obtained by searching the knowledge database based on the current voice request may also include currently popular characters, words, and sentences. In one example, based on the above "play Teacher A's X", the first search result obtained by searching the knowledge database may include a combination of one or more of the above {"token":"X","label":"SONG"}, {"token":"X","label":"ALBUM"}, and {"token":"A","label":"SINGER"}.

[0077] Furthermore, since the first search result can also be input into the large language model, and the first search result can be used for reasoning of the current voice request, in order to ensure that the large language model can reasonably understand the first search result after it is input into the large language model, and can reliably complete the reasoning work of the current voice request, the server in this embodiment can also call a pre-configured prompt information template and input the prompt information template into the large language model, so that the large language model can determine the reasoning task and complete the reasoning work according to the prompt information template.

[0078] It should also be noted that the prompt template information is used to assist the large language model in reasoning for the current voice request, such as helping the large language model understand the reasoning task corresponding to the current voice request, and / or helping the large language model understand the role of the first search result.

[0079] It is understood that the prompt template information is content that can be set according to the actual situation. For example, in some embodiments of this application, the prompt information template may include: "The following is a dialogue that takes place in the car, with participants including passengers, drivers, intelligent voice assistants, and agents. Passengers and drivers will chat with each other in the car and also send commands to the intelligent voice assistant; the intelligent voice assistant will listen to all the conversations between passengers and drivers, and when it determines that a command is being sent to itself, it needs to understand the content of the command, break down the task, and provide a series of APIs that can be called and their corresponding parameters; the agent will receive the APIs and their corresponding parameters provided by the voice assistant, execute them, and then provide the execution results. The name of each API is composed of multiple English words and their abbreviations to express its meaning. Next, I will give an instruction received by the system, and at the same time, I will also give the result based on this instruction." Please use the relevant external information retrieved from the external resource library as a reference to provide the final response from the intelligent voice assistant. The response consists of two parts: ACTIONS, indicating the API to be called, and ARGUMENTS, indicating the parameters corresponding to this API. For example, if the user command is "I want to go to a restaurant", please output the result in the following JSON format: {"ACTIONS":"NavigationSearchPoi","ARGUMENTS":[('POI_NAME','restaurant')]}. This indicates that the API to be called is "NavigationSearchPoi", and the type and value of one of the parameters are "POI_NAME" and "restaurant".

[0080] Optionally, in some embodiments of this application, the server may merge the prompt message template, the current voice request, and the first search result to input the merged information into the large language model.

[0081] As an example, in one instance, the merged information might include: "The following is a conversation taking place inside the car, involving a passenger, a driver, a smart voice assistant, and an Agent. The passenger and driver will chat with each other and send commands to the smart voice assistant; the smart voice assistant will listen to all the conversations between the passenger and driver, and when it determines that a command is being sent to itself, it needs to understand the command content, break down the task, and provide a series of callable APIs and corresponding parameters; the Agent will receive the APIs and corresponding parameters provided by the voice assistant, execute them, and then provide the execution results. Each API name is composed of multiple English words and their abbreviations to express its meaning. Next, I will provide a command received by the system, along with relevant external information retrieved from external resource libraries based on this command. Please use this information as a reference to provide the final response from the smart voice assistant, which includes two parts: ACTION." S represents the API to be called, and ARGUMENTS represents the parameters corresponding to this API. For example, if the user command is "I want to go to the restaurant", please output the result in the following JSON format: {"ACTIONS":"NavigationSearchPoi","ARGUMENTS":[('POI_NAME','restaurant')]}. This indicates that the API to be called is "NavigationSearchPoi", and the type and value of one of the parameters is "POI_NAME" and "restaurant". The user command is: Play Teacher A's X. The external information is: [{"token":"X","label":"SONG"},{"token":"X","label":"ALBUM"},{"token":"Teacher A","label":"SINGER"}]].

[0082] In the above merged information, "user instruction" can be understood as the current voice request. "External information" can be understood as the first search result.

[0083] Furthermore, it should be understood that the embodiments of this application can input the current voice request, the first search result, and the prompt information template into the large language model, thereby performing slot identification, API (Application Programming Interface) prediction, filling the entities obtained from the corresponding slot identification into the corresponding parameters of the application interface, and outputting the result of parameter filling.

[0084] As an example, in one instance, the current voice request is "pause movement of the right-side sliding door in the second row". The API prediction result obtained based on the large language model can be 'ControlSet', and the AF result can be [('DEVICE','sliding door'),('SET_TYPE','pause'),('POSITION','right-side of the second row')].

[0085] Furthermore, in some embodiments of this application, the execution result of parameter filling can be output in JSON (JavaScript Object Notation) format, for example: the output format is: {'ACTIONS':'ControlSet','ARGUMENTS':[('DEVICE','Sliding Door'),('SET_TYPE','Pause'),('POSITION','Second Row Right Side')]}.

[0086] In summary, in the embodiments of this application, the server can retrieve the knowledge database based on the current voice request during the model reasoning process to obtain the first retrieval result, so that the model can complete the reasoning work of the current voice request based on the first retrieval result and the prompt information template. In this way, the timeliness of the model reasoning result can be related to the information in the knowledge database, thereby avoiding the situation where the model gives an invalid or outdated answer due to the expiration of the model's training data to a certain extent, and ensuring the reasoning accuracy of the model.

[0087] Furthermore, if the knowledge database maintains currently popular corpus information or natural language information, the implementation method of this application can be based on the knowledge database, so that the large language model can generate a more accurate, timely and credible answer based on the first retrieval result corresponding to the current voice request. Therefore, it can effectively improve the usability of the vehicle voice recognition solution at the same time.

[0088] Moreover, compared to Figure 1 or Figure 2 As shown in the architecture, the implementation of this application simplifies the architecture of the voice interaction system and utilizes a pre-trained large language model to directly complete tasks including slot recognition, application interface prediction, and application interface parameter filling, thereby directly processing voice requests and completing voice interaction. The implementation of this application simplifies the architecture of the voice interaction system and utilizes a pre-trained large language model, reducing the need to call related models in different domains, effectively reducing latency, improving the response speed to user voice requests, and simplifying the architecture also facilitates subsequent maintenance and saves costs.

[0089] Please see Figure 5In some embodiments of this application, the knowledge database includes an index and corresponding data blocks, and therefore step 02 includes:

[0090] 020: Based on the current voice request, retrieve the index from the knowledge database to obtain the target index;

[0091] 021: Determine the first search result based on the data block in the knowledge database that corresponds to the target index.

[0092] The retrieval module of this application embodiment is further configured to retrieve the index in the knowledge database according to the current voice request to obtain the target index, and to determine the first retrieval result according to the data block in the knowledge database corresponding to the target index.

[0093] The processor in this embodiment is further configured to retrieve an index from a knowledge database based on a current voice request to obtain a target index, and to determine a first retrieval result based on a data block in the knowledge database corresponding to the target index.

[0094] Specifically, to improve retrieval efficiency, the knowledge database of the server in this application embodiment stores data blocks and corresponding indexes. Thus, when the server retrieves the knowledge database based on the current voice request, it can avoid directly retrieving data blocks with large data volumes through the current voice request, and instead retrieve indexes with smaller data volumes through the current voice request, thereby improving retrieval efficiency and reducing the resources and load required for retrieval.

[0095] In this application, the data blocks in the embodiments can be understood as the aforementioned {"token":"X","label":"SONG"}, {"token":"X","label":"ALBUM"} and {"token":"A","label":"SINGER"}.

[0096] Furthermore, in the embodiments of this application, the database index can be understood as information that can characterize or refer to data blocks to a certain extent. In some embodiments of this application, the information obtained by compressing data blocks can be understood as the index corresponding to the data blocks. In other embodiments, the information obtained by extracting features from data blocks can be understood as the index corresponding to the data blocks, such as the hash code of the data blocks.

[0097] Understandably, compared to directly retrieving a large data block through the current voice request, the server requires fewer computing resources and has higher execution efficiency when retrieving the index information corresponding to the data block through the current voice request.

[0098] It is also understandable that, in the case of retrieving an index from the knowledge database based on the current voice request and obtaining an index related to or matching the current voice request, or in other words, in the case of obtaining the target index, the server in this application embodiment can use the data block referred to or represented by the target index as the retrieval result of the current voice request.

[0099] Thus, in this embodiment of the application, the server can use the current voice request to retrieve the index corresponding to the data block in the knowledge database, thereby avoiding the situation where the data block is retrieved directly through the current voice request. This reduces the computing resources required to retrieve the knowledge database and ensures the efficiency of the retrieval operation.

[0100] In some embodiments of this application, the voice interaction method further includes:

[0101] The data blocks are embedded and encoded to obtain the index corresponding to the data blocks.

[0102] The voice interaction device according to the embodiments of this application further includes an encoding module. The encoding module is used to perform embedded encoding processing on data blocks to obtain an index corresponding to the data blocks.

[0103] The processor in this embodiment is also used to perform embedded encoding processing on the data block to obtain an index corresponding to the data block.

[0104] Specifically, in order to establish a clear pointing relationship between data blocks and their corresponding indexes, and to reduce the storage space required to store data blocks and indexes, the server in this embodiment can also perform embedding encoding processing on each data block in the knowledge database, or encode each data block into a corresponding embedding vector, thereby obtaining the index corresponding to each data block.

[0105] Thus, in this embodiment of the application, the index of the data block can be obtained by embedding the data block, so that the pointing relationship between the data block and the index corresponding to the data block is clear and unambiguous.

[0106] Please see Figure 6 In some embodiments of this application, step 020 includes:

[0107] 0200: Perform embedding encoding on the current voice request to obtain the encoding result;

[0108] 0201: Retrieve the index in the knowledge database based on the encoding results to obtain the target index.

[0109] The retrieval module in this embodiment is also used to perform embedded encoding processing on the current voice request to obtain the encoding result, and is also used to retrieve the index in the knowledge database based on the encoding result to obtain the target index.

[0110] The processor in this embodiment is also used to perform embedded encoding processing on the current voice request to obtain an encoding result, and is also used to retrieve the index in the knowledge database based on the encoding result to obtain the target index.

[0111] Specifically, to ensure the reliable execution of the retrieval operation, the embodiments of this application may also perform embedded encoding processing on the current voice request before retrieving the knowledge database, so that the current voice request and the data block can be in the same space or the same dimension, thereby ensuring the reliable execution of the retrieval operation.

[0112] Specifically, in the embodiments of this application, the server performs embedding encoding processing on each data block to obtain the index corresponding to each data block, and maintains each data block and its corresponding index based on the knowledge database. After obtaining the current voice request forwarded by the vehicle, the server performs embedding encoding processing on the current voice request to obtain the encoding result of the current voice request. Then, using the encoding result of the current voice request, the server retrieves the index corresponding to each data block in the knowledge database, thereby obtaining a target index that is related to or similar to the encoding result.

[0113] Optionally, in some embodiments of this application, the server performs embedding encoding on each data block to obtain the embedding vector of each data block, which is also an index; and, when the server receives the current voice request, it performs embedding encoding on the current voice request to obtain the corresponding embedding vector, which is also an encoding result.

[0114] Furthermore, since both the encoding result and the index are based on the embedding vector obtained from the embedding encoding process, the encoding result and the index can reside in the same vector space. Thus, the server can determine one or more indices that are closest to the encoding result based on the distance between the encoding result and the index in the vector space, thereby obtaining the target index.

[0115] Thus, in this embodiment of the application, the server can perform embedded encoding processing on the current voice request to obtain an encoding result that is in the same space or dimension as the index, so that the target index can be reliably obtained by retrieving the encoding result.

[0116] In some embodiments of this application, the knowledge database includes multiple indexes and a data block corresponding to each index, and step 0201 includes:

[0117] The target index is determined from multiple indexes based on the similarity between the encoding result and each index.

[0118] The retrieval module in this application embodiment is also used to determine the target index from multiple indexes based on the similarity between the encoding result and each index.

[0119] The processor in this embodiment is also used to determine a target index from multiple indexes based on the similarity between the encoding result and each index.

[0120] Specifically, to ensure the reliability of the search results, the embodiments of this application can retrieve knowledge databases based on similarity search.

[0121] Specifically, to more clearly illustrate the implementation methods of this application, please participate in... Figure 7 , Figure 7 This is a flowchart illustrating the voice interaction method in some embodiments of this application. For example... Figure 7 As shown, in this embodiment of the application, after receiving the current voice request forwarded by the vehicle, the server can search the index in the knowledge database based on the similarity search method. In other words, after embedding the current voice request to obtain the encoding result, the server can retrieve the index vector warehouse in the knowledge database based on the encoding result to obtain one or more indexes similar to the encoding result, i.e., the target index.

[0122] Furthermore, such as Figure 7 As shown, the server can send the current voice request and the data block corresponding to the target index (i.e., Figure 7 The "indexed data block" and the prompt message template (not in) Figure 7 (As shown in the image) are input into the large language model, so that the large language model can complete the reasoning work of the current voice request and output the corresponding answer based on the data block corresponding to the target index and the prompt information template.

[0123] It is understood that the server in this application uses a similarity search method to search the knowledge database for target indexes similar to the encoded result. Therefore, in some embodiments of this application, the server can determine the similarity between the encoded result and each index, so as to determine the index with the highest similarity as the target index.

[0124] Thus, in this embodiment of the application, the server can determine the target index from the indexes of each data block based on the similarity between the encoding result and the indexes of each data block, after determining the encoding result of the current voice request.

[0125] Optionally, in other embodiments of this application, the server may determine the top n indexes with the highest similarity among all indexes as the target indexes based on the similarity between the encoding result and each index. It is understood that the value of n can be set as needed.

[0126] For example, in some embodiments of the present application, the input information received by the large language model includes the current voice request, the prompt information template, and the data block corresponding to the target index. Since the prompt information template contains a large number of characters or semantic units (i.e., tokens), in order to avoid the input information being too long and affecting the inference performance of the large language model, the value of n above is 3.

[0127] Exemplarily, in some embodiments of the present application, the prompt information template further includes the description information of multiple APIs. Furthermore, the input information composed of the prompt information template, the current voice request, and the data block corresponding to the target index may include: "The following is a conversation that occurs in a car. The participants are a passenger, a driver, a smart voice assistant, and an Agent. The passenger and the driver will chat with each other in the car and also send instructions to the smart voice assistant. The smart voice assistant will listen to all the conversations between the passenger and the driver. When it judges that an instruction is sent to itself, it needs to understand the instruction content and then perform task decomposition, giving a series of callable APIs and corresponding parameters. The Agent will receive the APIs and corresponding parameters given by the voice assistant, execute them, and then give the execution results. The name of each API is composed of multiple English words and their abbreviations to express its meaning. Next, I will first give you several APIs and their meanings. Please read the API definitions, understand the meanings they constitute, and at the same time understand the connection between the API definitions and the API names. <API definition> (this part has been omitted). Finally, I will give an instruction received by the system. At the same time, I will also give the relevant external information in the external resource library retrieved based on this instruction. Please use this part of the information as a reference to give the final reply of the smart voice assistant. The reply includes two parts. ACTIONS represents the API to be called, and ARGUMENTS represents the parameters corresponding to this API. For example, if the user instruction is \"I want to go to Zhongguancun\", please output the result in the following JSON format: {\"ACTIONS\":\"NavigationSearchPoi\",\"ARGUMENTS\":[('POI_NAME','Zhongguancun')]}. It means that the API to be called is \"NavigationSearchPoi\", and the type and value of a corresponding parameter are \"POI_NAME\" and \"Zhongguancun\". The user instruction is: Play song Q of Teacher Z. The external information is: [{\"token\":\"song Q\",\"label\":\"SONG\"},{\"token\":\"song Q\",\"label\":\"ALBUM\"},{\"token\":\"Teacher Z\",\"label\":\"SINGER\"}]."

[0128] In some embodiments of this application, the knowledge database includes multiple indexes and a data block corresponding to each index, and the voice interaction method further includes:

[0129] The predetermined knowledge data is split into multiple data blocks.

[0130] The voice interaction device according to the embodiments of this application further includes a splitting module. The splitting module is used to split the predetermined knowledge data into multiple data blocks.

[0131] The processor in this embodiment is also used to split the predetermined knowledge data to obtain multiple data blocks.

[0132] Specifically, when the server in this application collects currently popular corpus information such as music titles, singer names, album names, place names, etc., which are highly time-sensitive, or when it collects highly time-sensitive knowledge data, it can split the complete knowledge data into multiple parts, that is, into multiple data blocks.

[0133] Understandably, when inputting complete knowledge data into a large language model to prompt it to perform inference for the current voice request, the sheer number of semantic units in the complete knowledge data might negatively impact the inference execution rate. Furthermore, the complete knowledge data may only be partially relevant to the current voice request; therefore, inputting complete knowledge data into the large language model could introduce noise into its inference process.

[0134] Therefore, the server in this application embodiment can, based on the splitting and processing of knowledge data, input a small amount of data block that is related to the current voice request into the large language model when the large language model needs to perform reasoning work for the current voice request. This allows the large language model to perform reasoning work for the current voice request based on the prompts of the data block, and avoids the situation where the large language model may output incorrect results due to noise in the complete knowledge data when performing reasoning work based on the complete knowledge data.

[0135] Thus, in this embodiment of the application, the server can split the complete knowledge data into multiple data blocks after obtaining the knowledge data, thereby avoiding the situation where the complete knowledge data is input into the large language model to prompt the large language model to perform reasoning work.

[0136] Furthermore, it can be understood that the data size of each data block in the embodiments of this application is set according to the actual situation.

[0137] For example, in some embodiments of this application, since the prompt message template contains descriptions of multiple APIs, in order to avoid the input information consisting of "prompt message template, current voice request and data block corresponding to target index" being too long, thus affecting the reasoning work of the large language model, each data block can be composed of 20 to 30 semantic units (i.e., tokens).

[0138] Please see Figure 8 In some embodiments of this application, the training steps of the large language model include:

[0139] 05: Obtain the voice request sample and the corresponding sample label;

[0140] 06: Based on the voice request sample, a search is performed on the knowledge database to obtain the second search result;

[0141] 07: Based on the voice request samples, sample tags, second search results, and pre-configured prompt information templates, train the reference model to obtain the large language model.

[0142] The voice interaction device according to this application further includes an acquisition module, a database retrieval module, and a training module. The acquisition module acquires voice request samples and corresponding sample tags. The database retrieval module retrieves a knowledge database based on the voice request samples to obtain a second retrieval result. The training module trains a reference model to obtain a large language model based on the voice request samples, sample tags, the second retrieval result, and pre-configured prompt information templates.

[0143] The processor in this embodiment is also used to obtain a voice request sample and a sample tag corresponding to the voice request sample, and to search the knowledge database according to the voice request sample to obtain a second search result, and to train a reference model to obtain a large language model according to the voice request sample, the sample tag, the second search result and the pre-configured prompt information template.

[0144] It is understood that the large language model in the embodiments of this application can be understood as a natural language processing model specifically trained for performing and processing specific downstream tasks. In other words, the large language model in the embodiments of this application has better processing performance on specific downstream tasks compared to other general models or basic models, such as being able to more accurately identify the slot corresponding to the voice request and predict the API corresponding to the voice request, and being able to complete the AF work more accurately.

[0145] Furthermore, it should be noted that the reference model in the embodiments of this application can be obtained by training a base model (or foundation model). It should also be noted that, compared to the base model, the reference model possesses more knowledge in a specific vertical domain. However, the reference module still lacks the downstream task processing capabilities required by the embodiments of this application. Therefore, the reference model needs to be adjusted and updated so that the updated reference module can be applied to the aforementioned downstream tasks, thereby obtaining the large language model required by the embodiments of this application.

[0146] It should also be noted that the voice request samples and corresponding sample labels in the embodiments of this application can be understood as training data for the reference model. It is understood that the method of obtaining the voice request samples and their corresponding sample labels is customizable according to actual circumstances. For example, in some embodiments, the voice request samples and their corresponding sample labels are derived from historically collected user logs, while in other embodiments, the voice request samples and their corresponding sample labels are manually set or constructed data.

[0147] Furthermore, it is also understandable that the specific form of the voice request sample and its corresponding sample label can be set according to the actual situation. For example, in one example, the voice request sample can be "pause movement of the right-side sliding door in the second row". The sample label corresponding to this voice request sample can include the API label and the AF label. The API label can be 'ControlSet', and the AF label can be [('DEVICE','sliding door'),('SET_TYPE','pause'),('POSITION','right-side of the second row')].

[0148] Furthermore, in order to ensure that the large language model can accurately and reliably complete the reasoning work of voice requests (referring to the current voice request and the voice request sample) in both the early training and later application, the embodiments of this application, while inputting the voice request sample into the reference model, will also use the voice request sample to search the knowledge database to obtain the corresponding second retrieval result, and input the second retrieval result and the voice request sample together into the reference model, so that when the reference model performs the reasoning work of the voice request sample, it can output an appropriate answer based on the prompts of the second retrieval result.

[0149] Furthermore, to ensure that the large language model can accurately understand the processing method of voice requests and reliably complete the reasoning work of voice requests during the early training and later application, the server in the embodiments of this application can also input a pre-configured prompt information template into the reference model while inputting the voice request sample and the second retrieval result into the reference model, so that the reference model can understand and complete the reasoning work of the voice request sample based on the prompt information template.

[0150] Furthermore, to enable the reference model to be applicable to downstream users requiring processing in the embodiments of this application, the server in the embodiments of this application may employ Supervised Fine Tuning (SFT) to train the reference model. That is, after the reference model outputs the prediction results of the voice request samples (such as API prediction results, slot recognition results, parameter filling results, etc.), the training parameters (such as weight parameters and bias parameters) in the reference model are updated in reverse using the difference between the prediction results and the sample labels of the voice request samples, so that the trained reference model can reliably complete the processing of the current voice request.

[0151] Thus, in this embodiment of the application, the server can train the reference model based on the voice request sample, the sample tag corresponding to the voice request sample, the second retrieval result corresponding to the voice request sample, and the pre-configured prompt information template, so as to obtain a large language model that can reliably process the downstream tasks required.

[0152] Please see Figure 9 In some embodiments of this application, the voice interaction method further includes:

[0153] 08: Inject pre-configured vehicle domain knowledge information into the base model;

[0154] 09: Train the base model to obtain the reference model.

[0155] The voice interaction device according to this application embodiment further includes an injection module and a basic model training module. The injection module is used to inject pre-configured vehicle domain knowledge information into the basic model. The basic model training module is used to train the basic model to obtain a reference model.

[0156] The processor in this embodiment is also used to inject pre-configured vehicle domain knowledge information into the base model, and to train the base model to obtain a reference model.

[0157] It is understandable that while a basic model has certain general natural language processing capabilities, it may misunderstand the meaning of words and characters in a specific vertical domain due to a lack of knowledge of that domain, thus making it difficult to apply to natural language processing tasks in that domain.

[0158] Therefore, in this embodiment of the application, the server can inject knowledge into the basic model with a certain natural language processing capability, so as to inject relevant knowledge information in the vehicle domain into the basic model, thereby enabling the basic model to correctly understand the information in the vehicle vertical domain.

[0159] It should be noted that the term "vehicle field" in this application's embodiments should be broadly understood to refer to knowledge and information that is directly or indirectly related to vehicles. For example, information directly related to vehicles may include vehicle components and their executable functions. Information indirectly related to vehicles may include multimedia functions, navigation functions, etc., related to the vehicle's cockpit.

[0160] Furthermore, it should be noted that injecting the above information into the base model aims to enhance the base model's cognition and understanding of vehicle-specific knowledge, thereby reducing the reliance on querying external knowledge of the vehicle-specific domain during the reasoning process.

[0161] Furthermore, it should be clarified that in the embodiments of this application, the basic model can be trained based on pre-configured vehicle knowledge information, so that the basic model can learn various knowledge directly or indirectly related to the vehicle, and thus reliably understand the semantics of words in the voice request during the reasoning process, thereby filling the entity into the correct interface parameters.

[0162] It's also understandable that a reference model can be obtained after training the base model, and the training process for generating the reference model is usually unsupervised. As mentioned earlier, the update of the reference model is based on SFT, that is, it updates the pre-trained language model based on labeled sample data. The labeled sample data refers to the voice request samples and their corresponding labels in the aforementioned example.

[0163] In other words, the reference model obtained after training can provide general, universal answers, but may deviate from the requirements of the task. For example, in vehicle-related fields, the parameter types of some interfaces differ from the usual ones. During supervised fine-tuning, the model weights are adjusted based on the differences from the true labels. Through this adjustment process, the model can capture task-specific patterns and characteristics in the labeled data, making the model more accurate and better adapted to a particular task.

[0164] In this way, by injecting knowledge from the vehicle domain into the base model and training the base model, the model can gain a certain understanding of knowledge from the vehicle domain, thereby improving the model's ability to process downstream tasks in the vehicle domain.

[0165] This application also provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the above-described voice interaction method.

[0166] In this specification, the terms "specifically," "furthermore," "particularly," "understandably," etc., refer to specific features, structures, materials, or characteristics described in connection with embodiments or examples that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0167] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0168] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A voice interaction method, characterized in that, include: Receive the current voice request forwarded by the vehicle; The current voice request is subjected to embedding encoding processing to obtain the encoding result; The index in the knowledge database is retrieved based on the encoding result to obtain the target index. The knowledge database includes multiple indexes and data blocks corresponding to each index. The index is obtained by compressing or extracting features from the data blocks. The index is obtained by embedding encoding of the data blocks. The data blocks are composed of semantic units. The first search result is determined based on the data block in the knowledge database that corresponds to the target index; Based on the large language model, according to the pre-configured prompt information template and the first retrieval result, slot recognition, application interface prediction and application interface parameter filling are performed on the current voice request to obtain the execution result of application interface parameter filling. The large language model is pre-trained and can generate the execution result of application interface parameter filling according to the voice request. The execution result is output and sent to the vehicle to complete the voice interaction.

2. The method according to claim 1, characterized in that, The step of retrieving the index in the knowledge database based on the encoding result to obtain the target index includes: The target index is determined from the plurality of indexes based on the similarity between the encoding result and each of the indexes.

3. The method according to claim 1, characterized in that, The method further includes: The predetermined knowledge data is split into multiple data blocks.

4. The method according to claim 1, characterized in that, The training steps of the large language model include: Obtain the voice request sample and the sample label corresponding to the voice request sample; Based on the voice request sample, the knowledge database is searched to obtain a second search result; The reference model is trained based on the voice request sample, the sample label, the second retrieval result, and the pre-configured prompt information template to obtain the large language model.

5. The voice interaction method according to claim 4, characterized in that, The method further includes: Inject pre-configured vehicle domain knowledge into the base model; The base model is trained to obtain the reference model.

6. A server, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the method of any one of claims 1-5.

Images