Generative large language model training method, model-based human-computer speech interaction method
By building a training set of service interface call sequence and user preference sorting, supervised fine-tuning and reinforcement learning of the generative large language model is solved, and the problem of inaccurate reply in task solution is realized. The generative large language model is realized In-depth understanding of user needs and generation of accurate answers.
Patent Information
- Application Number
- CN202310233922.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-03
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-03-03
AI Technical Summary
When used in real-world task resolution, the existing generative large language model is difficult to generate replies that match the actual situation, and it is impossible to use external knowledge to generate real-time information, resulting in inaccurate replies.
By constructing a training set of service interface call sequences based on user input text, supervised fine-tuning of the pre-trained model, combining user preference sorting and preset templates, a reward model is built, and a reinforcement learning method is used to train the target generation large language model, so that the It can understand user needs and generate accurate interface calling instructions.
The generative large language model is implemented to deeply understand user needs, build interface call instructions on its own, generate answers that are more in line with user actual needs and expectations, and avoid relying solely on natural language input and model parameter knowledge.
Smart Images

Figure CN116244416B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, specifically to artificial intelligence technical fields such as generative models, intelligent speech, and human-computer interaction, and in particular to a generative large language model training and a human-computer speech interaction method based on the generative large language model, as well as corresponding devices, electronic devices, computer-readable storage media, and computer program products. Background Art
[0002] Large language models (LLM, Large Language Model, which is essentially a generative model), such as ChatGPT (Chat Generative Pre-trained Transformer, a chatbot program developed by OpenAI), can generate fluent human-like responses for many downstream tasks (such as task-oriented dialogues and question answering).
[0003] However, applying LLMs to real-world task-solving applications remains challenging, mainly due to their potential to generate responses that are inconsistent with reality and their inability to use external knowledge as reference for response generation.
[0004] For example, questions such as "What is the current weather in City A?", "What is the exchange rate of currency B to currency C?", and "How long does it take to get from location C to location D?" all require real-time information when answering them. However, this real-time information (such as real-time time, real-time exchange rate, real-time road conditions, and real-time traffic information) cannot be generated simply by relying on the knowledge contained in the LLM parameters. It is necessary to rely on external knowledge to generate accurate results. Summary of the invention
[0005] The embodiments of the present disclosure provide a generative large language model training method and a human-computer speech interaction method based on the generative large language model, as well as a device, an electronic device, a computer-readable storage medium, and a computer program product supporting the method.
[0006] In a first aspect, an embodiment of the present disclosure proposes a generative large language model training method, including: constructing a first training set based on user input text and a matching service interface call sequence; wherein the user input text is obtained by converting the user input voice, and the service interface constituting the service interface call sequence corresponds to the functional usage intention expressed by the user input text; using the first training set to perform supervised fine-tuning training on a pre-trained first generative large language model to obtain a second generative large language model; constructing a second training set based on user preference ranking and a preset template set between the same user input text and different candidate outputs; using the second training set to perform supervised training on a pre-trained third generative large language model to obtain a reward model; training the second generative large language model in a reinforcement learning manner based on the score returned by the reward model to obtain a target generative large language model.
[0007] In a second aspect, an embodiment of the present disclosure proposes a generative large language model training device, comprising: a first training set construction unit, configured to construct a first training set based on a user input text and a matching service interface call sequence; wherein the user input text is obtained by converting the user input voice, and the service interface constituting the service interface call sequence corresponds to the functional usage intention expressed by the user input text; a supervised fine-tuning training unit, configured to use the first training set to perform supervised fine-tuning training on a pre-trained first generative large language model to obtain a second generative large language model; a second training set construction unit, configured to construct a second training set based on a user preference ranking and a preset template set between the same user input text and different candidate outputs; a reward model training unit, configured to use the second training set to perform supervised training on a pre-trained third generative large language model to obtain a reward model; a reinforcement learning training unit, configured to train the second generative large language model in a reinforcement learning manner based on the score returned by the reward model to obtain a target generative large language model.
[0008] In a third aspect, an embodiment of the present disclosure proposes a human-computer speech interaction method based on a generative large language model, including: obtaining a user input speech emitted by a user in the form of speech; inputting a user input text corresponding to the user input speech as input data into a target generative large language model; wherein the target generative large language model is obtained according to the generative large language model training method described in the first aspect above; executing an interface call sequence output by the target generative large language model to obtain an output text; and converting the output text into a machine reply speech in the form of speech.
[0009] In the fourth aspect, the embodiments of the present disclosure propose a human-computer speech interaction device based on a generative large language model, comprising: a user input speech acquisition unit, configured to acquire a user input speech emitted by a user in the form of speech; a model calling unit, configured to input a user input text corresponding to the user input speech as input data into a preset target generative large language model; wherein the target generative large language model is obtained according to the generative large language model training device as described in the second aspect; an interface call sequence execution unit, configured to execute the interface call sequence output by the target generative large language model to obtain an output text; and a machine reply speech generation unit, configured to convert the output text into a machine reply speech in speech form.
[0010] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the generative large language model training method as described in the first aspect or the human-computer speech interaction method based on the generative large language model as described in the third aspect.
[0011] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the generative large language model training method as described in the first aspect or the human-computer speech interaction method based on the generative large language model as described in the third aspect when executed.
[0012] In the seventh aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps of the generative large language model training method as described in the first aspect or the steps of the human-computer speech interaction method based on the generative large language model as described in the third aspect.
[0013] The generative large language model training scheme provided by the embodiment of the present disclosure has a first training set constructed by user input text (obtained by converting user input speech) and a service interface call sequence expressing the intention to use the corresponding function, so that the first training set is used to perform supervised fine-tuning training on the pre-trained first generative large language model, so that the trained second generative large language model has the ability to rewrite the user input text into a matching service interface call sequence, and the second training set constructed by user preference ranking and a preset template set between the same user input text and different candidate outputs is used to perform supervised training on the pre-trained third generative large language model, so that the trained reward model has the ability to perform quality scoring for different candidates generated by the generative large language model for the user input text, so that under the training of the reinforcement learning method, the target generative large language model finally obtained not only has the ability to rewrite the user input text into a matching service interface call sequence, but also has the ability to output results that better meet user needs. That is, through the above training scheme, the trained target generative large language model has the ability to deeply understand user needs and construct corresponding interface call instructions to query and return accurate answers. It can not only generate natural language output based on natural language input and the knowledge contained in the large model parameters, but also call the corresponding functions with the help of application program interfaces that provide professional capabilities to more effectively solve user needs. Under the influence of the reward model, the returned results will be more in line with the actual needs and expectations of users.
[0014] Based on the beneficial effects of the generative large language model training scheme provided in the above-mentioned embodiments, the human-computer voice interaction scheme based on the generative large language model provided in the embodiments of the present disclosure is specifically aimed at the scenario in which the voice assistant acts as the executor. The target generative large language model trained according to the above-mentioned training scheme is applied to the human-computer voice interaction scenario, so that the voice assistant uses the target generative large language model to convert the user input text corresponding to the user input voice into a matching service interface call sequence, and then obtains the output text by sequentially executing the service interface call sequence, and finally converts the output text into a machine reply voice and returns it to the user to complete the human-computer voice interaction, thereby improving the human-computer voice interaction experience.
[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0017] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;
[0018] Figure 2 A flowchart of a generative large language model training method provided in an embodiment of the present disclosure;
[0019] Figure 3 A flowchart of a method for constructing a first training set provided in an embodiment of the present disclosure;
[0020] Figure 4 A flowchart of a method for determining a service interface calling sequence provided by an embodiment of the present disclosure;
[0021] Figure 5 A flowchart of a method for constructing a second training set provided in an embodiment of the present disclosure;
[0022] Figure 6 A flowchart of a human-computer speech interaction method based on a generative large language model provided in an embodiment of the present disclosure;
[0023] Figure 7 A schematic diagram of a human-computer voice interaction process for a specific example provided in an embodiment of the present disclosure;
[0024] Figure 8 A structural block diagram of a generative large language model training device provided in an embodiment of the present disclosure;
[0025] Fig. 9 A structural block diagram of a human-computer voice interaction device based on a generative large language model provided in an embodiment of the present disclosure;
[0026] Fig.10 A schematic diagram of the structure of an electronic device suitable for executing a generative large language model training method and / or a human-computer voice interaction method based on a generative large language model, provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.
[0028] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0029] Figure 1 An exemplary system architecture 100 of an embodiment of the generative large language model training method of the present application, a human-computer speech interaction method based on a generative large language model, and supporting devices, electronic devices, and computer-readable storage media is shown.
[0030] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0031] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various applications for realizing information communication between the terminal devices 101, 102, 103 and the server 105 can be installed, such as human-computer voice interaction applications (such as navigation voice assistant applications, smart terminal voice assistant applications, etc.), model training applications, etc.
[0032] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.
[0033] The server 105 can provide various services through various built-in applications. For example, a human-computer voice interaction application that can provide human-computer voice interaction services to users can achieve the following effects when the server 105 runs the human-computer voice interaction application: first, the user input voice in the form of voice recorded by the terminal devices 101, 102, and 103 transmitted through the network 104; then, the user input text corresponding to the user input voice is input as input data into the pre-trained target generative large language model; next, the interface call sequence output by the target generative large language model is executed to obtain the output text; finally, the output text is converted into a machine reply voice in the form of voice. Furthermore, the machine reply voice can also be retransmitted back to the terminal devices 101, 102, and 103 through the network 104, so that the terminal devices 101, 102, and 103 can reply to the corresponding user by playing the machine reply voice.
[0034] Among them, the target generative large language model can be trained by the model training application built into the server 105 according to the following steps: first, based on the user input text and the matching service interface call sequence, a first training set is constructed, the user input text is converted from the user input voice, and the service interface constituting the service interface call sequence corresponds to the functional usage intention expressed by the user input text; then, the first training set is used to perform supervised fine-tuning training on the pre-trained first generative large language model to obtain a second generative large language model; then, based on the user preference ranking and preset template set between the same user input text and different candidate outputs, a second training set is constructed; next, the second training set is used to perform supervised training on the pre-trained third generative large language model to obtain a reward model; finally, the second generative large language model and the reward model are trained in a reinforcement learning manner to obtain a target generative large language model.
[0035] Since it takes more computing resources and stronger computing power to train the target generative large language model, the generative large language model training method provided in the subsequent embodiments of the present application is generally performed by a server 105 with stronger computing power and more computing resources. Correspondingly, the generative large language model training device is generally also arranged in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned various operations handed over to the server 105 through the model training application installed thereon, and then output the same result as the server 105. Correspondingly, the generative large language model training device can also be arranged in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.
[0036] Of course, the server used to train the target generative large language model can be different from the server that calls the trained target generative large language model for use. In particular, the target generative large language model trained by the server 105 can also be used to obtain a lightweight target generative large language model suitable for being placed in the terminal devices 101, 102, and 103 by model distillation, that is, it can be flexibly selected to use the lightweight target generative large language model in the terminal devices 101, 102, and 103 according to the recognition accuracy of actual needs, or to choose to use the more complex target generative large language model in the server 105.
[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0038] Please refer to Figure 2 , Figure 2 A flowchart of a generative large language model training method provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:
[0039] Step 201: construct a first training set based on the user input text and the matching service interface call sequence;
[0040] This step is intended to be performed by the execution subject of the generative large language model training method (e.g. Figure 1 The server 105 shown in the figure constructs a first training set based on a large number of sample pairs of "user input text-service interface call sequence" consisting of a user input text (converted from the user input voice sent by the user in voice form) and a matching service interface call sequence. The user input voice can be obtained from various channels, and can come from a user voice data set collected and recorded from a large number of authorized users over a long period of time in the past, or from a historical human-computer interaction sequence set; and in order to realize the conversion of user input voice into user input text, it can be achieved with the help of Automatic Speech Recognition (ASR) technology, and in order to improve the accuracy of the conversion result, it can also be carried out with the help of a speech-to-text model based on deep learning.
[0041] Among them, the service interfaces that constitute the service interface call sequence correspond to the functional usage intention expressed by the user input text, that is, the service interface call sequence can meet the expected needs expressed by the user input text, so as to realize the user needs or obtain a reply that satisfies the user's functional usage intention by executing the service interface call sequence to obtain a response.
[0042] The "service interface" described in the present disclosure refers to an application programming interface (API) for providing services. Different APIs may be provided by related applications installed on the operating system (including system-related applications that support the normal operation of the operating system and other applications built on the normal operation of the operating system), or may be directly provided by the operating system or some system components running on the operating system, such as Navi API for implementing navigation functions (usually provided by map applications), FindPOI API for implementing point of interest positioning functions (usually provided by map applications, the full name of the English abbreviation POI is: Point of Interest), Option API for implementing selection functions (usually provided by specific applications that provide multiple alternatives, such as map applications when seeking a preferred choice among multiple alternative routes to the same destination), Date API for implementing a specific date acquisition function (usually provided by clock applications or time components), FindSong API for implementing a song search function (usually provided by song applications or retrieval applications), in addition, it also includes but is not limited to the corresponding APIs for implementing the following functions:
[0043] Time query, date query, registered license plate number query, user search, friend query, song recognition, picture search, travel route planning, weather query, task scheduling, location query, restricted license plate number query, exchange rate conversion, base conversion, translation, surrounding environment query, encryption and decryption, data download, data transmission, problem query, picture editing, etc.
[0044] Among them, the service interface call sequence can be composed of at least a service interface call instruction corresponding to only one service interface, and usually it is composed of multiple (at least two) different service interface call instructions that exist in an execution order. Each service interface call instruction generally includes the interface name of the target service interface and specific interface call parameters, and the two are combined in the required format. However, for some special APIs, their interface call parameters can be empty; for some other special APIs, their interface call parameters can also be another service interface call instruction. In addition, the specific interface call parameters can be directly served by a text fragment in the user input text, or by other forms of content that express the same meaning as the corresponding text fragment.
[0045] To facilitate understanding of the solution described in this step, the following will be explained through a specific example:
[0046] Assume that after the speech-to-text operation is performed on the user input voice, the user input text obtained is: "Navigate to Z on Y street at X place, take the fastest route", then the service interface call sequence matching the user input text can be expressed as: Navi (FindPOI (Z on Y street at X place), Option (fastest route)). The reason for the corresponding service interface call sequence is that the final function call intention that the user wants to express can be determined as navigation based on the user input text, but in order to realize the final function call intention, it is also necessary to accurately determine the inaccurate description of the point of interest "X place Y street Z" that serves as the destination, and select the shortest route when the destination is accurately determined. Therefore, there are actually three function intention calls: point of interest determination function (corresponding to FindPOI API), route selection function (corresponding to Option API) and navigation function (corresponding to Navi API), among which the accurate point of interest determined by calling the point of interest determination function and the route selection preference determined by calling the route selection function should be used as input information of the navigation function (that is, FindPOI (X place Y street Z) and Option (fastest route) are both used as interface call parameters of Navi API at this time), that is, this execution order exists. In other words, if the service interface call sequence is composed of multiple service interface call instructions, these service interface call instructions need to be organized according to the execution order, so that each service interface call instruction contained in the service interface call sequence can be correctly executed.
[0047] It should be noted that the above-mentioned service interface call sequence that matches the user input text can be obtained by manual annotation based on one's own understanding of the user input text and knowledge of the relevant usage of the service interface, or it can be obtained by non-artificial objects based on concentrated extraction of data samples that imply the correspondence between the above-mentioned two, so as to achieve efficient construction of sample pairs of "user input text-service interface call sequence".
[0048] In some understandings, if the user input text is understood as a query, the service interface call sequence is actually a query' consisting of an interface call instruction consisting of an API and API parameters that can meet corresponding requirements.
[0049] Step 202: Perform supervised fine-tuning training on the pre-trained first generative large language model using the first training set to obtain a second generative large language model;
[0050] On the basis of step 201, this step aims to use the first training set as a fine-tuning training set by the above-mentioned execution subject to perform supervised fine-tuning training (Supervised Fine-Tuning, SFT) on the first pre-trained generative large language model, so as to further obtain a second generative large language model.
[0051] The SFT technology used in this step means that the training object is not the initial model that has not undergone any training, but the general generative large language model obtained through training with basic training samples, that is, the general generative large language model obtained through training with basic training samples usually only generates a text sequence as a result based on the predicted probability of the language unit (token) according to the knowledge contained in the text input and the large model parameters, and does not have the ability to rewrite it into a service interface call sequence including the target API and API parameters. With the help of this SFT technology, the huge time-consuming effort required to train the model from scratch can be avoided. It only needs to construct a first training set containing a small amount of training samples (the small amount here is the order of magnitude relative to the basic training samples), and then the general generative large language model can be trained twice to enable it to rewrite the user input text into a service interface call sequence including the target API and API parameters.
[0052] Of course, in some application scenarios, the general generative large language model trained with basic training samples may not be suitable as the basic model for using SFT technology (for example, the knowledge learned by the general generative large language model conflicts with the knowledge implicit in the newly constructed training samples). The initial generative language model can also be trained based on the newly constructed training samples, in order to make the trained target generative large language model have the desired capabilities.
[0053] Step 203: constructing a second training set based on the user preference ranking between the same user input text and different candidate outputs and the preset template set;
[0054] On the basis of step 202, this step aims to construct a second training set by the above-mentioned execution subject based on the user preference ranking between the same user input text and different candidate outputs and the preset templates that record the correspondence between the input text and the corresponding interface call instructions. The preset template can be specifically expressed as: "I want to find [String]—FindPOI(POIName)", "Navigate to [String]—Navi(POIName)", that is, the preset template is used to indicate what style of text fragment corresponds to what interface call instruction, where "String" is used to refer to the corresponding part and can be filled with any string. Specifically, the preset template can be extracted from the "user input text—service interface call sequence" sample pairs used to constitute the first training set, and can also be extracted from other data sets containing the above-mentioned correspondence.
[0055] That is, first, for each user input text, a sample pair formed by the user input text and each candidate output is constructed, and these sample pairs are sorted according to the user preference ranking of the candidate output in each sample pair; then, the preset templates (the preset template set contains multiple preset templates) that record the correspondence between the input text and the corresponding interface call instruction are combined to jointly construct the second training set, so that by using the second training set constructed in this way, the model can learn which results are more in line with the actual needs of the user.
[0056] It should be noted that the training samples described above for constructing the second training set can be manually labeled and generated by humans based on their own subjective understanding, or they can be extracted by non-artificial objects based on a data sample set that implies the correspondence between the above two (for example, a retrieval log that reflects the search terms and retrieval results in the user's retrieval operation) to achieve efficient construction of corresponding sample pairs.
[0057] When the retrieval log is used as the data sample set, the user preference ranking can be specifically expressed as the cumulative number of clicks on each candidate output in the candidate output list returned for the same user input text recorded in the retrieval log within a period of time, that is, the more cumulative clicks, the more the candidate output meets the user's actual needs. The length of the time period can be set by yourself, such as one week, half a month or a whole month. When the data sample set is served by other databases, the user preference ranking can also be expressed as other similar parameters that can express user preferences, such as the number of collections, the number of comments, etc., which will not be listed here one by one.
[0058] Step 204: Use the second training set to perform supervised training on the pre-trained third generative large language model to obtain a reward model.
[0059] On the basis of step 203, this step aims to use the second training set by the above-mentioned execution subject to perform supervised training on the pre-trained third generative large language model to obtain a reward model (Reward Model, RM). The reward model described here refers to learning how to give different rewards (generally returned in the form of high and low scores) to different alternative outputs under the same input by learning the user preference ranking between different sample pairs under each training sample in the second training set, so as to make the model learn in the direction of obtaining higher rewards, so as to be able to output results that better meet the actual needs of users.
[0060] Step 205: The second generative large language model is trained in a reinforcement learning manner based on the score returned by the reward model to obtain a target generative large language model.
[0061] On the basis of step 202 and step 204, this step aims to train the second generative large language model by the above-mentioned execution subject based on the score returned by the reward model in a reinforcement learning (RL) manner to obtain a target generative large language model.
[0062] The brief process of training the target generative large language model based on RL can be referred to the following description:
[0063] 1) Initializing a new generative large language model M based on the parameters of the second generative large language model;
[0064] 2) Based on the new prompt (prompt words, usually natural language input from the user), let M generate a response for each prompt and then input the response to the reward model (RM);
[0065] 3) RM will calculate a score for each reply as a scalar reward, and the score indicates the quality of the reply;
[0066] 4) Using RL (reinforcement learning) to continuously update the generation strategy (corresponding to English: policy) of the model M based on the total reward score obtained until convergence. The M trained to this point is the target generative large language model that meets the requirements.
[0067] To facilitate understanding of the “generative large language model” described and used in the present disclosure, the working principle of the generative model corresponding to the prefix “generative” is briefly introduced here:
[0068] The reason why it is called a generative model is that the working process of this type of model is similar to "text chain", that is, based on the user input and the previous output of the model, it continuously predicts the next token (language unit) and finally generates a text sequence. See the following example:
[0069] Assuming that the target generative large language model has learned the ability to output the user input text as a service interface call sequence according to the above scheme, then taking "navigation to Z on Y street in X place, take the fastest route" (for the convenience of the following examples, "query" is used to refer to it) as an example, the process of the target generative model outputting the corresponding service interface call sequence is carried out in the following steps:
[0070] query-->Navi;
[0071] query+Navi-->(;
[0072] query+Navi+(-->FindPOI;
[0073] query+Navi+(+FindPOI-->(;
[0074] query+Navi+(+FindPOI+(-->X;
[0075] query+Navi+(+FindPOI+(+X-->地;
[0076] query+Navi+(+FindPOI+(+X+地-->Y;
[0077] query+Navi+(+FindPOI+(+X+地+Y-->街;
[0078] query+Navi+(+FindPOI+(+X+地+Y+街-->的;
[0079] query+Navi+(+FindPOI+(+X+地+Y+街+的-->Z;
[0080] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z-->);
[0081] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z+)-->,;
[0082] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z+)+,-->Option;
[0083] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z+)+,+Option-->(;
[0084] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z+)+,+Option+(-->走;
[0085] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z+)+,+Option+(+走-->最;
[0086] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z+)+,+Option+(+走+最-->快;
[0087] query+Navi+(+FindPOI+(+X+地+Y+街+的+Z+)+,+Option+(+走+最+快-->的;
[0088] query+Navi+(+FindPOI+(+X+place+Y+street+of+Z+)+,+Option+(+take+the+fastest+way-->;
[0089] query+Navi+(+FindPOI+(+X+place+Y+street+of+Z+)+,+Option+(+take+the+fastest+route-->route;
[0090] query+Navi+(+FindPOI+(+X+place+Y+street+of+Z+)+,+Option+(+take+the+fastest+route-->);
[0091] query+Navi+(+FindPOI+(+X+place+Y+street+of+Z+)+,+Option+(+take+the+fastest+route+);
[0092] query+Navi+(+FindPOI+(+X+place+Y+street+of+Z+)+,+Option+(+take+the+fastest+route+)-->);
[0093] query+Navi+(+FindPOI+(+X+place+Y+street+of+Z+)+,+Option+(+take+the+fastest+route+)+)-->END.
[0094] That is, a service interface calling sequence of Navi (FindPOI (Z at Y street in X place), Option (take the fastest route)) can be finally obtained.
[0095] In Chinese, a token is usually a Chinese character; in English, a token usually refers to a sub-word (literally translated as "sub-word" in Chinese) rather than a word. The corresponding sub-word granularity is a form between word granularity and character granularity. Sub-words can retain the meaning of the language while effectively reducing the size of the vocabulary (i.e., the number of words in the vocabulary) and effectively solving the problem of out-of-vocabulary words. Taking the three English words "loved", "loving" and "loves" as examples, the following four sub-words can be obtained: "lov", "ed", "ing" and "es". In this way, the meaning and tense of the word itself can be separated to obtain sub-words with smaller granularity. In the above example, "FindPOI, Option and Navi" representing the API name will be used as an indivisible whole as a token.
[0096] In the generative large language model training method provided by the embodiment of the present disclosure, since the first training set is constructed by user input text (obtained by converting user input speech) and a service interface call sequence expressing the intention to use the corresponding function, the first training set is used to perform supervised fine-tuning training on the pre-trained first generative large language model, so that the trained second generative large language model has the ability to rewrite the user input text into a matching service interface call sequence, and the second training set constructed by user preference ranking between the same user input text and different candidate outputs is used to perform supervised training on the pre-trained third generative large language model, so that the trained reward model has the ability to perform quality scoring for different candidates generated by the generative large language model for the user input text, so that under the training of the reinforcement learning method, the target generative large language model finally obtained not only has the ability to rewrite the user input text into a matching service interface call sequence, but also has the ability to output results that better meet user needs. That is, through the above training scheme, the trained target generative large language model has the ability to deeply understand user needs and construct corresponding interface call instructions to query and return accurate answers. It can not only generate natural language output based on natural language input and the knowledge contained in the large model parameters, but also call the corresponding functions with the help of application program interfaces that provide professional capabilities to more effectively solve user needs. Under the influence of the reward model, the returned results will be more in line with the actual needs and expectations of users.
[0097] On the basis of the above embodiment, in the supervised fine-tuning training process using SFT technology, for special API-related symbols that do not appear in the first generative large language model, it is necessary to expand the language unit (i.e., token), which can be called a new token. The initialization of the features of the new token can be performed by sampling the multivariate normal distribution parameters of the features of the existing language unit.
[0098] Specifically, since the inserted API and API format-related special symbols do not exist in the vocabulary of the first generative large language model, the vocabulary of the first generative large language model also needs to be expanded. Regarding the initialization problem of the embedding (usually translated as word vector or word embedding) corresponding to the new token after the vocabulary expansion, a parameter sampling initialization method can also be used: that is, for the embedding corresponding to the newly added token, if it is randomly initialized or initialized with 0, it will usually increase the difficulty of the model to predict the new token. This is because the initialization embedding of the new token is too different from the embedding distribution of the existing token obtained after a large amount of pre-training of the general generative large language model. In order to reduce this difference, the embedding of each new token can be sampled according to the multivariate normal distribution (also called multivariate normal distribution or multivariate Gaussian distribution) of the embedding of the existing token. Formally, for each new token corresponding to the embedding—e n+i , sampling can be performed as follows:
[0099]
[0100] Among them, E∈R n×d It is the embedding [e1; ...; e n ] is a matrix composed of .
[0101] Please refer to Figure 3 , Figure 3 A flowchart of a method for constructing a first training set provided in an embodiment of the present disclosure, that is, for Figure 2 Step 201 in the process 200 shown provides a specific implementation method, and the other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 201 to obtain a new complete embodiment. The process 300 includes the following steps:
[0102] Step 301: obtaining a user input voice sent by a user in the form of voice, converting the user input voice into a user input text, and determining the functional usage intention expressed by the user input text;
[0103] This step is intended to have the above-mentioned execution subject first obtain the user input voice, and convert it into an easy-to-understand user input text through voice recognition technology, and then further determine the functional use intention that the user wants to express by understanding the meaning of the user input text. Specifically, when determining the functional use intention that the user wants to express based on the user input text, it can be determined by the text fragments that directly refer to certain functions in the user input text, or it can be further determined by understanding the meaning of the text fragments that indirectly refer to certain functions, which is not specifically limited here.
[0104] Step 302: Determine the target service interface and specific interface parameters for realizing the corresponding intent according to the functional usage intent.
[0105] Based on step 301, this step aims to allow the above-mentioned execution subject to determine the target service interface and specific interface parameters for realizing the corresponding intention according to the functional usage intention.
[0106] Taking the simple user input text "Navigate to People's Park" as an example, it only has one functional usage intention, which is navigation to the preset destination. The target service interface corresponding to this functional usage intention may be NaviAPI referring to the navigation function, and its specific interface parameters should be the destination served by "People's Park".
[0107] Step 303: construct a service interface call sequence according to the target service interface and specific interface parameters;
[0108] Based on step 302, this step aims to construct a service interface call sequence according to the target service interface and specific interface parameters. Continuing with the previous example, the interface call instruction of Navi (People's Park) can be generated. Since this example only contains one functional use intention, the only interface call instruction will exist as a service interface call sequence at this time.
[0109] Step 304: Construct a first training set based on sample pairs consisting of user input text and matching service interface call sequences.
[0110] On the basis of step 303, this step aims to construct the first training set by the above-mentioned execution subject from multiple sample pairs of "user input text-service interface call sequence".
[0111] Please refer to Figure 4 , Figure 4 A flowchart of a method for generating a service interface call sequence provided in an embodiment of the present disclosure, that is, for Figure 3Steps 302 and 303 in the process 300 shown provide a specific implementation method, mainly for the case where the expression of the user input text contains at least two functional usage intentions, and the other steps in the process 300 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 302 and step 303 to obtain a new complete embodiment. The process 400 includes the following steps:
[0112] Step 401: respectively determining the target service interface and specific interface parameters corresponding to each functional usage intention;
[0113] Take the complex user input text "Navigate to X place Y street Z, take the fastest route" as an example. It contains three functional usage intentions: point of interest determination, route selection, and navigation. There are also nested and dependent relationships between different functional usage intentions. At this time, it is necessary to first determine the target service interface and specific interface parameters corresponding to each functional usage intention:
[0114] For the point of interest determination function, the corresponding target service interface is: FindPOI API, and its specific interface parameters are: Z of Y street at X place;
[0115] For the route selection function, the corresponding target service interface is: Option API, and its specific interface parameters are: take the fastest route;
[0116] As for the navigation function, the corresponding target service interface is: Navi API, and the specific interface parameters are: FindPOI (Z of Y street at X place) and Option (take the fastest route), or the values of the two.
[0117] Step 402: Generate a corresponding interface call instruction according to the target service interface and specific interface parameters corresponding to each function usage intention;
[0118] Based on the example given in step 401, this step aims to generate corresponding interface call instructions according to the target service interface and specific interface parameters corresponding to each functional usage intention. The following three interface call instructions can be obtained:
[0119] 1) Find POI (Z of Y street in X place);
[0120] 2) Option (take the fastest route);
[0121] 3) Navi (Z1, fastest route), in this case, it is assumed that Z1 is the result of calling FindPOI (Z at Y street in X place), and the fastest route is the result of calling Option (fastest route).
[0122] Step 403: determining the execution order of different interface call instructions according to the execution dependency relationship between different functional usage intentions;
[0123] Continuing with the above example, it can be seen that there is an obvious execution dependency between the POI determination function, the route selection function and the navigation function, that is, the accurate POI determined by calling the POI determination function and the route selection preference determined by calling the route selection function should be used together as input information for the navigation function. Therefore, the execution order of the above three interface call instructions should be: first execute FindPOI (Z on Y street in X place) and Option (take the fastest route) (since there is no execution dependency between the two, when only serial is supported, one can be executed first and then the other; when parallel is supported, the two instructions can be executed concurrently), and then, when the results of the two instructions are obtained, the Navi (Z1, the fastest route) instruction is executed based on the two instruction results.
[0124] Step 404: Organize different interface calling instructions in execution order to obtain a service interface calling sequence.
[0125] Based on step 403, this step aims to organize different interface calling instructions according to the execution order by the above-mentioned execution subject to obtain a service interface calling sequence.
[0126] Still following the above example, the service interface call sequence obtained by organizing different interface call instructions in execution order can be expressed as:
[0127] Navi (FindPOI (Z on Y street in X place), Option (fastest route));
[0128] Or it may appear as:
[0129] FindPOI (Z at Y street in X place) --> Z1; Option (take the fastest route) --> the fastest route; Navi (Z1, the fastest route).
[0130] That is, the execution order can be expressed in a multi-layer nested manner, or can be expressed through a timing chain, or can be expressed in other ways that can achieve similar effects, which are not specifically limited here.
[0131] Please refer to Figure 5 , Figure 5 A flowchart of a method for constructing a second training set provided in an embodiment of the present disclosure, that is, for Figure 2Step 203 in the process 200 shown provides a specific implementation method, and the other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 203 to obtain a new complete embodiment. The process 500 includes the following steps:
[0132] Step 501: Obtain search terms from the search log;
[0133] Step 502: confirm all search results returned for the same search term from the search log, and determine the cumulative number of clicks on each search result within a time window;
[0134] Step 503: Generate multiple sample pairs for the same search term and each search result, and determine the ranking information between the corresponding different sample pairs according to the accumulated number of clicks corresponding to the different search results;
[0135] Step 504: construct a second training set based on the sample pairs sorted by user preference according to the sorting information and the preset templates recording the corresponding relationship between the input text and the corresponding interface calling instruction.
[0136] To facilitate understanding of the solution provided by this embodiment, the following will be described in detail through another example:
[0137] Assume that there is a query (i.e. the search term mentioned above) called "XX", and aggregate the number of clicks on different results of this query within a period of time (for example, 30 days). Assume that the following results are obtained as shown in Table 1:
[0138] Table 1 Statistics of cumulative click counts
[0139] Result No. result Total number of clicks r1 XX University 100 r2 East Gate of XX University 10 r3 XX University West Gate 9 r4 XX University Teachers Building 8 r5 XX University Library 0
[0140] Then the paired samples that can be constructed according to the above table can be:
[0141] rel(q,r1)>rel(q,r2);
[0142] rel(q,r1)>rel(q,r3);
[0143] rel(q,r1)>rel(q,r4);
[0144] rel(q,r1)>rel(q,r5);
[0145] rel(q,r2)>rel(q,r3);
[0146] rel(q,r2)>rel(q,r4);
[0147] rel(q,r2)>rel(q,r5);
[0148] rel(q,r3)>rel(q,r4);
[0149] rel(q,r3)>rel(q,r5);
[0150] rel(q,r4)>rel(q,r5), rel is a function used to express the user preference between the two.
[0151] Based on the above information, the query-query' template composed of the user input text and the matching service interface call sequence (for example: I want to find [String]—FindPOI(POIName) and navigate to [String]—Navi(POIName), as well as the pairwise relationship given above, is automatically constructed to construct the second training set:
[0152] rel(I want to find XX University, FindPOI(XX University))>rel(I want to find XX University, FindPOI(East Gate of XX University));
[0153] rel(I want to find XX University, FindPOI(XX University))>rel(I want to find XX University, FindPOI(XX University West Gate));
[0154] rel(I want to find XX University, FindPOI(XX University))>rel(I want to find XX University, FindPOI(XX University Teachers Building));
[0155] rel(I want to find XX University, FindPOI(XX University))>rel(I want to find XX University, FindPOI(XX University Library));
[0156] rel(I want to find XX University, FindPOI(East Gate of XX University))>rel(I want to find XX University, FindPOI(West Gate of XX University));
[0157] rel(I want to find XX University, FindPOI(XX University East Gate))>rel(I want to find XX University, FindPOI(XX University Teacher Building));
[0158] rel(I want to find XX University, FindPOI(East Gate of XX University))>rel(I want to find XX University, FindPOI(Library of XX University));
[0159] rel(I want to find XX University, FindPOI(XX University West Gate))>rel(I want to find XX University, FindPOI(XX University Teacher Building));
[0160] rel(I want to find XX University, FindPOI(XX University West Gate))>rel(I want to find XX University, FindPOI(XX University Library));
[0161] rel(I want to find XX University, FindPOI(XX University Teacher Building))>rel(I want to find XX University, FindPOI(XX University Library)).
[0162] For similar navigation to XX University, the situation is similar to the above and will not be listed here one by one.
[0163] By automatically generating the second training set using the data recorded in the retrieval log, not only a lot of manual labeling costs are saved, but the massive click and skip (non-click) data in the retrieval log is a completely real reflection of user needs, and it truly reflects the distribution of user preferences for different results, thereby making the constructed second training set more in line with the actual needs of users.
[0164] The above embodiments explain how to train a target generative large language model from various aspects. In order to highlight the effect of the trained target generative large language model from the actual usage scenario as much as possible, the present disclosure also specifically provides a solution for using the trained target generative large language model to solve practical problems, such as Figure 6 A human-computer speech interaction method based on a generative large language model is shown, comprising the following steps:
[0165] Step 601: Acquire user input voice sent by the user in voice form;
[0166] This step is intended to be performed by the execution subject of the human-computer speech interaction method based on the generative large language model (for example, installed in Figure 1 The voice assistant application on the terminal devices 101, 102, and 103 shown in the figure obtains the user input voice issued by the user in the form of voice.
[0167] Step 602: inputting the user input text corresponding to the user input speech as input data into a preset target generative large language model;
[0168] On the basis of step 601, this step aims to have the above-mentioned execution subject input the user input text corresponding to the user input voice as input data into the preset target generative large language model, so as to call the target generative large language model to process the user input text to obtain a rewritten interface call sequence.
[0169] Step 603: Execute the interface call sequence output by the target generative large language model to obtain output text;
[0170] On the basis of step 602, this step aims to cause the above-mentioned execution subject to execute the interface call sequence output by the target generative large language model to obtain the output text.
[0171] Taking the interface call sequence - "Navi(FindPOI(Z at Y street in X place), Option(fastest route))" as an example, after executing the interface call sequence, you can get output text such as "The fastest route to Z1 found for you is: 1) first pass through road 1...; 2) then pass through road 2...; 3) arrive at Z1". It should be noted that the output text is the navigation result information returned by the interface call module after calling the navigation API provided by the map application. Furthermore, if the navigation result information contains travel navigation information in other formats (such as images, hypertext containing links, etc.) in addition to text information, the format set of supported output results can also be expanded so that multi-format output information including text format can be returned.
[0172] Step 604: Convert the output text into a machine reply speech in speech form.
[0173] Based on step 603, this step aims to convert the output text into a machine reply voice in voice form by the above-mentioned execution subject, so as to make a voice response to the user input voice initiated by the user through the voice assistant.
[0174] Specifically, the voice assistant that serves as the executor of this embodiment and returns a machine reply voice to the user's input voice may specifically include: a map voice assistant from a map application, a voice assistant from a mobile application, and a home voice assistant from a smart terminal device (such as a smart speaker, a smart refrigerator, a smart wearable device, etc.), so as to meet the user's human-computer interaction needs in all scenarios.
[0175] The human-computer voice interaction method based on the generative large language model provided in this embodiment is specifically aimed at the scenario where the voice assistant acts as the executor. The target generative large language model trained according to the above training scheme is applied to the human-computer voice interaction scenario, so that the voice assistant uses the target generative large language model to convert the user input text corresponding to the user input voice into a matching service interface call sequence, and then obtains the output text by sequentially executing the service interface call sequence, and finally converts the output text into a machine reply voice and returns it to the user to complete the human-computer voice interaction, thereby improving the human-computer voice interaction experience.
[0176] In order to prevent the target generative large language model from outputting the same format of output results for user input texts with different expressions, the generated output texts can also be adjusted to different expressions that express the same meaning.
[0177] One expression method can be: adjusting the generated output text to a random expression method that expresses the same meaning, that is, randomly selecting an expression method from the expression library for expression adjustment each time the output is performed. The expression methods can be divided into: lively type, ordinary type, concise type, comprehensive type, etc.;
[0178] Another way of expression may be: adjusting the generated output text to express the same meaning and the same expression as the user input text, that is, making the output text use the same expression as the user input text to enhance the user's experience of the output text.
[0179] To further deepen the understanding of the solution provided by the present disclosure, the following will also be Figure 7 A schematic diagram of a human-computer voice interaction process for a specific example is shown below:
[0180] 1) The voice assistant receives the user's voice input: "Navigate to location X, street Y, Z, take the fastest route";
[0181] 2) The voice assistant uses speech-to-text technology to convert the text into the user input text: "Navigate to X place Y street Z, take the fastest route";
[0182] 3) The voice assistant inputs the user input text "Navigate to X place Y street Z, take the fastest route" into the pre-trained target generative language model;
[0183] 4) The target generative language model outputs the service interface call sequence of “Navi(FindPOI(Z of Y street in X place),Option(fastest route))”;
[0184] 5) The voice assistant executes the service interface call sequence, first executing FindPOI(String) to find the POI, executing Option(Type) to obtain the navigation mode, and finally executing Navi(POI, Opt) to achieve navigation;
[0185] 5) The voice assistant obtains the output text of the execution result: obtains the route: Road 1, Road 2, Road 3, ...;
[0186] 6) The voice assistant converts it into speech and issues the corresponding voice output: "The following route is found for you: ..."
[0187] Further references Figure 8 and Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a generative large language model training device and an embodiment of a human-computer speech interaction device based on a generative large language model. Figure 2Corresponding to the embodiment of the generative large language model training method shown in FIG, the embodiment of the human-computer voice interaction device based on the generative large language model is Figure 6 The above device can be applied to various electronic devices.
[0188] like Figure 8 As shown, the generative large language model training device 800 of this embodiment may include: a first training set construction unit 801, a supervised fine-tuning training unit 802, a second training set construction unit 803, a reward model training unit 804, and a reinforcement learning training unit 805. Among them, the first training set construction unit 801 is configured to construct a first training set based on the user input text and the matching service interface call sequence; wherein the user input text is obtained by converting the user input voice, and the service interface constituting the service interface call sequence corresponds to the functional usage intention expressed by the user input text; the supervised fine-tuning training unit 802 is configured to use the first training set to perform supervised fine-tuning training on the pre-trained first generative large language model to obtain a second generative large language model; the second training set construction unit 803 is configured to construct a second training set based on the user preference ranking and preset template set between the same user input text and different candidate outputs; the reward model training unit 804 is configured to use the second training set to perform supervised training on the pre-trained third generative large language model to obtain a reward model; the reinforcement learning training unit 805 is configured to train the second generative large language model in a reinforcement learning manner based on the score returned by the reward model to obtain a target generative large language model.
[0189] In this embodiment, in the generative large language model training device 800, the specific processing of the first training set construction unit 801, the supervised fine-tuning training unit 802, the second training set construction unit 803, the reward model training unit 804, and the reinforcement learning training unit 805 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 - 205 in the corresponding embodiment are not repeated here.
[0190] In some optional implementations of this embodiment, the first training set construction unit 801 may include:
[0191] A function use intention determination subunit is configured to obtain a user input voice issued by a user in the form of voice, convert the user input voice into a user input text, and determine the function use intention expressed by the user input text;
[0192] The target interface and interface parameter determination subunit is configured to determine the target service interface and specific interface parameters for realizing the corresponding intention according to the functional use intention;
[0193] The service interface call sequence construction subunit is configured to construct a service interface call sequence according to the target service interface and specific interface parameters;
[0194] The first training set construction subunit is configured to construct a first training set based on sample pairs consisting of user input text and matching service interface call sequences.
[0195] In some optional implementations of this embodiment, the target interface and interface parameter determination subunit may be further configured as follows:
[0196] In response to the expression of the user input text containing at least two functional use intentions, respectively determining a target service interface and specific interface parameters corresponding to each functional use intention;
[0197] Correspondingly, the service interface call sequence construction subunit can be further configured as follows:
[0198] Generate corresponding interface call instructions according to the target service interface and specific interface parameters corresponding to each function usage intention;
[0199] Determine the execution order of different interface call instructions based on the execution dependency between different functional usage intentions;
[0200] Different interface calling instructions are organized in execution order to obtain a service interface calling sequence.
[0201] In some optional implementations of the present embodiment, under a supervised fine-tuning training method, for new language units representing service interfaces that do not appear in the first generative large language model, parameter sampling of features of the new language units is initialized according to the multivariate normal distribution of features of existing language units.
[0202] In some optional implementations of this embodiment, the second training set construction unit 803 may be further configured to:
[0203] Get search terms from search logs;
[0204] Confirm all search results returned for the same search term from the search log, and determine the cumulative number of clicks on each search result within a time window;
[0205] Generate multiple sample pairs with the same search term and each search result, and determine the ranking information between the corresponding different sample pairs according to the accumulated number of clicks corresponding to the different search results;
[0206] A second training set is constructed based on each sample pair sorted by user preference according to the sorting information and each preset template recording the corresponding relationship between the input text and the corresponding interface calling instruction.
[0207] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The generative large language model training apparatus provided by this embodiment, since the first training set is constructed by the user input text (obtained by converting the user input voice) and the service interface call sequence expressing the corresponding function usage intention, the pre-trained first generative large language model is subjected to supervised fine-tuning training through the first training set, so that the trained second generative large language model has the ability to rewrite the user input text into a matching service interface call sequence, and the pre-trained third generative large language model is subjected to supervised training by the second training set constructed by the user preference ranking between the same user input text and different candidate outputs, so that the trained reward model has the ability to perform quality scoring for different candidates generated by the generative large language model for the user input text, so that under the training of the reinforcement learning method, the target generative large language model finally obtained not only has the ability to rewrite the user input text into a matching service interface call sequence, but also has the ability to output results that better meet user needs. That is, through the above training scheme, the trained target generative large language model has the ability to deeply understand user needs and construct corresponding interface call instructions to query and return accurate answers. It can not only generate natural language output based on natural language input and the knowledge contained in the large model parameters, but also call the corresponding functions with the help of application program interfaces that provide professional capabilities to more effectively solve user needs. Under the influence of the reward model, the returned results will be more in line with the actual needs and expectations of users.
[0208] like Fig. 9 As shown, the human-machine speech interaction device 900 based on the generative large language model of this embodiment may include: a user input speech acquisition unit 901, a model calling unit 902, an interface call sequence execution unit 903, and a machine reply speech generation unit 904. Among them, the user input speech acquisition unit 901 is configured to acquire the user input speech issued by the user in the form of speech; the model calling unit 902 is configured to input the user input text corresponding to the user input speech as input data into the preset target generative large language model; the interface call sequence execution unit 903 is configured to execute the interface call sequence output by the target generative large language model to obtain the output text; the machine reply speech generation unit 904 is configured to convert the output text into a machine reply speech in the form of speech.
[0209] In this embodiment, in the human-machine speech interaction device 900 based on the generative large language model, the specific processing of the user input speech acquisition unit 901, the model calling unit 902, the interface calling sequence execution unit 903, and the machine reply speech generation unit 904 and the technical effects brought by them can correspond to Figure 6 The relevant descriptions in the illustrated method embodiment will not be repeated here.
[0210] In some optional implementations of this embodiment, the interface call sequence execution unit 903 may be further configured to:
[0211] Determine the execution order of each interface call instruction and different interface call instructions contained in the service interface call sequence output by the target generative large language model;
[0212] Execute the corresponding interface call instructions in order.
[0213] In some optional implementations of this embodiment, the human-computer speech interaction device 900 based on the generative large language model may further include:
[0214] The expression adjustment unit is configured to adjust the output text into different expressions that express the same meaning.
[0215] In some optional implementations of this embodiment, the expression adjustment unit may be further configured to:
[0216] Adjust the output text to a random expression that expresses the same meaning.
[0217] In some optional implementations of this embodiment, the expression adjustment unit may be further configured to:
[0218] Adjust the output text to express the same meaning as the user input text.
[0219] In some optional implementations of this embodiment, the voice assistant that returns a machine reply voice to the user input voice includes: a map voice assistant from a map application, a voice assistant from a mobile application, and a home voice assistant from a smart terminal device.
[0220] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The human-computer voice interaction solution based on the generative large language model provided by this embodiment is specifically for the scenario in which the voice assistant acts as the executor. The target generative large language model trained according to the above-mentioned training scheme is applied to the human-computer voice interaction scenario, so that the voice assistant can use the target generative large language model to convert the user input text corresponding to the user input voice into a matching service interface call sequence, and then obtain the output text by sequentially executing the service interface call sequence, and finally convert the output text into a machine reply voice and return it to the user to complete the human-computer voice interaction, thereby improving the human-computer voice interaction experience.
[0221] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, the generative large language model training method described in any of the above embodiments and / or the human-computer speech interaction method based on the generative large language model can be implemented.
[0222] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the generative large language model training method described in any of the above embodiments and / or the human-computer speech interaction method based on the generative large language model when executed.
[0223] The embodiments of the present disclosure provide a computer program product, which, when executed by a processor, can implement the generative large language model training method and / or the human-computer speech interaction method based on the generative large language model described in any of the above embodiments.
[0224] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0225] like Fig.10As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0226] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0227] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a generative large language model training method and / or a human-computer voice interaction method based on a generative large language model. For example, in some embodiments, a generative large language model training method and / or a human-computer voice interaction method based on a generative large language model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the generative large language model training method and / or the human-computer speech interaction method based on the generative large language model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the generative large language model training method and / or the human-computer speech interaction method based on the generative large language model by any other appropriate means (e.g., by means of firmware).
[0228] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0229] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0230] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0231] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0232] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0233] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.
[0234] The generative large language model training scheme provided by the embodiment of the present disclosure has a first training set constructed by user input text (obtained by converting user input speech) and a service interface call sequence expressing the intention to use the corresponding function, so that the first training set is used to perform supervised fine-tuning training on the pre-trained first generative large language model, so that the trained second generative large language model has the ability to rewrite the user input text into a matching service interface call sequence, and the second training set constructed by user preference ranking and a preset template set between the same user input text and different candidate outputs is used to perform supervised training on the pre-trained third generative large language model, so that the trained reward model has the ability to perform quality scoring for different candidates generated by the generative large language model for the user input text, so that under the training of the reinforcement learning method, the target generative large language model finally obtained not only has the ability to rewrite the user input text into a matching service interface call sequence, but also has the ability to output results that better meet user needs. That is, through the above training scheme, the trained target generative large language model has the ability to deeply understand user needs and construct corresponding interface call instructions to query and return accurate answers. It can not only generate natural language output based on natural language input and the knowledge contained in the large model parameters, but also call the corresponding functions with the help of application program interfaces that provide professional capabilities to more effectively solve user needs. Under the influence of the reward model, the returned results will be more in line with the actual needs and expectations of users.
[0235] Based on the beneficial effects of the generative large language model training scheme provided in the above-mentioned embodiments, the human-computer voice interaction scheme based on the generative large language model provided in the embodiments of the present disclosure is specifically aimed at the scenario in which the voice assistant acts as the executor. The target generative large language model trained according to the above-mentioned training scheme is applied to the human-computer voice interaction scenario, so that the voice assistant uses the target generative large language model to convert the user input text corresponding to the user input voice into a matching service interface call sequence, and then obtains the output text by sequentially executing the service interface call sequence, and finally converts the output text into a machine reply voice and returns it to the user to complete the human-computer voice interaction, thereby improving the human-computer voice interaction experience.
[0236] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0237] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A generative large language model training method, comprising: Acquire a user input voice issued by a user in the form of voice, convert the user input voice into a user input text, and determine the functional use intention expressed by the user input text; In response to the expression of the user input text containing at least two of the function usage intentions, respectively determine the target service interface and specific interface parameters corresponding to each of the function usage intentions; generate corresponding interface call instructions according to the target service interface and specific interface parameters corresponding to each of the function usage intentions; determine the execution order of different interface call instructions according to the execution dependency relationship between different function usage intentions; organize different interface call instructions according to the execution order to obtain a service interface call sequence; and construct a first training set based on sample pairs consisting of the user input text and the matching service interface call sequence; Using the first training set to perform supervised fine-tuning training on the pre-trained first generative large language model, a second generative large language model is obtained; in the supervised fine-tuning training mode, the features of the new language unit representing the service interface that does not appear in the first generative large language model are initialized by parameter sampling according to the multivariate normal distribution of the features of the existing language units; Constructing a second training set based on user preference rankings between the same user input text and different candidate outputs and a preset template set, wherein the candidate output is an output that matches the user input text, and the preset template set includes preset templates that record the corresponding relationship between the input text and the corresponding interface call instruction; The pre-trained third generative large language model is supervisedly trained using the second training set to obtain a reward model; The second generative large language model is trained in a reinforcement learning manner based on the score returned by the reward model to obtain a target generative large language model.
2. The method according to claim 1, wherein: The second training set is constructed based on the user preference ranking between the same user input text and different candidate outputs and the preset template set, including: Get search terms from search logs; Confirming all search results returned for the same search term from the search log, and determining the cumulative number of clicks on each search result within a time window; Generate multiple sample pairs with the same search term and each search result, and determine the ranking information between the corresponding different sample pairs according to the accumulated number of clicks corresponding to the different search results; A second training set is jointly constructed based on each sample pair sorted by user preference according to the sorting information and each preset template recording the correspondence between the input text and the corresponding interface call instruction; wherein the preset template set is composed of each preset template.
3. A human-computer speech interaction method based on a generative large language model, comprising: Get the user input voice issued by the user in voice form; Inputting the user input text corresponding to the user input speech as input data into a target generative large language model; wherein the target generative large language model is obtained based on the generative large language model training method according to claim 1 or 2; Execute the interface call sequence output by the target generative large language model to obtain output text; The output text is converted into a machine reply speech in speech form.
4. The method according to claim 3, wherein: The interface call sequence for executing the target generative large language model output includes: Determining the execution order of each interface calling instruction and different interface calling instructions contained in the service interface calling sequence output by the target generative large language model; Execute the corresponding interface calling instructions in sequence according to the execution order.
5. The method according to claim 3, further comprising: The output text is adjusted to different expressions that express the same meaning.
6. The method according to claim 5, wherein: The step of adjusting the output text to different expressions expressing the same meaning includes: The output text is adjusted to a random expression that expresses the same meaning.
7. The method according to claim 5, wherein: The step of adjusting the output text to different expressions expressing the same meaning includes: The output text is adjusted to express the same meaning and in the same expression as the user input text.
8. The method according to any one of claims 3 to 7, wherein: The voice assistants that return the machine reply voice to the user input voice include: a map voice assistant from a map application, a voice assistant from a mobile application, and a home voice assistant from a smart terminal device.
9. A generative large language model training device, comprising: A first training set construction unit is configured to obtain a user input speech uttered by a user in the form of speech, convert the user input speech into a user input text, and determine a functional use intention expressed by the user input text; In response to the expression of the user input text containing at least two of the function usage intentions, respectively determine the target service interface and specific interface parameters corresponding to each of the function usage intentions; generate corresponding interface call instructions according to the target service interface and specific interface parameters corresponding to each of the function usage intentions; determine the execution order of different interface call instructions according to the execution dependency relationship between different function usage intentions; organize different interface call instructions according to the execution order to obtain a service interface call sequence; and construct a first training set based on sample pairs consisting of the user input text and the matching service interface call sequence; The supervised fine-tuning training unit is configured to perform supervised fine-tuning training on the pre-trained first generative large language model using the first training set to obtain a second generative large language model; in the supervised fine-tuning training mode, the features of the new language unit representing the service interface that does not appear in the first generative large language model are initialized by parameter sampling according to the multivariate normal distribution of the features of the existing language units; A second training set construction unit is configured to construct a second training set based on a user preference ranking between the same user input text and different candidate outputs and a preset template set, wherein the candidate output is an output matching the user input text, and the preset template set includes preset templates recording a corresponding relationship between the input text and the corresponding interface call instruction; A reward model training unit is configured to perform supervised training on the pre-trained third generative large language model using the second training set to obtain a reward model; The reinforcement learning training unit is configured to train the second generative large language model in a reinforcement learning manner based on the score returned by the reward model to obtain a target generative large language model.
10. The device according to claim 9, wherein: The second training set construction unit is further configured to: Get search terms from search logs; Confirming all search results returned for the same search term from the search log, and determining the cumulative number of clicks on each search result within a time window; Generate multiple sample pairs with the same search term and each search result, and determine the ranking information between the corresponding different sample pairs according to the accumulated number of clicks corresponding to the different search results; A second training set is jointly constructed based on each sample pair sorted by user preference according to the sorting information and each preset template recording the correspondence between the input text and the corresponding interface call instruction; wherein the preset template set is composed of each preset template.
11. A human-computer speech interaction device based on a generative large language model, comprising: A user input voice acquisition unit is configured to acquire a user input voice issued by a user in the form of voice; A model calling unit is configured to input the user input text corresponding to the user input voice as input data into a preset target generative large language model; wherein the target generative large language model is obtained based on the generative large language model training device according to claim 9 or 10; An interface call sequence execution unit is configured to execute the interface call sequence output by the target generative large language model to obtain an output text; The machine reply speech generating unit is configured to convert the output text into a machine reply speech in speech form.
12. The device according to claim 11, wherein The interface call sequence execution unit is further configured to: Determining the execution order of each interface calling instruction and different interface calling instructions contained in the service interface calling sequence output by the target generative large language model; Execute the corresponding interface calling instructions in sequence according to the execution order.
13. The apparatus according to claim 12, further comprising: The expression adjustment unit is configured to adjust the output text into different expressions that express the same meaning.
14. The device according to claim 13, wherein: The expression adjustment unit is further configured to: The output text is adjusted to a random expression that expresses the same meaning.
15. The device according to claim 13, wherein: The expression adjustment unit is further configured to: The output text is adjusted to express the same meaning and in the same expression as the user input text.
16. The device according to any one of claims 11 to 15, wherein: The voice assistants that return the machine reply voice to the user input voice include: a map voice assistant from a map application, a voice assistant from a mobile application, and a home voice assistant from a smart terminal device.
17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the generative large language model training method described in claim 1 or 2 and / or the human-computer speech interaction method based on the generative large language model described in any one of claims 3-8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the generative large language model training method described in claim 1 or 2 and / or the human-computer speech interaction method based on the generative large language model described in any one of claims 3-8.
19. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the generative large language model training method according to claim 1 or 2 and / or the steps of the human-computer speech interaction method based on the generative large language model according to any one of claims 3-8.
