Interactive model training method and method, device and medium for answer information generation
By integrating supervised fine-tuning and reinforcement learning with artificial feedback, the dialogue model effectively addresses the challenge of understanding user intentions, resulting in enhanced response accuracy.
Patent Information
- Application Number
- JP2024098979
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-30
- Filing Date
- 2024-06-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-06-19
AI Technical Summary
Existing dialogue models struggle to accurately understand user intentions and generate relevant responses, particularly in task-based interactions, due to the limitations of current training methods.
A method involving supervised fine-tuning training with reinforcement learning, utilizing a dialogue model trained with a combination of supervised and reinforcement learning phases, incorporating artificial feedback to enhance the model's ability to understand user intentions and improve response accuracy.
The approach enhances the dialogue model's factual accuracy and ability to understand user intentions, leading to improved response information generation.
Smart Images

Figure 2024163990000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of natural language processing and intelligent dialogue technology, and more particularly to a dialogue model training method, a response information generating method realized based on the dialogue model, a device, an electronic device, a computer-readable storage medium, and a computer program. [Background technology]
[0002] Artificial intelligence is a field that studies how computers can imitate some of the human thought processes and intellectual behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. AI hardware technologies generally include sensors, AI-dedicated chips, cloud computing, distributed storage, big data processing, etc., while AI software technologies mainly include computer vision technology, voice recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0003] Task-based dialogue generation technology based on ultra-large-scale language models is currently one of the research focuses in the field of artificial intelligence. This technology utilizes the natural language generation capabilities of large-scale language models to generate dialogue content that meets the requirements of specific tasks by combining it with the specific needs of task-based dialogue.
[0004] The methods described in this section are not necessarily methods that have been previously conceived or adopted. Unless otherwise noted, any methods described in this section should not be considered prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise noted, any problem addressed in this section should not be considered an admission of prior art. Summary of the Invention
[0005] The present disclosure provides a dialogue model training method, an answer information generating method implemented based on a dialogue model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program.
[0006] According to an aspect of the present disclosure, a method for training a dialogue model is provided, comprising: obtaining a first sample data set including at least one first sample data and at least one second sample data, each of the at least one first sample data including a first question text and a first answer text, and each of the at least one second sample data including a second question text; using the first sample data set to train a dialogue model, inputting at least one first question text corresponding to the at least one first sample data into the dialogue model respectively to obtain at least one corresponding first answer prediction result output by the dialogue model; for a second question text corresponding to each of the at least one second sample data, inputting the second question text into the dialogue model; The method includes performing an operation of obtaining a second answer prediction result output by the dialogue model, inputting the second answer prediction result into an incentive model obtained by training based on at least one sample question, a plurality of answer texts corresponding to each of the at least one sample question, and each label in the plurality of answer texts to obtain a score of the second answer prediction result output by the incentive model, the label indicating a user satisfaction level of the corresponding answer text, determining a total loss based on the at least one first answer prediction result, each of the first answer texts in the at least one first sample data, and each of the scores corresponding to each of the first sample data, and performing a first training process of adjusting at least one parameter of the dialogue model based on the total loss.
[0007] According to one aspect of the present disclosure, there is provided a method for generating answer information implemented based on a dialogue model, the method including obtaining a user's question text, inputting the question text into a dialogue model, and obtaining an answer text generated by the dialogue model, the dialogue model being trained according to the dialogue model training method.
[0008] According to an aspect of the present disclosure, an apparatus for training a dialogue model is provided, the apparatus including a first acquiring unit configured to acquire a first sample data set including at least one first sample data and at least one second sample data, each of the at least one first sample data including a first question text and a first answer text, and each of the at least one second sample data including a second question text; training the dialogue model using the first sample data set, inputting at least one first question text corresponding to the at least one first sample data into the dialogue model respectively to obtain at least one corresponding first answer prediction result output by the dialogue model; and for a second question text corresponding to each of the at least one second sample data, inputting the second question text into the dialogue model; and a first training unit configured to perform a first training process of: acquiring a second answer prediction result output by the dialogue model; inputting the second answer prediction result into an incentive model obtained by training based on at least one sample question, a plurality of answer texts corresponding to each of the at least one sample question, and each label in the plurality of answer texts to acquire a score of the second answer prediction result output by the incentive model, wherein the label indicates a user satisfaction level of the corresponding answer text; determining a total loss based on the at least one first answer prediction result, each of the first answer texts in the at least one first sample data, and each of the scores corresponding to each of the first answer texts in the at least one second sample data; and adjusting at least one parameter of the dialogue model based on the total loss.
[0009] According to another aspect of the present invention, there is provided an answer information generation device implemented based on a dialogue model, comprising: an acquisition unit configured to acquire a user's question text; and a generation unit configured to input the question text into the dialogue model to acquire an answer text generated by the dialogue model, wherein the dialogue model is trained according to the above dialogue model training method.
[0010] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method for training the dialogue model or a method for generating answer information implemented based on the dialogue model.
[0011] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions, wherein the computer instructions are used to cause a computer to execute a method for training the dialogue model or a method for generating answer information implemented based on the dialogue model.
[0012] According to another aspect of the present disclosure, there is provided a computer program that, when executed by a processor, realizes the above-mentioned dialogue model training method or the answer information generating method realized based on the dialogue model.
[0013] According to one or more embodiments of the present disclosure, in the reinforcement learning training phase based on artificial feedback of the dialogue model, a loss of supervised fine-tuning training is introduced, so that the ability to solve the dialogue task learned during the supervised fine-tuning training is not forgotten in the reinforcement learning phase, and the factual accuracy and the ability to understand user intentions of the dialogue model are improved, thereby overall improving the answer information generation effect of the dialogue model.
[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. Other features of the present disclosure will be easily understood from the following description. [Brief description of the drawings]
[0015] The drawings illustratively illustrate the examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and are not intended to limit the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements.
[0016] [Figure 1] 1 shows a schematic diagram of an exemplary system capable of implementing the methods described herein, according to an embodiment of the present disclosure. [Diagram 2] 1 illustrates a flowchart of a method for training a dialogue model according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates a flowchart for determining total loss according to an embodiment of the present disclosure. [Figure 4] 1 illustrates a flowchart for determining a second loss according to an embodiment of the present disclosure. [Diagram 5] 1 illustrates a flowchart of a method for training a dialogue model according to an embodiment of the present disclosure. [Figure 6] 1 illustrates a flowchart of a method for generating answer information implemented based on a dialogue model according to an embodiment of the present disclosure. [Figure 7] FIG. 1 shows a block diagram of a dialogue model training device according to an embodiment of the present disclosure. [Figure 8] FIG. 1 shows a configuration block diagram of a response information generating device realized based on a dialogue model according to an embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates a block diagram of an exemplary electronic device for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, and various details in the embodiments of the present disclosure contained therein are to be considered as merely illustrative, in order to facilitate understanding. Therefore, it should be recognized that those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description omits descriptions of known functions and structures.
[0018] In this disclosure, unless otherwise stated, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of an element, or in some cases, may refer to different instances based on the contextual description.
[0019] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless specifically limited in number. It should be noted that the term "and / or" as used in the present disclosure covers any and all possible combinations of the listed items.
[0020] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
[0021] 1 illustrates a schematic diagram of an example system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. With reference to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
[0022] In an embodiment of the present disclosure, the server 120 may execute one or more services or software applications that enable the execution of the method for training the dialogue model or the method for generating answer information implemented based on the dialogue model.
[0023] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software-as-a-Service (SaaS) model.
[0024] In the configuration shown in FIG. 1, the server 120 may include one or more assemblies that realize the functions performed by the server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed on one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 may in turn use one or more client applications to interact with the server 120 to utilize the services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from the system 100. Thus, FIG. 1 is an example of a system for implementing the various methods described herein and is not intended to be limiting.
[0025] A user may input dialogue text using client devices 101, 102, 103, 104, 105, and / or 106. The client devices may provide an interface through which a user of the client device interacts with the client device. The client devices may also output information to the user through the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.
[0026] The client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., GOOGLE Chrome OS), and may include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android, and the like. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.
[0027] Network 110 may be any type of network known to those skilled in the art that may use any one of a number of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.
[0028] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable configuration and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage devices). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.
[0029] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercially available server operating system. Server 120 may also run any one of a variety of additional server applications and / or mid-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.
[0030] In some embodiments, server 120 may include one or more applications for analyzing and synthesizing data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.
[0031] In some embodiments, the server 120 may be a server of a distributed system or a server incorporating blockchain. The server 120 may be a cloud server, or an intelligent cloud computing server or an intelligent cloud host equipped with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, and solves the defects of the traditional physical host and virtual private server (VPS) services, such as high management difficulty and low business scalability.
[0032] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files and video files. Databases 130 may be configured in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or a dedicated connection. Databases 130 may be of a variety of types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the database and from the database in response to instructions.
[0033] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be various types of databases, such as a key-value repository, an object repository, a general-purpose repository supported by a file system, etc.
[0034] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the application of the various methods and apparatus described in accordance with this disclosure.
[0035] According to an embodiment of the present disclosure, as shown in FIG. 2, a method for training a dialogue model is provided, comprising: A step S201 of obtaining a first sample data set including at least one first sample data and at least one second sample data, each of the at least one first sample data including a first question text and a first answer text, and each of the at least one second sample data including a second question text; Use the first sample dataset to train the dialogue model. A step S2021 of inputting at least one first question text corresponding to at least one first sample data into a dialogue model, and obtaining at least one corresponding first answer prediction result output by the dialogue model; For each second question text in at least one second sample data, A step S2022 of inputting the second question text into a dialogue model to obtain a second answer prediction result output by the dialogue model; Step S2023 of inputting the second answer prediction result into an incentive model obtained by training based on at least one sample question, a plurality of answer texts corresponding to each of the at least one sample question, and each label of the plurality of answer texts to obtain a score of the second answer prediction result output by the incentive model, the label indicating the user satisfaction level of the corresponding answer text; A step S2024 of determining a total loss based on at least one first answer prediction result, each first answer text in at least one first sample data, and each corresponding score in at least one second sample data; The method includes performing a first training process S202 and a step S2025 of adjusting at least one parameter of a dialogue model based on the overall loss.
[0036] Therefore, by introducing the loss of supervised fine-tuning training in the reinforcement learning training stage based on artificial feedback of the dialogue model, the ability to solve dialogue tasks learned during supervised fine-tuning training is not forgotten in the reinforcement learning stage, and the factual accuracy and user intention understanding ability of the dialogue model are improved, thereby overall improving the answer information generation effect of the dialogue model.
[0037] In some embodiments, the first sample data set may include two types of sample data, where each of the first sample data includes a first question text and a corresponding first answer text, and each of the second sample data includes a second question text.
[0038] In some embodiments, the second question text may be the same as a first question text in the first sample data set. In some embodiments, the first question text and the second question text in the first sample data set may be different from each other.
[0039] In some embodiments, each of the first question text and the second question text can be input into a current dialogue model, and a corresponding first answer prediction result and a corresponding second answer prediction result can be obtained. Then, each of the second answer prediction results and the corresponding second question text can be input into a pre-trained incentive model, and a score of the second answer prediction result output by the incentive model can be obtained. Here, the score of the second answer prediction result can be used to indicate the user's satisfaction with the answer information.
[0040] In some embodiments, the incentive model can be obtained by training in the following manner: first, obtain one or more sample questions, and input each sample question into the current dialogue model in turn, thereby generating multiple answer texts for each sample question, and obtaining a label for each answer text based on artificial marking, where the label can indicate the user satisfaction of the corresponding answer text; then, input each answer text and its corresponding sample question into an initial incentive model, thereby obtaining a prediction result of the model; then, calculate a loss based on the prediction result and the corresponding label, and adjust model parameters until the incentive model converges.
[0041] In some embodiments, the incentive model may be established based on architectures such as multi-layer perceptrons, neural networks, and the like.
[0042] In some embodiments, the total loss can be calculated based on at least one first answer prediction result, each first answer text in at least one first sample data, and each corresponding score in at least one second sample data. For example, when the first question text and the second question text are the same, the difference between each first answer prediction result and the corresponding first answer text is calculated, and then a weighting factor for the difference is determined based on the corresponding score, and the total loss can be calculated based on the difference multiplied by one or more weighting factors.
[0043] In some embodiments, as shown in FIG. 3, determining the overall loss based on at least one first answer prediction result, each first answer text in at least one first sample data, and each corresponding score in at least one second sample data includes: A step S301 of determining a first loss based on each first answer text and a corresponding first answer prediction result in at least one first sample data; determining a second loss based on at least one score corresponding to at least one second sample data; and determining a total loss based on the first loss and the second loss in step S303.
[0044] In this way, by calculating the losses of the two parts respectively and obtaining the overall loss, federated modeling of two-stage models, namely, the dialogue task-oriented supervised fine-tuning training stage and the reinforcement learning stage with artificial feedback, can be realized, and mutually promoted, improving the model's modeling ability for user preferences while maintaining the model's understanding and satisfaction ability for user instructions, and further improving the overall effect of generating answer information of the dialogue model.
[0045] In some embodiments, the first loss may be determined based on a difference between a first answer prediction result for each of the at least one first sample data and a corresponding first answer text.
[0046] In some embodiments, the difference may be measured based on a cross-entropy error function or a mean squared error function.
[0047] In some embodiments, the second loss may be an average or expectation of at least one score corresponding to at least one second answer prediction result.
[0048] In some embodiments, the total loss may be obtained by a combination of mixed losses, alternating minimized losses, and the like.
[0049] In some embodiments, determining the overall loss based on the first loss and the second loss may include weighting the first loss and the second loss based on a first predetermined weight corresponding to the first loss and a second predetermined weight corresponding to the second loss to obtain the overall loss.
[0050] This allows us to control the impact of the two training methods on the overall model training by weighting the two losses, thereby ensuring the effectiveness of the training.
[0051] In some embodiments, in the training phase of the dialogue model, multiple first sample data sets may be obtained sequentially, and multiple rounds of model training may be performed based on the multiple first sample data sets.
[0052] In some embodiments, as the number of training rounds increases, the first and second predetermined weights can be adjusted correspondingly to adjust the proportion of the loss function in different training rounds under the condition of ensuring that the sum is constant, thereby controlling the model training process. For example, a fluctuation factor is set, and the first predetermined weight is gradually increased while the second predetermined weight is gradually decreased according to the fluctuation factor as the number of training rounds increases, thereby ensuring the training effect of the model and improving the training efficiency of the model.
[0053] In some embodiments, as shown in FIG. 4, determining the second loss based on at least one score corresponding to the at least one second sample data includes: A step S401 of determining a mean and a variance of at least one score based on the at least one score; S402, for each score of the at least one score, normalizing the score based on its mean and variance to obtain an updated score; and determining a second loss based on the at least one updated score (S403).
[0054] In this way, by normalizing the scores and then calculating the losses, the score distribution in the training process can be further optimized, which can avoid the problem of weakening the impact of reinforcement learning losses on the model due to the introduction of other losses in the reinforcement learning stage, improve the stability of reinforcement learning, and ensure the effect of reinforcement learning losses on model training under the framework of federated optimization.
[0055] In the reinforcement learning stage, by introducing extra losses and performing associative training, the increase in loss in the reinforcement learning part is suppressed and the role that reinforcement learning should play is weakened.
[0056] In some embodiments, before calculating the second loss, the scores corresponding to all the second answer prediction results in the first sample dataset of the current round may be averaged and variance-calculated, and then all the scores in the current round may be normalized based on the average and variance to obtain normalized scores, and the second loss may be calculated based on each normalized score.
[0057] In some exemplary embodiments, the score r i The normalization operation for can be expressed by the following equation:
number
number
number
[0058] This allows the scores in the round to be under a dynamic standard normal distribution, thereby improving the stability of reinforcement learning and ensuring that the loss of reinforcement learning increases normally under the framework of federated optimization.
[0059] In some embodiments, the second loss may be the average or expectation of the normalized scores.
[0060] In some embodiments, the method for training the dialogue model may further include: obtaining a pre-trained language model and a second sample dataset including at least one third sample data, each of the at least one third sample data including a third question text and a third answer text, and obtaining the pre-trained language model by training based on a predetermined number of unsupervised sample corpora; and repeatedly performing a training operation on the pre-trained language model, so as to obtain an initial dialogue model before using the first sample dataset, based on each third sample data in the second sample dataset, inputting a third question text corresponding to the third sample data into the pre-trained language model until the pre-trained language model converges, and obtaining a third answer prediction result output by the pre-trained language model; and adjusting parameters of the pre-trained language model based on the third answer prediction result and the third answer text corresponding to the third sample data to update the pre-trained language model.
[0061] Thus, before performing the first training process, first perform supervised fine-tuning training based on the pre-trained language model to obtain an initial dialogue model, and then further perform joint training of supervised fine-tuning training and reinforcement learning training based on artificial feedback based on the initial dialogue model, so that the model has the ability to solve dialogue tasks, and further obtains prediction ability for user preferences, thereby improving the overall performance of the model.
[0062] FIG. 5 illustrates a flowchart of a method for training a dialogue model according to an embodiment of the present disclosure.
[0063] According to some embodiments, as shown in FIG. 5, the training process of the dialogue model may include: a step S501 of training a generic pre-trained language model based on an unsupervised corpus of a predetermined size to obtain a generic pre-trained language model; a step S502 of performing supervised fine-tuning training on the generic pre-trained language model, specifically, first obtaining a second sample data set, applying third sample data therein, and performing supervised fine-tuning training on the generic pre-trained language model to obtain an initial dialogue model that can understand the intention contained in the question or command input by a user and provide a relatively high-quality answering ability based on the intention; and a step S503 of training the initial dialogue model based on reinforcement learning to obtain a final dialogue model, in which the step includes obtaining multiple lots of first sample data sets based on the above-mentioned training method, and applying two different sample data in the first sample data set in each round to jointly train the model to obtain a final dialogue model.
[0064] In some embodiments, the third sample data in the second sample data set and the first sample data in the first sample data set can be obtained from the same pre-prepared sample data set, and each sample data in the sample data set has sample questions and corresponding answer information, which allows the model to better predict user preferences while maintaining its ability to understand questions or commands input by users and generate high-quality answer information.
[0065] In some embodiments, the dialogue model is obtained through at least one first training process based on an initial dialogue model, and the training method for the dialogue model further includes inputting the second question text into the initial dialogue model to obtain a fourth answer prediction result output by the initial dialogue model, where determining the second loss based on at least one score corresponding to the at least one second sample data includes determining the second loss based on the at least one score, the second question text corresponding to each of the at least one second sample data, the second answer prediction result, and the fourth answer prediction result.
[0066] As a result, in the second loss calculation process, the stability of model training can be further improved by introducing a regularization term calculated based on the second question text, the second answer prediction result, and the fourth answer prediction result.
[0067] In some embodiments, a normalization term can be determined based on the difference between the second question text of each second sample data corresponding to the second answer prediction result and the fourth answer prediction result, and a second loss is determined based on the normalization term and the score (or normalized score) corresponding to the second answer prediction result.
[0068] In some embodiments, the regularization term mentioned above may be calculated based on the KL divergence.
[0069] In some embodiments, the total loss may be expressed by the following formula:
number
number
number
number
number
number
number
number
[0070] In some embodiments, the first number of the at least one first sample data and the second number of the at least one second sample data are both plural, and the first number and the second number correspond to a predetermined ratio.
[0071] In this way, by controlling the proportion of the two types of sample data, the extent to which the two training methods in the joint training affect the model during the training process can be controlled, and the training effect of the model can be optimized overall.
[0072] In some embodiments, the predetermined ratio between the first number and the second number may be, for example, 1:7.
[0073] In some embodiments, for a plurality of first sample data sets in a plurality of rounds, the proportion of the first number and the second number can be increased with the increase in the number of training rounds, for example, the proportion of the first number can be gradually increased, thereby ensuring the training effect of the model and at the same time improving the training efficiency of the model.
[0074] In some demonstrative embodiments, the dialogue model can be established based on, for example, a knowledge-expanded large language model for dialogue (eg, ERNIE bot, etc.).
[0075] In some embodiments, as shown in FIG. 6, a method for generating answer information is provided that is implemented based on a dialogue model, and includes step S601 of obtaining a user's question text, and step S602 of inputting the question text into the dialogue model to obtain an answer text generated by the dialogue model, where the dialogue model is trained according to the above-mentioned dialogue model training method.
[0076] In this way, by using the dialogue model obtained by training using the above-mentioned training method, it is possible to provide a dialogue model with greater accuracy in understanding facts and greater ability to understand user intent, thereby enabling the generation of answer information that is more in line with the user's expectations.
[0077] In some embodiments, as shown in FIG. 7, an apparatus 700 for training a dialogue model is provided, a first acquiring unit 710 configured to acquire a first sample data set including at least one first sample data and at least one second sample data, each of the at least one first sample data including a first question text and a first answer text, and each of the at least one second sample data including a second question text; Use the first sample dataset to train the dialogue model. Inputting at least one first question text corresponding to at least one first sample data into a dialogue model, respectively, to obtain at least one corresponding first answer prediction result output by the dialogue model; For each second question text in at least one second sample data, inputting the second question text into a dialogue model to obtain a second answer prediction result output by the dialogue model; inputting the second answer prediction result into an incentive model obtained by training based on at least one sample question, a plurality of answer texts corresponding to each of the at least one sample question, and each label of the plurality of answer texts, to obtain a score of the second answer prediction result output by the incentive model, the label indicating a user satisfaction level of the corresponding answer text; Determine a total loss based on at least one first answer prediction result, each first answer text in at least one first sample data, and each corresponding score in at least one second sample data; and a first training unit 720 configured to perform a first training process of adjusting at least one parameter of a dialogue model based on the overall loss.
[0078] In some embodiments, determining a total loss based on the at least one first answer prediction result, each first answer text in the at least one first sample data, and each corresponding score in the at least one second sample data may include determining a first loss based on each first answer text and corresponding first answer prediction result in the at least one first sample data, determining a second loss based on at least one score corresponding to the at least one second sample data, and determining a total loss based on the first loss and the second loss.
[0079] In some embodiments, determining the second loss based on the at least one score corresponding to the at least one second sample data may include determining a mean and a variance of the at least one score based on the at least one score, and for each score in the at least one score, normalizing the score based on the mean and variance to obtain an updated score, and determining the second loss based on the at least one updated score.
[0080] In some embodiments, determining the overall loss based on the first loss and the second loss may include weighting the first loss and the second loss based on a first predetermined weight corresponding to the first loss and a second predetermined weight corresponding to the second loss to obtain the overall loss.
[0081] In some embodiments, the training apparatus may further include: a second acquiring unit configured to acquire a pre-trained language model and a second sample dataset including at least one third sample data, each of the at least one third sample data including a third question text and a third answer text, and the pre-trained language model is trained based on a predetermined number of unsupervised sample corpora; and a second training unit configured to repeatedly perform a training operation on the pre-trained language model, such as inputting a third question text corresponding to the third sample data into the pre-trained language model based on each third sample data in the second sample dataset until the pre-trained language model converges, and obtaining a third answer prediction result output by the pre-trained language model, and adjusting parameters of the pre-trained language model based on the third answer prediction result and the third answer text corresponding to the third sample data, to update the pre-trained language model, so as to acquire an initial dialogue model before training the dialogue model using the first sample dataset.
[0082] In some embodiments, the dialogue model is obtained through at least one first training process based on an initial dialogue model, and the training apparatus further includes a third obtaining unit configured to input the second question text into the initial dialogue model to obtain a fourth answer prediction result output by the initial dialogue model, and wherein determining the second loss based on at least one score corresponding to the at least one second sample data includes determining the second loss based on the at least one score, the second question text, the second answer prediction result, and the fourth answer prediction result corresponding to each of the at least one second sample data.
[0083] In some embodiments, the first number of the at least one first sample data and the second number of the at least one second sample data are plural, and the first number and the second number correspond to a predetermined ratio.
[0084] In some embodiments, as shown in FIG. 8 , an answer information generating device 800 realized based on a dialogue model is further provided, including an obtaining unit 810 configured to obtain a user's question text, and a generating unit 820 configured to input the question text into the dialogue model to obtain an answer text generated by the dialogue model, where the dialogue model is trained according to the above dialogue model training method.
[0085] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, and a computer program are further provided.
[0086] With reference to FIG. 9, a configuration block diagram of an electronic device 900 functioning as a server or client of the present disclosure will be described, which is an example of a hardware device applicable to each aspect of the present disclosure. The electronic device may represent various forms of digital electronic computers, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.
[0087] 9, the electronic device 900 includes a computing unit 901 that can perform various appropriate operations and processes according to a computer program stored in a read only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the electronic device 900. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0088] A number of components in the electronic device 900, including an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909, are connected to the I / O interface 905. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input numeric or character information and generate key signal inputs for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a trackboard, a trackball, a control lever, a microphone, and / or a remote control. The output unit 907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 may include, but is not limited to, a magnetic disk, an optical disk. The communications unit 909 enables the electronic device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communications device, a wireless communications transceiver, and / or a chipset (e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communications device, and / or the like).
[0089] The computing unit 901 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 executes various methods and processes described above, such as the training method of the dialogue model or the answer information generation method implemented based on the dialogue model. For example, in some embodiments, the training method of the dialogue model or the answer information generation method implemented based on the dialogue model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, the training method of the dialogue model or the answer information generation method implemented based on the dialogue model can be executed. Alternatively, in other embodiments, the computation unit 901 may be configured to execute the method for training the dialogue model or the method for generating answer information realized based on the dialogue model in any other suitable manner (e.g., by firmware).
[0090] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may include receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0091] The program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations specified in the flow charts and / or block diagrams. The program code may be entirely executed on a machine, partly executed on a machine, partly executed on a machine and partly executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.
[0092] In the context of the present disclosure, a machine-readable medium may be a tangible medium, which may include or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0093] To provide for interaction with a user, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback) and may receive input from the user in any form (including sound input, speech input, or tactile input).
[0094] The systems and techniques described herein may be implemented in a computing system including backstage components (e.g., as a data server), middleware components (e.g., as an application server), front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique embodiments), or any combination of backstage, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0095] The computer system may include a client side and a server. The client side and the server are generally remote from each other and usually interact with each other via a communication network. The relationship between the client side and the server is generated by running computer programs having a client side-server relationship with each other on corresponding computers. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0096] It should be understood that the steps may be reordered, added, or removed using the various forms of flow described above. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, and the present disclosure is not limited thereto as long as the technical solutions disclosed in this disclosure can achieve the desired results.
[0097] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents after grant. Various elements of the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.
Claims
1. A method for training a dialogue model, comprising: Obtaining a first sample data set including at least one first sample data and at least one second sample data, each of the at least one first sample data including a first question text and a first answer text, and each of the at least one second sample data including a second question text; training the dialogue model using the first sample data set; inputting at least one first question text corresponding to the at least one first sample data into the dialogue model, respectively, to obtain at least one corresponding first answer prediction result output by the dialogue model; For each second question text in the at least one second sample data, inputting the second question text into the dialogue model to obtain a second answer prediction result output by the dialogue model; inputting the second answer prediction result into an incentive model obtained by training based on at least one sample question, a plurality of answer texts corresponding to the at least one sample question, and each label of the plurality of answer texts, to obtain a score of the second answer prediction result output by the incentive model, the label indicating a user satisfaction level of the corresponding answer text; determining a total loss based on the at least one first answer prediction result, each first answer text in the at least one first sample data, and each corresponding score in the at least one second sample data; performing a first training process of adjusting at least one parameter of the dialogue model based on the overall loss.
2. determining a total loss based on the at least one first answer prediction result, each first answer text in the at least one first sample data, and each corresponding score in the at least one second sample data, determining a first loss based on each first answer text and a corresponding first answer prediction result in the at least one first sample data; determining a second loss based on at least one score corresponding to the at least one second sample data; and determining the total loss based on the first loss and the second loss.
3. Determining a second loss based on at least one score corresponding to the at least one second sample data includes: determining a mean and a variance of the at least one score based on the at least one score; for each score in the at least one score, normalizing the score based on the mean and the variance to obtain an updated score; and determining the second loss based on at least one updated score.
4. determining the total loss based on the first loss and the second loss, 4. The method of claim 3, comprising weighting the first loss and the second loss based on a first predetermined weight corresponding to the first loss and a second predetermined weight corresponding to the second loss to obtain the total loss.
5. Obtaining a pre-trained language model and a second sample data set including at least one third sample data, each of the at least one third sample data including a third question text and a third answer text, and obtaining the pre-trained language model by training the pre-trained language model based on a predetermined number of unsupervised sample corpora; Before training the dialogue model using the first sample data set, an initial dialogue model is obtained based on each third sample data in the second sample data set until the pre-trained language model converges. inputting a third question text corresponding to the third sample data into the pre-trained language model to obtain a third answer prediction result output by the pre-trained language model; 5. The method of claim 4, further comprising: repeatedly performing a training operation on the pre-trained language model to update the pre-trained language model by adjusting parameters of the pre-trained language model based on the third answer prediction result and a third answer text corresponding to the third sample data.
6. The dialogue model is obtained by performing at least one first training process based on the initial dialogue model, and the method includes: inputting the second question text into the initial dialogue model to obtain a fourth answer prediction result output by the initial dialogue model; Determining a second loss based on at least one score corresponding to the at least one second sample data includes:
6. The method of claim 5, further comprising determining the second loss based on the at least one score, a corresponding second question text in the at least one second sample data, a second answer prediction result, and a fourth answer prediction result.
7. 7. The method of claim 6, wherein the first number of the at least one first sample data and the second number of the at least one second sample data are each plural, and the first number and the second number correspond to a predetermined ratio.
8. A method for generating answer information based on a dialogue model, comprising: Obtaining a user's question text; A method for generating answer information based on a dialogue model, comprising inputting the question text into the dialogue model and obtaining an answer text generated by the dialogue model, the dialogue model being trained according to a training method described in any one of claims 1 to 7.
9. A dialogue model training device, comprising: a first acquisition unit configured to acquire a first sample data set including at least one first sample data and at least one second sample data, each of the at least one first sample data including a first question text and a first answer text, and each of the at least one second sample data including a second question text; training the dialogue model using the first sample data set; inputting at least one first question text corresponding to the at least one first sample data into the dialogue model, respectively, to obtain at least one corresponding first answer prediction result output by the dialogue model; For each second question text in the at least one second sample data, inputting the second question text into the dialogue model to obtain a second answer prediction result output by the dialogue model; inputting the second answer prediction result into an incentive model obtained by training based on at least one sample question, a plurality of answer texts corresponding to the at least one sample question, and each label of the plurality of answer texts, to obtain a score of the second answer prediction result output by the incentive model, the label indicating a user satisfaction level of the corresponding answer text; determining a total loss based on the at least one first answer prediction result, each first answer text in the at least one first sample data, and each corresponding score in the at least one second sample data; a first training unit configured to perform a first training process of adjusting at least one parameter of the dialogue model based on the overall loss.
10. determining a total loss based on the at least one first answer prediction result, each first answer text in the at least one first sample data, and each corresponding score in the at least one second sample data, determining a first loss based on each first answer text and a corresponding first answer prediction result in the at least one first sample data; determining a second loss based on at least one score corresponding to the at least one second sample data; and determining the total loss based on the first loss and the second loss.
11. Determining a second loss based on at least one score corresponding to the at least one second sample data includes: determining a mean and a variance of the at least one score based on the at least one score; for each score in the at least one score, normalizing the score based on the mean and the variance to obtain an updated score; and determining the second loss based on at least one updated score.
12. determining the total loss based on the first loss and the second loss, 12. The apparatus of claim 11, further comprising weighting the first loss and the second loss based on a first predetermined weight corresponding to the first loss and a second predetermined weight corresponding to the second loss to obtain the total loss.
13. a second acquiring unit configured to acquire a pre-trained language model and a second sample data set including at least one third sample data, each of the at least one third sample data including a third question text and a third answer text, and the pre-trained language model is trained based on a predetermined amount of unsupervised sample corpora; Before training the dialogue model using the first sample data set, an initial dialogue model is obtained based on each third sample data in the second sample data set until the pre-trained language model converges. inputting a third question text corresponding to the third sample data into the pre-trained language model to obtain a third answer prediction result output by the pre-trained language model; and a second training unit configured to iteratively perform a training operation on the pre-trained language model to update the pre-trained language model by adjusting parameters of the pre-trained language model based on the third answer prediction result and a third answer text corresponding to the third sample data.
14. The dialogue model is obtained by performing the first training process at least once based on the initial dialogue model, and the device: a third obtaining unit configured to input the second question text into the initial dialogue model to obtain a fourth answer prediction result output by the initial dialogue model; Determining a second loss based on at least one score corresponding to the at least one second sample data includes: The apparatus of claim 13, further comprising determining the second loss based on the at least one score, a second question text corresponding to each of the at least one second sample data, a second answer prediction result, and a fourth answer prediction result.
15. 15. The apparatus of claim 14, wherein the first number of the at least one first sample data and the second number of the at least one second sample data are each plural, and the first number and the second number correspond to a predetermined ratio.
16. An answer information generating device realized based on a dialogue model, a capturing unit configured to capture a user's question text; and a generation unit configured to input the question text into the dialogue model and obtain an answer text generated by the dialogue model, wherein the dialogue model is trained according to the training method described in any one of claims 1 to 7.
17. An electronic device, At least one processor; a memory communicatively coupled to the at least one processor; An electronic device, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.
18. A non-transitory computer readable storage medium having stored thereon computer instructions, the computer instructions being used to cause a computer to perform the method of any one of claims 1 to 7.
19. A computer program which, when executed by a processor, performs the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Question and answer model training method, question and answer model training device, question and answer method, and question and answer device
CN113961686A
Answer selection device, model learning device, method for selecting answer, method for learning model, and program
JP2019192073A
Language model pre-training method, apparatus, device, and storage medium
JP2023012493A
Cited By
Classification and question and answer large model joint parameter adjustment method and device
CN119621912A
Big language model data illusion identification method and device and storage medium
CN121233988A
