Model training method and data processing method

Abstract

The application discloses a model training method and a data processing method, relates to the technical field of artificial intelligence, and comprises the following steps: acquiring a training sample set; if the question type of a sample question is a restrictive question, training an initial natural language model according to a loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model; and if the question type of the sample question is a question other than the restrictive question, training the initial natural language model according to a loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model. The application solves the technical problem that the accuracy of a natural language model in replying to a question is relatively low when the same loss function is used for different types of sample questions during DPO training of the natural language model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a model training method and a data processing method. Background Art

[0002] With the rapid development of artificial intelligence (AI), reinforcement learning (RL) has been widely applied to numerous tasks. Among the many RL algorithms, Direct Preference Optimization (DPO) has garnered widespread attention and adoption due to its low training cost. However, from a practical perspective, DPO uses the same loss function for different types of sample questions when training natural language models. Consequently, its performance on non-(semi-)open-ended tasks is unsatisfactory. In particular, once the model reaches a relatively optimal solution, its performance rapidly degrades.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a model training method and a data processing method to at least solve the technical problem that when training a natural language model through DPO, the same loss function is used for different types of sample questions, resulting in relatively low accuracy of the natural language model in answering questions.

[0005] According to one aspect of an embodiment of the present application, a model training method is provided, comprising: obtaining a training sample set, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include a sample question, first answer information and second answer information corresponding to the sample question, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, then the initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, then the initial natural language model is trained according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model.

[0006] Optionally, the training of the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model comprises: obtaining a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information by replying to a sample question in the training sample set by the initial natural language model; obtaining a third generation probability by calculating according to the second generation probability, wherein the third generation probability is a probability of occurrence of an opposite event of an event corresponding to the second generation probability; performing normalization processing on the first generation probability to obtain a processed first generation probability, and performing normalization processing on the third generation probability to obtain a processed third generation probability; obtaining a loss function corresponding to the question other than the restrictive question by calculating according to the processed first generation probability and the processed third generation probability, and training the initial natural language model according to the loss function corresponding to the question other than the restrictive question to obtain the target natural language model.

[0007] Optionally, the normalization processing on the first generation probability to obtain a processed first generation probability comprises: determining a total number of characters corresponding to the first answer information; and performing normalization processing on the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0008] Optionally, the training of the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model comprises: obtaining a fourth generation probability corresponding to the first answer information and a fifth generation probability corresponding to the second answer information by replying to a sample question in the training sample set by the initial natural language model; obtaining a sixth generation probability by calculating according to the fifth generation probability, wherein the sixth generation probability is a probability of occurrence of an opposite event of an event corresponding to the fifth generation probability; obtaining a loss function corresponding to the restrictive question by calculating according to the fourth generation probability and the sixth generation probability, and training the initial natural language model according to the loss function corresponding to the restrictive question to obtain the target natural language model.

[0009] Optionally, calculations are performed based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: answering the sample questions in the training sample set through the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; calculations are performed based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; calculations are performed based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0010] Optionally, calculations are performed based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; obtaining a weight value and a preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive question.

[0011] Optionally, obtaining a training sample set includes: obtaining multiple sample questions; answering the multiple sample questions a preset number of times through a first large language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; and obtaining the training sample set based on scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0012] Optionally, obtaining the training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information includes: scoring the multiple answer information corresponding to the target sample question through the second largest language model to obtain the scoring results corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; obtaining the target generation probability corresponding to the answer information in the multiple answer information; determining the first answer information and the second answer information from the multiple answer information based on the scoring results and the target generation probability; and obtaining the training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0013] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a target question input by a target object; answering the target question through a target natural language model to obtain target answer information, wherein the target natural language model is trained using any one of the model training methods described above; and returning the target answer information to the target object.

[0014] According to another aspect of an embodiment of the present application, a model training device is also provided, including: an acquisition unit for acquiring a training sample set, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; a first training unit for training the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model if the question type of the sample question is a restrictive question; a second training unit for training the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model if the question type of the sample question is a question other than a restrictive question.

[0015] The first training unit includes: a first answer module, which is used to answer the sample questions in the training sample set through the initial natural language model, and obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; a first generation module, which is used to calculate based on the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; a first processing module, which is used to normalize the first generation probability to obtain the processed first generation probability, and normalize the third generation probability to obtain the processed third generation probability; a first calculation module, which is used to calculate based on the processed first generation probability and the processed third generation probability to obtain a loss function corresponding to questions other than restrictive questions, and train the initial natural language model based on the loss function corresponding to questions other than restrictive questions to obtain the target natural language model.

[0016] Optionally, the first processing module includes: a determination submodule, used to determine the total number of characters corresponding to the first answer information; and a processing submodule, used to normalize the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0017] Optionally, the second training unit includes: a second answer module, used to answer the sample questions in the training sample set through the initial natural language model, and obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information; a second calculation module, used to calculate based on the fifth generation probability to obtain the sixth generation probability, wherein the sixth generation probability is the probability of occurrence of the opposite event of the event corresponding to the fifth generation probability; a third calculation module, used to calculate based on the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question, and train the initial natural language model based on the loss function corresponding to the restrictive question to obtain the target natural language model.

[0018] Optionally, the third calculation module includes: a reply submodule, used to reply to the sample questions in the training sample set through the initial natural language model, and obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; a first calculation submodule, used to perform calculations based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; a second calculation submodule, used to perform calculations based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0019] Optionally, the second calculation sub-module includes: a first calculation sub-module, used to calculate the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; a second calculation sub-module, used to calculate the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; an acquisition sub-module, used to obtain the weight value and the preset scaling factor corresponding to the second answer information; a third calculation sub-module, used to calculate based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculate based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; a fourth calculation sub-module, used to calculate based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive problem.

[0020] Optionally, the acquisition unit includes: an acquisition subunit, used to acquire multiple sample questions; a reply subunit, used to reply to the multiple sample questions a preset number of times through the first large language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; and a processing subunit, used to obtain the training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0021] Optionally, the processing subunit includes: a scoring module, used to score multiple answer information corresponding to the target sample question through the second largest language model, and obtain a scoring result corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; an acquisition module, used to obtain the target generation probability corresponding to the answer information in the multiple answer information; a determination module, used to determine the first answer information and the second answer information from the multiple answer information based on the scoring result and the target generation probability; a second processing module, used to obtain the training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned model training methods or the above-mentioned data processing methods.

[0023] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including a memory storing an executable program; and a processor for running the program, wherein when the program is running, any one of the above-mentioned model training methods or the above-mentioned data processing method is executed.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is also provided, which includes a stored computer program, and when the computer program is executed by a processor, it implements any one of the above-mentioned model training methods or the above-mentioned data processing methods.

[0025] In an embodiment of the present application, a method is adopted in which a training sample set is obtained and an initial natural language model is trained based on a loss function corresponding to the question type of the sample question and the training sample set. By using the loss function corresponding to the question type of the sample question, the purpose of targeted training of the natural language model is achieved, and the technical problem of using the same loss function for different types of sample questions when training the natural language model through DPO, resulting in relatively low accuracy in answering questions by the natural language model, is solved. By selecting different loss functions according to different types of sample questions, the rationality of the gradient during the model training process can be ensured, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model in answering questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0027] Figure 1 is a hardware structure block diagram of a computer terminal provided according to Embodiment One of the application;

[0028] Figure 2 is a flow of a model training method provided according to Embodiment One of the application Figure 1 ;

[0029] Figure 3 is a flow of a model training method provided according to Embodiment One of the application Figure 2 ;

[0030] Figure 4 is a flowchart of a data processing method provided according to Embodiment Two of the application;

[0031] Figure 5 is a schematic diagram of a model training device provided according to Embodiment Three of the application;

[0032] Figure 6 is a structural block diagram of an electronic device provided according to Embodiment Three of the application. DETAILED DESCRIPTION

[0033] In order to make the personnel in the art better understand the application scheme, the technical solutions in the embodiments of the application will be described clearly and completely below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of protection of the application.

[0034] It should be noted that the terms "first", "second", and the like in the specification and claims of the application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0035] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0036] DPO: Direct Preference Optimization, direct preference optimization;

[0037] Questions other than restricted questions: open-ended questions, which usually refer to tasks with many possible answers and require a high degree of creativity, reasoning, or comprehensive analytical skills. Examples include text summarization and question answering.

[0038] Restricted questions: These are closed-ended questions, typically tasks where the answers are relatively fixed or unique, such as classification and entity recognition. Semi-open questions, which are between open-ended and closed-ended, typically refer to tasks where the answers may vary to some extent but are relatively fixed or adhere to certain rules, such as translation.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] Example 1

[0041] According to an embodiment of the present application, a model training method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a model training method. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, and the processor set 102 may include a processor set, Figure 1102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." This data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be fully or partially integrated into any of the other components of the computer terminal 10 (or mobile device). As discussed in the embodiments of this application, this data processing circuitry functions as a processor control (e.g., selecting a variable resistor terminal path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned model training method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0045] Transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of computer terminal 10. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0046] The display may be a touch screen liquid crystal display that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0047] Under the above operating environment, this application provides Figure 2 The model training method shown. Figure 2 This is the process of the model training method according to the first embodiment of the present application Figure 1 The model training method includes:

[0048] Step S201: Acquire a training sample set, wherein the training sample set includes at least: a plurality of training samples, wherein the training samples in the plurality of training samples include a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer and the second answer information is a non-recommended answer;

[0049] Step S202: If the sample question is a restrictive question, the initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model.

[0050] Step S203: If the question type of the sample question is a question other than a restrictive question, the initial natural language model is trained according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model.

[0051] Alternatively, the Determined Proof of Score (DPO) algorithm has received widespread attention and adoption in the prior art due to its low training cost. However, DPO exhibits an intuitive paradox during backpropagation. When calculating gradients, DPO uses the same method for both positive and negative examples. The only difference is that parameters for positive examples are updated along the gradient direction, while parameters for negative examples are updated in the opposite direction. It is known that for a piece of data y, as the probability of model generation increases, the corresponding gradient for that data decreases until it approaches 0. It is found in the gradient of DPO that for positive examples, when the generation probability increases, the gradient will become smaller. When the generation probability remains unchanged, it means that its impact on model updates is constantly decreasing, which is in line with common sense; but for negative examples, when the generation probability decreases, this is first of all beneficial to the model, but the gradient is constantly increasing. The method of updating the model in the opposite direction for negative examples in DPO has certain problems, especially for some non-open and semi-open tasks. It is often the case that the model is updated normally in the early stage of training (the generation probability of positive examples continues to increase, and the generation probability of negative examples continues to decrease), but in the later stage of training, when the negative example gradient drops to a certain level, the first situation occurs, where the generation probability of positive and negative examples decreases at the same time, causing the model to quickly deviate before reaching the theoretically optimal solution.

[0052] In order to solve the above problems, in the model training method provided in Example 1 of the present application, the above-mentioned training sample set is first constructed based on the natural language model of the required field. It should be noted that the training sample set includes at least: multiple training samples, and the training samples in the multiple training samples include sample questions. The sample questions include corresponding first answer information and second answer information. The first answer information is chosen (selected response), that is, the first answer information is the information expected to be output by the model, that is, the recommended answer, and the second answer information is rejected (rejected response). That is, the second answer information is information that the expected model will not output, that is, a non-recommended answer.

[0053] After obtaining the sample questions, the question types of the sample questions can be identified. For example, the sample questions can be directly classified according to their meaning to obtain the corresponding question types. It should be noted that question types can be divided into open questions (i.e., questions other than the restrictive questions mentioned above) and non-(semi-) open questions (i.e., the restrictive questions mentioned above). These questions have a unique answer or questions whose answers may contain a certain degree of variation, but the answers are relatively fixed or follow certain rules and cannot diverge arbitrarily.

[0054] The initial natural language model is trained based on the loss function and training sample set corresponding to the question type of different sample questions to obtain the target natural language model. For example, for open-ended questions, a binary cross-entropy loss function can be selected. This loss function increases the probability of the model generating a preferred response and reduces the probability of generating a non-preferred response. For non-(semi-) open-ended questions, a binary cross-entropy loss function can also be selected, but this loss function needs to be set to increase the probability of the model generating a preferred response and increase the probability of the opposite event of generating a non-preferred response.

[0055] In summary, the method of obtaining a training sample set and training the initial natural language model based on the loss function corresponding to the problem type of the sample question and the training sample set is adopted. The purpose of targeted training of the natural language model is achieved through the loss function corresponding to the problem type of the sample question, and the technical problem of using the same loss function for different types of sample questions when training the natural language model through DPO, resulting in relatively low accuracy of the natural language model in answering questions, is solved. Different loss functions are selected according to different types of sample questions to train the natural language model, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model in answering questions.

[0056] In order to improve the effectiveness of training, in the model training method provided in Example 1 of the present application, the initial natural language model is trained based on the loss function and training sample set corresponding to the question type of the sample question to obtain the target natural language model, including: identifying the question type of the sample question to obtain the question type, wherein the question type is one of the following: restrictive questions and questions other than restrictive questions; training the initial natural language model based on the loss function and training sample set corresponding to the restrictive questions to obtain the target natural language model, or training the initial natural language model based on the loss function and training sample set corresponding to questions other than restrictive questions to obtain the target natural language model.

[0057] For non-(semi-) open-ended questions, in the model training method provided in Example 1 of the present application, the initial natural language model is trained based on the loss function and training sample set corresponding to the restrictive questions to obtain the target natural language model, including: answering the sample questions in the training sample set through the initial natural language model to obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information; calculating based on the fifth generation probability to obtain the sixth generation probability, wherein the sixth generation probability is the probability of occurrence of the opposite event of the event corresponding to the fifth generation probability; calculating based on the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive questions, and training the initial natural language model based on the loss function corresponding to questions other than the restrictive questions to obtain the target natural language model.

[0058] Optionally, for non-(semi-) open-ended questions, the sample questions in the training sample set and the corresponding first answer information and second answer information are input into the initial natural language model, and the sample questions in the training sample set are answered by the initial natural language model to obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information. For example, represents the output probability distribution of the initial natural language model, Indicates the first answer information, Indicates the second answer information, represents the fourth generation probability, Represents the fifth generation probability.

[0059] In order to stabilize the model training, it is usually hoped that the weight change of the natural language model is smooth when approaching a better solution. The characteristic of is that the gradient changes explosively when it approaches 0 and changes slowly when it approaches 1. This characteristic makes the function more suitable for maximization rather than minimization. Considering that minimizing the probability of negative examples is equivalent to maximizing the probability of the opposite event, the sixth generation probability is calculated based on the fifth generation probability. The sixth generation probability is the probability of the opposite event corresponding to the fifth generation probability. For example, the sixth generation probability is .

[0060] Finally, the loss function corresponding to the problem other than the restrictive problem is calculated based on the fourth generation probability and the sixth generation probability, that is, the loss function is a loss function that increases the probability of the model generating a preferred response and increases the probability of generating an event that is not a preferred response. For example, the loss function is

[0061] ,in, Denotes variance, D is the above training sample set, and x is the sample problem. is a sigmoid function that maps log probability to the interval (0, 1). The above loss function is used to train the natural language model, making the training of the natural language model more effective.

[0062] For non-(semi-) open-ended questions, by increasing the probability of the model generating a preferred response and increasing the loss function of the probability of generating a non-preferred response, the relative size of the gradient can increase when the probability of generating a non-preferred response increases, and when the probability of generating a non-preferred response decreases, the relative size of the gradient can decrease, thereby achieving the purpose of improving the training effect of the natural language model.

[0063] In order to further improve the accuracy of the model's answers to questions, in the model training method provided in Example 1 of the present application, calculations are performed based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: answering the sample questions in the training sample set through the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; calculating based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; calculating based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0064] Optionally, calculating the loss function corresponding to questions other than restrictive questions based on the fourth generation probability and the sixth generation probability includes the following steps: answering sample questions in the training sample set through the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information, for example, Generate probabilities for the first reference, The second reference generation probability is calculated based on the second reference generation probability to obtain the third reference generation probability. For example, the third reference generation probability is Finally, the loss function corresponding to the restrictive problem is obtained by calculating the first reference generation probability, the third reference generation probability, the fourth generation probability, and the sixth generation probability.

[0065] In summary, when updating the model parameters according to the fourth generation probability and the sixth generation probability, the first reference generation probability and the third reference generation probability can impose some constraints on the update of the model parameters, thereby avoiding large changes in the model parameters that affect the generalization ability of the model.

[0066] In the model training method provided in Example 1 of the present application, calculations are performed based on the first reference generation probability, the third reference generation probability, the fourth generation probability, and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; obtaining a weight value and a preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor, and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive question.

[0067] Optionally, the following steps are used to calculate the loss function corresponding to the problem other than the restrictive problem: calculate according to the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio, for example, the first ratio is . Calculate the ratio of the third reference generation probability to the sixth generation probability to obtain a second ratio, for example, the second ratio is . In order to better control the role of the second answer information (i.e., negative example) in the model update process, a corresponding weight value is set for the second ratio, and a preset scaling factor is obtained. In order to increase the training speed of the model and better reduce the probability of generating negative examples, the weight value is generally set to be greater than 1. Finally, based on the preset scaling factor and the first ratio, as well as the weight value, the preset scaling factor, and the second ratio, the loss function corresponding to the above-mentioned problem other than the restrictive problem is obtained.

[0068] For example, the loss function for problems other than the restrictive problem is ,in, is the preset scaling factor, is the weight value.

[0069] From the above loss function expression, we can see that the loss function is mainly affected by The impact of the loss function is eliminated, and the problem of the model generating both positive and negative examples with a very low probability in order to achieve lower loss is avoided. Through the above loss function, when the probability of generating negative examples increases, the relative size of the gradient increases, and when the probability of generating negative examples decreases, the relative size of the gradient decreases. When it is unfavorable to the model (the probability of generating negative examples is high), the relative size of the negative gradient increases. When the model is well fitted (the probability of generating negative examples decreases), the relative size of the negative gradient gradually approaches 0, thereby achieving the effect of improving the accuracy of the target natural language model's response to questions.

[0070] How to train the initial natural language model based on the loss function and training sample set corresponding to questions other than restrictive questions is also crucial. Therefore, in the model training method provided in Example 1 of the present application, obtaining the target natural language model includes: answering the sample questions in the training sample set through the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; calculating based on the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; normalizing the first generation probability to obtain a processed first generation probability, normalizing the third generation probability to obtain a processed third generation probability; calculating based on the processed first generation probability and the processed third generation probability to obtain a loss function corresponding to questions other than restrictive questions, and training the initial natural language model based on the loss function corresponding to questions other than restrictive questions to obtain a target natural language model.

[0071] Optionally, normalizing the first generation probability to obtain the processed first generation probability includes: determining the total number of characters corresponding to the first answer information; and normalizing the first generation probability based on the total number of characters to obtain the processed first generation probability.

[0072] Optionally, the initial natural language model is used to answer the sample questions in the training sample set to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information. For example, the first generation probability is , the second generation probability is Similarly, in order to avoid the problem that the relative size of the gradient will become lower when the generation probability of the unpreferred response increases, the third generation probability is calculated based on the second generation probability. It should be noted that the third generation probability is the probability of the opposite event to the event corresponding to the second generation probability. For example, the third generation probability is .

[0073] Since the probability of generating open tasks is generally low, This may result in information being lost, for example When the loss function is reduced from 10e-10 to 10e-20, it is basically imperceptible. To this end, we can normalize the probabilities to expand the model's perception of changes in generation probabilities in open-ended tasks. That is, we normalize the first generation probability to obtain the processed first generation probability, and normalize the third generation probability to obtain the processed third generation probability.

[0074] In an optional embodiment, the normalization of the first generation probability comprises determining a total number of characters corresponding to the first answer information, and then normalizing the first generation probability according to the total number of characters to obtain a processed first generation probability, for example, the normalization of the first generation probability is to , wherein, represents the number of words of the answer information.

[0075] When updating the model parameters, in order to constrain the updating process of the model parameters, the initial natural language model can be used to reply to the sample questions in the training sample set to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information, for example, is the first reference generation probability, is the second reference generation probability, and a third reference generation probability is calculated according to the second reference generation probability, for example, the third reference generation probability is .

[0076] Then, the processed first generation probability and the processed third generation probability, and the first reference generation probability and the second reference generation probability are calculated to obtain a loss function corresponding to the restrictive question, for example, the loss function corresponding to the restrictive question is

[0077] , wherein, represents the total number of characters of the first answer information, represents the total number of characters of the second answer information. The initial generation probability (i.e., the first reference generation probability and the second reference generation probability described above) output by the initial natural language model can constrain the gradient when calculating the gradient of the model update, and avoid over updating the parameters of the model.

[0078] Finally, the initial natural language model is trained according to the loss function corresponding to the question other than the restrictive question to obtain a target natural language model. Through the loss function described above, the generation probability can be perceived, and the effectiveness of the model training can be improved.

[0079] In order to improve the efficiency of obtaining the training sample set, in the model training method provided in Embodiment One of the present application, obtaining the training sample set comprises: obtaining a plurality of sample questions; replying to the plurality of sample questions a predetermined number of times by a first large language model to obtain a plurality of answer information corresponding to the sample questions in the plurality of sample questions; and obtaining the training sample set according to the scoring results corresponding to the answer information in the plurality of answer information.

[0080] Optionally, a plurality of sample questions can be obtained according to requirements, and then the plurality of sample questions are input to the first large language model, and a predetermined number of answers to the plurality of sample questions are obtained by the first large language model, for example, 10 answers to the sample questions are obtained by the first large language model, and 10 corresponding answer information is obtained. Then, the first answer information and the second answer information are determined in the answer information, and finally the training sample set is constructed according to the plurality of sample questions and the first answer information and the second answer information.

[0081] The training sample set can be accurately and quickly obtained by the first large language model.

[0082] In order to improve the accuracy of determining the first answer information and the second answer information, in the model training method provided in Embodiment One of the present application, the training sample set is obtained according to the score results of the plurality of sample questions and the answer information in the plurality of answer information, including: scoring the plurality of answer information corresponding to the target sample question by the second large language model to obtain the score result corresponding to the answer information in the plurality of answer information, wherein the target sample question is any one of the plurality of sample questions; obtaining the target generation probability corresponding to the answer information in the plurality of answer information; determining the first answer information and the second answer information from the plurality of answer information according to the score result and the target generation probability; and obtaining the training sample set according to the plurality of sample questions, the first answer information and the second answer information.

[0083] Optionally, the second large language model is used to score the plurality of answer information corresponding to the target sample question, that is, the second large language model scores the answer information according to whether the answer information can answer the corresponding target sample question to obtain the score result corresponding to the answer information. Then the target generation probability corresponding to the answer information obtained by the first large language model to answer the sample question is obtained, and the first answer information and the second answer information are determined from the plurality of answer information according to the score result and the target generation probability. For example, the plurality of answer information can be sorted according to the score result and the target generation probability, and finally the first answer information and the second answer information are determined according to the sorting result. Finally, the training sample set is obtained according to the plurality of sample questions, the first answer information and the second answer information.

[0084] The reliability of the answer information can be accurately evaluated by the score result and the target generation probability, thereby improving the accuracy of determining the first answer information and the second answer information.

[0085] In an optional embodiment, as Figure 3The flowchart shown implements the training of the natural language model, which specifically includes the following steps: constructing a training sample set based on sample questions and the positive examples (i.e., the first answer information mentioned above) and negative examples (i.e., the second answer information mentioned above) corresponding to the sample questions, determining the loss function corresponding to open questions and the loss function corresponding to non-(semi-) open questions. For example, the loss function corresponding to open questions is , the loss function corresponding to the non-(semi-) open problem is The target natural language model is obtained by training the natural language model using the loss function corresponding to the restrictive question and the loss function corresponding to the non-(semi-)restrictive question as well as the training sample set.

[0086] In the model training method provided in Example 1 of the present application, a training sample set is obtained, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, the initial natural language model is trained according to the loss function and training sample set corresponding to the restrictive question to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, the initial natural language model is trained according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model, which solves the technical problem that the same loss function is used for different types of sample questions when training the natural language model through DPO, resulting in relatively low accuracy of the natural language model in answering questions. Different loss functions are selected according to different types of sample questions to train the natural language model, which can ensure the rationality of the gradient during the model training process, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model in answering questions.

[0087] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0088] Through the description of the foregoing embodiments, those skilled in the art can clearly understand that the method according to the foregoing embodiments can be implemented by means of software in addition to a necessary universal hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disc), and includes a plurality of instructions to cause a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method of each embodiment of the present application.

[0089] Embodiment 2

[0090] According to the embodiments of the present application, a data processing method is also provided, Figure 4 is a flowchart of the data processing method according to Embodiment Two of the present application. The data processing method comprises:

[0091] Step S401: obtaining a target question input by a target object;

[0092] Step S402: replying to the target question by a target natural language model to obtain target reply information, wherein the target natural language model is trained by any one of the model training methods described above;

[0093] Step S403: returning the target reply information to the target object.

[0094] It should be noted that, for each of the foregoing method embodiments, in order to simply describe, each is described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0095] Through the description of the foregoing embodiments, those skilled in the art can clearly understand that the method according to the foregoing embodiments can be implemented by means of software in addition to a necessary universal hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disc), and includes a plurality of instructions to cause a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method of each embodiment of the present application.

[0096] Example 3

[0097] According to an embodiment of the present application, a model training device for implementing the above-mentioned model training method is also provided. Figure 5 As shown, the apparatus includes: an acquisition unit 501 , a first training unit 502 and a second training unit 503 .

[0098] An acquiring unit 501 is configured to acquire a training sample set, wherein the training sample set includes at least: a plurality of training samples, wherein the training sample in the plurality of training samples includes a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer and the second answer information is a non-recommended answer;

[0099] A first training unit 502 is configured to train the initial natural language model according to a loss function corresponding to the restrictive question and a training sample set to obtain a target natural language model if the question type of the sample question is a restrictive question;

[0100] The second training unit 503 is used to train the initial natural language model according to the loss function and training sample set corresponding to the questions other than restrictive questions to obtain the target natural language model if the question type of the sample question is a question other than restrictive questions.

[0101] In the model training device provided in Example 3 of the present application, a training sample set is obtained by an acquisition unit 501, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, the first training unit 502 trains the initial natural language model according to the loss function and training sample set corresponding to the restrictive question to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, the second training unit 503 trains the initial natural language model according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model, which solves the technical problem that the same loss function is used for different types of sample questions when training the natural language model through DPO, resulting in relatively low accuracy of the natural language model in answering questions. Different loss functions are selected according to different types of sample questions to train the natural language model, which can ensure the rationality of the gradient during the model training process, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model in answering questions.

[0102] Optionally, in the model training device provided in Example 3 of the present application, the first training unit includes: a first answer module, which is used to answer sample questions in the training sample set through the initial natural language model, and obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; a first generation module, which is used to calculate based on the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; a first processing module, which is used to normalize the first generation probability to obtain the processed first generation probability, and normalize the third generation probability to obtain the processed third generation probability; a first calculation module, which is used to calculate based on the processed first generation probability and the processed third generation probability to obtain a loss function corresponding to questions other than restrictive questions, and train the initial natural language model based on the loss function corresponding to questions other than restrictive questions to obtain a target natural language model.

[0103] Optionally, in the model training device provided in Example 3 of the present application, the first processing module includes: a determination submodule, used to determine the total number of characters corresponding to the first answer information; and a processing submodule, used to normalize the first generation probability based on the total number of characters to obtain the processed first generation probability.

[0104] Optionally, in the model training device provided in Example 3 of the present application, the second training unit includes: a second answer module, used to answer sample questions in the training sample set through the initial natural language model, and obtain a fourth generation probability corresponding to the first answer information and a fifth generation probability corresponding to the second answer information; a second calculation module, used to calculate based on the fifth generation probability to obtain a sixth generation probability, wherein the sixth generation probability is the probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; a third calculation module, used to calculate based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, and train the initial natural language model based on the loss function corresponding to the restrictive question to obtain a target natural language model.

[0105] Optionally, in the model training device provided in Example 3 of the present application, the third calculation module includes: a reply submodule, which is used to reply to the sample questions in the training sample set through the initial natural language model, and obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; a first calculation submodule, which is used to perform calculations based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; and a second calculation submodule, which is used to perform calculations based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0106] Optionally, in the model training device provided in Example 3 of the present application, the second calculation sub-module includes: a first calculation sub-module, used to calculate the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; a second calculation sub-module, used to calculate the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; an acquisition sub-module, used to obtain the weight value and the preset scaling factor corresponding to the second answer information; a third calculation sub-module, used to calculate based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculate based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; a fourth calculation sub-module, used to calculate based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive problem.

[0107] Optionally, in the model training device provided in Example 3 of the present application, the acquisition unit includes: an acquisition sub-unit, used to acquire multiple sample questions; a reply sub-unit, used to reply to the multiple sample questions a preset number of times through the first large language model, and obtain multiple answer information corresponding to the sample questions in the multiple sample questions; a processing sub-unit, used to obtain a training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0108] Optionally, in the model training device provided in Example 3 of the present application, the processing subunit includes: a scoring module, which is used to score multiple answer information corresponding to the target sample question through the second largest language model, and obtain a scoring result corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; an acquisition module, which is used to obtain the target generation probability corresponding to the answer information in the multiple answer information; a determination module, which is used to determine the first answer information and the second answer information from the multiple answer information based on the scoring result and the target generation probability; a second processing module, which is used to obtain a training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0109] It should be noted that the acquisition unit 501, the first training unit 502, and the second training unit 503 described above correspond to steps S201 to S203 in Example 1. The examples and application scenarios implemented by the two units and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0110] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0111] Embodiment 4

[0112] The embodiments of the present application can provide an electronic device, which can be any one of electronic devices in an electronic device terminal group. Alternatively, in the embodiments, the electronic device can also be replaced by a terminal device such as a mobile terminal.

[0113] Alternatively, in the embodiments, the electronic device can be located in at least one network device of a plurality of network devices of a computer network.

[0114] In the embodiments, the computer terminal can execute program codes of the following steps in the model training method and the data processing method provided by the embodiments: obtaining a training sample set, wherein the training sample set at least includes a plurality of training samples, and each training sample in the plurality of training samples includes a sample question, first answer information corresponding to the sample question, and second answer information corresponding to the sample question, the first answer information being a recommended answer, and the second answer information being a non-recommended answer; if the sample question is a restrictive question, training an initial natural language model according to a loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model; and if the sample question is a question other than the restrictive question, training the initial natural language model according to a loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model.

[0115] The computer terminal can execute program codes of the following steps in the model training method and the data processing method provided by the embodiments: training the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model, including: replying to the sample question in the training sample set by the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; calculating according to the second generation probability to obtain a third generation probability, wherein the third generation probability is a probability of an opposite event of an event corresponding to the second generation probability; normalizing the first generation probability to obtain a processed first generation probability, and normalizing the third generation probability to obtain a processed third generation probability; calculating according to the processed first generation probability and the processed third generation probability to obtain the loss function corresponding to the question other than the restrictive question, and training the initial natural language model according to the loss function corresponding to the question other than the restrictive question to obtain the target natural language model.

[0116] The computer terminal can execute program codes of the following steps in the model training method and the data processing method provided by the above embodiments: performing normalization processing on the first generation probability to obtain the processed first generation probability, including: determining the total number of characters corresponding to the first answer information; and performing normalization processing on the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0117] The computer terminal can execute program codes of the following steps in the model training method and the data processing method provided by the above embodiments: training the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model, including: replying to the sample question in the training sample set through the initial natural language model to obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information; calculating the sixth generation probability according to the fifth generation probability, wherein the sixth generation probability is the occurrence probability of the opposite event of the event corresponding to the fifth generation probability; calculating the loss function corresponding to the restrictive question according to the fourth generation probability and the sixth generation probability, and training the initial natural language model according to the loss function corresponding to the restrictive question to obtain the target natural language model.

[0118] The computer terminal can execute program codes of the following steps in the model training method and the data processing method provided by the above embodiments: calculating the loss function corresponding to the restrictive question according to the fourth generation probability and the sixth generation probability, including: replying to the sample question in the training sample set through the initial natural language model to obtain the first reference generation probability corresponding to the first answer information and the second reference generation probability corresponding to the second answer information; calculating the third reference generation probability according to the second reference generation probability, wherein the third reference generation probability is the occurrence probability of the opposite event of the event corresponding to the second reference generation probability; and calculating the loss function corresponding to the restrictive question according to the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability.

[0119] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiment: calculating based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; obtaining the weight value and the preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive question.

[0120] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiment: Obtaining a training sample set includes: obtaining multiple sample questions; answering the multiple sample questions a preset number of times through the first large language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; obtaining a training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0121] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiment: obtaining a training sample set based on the scoring results corresponding to the answer information in the multiple sample questions and the multiple answer information, including: scoring the multiple answer information corresponding to the target sample question through the second largest language model to obtain the scoring results corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; obtaining the target generation probability corresponding to the answer information in the multiple answer information; determining the first answer information and the second answer information from the multiple answer information based on the scoring results and the target generation probability; obtaining the training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0122] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiments: obtaining the target question input by the target object; answering the target question through the target natural language model to obtain target answer information, wherein the target natural language model is trained using any of the above-mentioned model training methods; and returning the target answer information to the target object.

[0123] Optionally, Figure 6 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 6 As shown, the electronic device 60 may include: one or more ( Figure 6(only one is shown) processor 602, memory 604. The electronic device 60 may further include a memory controller to control and manage the memory 604; the electronic device 60 may further include a peripheral interface to connect to a radio frequency module, an audio module, and a display screen, etc.

[0124] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method and data processing method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned model training method and data processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the electronic device 60 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0125] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a training sample set, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, then the initial natural language model is trained according to the loss function and training sample set corresponding to the restrictive question to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, then the initial natural language model is trained according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model.

[0126] Optionally, the processor can further execute program codes of the following steps: training the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model, including: obtaining a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information by replying to the sample question in the training sample set by the initial natural language model; obtaining a third generation probability by calculating according to the second generation probability, wherein the third generation probability is the occurrence probability of the opposite event of the event corresponding to the second generation probability; performing normalization processing on the first generation probability to obtain a processed first generation probability, and performing normalization processing on the third generation probability to obtain a processed third generation probability; obtaining the loss function corresponding to the question other than the restrictive question by calculating according to the processed first generation probability and the processed third generation probability, and training the initial natural language model according to the loss function corresponding to the question other than the restrictive question to obtain the target natural language model.

[0127] Optionally, the processor can further execute program codes of the following steps: performing normalization processing on the first generation probability to obtain a processed first generation probability, including: determining the total number of characters corresponding to the first answer information; performing normalization processing on the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0128] Optionally, the processor can further execute program codes of the following steps: training the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model, including: obtaining a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information by replying to the sample question in the training sample set by the initial natural language model; obtaining a third generation probability by calculating according to the second generation probability, wherein the third generation probability is the occurrence probability of the opposite event of the event corresponding to the second generation probability; obtaining the loss function corresponding to the question other than the restrictive question by calculating according to the fourth generation probability and the sixth generation probability, and training the initial natural language model according to the loss function corresponding to the restrictive question to obtain the target natural language model.

[0129] Optionally, the processor may also execute the program code of the following steps: performing calculations based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: answering the sample questions in the training sample set through the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; performing calculations based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; performing calculations based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0130] Optionally, the processor may also execute the following program code steps: performing calculations based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; obtaining a weight value and a preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain a loss function corresponding to the restrictive question.

[0131] Optionally, the processor may also execute the program code of the following steps: obtaining a training sample set includes: obtaining multiple sample questions; answering the multiple sample questions a preset number of times through the first language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; obtaining a training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0132] Optionally, the processor may also execute the program code of the following steps: obtaining a training sample set based on scoring results corresponding to multiple sample questions and multiple answer information, including: scoring multiple answer information corresponding to the target sample question through the second largest language model to obtain scoring results corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; obtaining a target generation probability corresponding to the answer information in the multiple answer information; determining the first answer information and the second answer information from the multiple answer information based on the scoring results and the target generation probability; obtaining a training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0133] Optionally, the above-mentioned processor can also execute the program code of the following steps: obtaining the target question input by the target object; answering the target question through the target natural language model to obtain target answer information, wherein the target natural language model is trained using any of the above-mentioned model training methods; and returning the target answer information to the target object.

[0134] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the electronic device 60 may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), or a PAD. Figure 6 It does not limit the structure of the above electronic device. For example, the electronic device 60 may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 6 Different configurations shown.

[0135] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0136] Example 5

[0137] An embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the above-mentioned computer program product can be used to store the program code executed by the model training method and data processing method provided in the above-mentioned embodiment 1.

[0138] Example 6

[0139] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product can be used to store the program code executed by the model training method and data processing method provided in the first embodiment.

[0140] Optionally, in this embodiment, the computer program product may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0141] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0142] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0144] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0146] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program code.

[0147] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: Obtaining a training sample set, wherein the training sample set includes at least: a plurality of training samples, wherein the training samples in the plurality of training samples include a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer and the second answer information is a non-recommended answer; If the question type of the sample question is a restrictive question, the initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model; If the question type of the sample question is a question other than a restrictive question, training the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model; The initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model, including: answering the sample questions in the training sample set by using the initial natural language model to obtain a fourth generation probability corresponding to the first answer information and a fifth generation probability corresponding to the second answer information; Calculating based on the fifth generation probability to obtain a sixth generation probability, wherein the sixth generation probability is a probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; Calculating based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, and training the initial natural language model based on the loss function corresponding to the restrictive question to obtain the target natural language model; The loss function corresponding to the restrictive problem obtained by calculating according to the fourth generation probability and the sixth generation probability includes: Responding to sample questions in the training sample set using the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; Calculating based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is a probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; Calculating based on the first reference generation probability, the third reference generation probability, the fourth generation probability, and the sixth generation probability to obtain a loss function corresponding to the restrictive problem; The first reference generation probability and the second reference generation probability are used to generate constraints on the update of model parameters.

2. The method according to claim 1, characterized in that Training the initial natural language model according to the loss function corresponding to the questions other than the restrictive questions and the training sample set to obtain the target natural language model includes: Answering the sample questions in the training sample set using the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; Calculating based on the second generation probability to obtain a third generation probability, wherein the third generation probability is a probability of occurrence of an event opposite to the event corresponding to the second generation probability; Normalizing the first generation probability to obtain a processed first generation probability, and normalizing the third generation probability to obtain a processed third generation probability; Calculation is performed based on the first generation probability after processing and the third generation probability after processing to obtain loss functions corresponding to questions other than restrictive questions, and the initial natural language model is trained based on the loss functions corresponding to the restrictive questions to obtain the target natural language model.

3. The method according to claim 2, characterized in that Normalizing the first generation probability to obtain the processed first generation probability includes: Determining the total number of characters corresponding to the first answer information; The first generation probability is normalized according to the total number of characters to obtain the processed first generation probability.

4. The method according to claim 1, wherein Calculating based on the first reference generation probability, the third reference generation probability, the fourth generation probability, and the sixth generation probability, the loss function corresponding to the restrictive problem includes: Calculating a ratio of the first reference generation probability to the fourth generation probability to obtain a first ratio; Calculating a ratio of the third reference generation probability to the sixth generation probability to obtain a second ratio; Obtaining a weight value and a preset scaling factor corresponding to the second answer information; Calculating based on the preset scaling factor and the first ratio to obtain a first value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second value; Calculation is performed based on the first value and the second value to obtain a loss function corresponding to the restrictive problem.

5. The method according to claim 1, wherein Obtaining a training sample set includes: Get multiple sample questions; Responding to the plurality of sample questions a preset number of times using the first large language model to obtain a plurality of answer information corresponding to the sample questions among the plurality of sample questions; The training sample set is obtained according to the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

6. The method according to claim 5, characterized in that The training sample set is obtained based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information, including: Scoring multiple answer information corresponding to a target sample question using a second language model to obtain a scoring result corresponding to an answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; Obtaining a target generation probability corresponding to an answer information in the plurality of answer information; determining the first answer information and the second answer information from the plurality of answer information according to the scoring result and the target generation probability; The training sample set is obtained according to the multiple sample questions, the first answer information and the second answer information.

7. A data processing method, characterized in that: include: Get the target question input by the target object; Responding to the target question through a target natural language model to obtain target response information, wherein the target natural language model is trained using the model training method according to any one of claims 1 to 6; Return the target reply information to the target object.

8. A model training device, characterized in that: include: an acquiring unit, configured to acquire a training sample set, wherein the training sample set includes at least: a plurality of training samples, wherein a training sample in the plurality of training samples includes a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer and the second answer information is a non-recommended answer; A first training unit is configured to, if the question type of the sample question is a restrictive question, train the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model; a second training unit configured to, if the question type of the sample question is a question other than a restrictive question, train the initial natural language model according to a loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model; Among them, the second training unit includes: a second answer module, which is used to answer the sample questions in the training sample set through the initial natural language model, and obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information; a second calculation module, which is used to calculate based on the fifth generation probability to obtain a sixth generation probability, wherein the sixth generation probability is the probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; a third calculation module, which is used to calculate based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to a restrictive question, and train the initial natural language model based on the loss function corresponding to the restrictive question to obtain the target natural language model; Among them, the third calculation module includes: a reply submodule, which is used to reply to the sample questions in the training sample set through the initial natural language model, and obtain the first reference generation probability corresponding to the first answer information and the second reference generation probability corresponding to the second answer information; a first calculation submodule, which is used to calculate based on the second reference generation probability to obtain the third reference generation probability, wherein the third reference generation probability is the probability of occurrence of the opposite event of the event corresponding to the second reference generation probability; a second calculation submodule, which is used to calculate based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question, wherein the first reference generation probability and the second reference generation probability are used to generate constraints on the update of model parameters.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein, when the program is running, the device where the storage medium is located is controlled to execute the model training method described in any one of claims 1 to 6, or the data processing method described in claim 7.

10. An electronic device, characterized in that: include: a memory storing an executable program; A processor for running the program, wherein the program, when running, executes the model training method described in any one of claims 1 to 6, or the data processing method described in claim 7.

11. A computer program product, characterized in that The computer program product includes a stored computer program, and when the computer program is executed by a processor, it implements the model training method described in any one of claims 1 to 6, or the data processing method described in claim 7.

Images