Model training method and data processing method

Abstract

The invention discloses a model training method and a data processing method, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a training sample set; if the problem type of the sample problem is a restrictive problem, training the initial natural language model according to a loss function corresponding to the restrictive problem and the training sample set to obtain a target natural language model; and if the problem type of the sample problem is the problem except the restrictive problem, training the initial natural language model according to a loss function corresponding to the problem except the restrictive problem and the training sample set to obtain a target natural language model. The technical problem that when a natural language model is trained through DPO, the same loss function is adopted for different types of sample problems, so that the accuracy of question answering of the natural language model is low is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a model training method and a data processing method. Background Art

[0002] With the rapid development of artificial intelligence technology, reinforcement learning has been widely used in many task scenarios. Among many reinforcement learning algorithms, DPO (Direct Preference Optimization) has received widespread attention and use due to its low training cost. However, from the perspective of actual tasks, DPO uses the same loss function for different types of sample problems when training natural language models. Therefore, when faced with non-(semi-) open tasks, its performance is not satisfactory. Especially when the model reaches a better solution, the effect will drop rapidly.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a model training method and a data processing method to at least solve the technical problem that the same loss function is used for different types of sample questions when training a natural language model through DPO, resulting in relatively low accuracy of the natural language model's answers to questions.

[0005] According to one aspect of an embodiment of the present application, a model training method is provided, comprising: obtaining a training sample set, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include a sample question, first answer information and second answer information corresponding to the sample question, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, then the initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, then the initial natural language model is trained according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model.

[0006] Optionally, the initial natural language model is trained according to the loss function corresponding to the questions other than restrictive questions and the training sample set to obtain the target natural language model, including: answering the sample questions in the training sample set through the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; calculating according to the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; normalizing the first generation probability to obtain a processed first generation probability, and normalizing the third generation probability to obtain a processed third generation probability; calculating according to the processed first generation probability and the processed third generation probability to obtain a loss function corresponding to the questions other than restrictive questions, and training the initial natural language model according to the loss function corresponding to the questions other than restrictive questions to obtain the target natural language model.

[0007] Optionally, normalizing the first generation probability to obtain the processed first generation probability includes: determining the total number of characters corresponding to the first answer information; and normalizing the first generation probability based on the total number of characters to obtain the processed first generation probability.

[0008] Optionally, the initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model, including: answering the sample question in the training sample set through the initial natural language model to obtain a fourth generation probability corresponding to the first answer information and a fifth generation probability corresponding to the second answer information; calculating according to the fifth generation probability to obtain a sixth generation probability, wherein the sixth generation probability is the probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; calculating according to the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question, and training the initial natural language model according to the loss function corresponding to the restrictive question to obtain the target natural language model.

[0009] Optionally, calculations are performed based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: answering sample questions in the training sample set through the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; calculations are performed based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; calculations are performed based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0010] Optionally, calculations are performed based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability to the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability to the sixth generation probability to obtain a second ratio; obtaining a weight value and a preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive question.

[0011] Optionally, obtaining a training sample set includes: obtaining multiple sample questions; answering the multiple sample questions a preset number of times through a first large language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; and obtaining the training sample set based on scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0012] Optionally, obtaining the training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information includes: scoring the multiple answer information corresponding to the target sample question through the second largest language model to obtain the scoring results corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; obtaining the target generation probability corresponding to the answer information in the multiple answer information; determining the first answer information and the second answer information from the multiple answer information based on the scoring results and the target generation probability; and obtaining the training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0013] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a target question input by a target object; answering the target question through a target natural language model to obtain target answer information, wherein the target natural language model is trained using any one of the model training methods described above; and returning the target answer information to the target object.

[0014] According to another aspect of an embodiment of the present application, a model training device is also provided, including: an acquisition unit, used to acquire a training sample set, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include a sample question, a first answer information and a second answer information corresponding to the sample question, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; a first training unit, used to train the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model if the question type of the sample question is a restrictive question; a second training unit, used to train the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model if the question type of the sample question is a question other than a restrictive question.

[0015] The first training unit includes: a first answering module, which is used to answer the sample questions in the training sample set through the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; a first generation module, which is used to calculate based on the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; a first processing module, which is used to normalize the first generation probability to obtain a processed first generation probability, and normalize the third generation probability to obtain a processed third generation probability; a first calculation module, which is used to calculate based on the processed first generation probability and the processed third generation probability to obtain a loss function corresponding to questions other than restrictive questions, and train the initial natural language model based on the loss function corresponding to questions other than restrictive questions to obtain the target natural language model.

[0016] Optionally, the first processing module includes: a determination submodule, used to determine the total number of characters corresponding to the first answer information; and a processing submodule, used to normalize the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0017] Optionally, the second training unit includes: a second answer module, used to answer sample questions in the training sample set through the initial natural language model, and obtain a fourth generation probability corresponding to the first answer information and a fifth generation probability corresponding to the second answer information; a second calculation module, used to calculate based on the fifth generation probability to obtain a sixth generation probability, wherein the sixth generation probability is the probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; a third calculation module, used to calculate based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to a restrictive question, and train the initial natural language model based on the loss function corresponding to the restrictive question to obtain the target natural language model.

[0018] Optionally, the third calculation module includes: a reply submodule, which is used to reply to the sample questions in the training sample set through the initial natural language model, and obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; a first calculation submodule, which is used to calculate based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; and a second calculation submodule, which is used to calculate based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0019] Optionally, the second calculation sub-module includes: a first calculation sub-module, used to calculate the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; a second calculation sub-module, used to calculate the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; an acquisition sub-module, used to obtain a weight value and a preset scaling factor corresponding to the second answer information; a third calculation sub-module, used to calculate based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculate based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; a fourth calculation sub-module, used to calculate based on the first numerical value and the second numerical value to obtain a loss function corresponding to the restrictive problem.

[0020] Optionally, the acquisition unit includes: an acquisition subunit, used to acquire multiple sample questions; a reply subunit, used to reply to the multiple sample questions a preset number of times through the first large language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; and a processing subunit, used to obtain the training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0021] Optionally, the processing subunit includes: a scoring module, used to score multiple answer information corresponding to the target sample question through the second largest language model, and obtain a scoring result corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; an acquisition module, used to obtain a target generation probability corresponding to the answer information in the multiple answer information; a determination module, used to determine the first answer information and the second answer information from the multiple answer information based on the scoring result and the target generation probability; a second processing module, used to obtain the training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned model training methods, or the above-mentioned data processing method.

[0023] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising a memory storing an executable program; and a processor for running the program, wherein when the program is running, the model training method described in any one of the above, or the data processing method described above is executed.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is also provided, which includes a stored computer program, and when the computer program is executed by a processor, any one of the above-mentioned model training methods or the above-mentioned data processing methods is implemented.

[0025] In an embodiment of the present application, a training sample set is obtained, and an initial natural language model is trained based on a loss function corresponding to the question type of the sample question and the training sample set. The purpose of targeted training of the natural language model is achieved through the loss function corresponding to the question type of the sample question, and the technical problem of using the same loss function for different types of sample questions when training the natural language model through DPO, resulting in relatively low accuracy in answering questions by the natural language model, is solved. Different loss functions are selected according to different types of sample questions to ensure the rationality of the gradient during the model training process, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model in answering questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 is a hardware structure block diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0028] Figure 2 This is the process of the model training method provided in Example 1 of this application Figure 1 ;

[0029] Figure 3 This is the process of the model training method provided in Example 1 of this application Figure 2 ;

[0030] Figure 4 is a flow chart of a data processing method provided according to Embodiment 2 of the present application;

[0031] Figure 5 is a schematic diagram of a model training device provided according to Embodiment 3 of the present application;

[0032] Figure 6 It is a structural block diagram of an electronic device provided according to Embodiment 3 of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0036] DPO: Direct Preference Optimization, direct preference optimization;

[0037] Questions other than restrictive questions: open-ended questions usually refer to tasks that can have many possible answers and require a high degree of creativity, reasoning, or comprehensive analysis skills to complete. For example, text summarization and question answering;

[0038] Restrictive questions: Non-open questions usually refer to tasks whose answers are relatively fixed or unique, such as classification and entity recognition; Semi-open questions are between open and non-open questions, and usually refer to tasks whose answers may contain a certain degree of variation, but are relatively fixed or follow certain rules and cannot diverge arbitrarily, such as translation.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] Example 1

[0041] According to an embodiment of the present application, a model training method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a model training method. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, and the processor set 102 may include a processor set, Figure 1102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned model training method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] The display may be a touch screen type liquid crystal display, which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0047] Under the above operating environment, this application provides Figure 2 The model training method shown. Figure 2 This is the process of the model training method according to the first embodiment of the present application. Figure 1 The model training method includes:

[0048] Step S201, obtaining a training sample set, wherein the training sample set at least includes: a plurality of training samples, wherein the training samples in the plurality of training samples include a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer, and the second answer information is a non-recommended answer;

[0049] Step S202: if the question type of the sample question is a restrictive question, the initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model;

[0050] Step S203: If the question type of the sample question is a question other than a restrictive question, the initial natural language model is trained according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model.

[0051] Alternatively, in the prior art, the DPO algorithm has received widespread attention and use due to its low training cost. However, there is an intuitive paradox in the back-propagation process of DPO. When DPO calculates the gradient, the positive and negative examples are calculated in the same way. The only difference is that the positive examples update the parameters along the gradient direction, while the negative examples are updated in the opposite direction of the gradient. It is known that for a piece of data y, as the probability of model generation increases, the gradient corresponding to this data will continue to decrease until it approaches 0. It is found in the gradient of DPO that for positive examples, when the generation probability increases, the gradient will become smaller. When the generation probability remains unchanged, it means that its impact on model updating is constantly decreasing, which is in line with common sense; but for negative examples, when the generation probability decreases, this is first of all beneficial to the model, but the gradient is constantly increasing. The method of updating the model in the opposite direction for negative examples in DPO has certain problems, especially for some non-open and semi-open tasks. It is often the case that the model is updated normally in the early stage of training (the generation probability of positive examples continues to increase, and the generation probability of negative examples continues to decrease), but in the later stage of training, when the negative example gradient drops to a certain extent, the first situation occurs, the generation probability of positive and negative examples decreases at the same time, which causes the model to quickly deviate before reaching the theoretically optimal solution.

[0052] In order to solve the above problems, in the model training method provided in the first embodiment of the present application, the above training sample set is first constructed according to the natural language model of the required field. It should be noted that the training sample set at least includes: multiple training samples, and the training samples in the multiple training samples include sample questions. The sample questions include corresponding first answer information and second answer information. The first answer information is chosen (selected response), that is, the first answer information is the information expected to be output by the model, that is, the recommended answer, and the second answer information is rejected (rejected response). That is, the second answer information is the information that the expected model will not output, that is, a non-recommended answer.

[0053] After obtaining the sample questions, the question types of the sample questions can be identified. For example, the sample questions can be directly classified according to their meanings to obtain the question types corresponding to the sample questions. It should be noted that the question types can be divided into open questions (i.e., questions other than the restrictive questions mentioned above); non-(semi-) open questions (i.e., the restrictive questions mentioned above), questions with a unique answer or questions whose answers may contain a certain degree of variation, but the answers are relatively fixed or follow certain rules and cannot diverge arbitrarily.

[0054] The initial natural language model is trained according to the loss function and training sample set corresponding to the question type of different sample questions to obtain the target natural language model. For example, for open-ended questions, a binary cross entropy loss function can be selected, which is a loss function that increases the probability of the model generating a preferred response and reduces the probability of generating a non-preferred response. For non-(semi-) open-ended questions, a binary cross entropy loss function can also be selected, but this loss function needs to be set to a loss function that increases the probability of the model generating a preferred response and increases the probability of generating the opposite event of the non-preferred response.

[0055] In summary, the method of obtaining a training sample set and training the initial natural language model according to the loss function corresponding to the problem type of the sample question and the training sample set is adopted. The purpose of targeted training of the natural language model is achieved through the loss function corresponding to the problem type of the sample question, and the technical problem of using the same loss function for different types of sample questions when training the natural language model through DPO, resulting in relatively low accuracy of the natural language model in answering questions, is solved. Different loss functions are selected according to different types of sample questions to train the natural language model, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model in answering questions.

[0056] In order to improve the effectiveness of training, in the model training method provided in Example 1 of the present application, the initial natural language model is trained according to the loss function and training sample set corresponding to the question type of the sample question, and the target natural language model is obtained, including: identifying the question type of the sample question to obtain the question type, wherein the question type is one of the following: restrictive questions and questions other than restrictive questions; training the initial natural language model according to the loss function and training sample set corresponding to the restrictive questions to obtain the target natural language model, or, training the initial natural language model according to the loss function and training sample set corresponding to questions other than restrictive questions to obtain the target natural language model.

[0057] For non-(semi-) open-ended questions, in the model training method provided in Example 1 of the present application, the initial natural language model is trained according to the loss function and training sample set corresponding to the restrictive questions to obtain the target natural language model, including: answering the sample questions in the training sample set by the initial natural language model to obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information; calculating according to the fifth generation probability to obtain the sixth generation probability, wherein the sixth generation probability is the probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; calculating according to the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive questions, and training the initial natural language model according to the loss function corresponding to the questions other than the restrictive questions to obtain the target natural language model.

[0058] Optionally, for non-(semi-) open-ended questions, the sample questions in the training sample set and the corresponding first answer information and second answer information are input into the initial natural language model, and the sample questions in the training sample set are answered by the initial natural language model to obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information. For example, represents the output probability distribution of the initial natural language model, Indicates the first answer information. Indicates the second answer information, represents the fourth generation probability, Represents the fifth generation probability.

[0059] In order to stabilize the model training, it is usually hoped that the weight change of the natural language model is smooth when approaching a better solution. The characteristic of is that the gradient changes explosively when it approaches 0 and changes slowly when it approaches 1. This characteristic makes the function more suitable for maximization rather than minimization. Considering that minimizing the probability of negative examples is equivalent to maximizing the probability of opposing events, the sixth generation probability is calculated based on the fifth generation probability. The sixth generation probability is the probability of the opposite event of the event corresponding to the fifth generation probability. For example, the sixth generation probability is .

[0060] Finally, the loss function corresponding to the problem other than the restrictive problem is calculated based on the fourth generation probability and the sixth generation probability, that is, the loss function is a loss function that increases the probability of the model generating a preferred response and increases the probability of generating an event that is not a preferred response. For example, the loss function is

[0061] ,in, represents the variance, D is the training sample set mentioned above, and x is the sample problem. is a sigmoid function, which is used to map the logarithmic probability to the interval (0, 1). The natural language model is trained by the above loss function, making the training of the natural language model more effective.

[0062] For non-(semi-) open-ended questions, by increasing the probability of the model generating a preferred response and increasing the loss function of the probability of generating a non-preferred response, the relative size of the gradient will increase when the probability of generating a non-preferred response increases, and when the probability of generating a non-preferred response decreases, the relative size of the gradient will decrease, thereby achieving the purpose of improving the training effect of the natural language model.

[0063] In order to further improve the accuracy of the model's answers to questions, in the model training method provided in Example 1 of the present application, calculations are performed based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: answering the sample questions in the training sample set through the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; calculating based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; calculating based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0064] Optionally, the loss function corresponding to the question other than the restrictive question is calculated according to the fourth generation probability and the sixth generation probability, including the following steps: the sample question in the training sample set can be answered by the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information, for example, Generate probabilities for the first reference, is the second reference generation probability, and is calculated based on the second reference generation probability to obtain a third reference generation probability. For example, the third reference generation probability is Finally, the loss function corresponding to the restrictive problem is obtained by calculating the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability.

[0065] In summary, when the model parameters are updated according to the fourth generation probability and the sixth generation probability, the first reference generation probability and the third reference generation probability can impose some constraints on the update of the model parameters, thereby avoiding large changes in the model parameters that affect the generalization ability of the model.

[0066] In the model training method provided in Example 1 of the present application, calculations are performed based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability to the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability to the sixth generation probability to obtain a second ratio; obtaining a weight value and a preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive question.

[0067] Optionally, the loss function corresponding to the problem other than the restrictive problem is calculated by the following steps: a first ratio is obtained by calculating according to the ratio of the first reference generation probability to the fourth generation probability. For example, the first ratio is . Calculate the ratio of the third reference generation probability to the sixth generation probability to obtain a second ratio. For example, the second ratio is . In order to better control the role of the second answer information (i.e., negative example) in the model update process, a corresponding weight value is set for the second ratio, and a preset scaling factor is obtained. In order to increase the training speed of the model and better reduce the probability of generating negative examples, the weight value is generally set to be greater than 1. Finally, according to the preset scaling factor and the first ratio, as well as the weight value, the preset scaling factor and the second ratio, the loss function corresponding to the above-mentioned problem other than the restrictive problem is obtained.

[0068] For example, the loss function for problems other than the restrictive problem is ,in, is the preset scaling factor, is the weight value.

[0069] From the above loss function expression, we can see that the loss function is mainly affected by The impact of will not cause the problem that the probability of the model generating positive and negative examples becomes very low in order to make the loss lower. Through the above loss function, when the probability of generating negative examples increases, the relative size of the gradient will increase, and when the probability of generating negative examples decreases, the relative size of the gradient will decrease. When it is unfavorable to the model (the probability of generating is high), the relative size of the negative example gradient is increased. When the model is fitted (the probability of generating is reduced), the relative size of the negative example gradient gradually tends to 0, thereby achieving the effect of improving the accuracy of the target natural language model in answering questions.

[0070] How to train the initial natural language model according to the loss function and training sample set corresponding to questions other than restrictive questions is also crucial. Therefore, in the model training method provided in Example 1 of the present application, obtaining the target natural language model includes: answering the sample questions in the training sample set through the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; calculating according to the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; normalizing the first generation probability to obtain a processed first generation probability, normalizing the third generation probability to obtain a processed third generation probability; calculating according to the processed first generation probability and the processed third generation probability to obtain a loss function corresponding to questions other than restrictive questions, and training the initial natural language model according to the loss function corresponding to questions other than restrictive questions to obtain a target natural language model.

[0071] Optionally, normalizing the first generation probability to obtain the processed first generation probability includes: determining the total number of characters corresponding to the first answer information; and normalizing the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0072] Optionally, the initial natural language model is used to answer sample questions in the training sample set to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information. For example, the first generation probability is , the second generation probability is Similarly, in order to avoid the problem that the relative size of the gradient will become lower when the generation probability of the unpreferred response increases, the third generation probability is calculated based on the second generation probability. It should be noted that the third generation probability is the probability of the opposite event to the event corresponding to the second generation probability. For example, the third generation probability is .

[0073] Since the probability of generating open tasks is generally low, This may result in information being lost, for example 10e-10 is reduced to 10e-20, and the loss function is basically imperceptible. For this reason, the probability can be normalized to expand the model's perception of the change in generation probability in the open task, that is, the first generation probability is normalized to obtain the processed first generation probability, and the third generation probability is normalized to obtain the processed third generation probability.

[0074] In an optional embodiment, normalizing the first generation probability includes determining the total number of characters corresponding to the first answer information, and then normalizing the first generation probability according to the total number of characters to obtain the processed first generation probability. Normalize to ,in, Indicates the number of words in the answer message.

[0075] When updating the model parameters, in order to generate some constraints on the updating process of the model parameters, the sample questions in the training sample set can be answered by the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information, for example, Generate probabilities for the first reference, is the second reference generation probability, and is calculated based on the second reference generation probability to obtain a third reference generation probability. For example, the third reference generation probability is .

[0076] Then, according to the processed first generation probability and the processed third generation probability, as well as the first reference generation probability and the second reference generation probability, a loss function corresponding to the restrictive problem is obtained. For example, the loss function corresponding to the restrictive problem is:

[0077] ,in, Indicates the total number of characters in the first answer information. The total number of characters representing the second answer information. The initial generation probability output by the initial natural language model (i.e., the first reference generation probability and the second reference generation probability mentioned above) can constrain the gradient when calculating the gradient of the model update, thereby preventing the model from over-updating its own parameters.

[0078] Finally, the initial natural language model is trained according to the loss function corresponding to the problem other than the restrictive problem to obtain the target natural language model. The above loss function can make the generation probability perceived, thereby improving the effectiveness of model training.

[0079] In order to improve the efficiency of obtaining a training sample set, in the model training method provided in the first embodiment of the present application, obtaining a training sample set includes: obtaining a plurality of sample questions; answering the plurality of sample questions a preset number of times through a first large language model to obtain a plurality of answer information corresponding to the sample questions in the plurality of sample questions; and obtaining a training sample set based on scoring results corresponding to the plurality of sample questions and the answer information in the plurality of answer information.

[0080] Optionally, multiple sample questions can be obtained according to demand, and then the multiple sample questions can be input into the first large language model, and the multiple sample questions can be answered a preset number of times by the first large language model. For example, the sample questions can be answered 10 times by the first large language model to obtain 10 corresponding answer information. Then, the first answer information and the second answer information are determined from the answer information, and finally the above-mentioned training sample set is constructed based on the multiple sample questions and the first answer information and the second answer information.

[0081] By answering sample questions through the first language model, a training sample set can be obtained accurately and quickly.

[0082] In order to improve the accuracy of determining the first answer information and the second answer information, in the model training method provided in Example 1 of the present application, a training sample set is obtained based on the scoring results corresponding to the answer information in multiple sample questions and multiple answer information, including: scoring the multiple answer information corresponding to the target sample question through the second largest language model to obtain the scoring results corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; obtaining the target generation probability corresponding to the answer information in the multiple answer information; determining the first answer information and the second answer information from the multiple answer information based on the scoring results and the target generation probability; and obtaining the training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0083] Optionally, the second largest language model is used to score multiple answer information corresponding to the target sample question, that is, the second largest language model scores the answer information according to whether the answer information can answer the corresponding target sample question to obtain a scoring result corresponding to the answer information. Then, the target generation probability corresponding to the answer information of the first largest language model to the sample question is obtained, and the first answer information and the second answer information are determined from the multiple answer information according to the scoring result and the target generation probability. For example, the multiple answer information can be sorted according to the scoring result and the target generation probability, and finally the first answer information and the second answer information are determined according to the sorting result. Finally, a training sample set is obtained based on the multiple sample questions, the first answer information and the second answer information.

[0084] The reliability of the answer information can be accurately evaluated through the scoring results and the target generation probability, thereby improving the accuracy of determining the first answer information and the second answer information.

[0085] In an optional embodiment, the following may be used: Figure 3The flowchart shown implements the training of the natural language model, which specifically includes the following steps: constructing a training sample set based on sample questions and the positive examples (i.e., the first answer information mentioned above) and negative examples (i.e., the second answer information mentioned above) corresponding to the sample questions, determining the loss function corresponding to the open questions and the loss function corresponding to the non-(semi-) open questions. For example, the loss function corresponding to the open questions is , the loss function corresponding to the non-(semi-)open problem is

[0086] The target natural language model is obtained by training the natural language model using the loss function corresponding to the restrictive question and the loss function corresponding to the non-(semi-)restrictive question as well as the training sample set.

[0087] In the model training method provided in Example 1 of the present application, a training sample set is obtained, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, the initial natural language model is trained according to the loss function and training sample set corresponding to the restrictive question to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, the initial natural language model is trained according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model, thereby solving the technical problem that the same loss function is used for different types of sample questions when training the natural language model through DPO, resulting in relatively low accuracy of the natural language model answering questions. Different loss functions are selected according to different types of sample questions to train the natural language model, which can ensure the rationality of the gradient during the model training process, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model answering questions.

[0088] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0090] Example 2

[0091] According to an embodiment of the present application, a data processing method is also provided. Figure 4 : is a flow chart of a data processing method according to Embodiment 2 of the present application. The data processing method comprises:

[0092] Step S401, obtaining a target question input by a target object;

[0093] Step S402, answering the target question through a target natural language model to obtain target answer information, wherein the target natural language model is trained using any one of the above model training methods;

[0094] Step S403, returning the target reply information to the target object.

[0095] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0097] Example 3

[0098] According to an embodiment of the present application, a model training device for implementing the above-mentioned model training method is also provided, such as Figure 5 As shown, the device includes: an acquisition unit 501, a first training unit 502 and a second training unit 503.

[0099] An acquisition unit 501 is used to acquire a training sample set, wherein the training sample set at least includes: a plurality of training samples, wherein the training samples in the plurality of training samples include a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer, and the second answer information is a non-recommended answer;

[0100] A first training unit 502 is used to train the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model if the question type of the sample question is a restrictive question;

[0101] The second training unit 503 is used to train the initial natural language model according to the loss function and training sample set corresponding to the questions other than the restrictive questions to obtain the target natural language model if the question type of the sample question is a question other than the restrictive question.

[0102] In the model training device provided in Example 3 of the present application, a training sample set is obtained by an acquisition unit 501, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, the first training unit 502 trains the initial natural language model according to the loss function and training sample set corresponding to the restrictive question to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, the second training unit 503 trains the initial natural language model according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model, which solves the technical problem that the same loss function is used for different types of sample questions when training the natural language model through DPO, resulting in a relatively low accuracy of the natural language model answering questions. Different loss functions are selected according to different types of sample questions to train the natural language model, which can ensure the rationality of the gradient during the model training process, so that the natural language model can converge to a better solution more stably, thereby achieving the technical effect of improving the accuracy of the natural language model answering questions.

[0103] Optionally, in the model training device provided in Example 3 of the present application, the first training unit includes: a first answer module, which is used to answer sample questions in the training sample set through the initial natural language model, and obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; a first generation module, which is used to calculate based on the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; a first processing module, which is used to normalize the first generation probability to obtain the processed first generation probability, and normalize the third generation probability to obtain the processed third generation probability; a first calculation module, which is used to calculate based on the processed first generation probability and the processed third generation probability to obtain a loss function corresponding to questions other than restrictive questions, and train the initial natural language model based on the loss function corresponding to questions other than restrictive questions to obtain a target natural language model.

[0104] Optionally, in the model training device provided in Example 3 of the present application, the first processing module includes: a determination submodule, used to determine the total number of characters corresponding to the first answer information; and a processing submodule, used to normalize the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0105] Optionally, in the model training device provided in Example 3 of the present application, the second training unit includes: a second answer module, which is used to answer sample questions in the training sample set through the initial natural language model, and obtain a fourth generation probability corresponding to the first answer information and a fifth generation probability corresponding to the second answer information; a second calculation module, which is used to calculate based on the fifth generation probability to obtain a sixth generation probability, wherein the sixth generation probability is the probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; a third calculation module, which is used to calculate based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, and train the initial natural language model based on the loss function corresponding to the restrictive question to obtain a target natural language model.

[0106] Optionally, in the model training device provided in Example 3 of the present application, the third calculation module includes: a reply submodule, which is used to reply to sample questions in the training sample set through the initial natural language model, and obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; a first calculation submodule, which is used to calculate based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; a second calculation submodule, which is used to calculate based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0107] Optionally, in the model training device provided in Example 3 of the present application, the second calculation sub-module includes: a first calculation sub-module, used to calculate the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; a second calculation sub-module, used to calculate the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; an acquisition sub-module, used to obtain the weight value and the preset scaling factor corresponding to the second answer information; a third calculation sub-module, used to calculate based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculate based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; a fourth calculation sub-module, used to calculate based on the first numerical value and the second numerical value to obtain a loss function corresponding to the restrictive problem.

[0108] Optionally, in the model training device provided in Example 3 of the present application, the acquisition unit includes: an acquisition sub-unit, used to acquire multiple sample questions; a reply sub-unit, used to reply to the multiple sample questions a preset number of times through the first largest language model, and obtain multiple answer information corresponding to the sample questions in the multiple sample questions; a processing sub-unit, used to obtain a training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0109] Optionally, in the model training device provided in Example 3 of the present application, the processing subunit includes: a scoring module, which is used to score multiple answer information corresponding to the target sample question through the second largest language model, and obtain a scoring result corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; an acquisition module, which is used to obtain a target generation probability corresponding to the answer information in the multiple answer information; a determination module, which is used to determine the first answer information and the second answer information from the multiple answer information based on the scoring result and the target generation probability; a second processing module, which is used to obtain a training sample set based on the multiple sample questions, the first answer information, and the second answer information.

[0110] It should be noted that the acquisition unit 501, the first training unit 502 and the second training unit 503 described above correspond to steps S201 to S203 in Embodiment 1, and the examples and application scenarios implemented by the two units and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1 described above. It should be noted that the above modules as part of the device can be run in the computer terminal 10 provided in Embodiment 1.

[0111] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0112] Example 4

[0113] The embodiment of the present application may provide an electronic device, which may be any electronic device in an electronic device terminal group. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0114] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0115] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the model training method and the data processing method provided in the above-mentioned embodiment: obtaining a training sample set, wherein the training sample set at least includes: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, then the initial natural language model is trained according to the loss function and training sample set corresponding to the restrictive question to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, then the initial natural language model is trained according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model.

[0116] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiment: training the initial natural language model according to the loss function and training sample set corresponding to the questions other than the restrictive questions to obtain the target natural language model, including: answering the sample questions in the training sample set through the initial natural language model to obtain the first generation probability corresponding to the first answer information and the second generation probability corresponding to the second answer information; calculating according to the second generation probability to obtain the third generation probability, wherein the third generation probability is the probability of occurrence of the opposite event of the event corresponding to the second generation probability; normalizing the first generation probability to obtain the processed first generation probability, normalizing the third generation probability to obtain the processed third generation probability; calculating according to the processed first generation probability and the processed third generation probability to obtain the loss function corresponding to the questions other than the restrictive questions, and training the initial natural language model according to the loss function corresponding to the questions other than the restrictive questions to obtain the target natural language model.

[0117] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiments: normalizing the first generation probability to obtain the processed first generation probability includes: determining the total number of characters corresponding to the first answer information; normalizing the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0118] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiment: training the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model, including: answering the sample questions in the training sample set through the initial natural language model to obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information; calculating according to the fifth generation probability to obtain the sixth generation probability, wherein the sixth generation probability is the probability of occurrence of the opposite event of the event corresponding to the fifth generation probability; calculating according to the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question, and training the initial natural language model according to the loss function corresponding to the restrictive question to obtain the target natural language model.

[0119] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiment: calculating based on the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question includes: answering the sample questions in the training sample set through the initial natural language model to obtain the first reference generation probability corresponding to the first answer information and the second reference generation probability corresponding to the second answer information; calculating based on the second reference generation probability to obtain the third reference generation probability, wherein the third reference generation probability is the probability of occurrence of the opposite event of the event corresponding to the second reference generation probability; calculating based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question.

[0120] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and the data processing method provided in the above-mentioned embodiment: calculating based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability and the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability and the sixth generation probability to obtain a second ratio; obtaining the weight value and the preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive question.

[0121] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiment: obtaining a training sample set includes: obtaining multiple sample questions; answering the multiple sample questions a preset number of times through the first largest language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; obtaining a training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0122] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and the data processing method provided in the above-mentioned embodiment: obtaining a training sample set based on the scoring results corresponding to the answer information in the multiple sample questions and the multiple answer information, including: scoring the multiple answer information corresponding to the target sample question through the second largest language model to obtain the scoring results corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; obtaining the target generation probability corresponding to the answer information in the multiple answer information; determining the first answer information and the second answer information from the multiple answer information based on the scoring results and the target generation probability; obtaining the training sample set based on the multiple sample questions, the first answer information and the second answer information.

[0123] The above-mentioned computer terminal can execute the program code of the following steps in the model training method and data processing method provided in the above-mentioned embodiments: obtaining the target question input by the target object; answering the target question through the target natural language model to obtain target answer information, wherein the target natural language model is trained using any of the above-mentioned model training methods; and returning the target answer information to the target object.

[0124] Optionally, Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 6 As shown, the electronic device 60 may include: one or more ( Figure 6Only one is shown in the figure) processor 602, memory 604. The electronic device 60 may also include a storage controller, through which the memory 604 is controlled and managed; the electronic device 60 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0125] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method and data processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned model training method and data processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the electronic device 60 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a training sample set, wherein the training sample set includes at least: multiple training samples, the training samples in the multiple training samples include sample questions, first answer information and second answer information corresponding to the sample questions, the first answer information is a recommended answer, and the second answer information is a non-recommended answer; if the question type of the sample question is a restrictive question, then the initial natural language model is trained according to the loss function and training sample set corresponding to the restrictive question to obtain a target natural language model; if the question type of the sample question is a question other than a restrictive question, then the initial natural language model is trained according to the loss function and training sample set corresponding to the question other than the restrictive question to obtain a target natural language model.

[0127] Optionally, the processor may also execute the program code of the following steps: training the initial natural language model according to the loss function and training sample set corresponding to questions other than restrictive questions to obtain the target natural language model, including: answering the sample questions in the training sample set through the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; calculating according to the second generation probability to obtain a third generation probability, wherein the third generation probability is the probability of occurrence of an event opposite to the event corresponding to the second generation probability; normalizing the first generation probability to obtain a processed first generation probability, normalizing the third generation probability to obtain a processed third generation probability; calculating according to the processed first generation probability and the processed third generation probability to obtain the loss function corresponding to questions other than restrictive questions, and training the initial natural language model according to the loss function corresponding to questions other than restrictive questions to obtain the target natural language model.

[0128] Optionally, the processor may also execute the following program code steps: normalizing the first generation probability to obtain the processed first generation probability, including: determining the total number of characters corresponding to the first answer information; normalizing the first generation probability according to the total number of characters to obtain the processed first generation probability.

[0129] Optionally, the processor may also execute the program code of the following steps: training the initial natural language model according to the loss function and training sample set corresponding to questions other than restrictive questions to obtain the target natural language model, including: answering the sample questions in the training sample set through the initial natural language model to obtain the fourth generation probability corresponding to the first answer information and the fifth generation probability corresponding to the second answer information; calculating according to the fifth generation probability to obtain the sixth generation probability, wherein the sixth generation probability is the probability of occurrence of the opposite event of the event corresponding to the fifth generation probability; calculating according to the fourth generation probability and the sixth generation probability to obtain the loss function corresponding to the restrictive question, and training the initial natural language model according to the loss function corresponding to the restrictive question to obtain the target natural language model.

[0130] Optionally, the processor may also execute the following steps of program code: performing calculations based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: answering the sample questions in the training sample set through the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; performing calculations based on the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the probability of occurrence of an event opposite to the event corresponding to the second reference generation probability; performing calculations based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question.

[0131] Optionally, the processor may also execute the following steps of program code: performing calculations based on the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, including: calculating the ratio of the first reference generation probability to the fourth generation probability to obtain a first ratio; calculating the ratio of the third reference generation probability to the sixth generation probability to obtain a second ratio; obtaining a weight value and a preset scaling factor corresponding to the second answer information; calculating based on the preset scaling factor and the first ratio to obtain a first numerical value, and calculating based on the weight value, the preset scaling factor and the second ratio to obtain a second numerical value; calculating based on the first numerical value and the second numerical value to obtain the loss function corresponding to the restrictive question.

[0132] Optionally, the processor may also execute the program code of the following steps: obtaining a training sample set includes: obtaining multiple sample questions; answering the multiple sample questions a preset number of times through the first largest language model to obtain multiple answer information corresponding to the sample questions in the multiple sample questions; obtaining a training sample set based on the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

[0133] Optionally, the processor may also execute program code for the following steps: obtaining a training sample set based on scoring results corresponding to multiple sample questions and multiple answer information, including: scoring multiple answer information corresponding to a target sample question through a second largest language model, and obtaining scoring results corresponding to the answer information in multiple answer information, wherein the target sample question is any one of the multiple sample questions; obtaining a target generation probability corresponding to the answer information in the multiple answer information; determining first answer information and second answer information from the multiple answer information based on the scoring results and the target generation probability; and obtaining a training sample set based on the multiple sample questions, the first answer information, and the second answer information.

[0134] Optionally, the processor may also execute the program code of the following steps: obtaining a target question input by the target object; answering the target question through a target natural language model to obtain target answer information, wherein the target natural language model is trained using any of the above-mentioned model training methods; and returning the target answer information to the target object.

[0135] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the electronic device 60 may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 6 The structure of the electronic device is not limited. For example, the electronic device 60 may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 6 Different configurations are shown.

[0136] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0137] Example 5

[0138] An embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the above-mentioned computer program product can be used to store the program code executed by the model training method and data processing method provided in the above-mentioned embodiment 1.

[0139] Example 6

[0140] The embodiment of the present application also provides a computer program product. Optionally, in this embodiment, the computer program product can be used to store the program code executed by the model training method and the data processing method provided in the first embodiment.

[0141] Optionally, in this embodiment, the computer program product may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0142] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0143] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0148] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, It is characterized in that include: Acquire a training sample set, wherein the training sample set at least includes: a plurality of training samples, wherein the training samples in the plurality of training samples include a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer, and the second answer information is a non-recommended answer; If the question type of the sample question is a restrictive question, training the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model; If the question type of the sample question is a question other than a restrictive question, the initial natural language model is trained according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model.

2. The method according to claim 1, It is characterized in that The initial natural language model is trained according to the loss function corresponding to the questions other than the restrictive questions and the training sample set to obtain the target natural language model, including: Replying to the sample questions in the training sample set by the initial natural language model to obtain a first generation probability corresponding to the first answer information and a second generation probability corresponding to the second answer information; Calculating according to the second generation probability to obtain a third generation probability, wherein the third generation probability is a probability of occurrence of an event opposite to the event corresponding to the second generation probability; Normalizing the first generation probability to obtain a processed first generation probability, and normalizing the third generation probability to obtain a processed third generation probability; Calculation is performed based on the first generation probability after processing and the third generation probability after processing to obtain loss functions corresponding to questions other than restrictive questions, and the initial natural language model is trained based on the loss functions corresponding to the restrictive questions to obtain the target natural language model.

3. The method according to claim 2, It is characterized in that Normalizing the first generation probability to obtain the processed first generation probability includes: Determine the total number of characters corresponding to the first answer information; The first generation probability is normalized according to the total number of characters to obtain the processed first generation probability.

4. The method according to claim 1, It is characterized in that The initial natural language model is trained according to the loss function corresponding to the restrictive question and the training sample set to obtain the target natural language model, including: Replying to the sample questions in the training sample set by the initial natural language model to obtain a fourth generation probability corresponding to the first answer information and a fifth generation probability corresponding to the second answer information; Calculating according to the fifth generation probability to obtain a sixth generation probability, wherein the sixth generation probability is a probability of occurrence of an event opposite to the event corresponding to the fifth generation probability; A calculation is performed based on the fourth generation probability and the sixth generation probability to obtain a loss function corresponding to the restrictive question, and the initial natural language model is trained based on the loss function corresponding to the restrictive question to obtain the target natural language model.

5. The method according to claim 4, It is characterized in that The loss function corresponding to the restrictive problem obtained by calculating according to the fourth generation probability and the sixth generation probability includes: Replying to the sample questions in the training sample set by the initial natural language model to obtain a first reference generation probability corresponding to the first answer information and a second reference generation probability corresponding to the second answer information; Calculating according to the second reference generation probability to obtain a third reference generation probability, wherein the third reference generation probability is the occurrence probability of an event opposite to the event corresponding to the second reference generation probability; Calculation is performed based on the first reference generation probability, the third reference generation probability, the fourth generation probability, and the sixth generation probability to obtain a loss function corresponding to the restrictive problem.

6. The method according to claim 5, It is characterized in that Calculating according to the first reference generation probability, the third reference generation probability, the fourth generation probability and the sixth generation probability, the loss function corresponding to the restrictive problem includes: Calculating a ratio of the first reference generation probability to the fourth generation probability to obtain a first ratio; Calculating a ratio of the third reference generation probability to the sixth generation probability to obtain a second ratio; Obtaining a weight value and a preset scaling factor corresponding to the second answer information; Calculating according to the preset scaling factor and the first ratio to obtain a first value, and calculating according to the weight value, the preset scaling factor and the second ratio to obtain a second value; A calculation is performed based on the first value and the second value to obtain a loss function corresponding to the restrictive problem.

7. The method according to claim 1, It is characterized in that Obtaining a training sample set includes: Get multiple sample questions; Replying to the plurality of sample questions a preset number of times through the first large language model to obtain a plurality of answer information corresponding to the sample questions among the plurality of sample questions; The training sample set is obtained according to the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information.

8. The method according to claim 7, It is characterized in that The training sample set is obtained according to the scoring results corresponding to the multiple sample questions and the answer information in the multiple answer information, including: Scoring multiple answer information corresponding to the target sample question by using the second largest language model to obtain a scoring result corresponding to the answer information in the multiple answer information, wherein the target sample question is any one of the multiple sample questions; Obtaining a target generation probability corresponding to the answer information among the plurality of answer information; Determining the first answer information and the second answer information from the plurality of answer information according to the scoring result and the target generation probability; The training sample set is obtained according to the multiple sample questions, the first answer information and the second answer information.

9. A data processing method, It is characterized in that include: Get the target question input by the target object; Responding to the target question through a target natural language model to obtain target response information, wherein the target natural language model is trained using the model training method described in any one of claims 1 to 8; The target reply information is returned to the target object.

10. A model training device, It is characterized in that include: an acquisition unit, configured to acquire a training sample set, wherein the training sample set at least includes: a plurality of training samples, wherein a training sample in the plurality of training samples includes a sample question, first answer information and second answer information corresponding to the sample question, wherein the first answer information is a recommended answer, and the second answer information is a non-recommended answer; A first training unit is used for training the initial natural language model according to the loss function corresponding to the restrictive question and the training sample set to obtain a target natural language model if the question type of the sample question is a restrictive question; The second training unit is used to train the initial natural language model according to the loss function corresponding to the question other than the restrictive question and the training sample set to obtain the target natural language model if the question type of the sample question is a question other than the restrictive question.

11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein, when the program is running, the device where the storage medium is located is controlled to execute the model training method described in any one of claims 1 to 8, or the data processing method described in claim 9.

12. An electronic device, It is characterized in that include: A memory storing an executable program; A processor for running the program, wherein the program, when running, executes the model training method described in any one of claims 1 to 8, or the data processing method described in claim 9.

13. A computer program product, It is characterized in that The computer program product includes a stored computer program, and when the computer program is executed by a processor, the model training method described in any one of claims 1 to 8, or the data processing method described in claim 9 is implemented.

Images