Data processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a computer readable storage medium. The method comprises the steps of obtaining a training sample comprising a sample question, at least two sample answers and a preference tag; determining the at least two sample answers as sample tags of the sample question, and generating at least two supervised training samples according to the sample question and the at least two sample tags; adjusting parameters in the initial question and answer model according to the at least two supervised training samples to obtain a first question and answer model; inputting the training sample into a first question and answer model, and generating first prediction probabilities corresponding to the at least two sample answers in the first question and answer model; according to the preference label and the at least two first prediction probabilities, parameters in the first question and answer model are adjusted, and a second question and answer model is obtained. By adopting the method, the model training stability and the generation diversity can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] In recent years, with the widespread use of large language models (LLMs), how to better align models with human preferences has become a core issue in model fine-tuning. Reinforcement learning from human feedback (RLHF) is the current mainstream alignment method. Direct Preference Optimization (DPO), a reinforcement learning alternative that does not require explicit training of a reward model, has been widely adopted due to its stability and simplicity.

[0003] DPO maximizes the model's predicted probability for the preferred response and minimizes the predicted probability for the dispreferred response, making the model output more consistent with human preferences. However, existing DPO methods have significant technical bottlenecks: because the model applies a large negative gradient to dispreferred responses with low prediction probabilities during training, this operation can easily lead to "weight bias" in the model's probability distribution—that is, the confidence of most answers generally decreases, and the generation probability is forcibly concentrated on a very small number of high-confidence outputs. This can lead to the following problems in the model:

[0004] 1. The model output lacks diversity, and the generated content tends to be repetitive, reducing practicality;

[0005] 2. The model training effect is unstable. After further optimization, the model may lose confidence in the optimal answer.

[0006] For large language models, the accumulation of probability compression effects will seriously hinder stability and diversity under long-term training. Summary of the Invention

[0007] The embodiments of the present application provide a data processing method, apparatus, device, and computer-readable storage medium that can improve model training stability and generation diversity.

[0008] On the one hand, an embodiment of the present application provides a data processing method, including:

[0009] Obtaining a training sample; the training sample includes a sample question, at least two sample answers to the sample question, and a preference label for indicating a preference relationship between the at least two sample answers;

[0010] Determine at least two sample answers as sample labels of the sample question, and generate at least two supervised training samples based on the sample question and the at least two sample labels; wherein one supervised training sample includes a sample question carrying one sample label;

[0011] Adjusting parameters of the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model;

[0012] Inputting the training sample into the first question-answering model, generating first predicted probabilities corresponding to at least two sample answers in the first question-answering model; wherein a first predicted probability refers to the probability of outputting a sample answer for the sample question;

[0013] According to the preference label and the first predicted probabilities corresponding to at least two sample answers, the parameters in the first question-answering model are adjusted to obtain a second question-answering model; the second question-answering model is used to generate an output answer to the input question.

[0014] In one aspect, an embodiment of the present application provides a data processing device, including:

[0015] An acquisition module is configured to acquire a training sample; the training sample includes a sample question, at least two sample answers to the sample question, and a preference label indicating a preference relationship between the at least two sample answers;

[0016] A generation module is configured to determine the at least two sample answers as sample labels of the sample question, and generate at least two supervised training samples based on the sample question and the at least two sample labels; wherein one supervised training sample includes a sample question carrying one sample label;

[0017] An adjustment module, configured to adjust parameters in the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model;

[0018] The generation module is further configured to input the training sample into the first question-answering model, and generate first predicted probabilities corresponding to at least two sample answers in the first question-answering model; wherein a first predicted probability refers to the probability of outputting a sample answer for the sample question;

[0019] The adjustment module is also used to adjust the parameters in the first question-answering model according to the preference label and the first predicted probabilities corresponding to at least two sample answers to obtain a second question-answering model; the second question-answering model is used to generate an output answer to the input question.

[0020] In one possible implementation, the adjustment module adjusts parameters in the first question-answering model based on the preference label and the first predicted probabilities corresponding to the at least two sample answers to obtain a second question-answering model for performing the following operations:

[0021] Generate a sample loss value corresponding to the training sample based on the preference label and the first predicted probabilities corresponding to the at least two sample answers;

[0022] Obtaining a poorly selected sample answer from at least two sample answers based on the preference label, and determining a training risk value for the training sample based on a first predicted probability corresponding to the poorly selected sample answer; the training risk value is used to indicate the intensity of the disturbance of the training sample on the model output distribution of the first question-answering model;

[0023] According to the training risk value, the sample loss value is optimized to obtain the sample optimization loss value corresponding to the training sample; the sample optimization loss value is less than the sample loss value;

[0024] According to the sample optimization loss value, the parameters in the first question-answering model are adjusted to obtain the question-answering model.

[0025] In one possible implementation, the adjustment module determines the training risk value of the training sample based on the first predicted probability corresponding to the answer of the inferior sample, and is configured to perform the following operations:

[0026] Perform logarithmic processing on the first predicted probability corresponding to the answer of the inferior sample to obtain the log-likelihood value;

[0027] The negative of the log-likelihood value is determined as the training risk value of the training sample; there is an inverse relationship between the training risk value and the first predicted probability corresponding to the answer of the inferior sample.

[0028] In a possible implementation, the number of the inferior sample answers is at least two, and the at least two inferior sample answers include the inferior sample answer C d , d is a positive integer, and d is less than or equal to the total number of at least two inferior sample answers;

[0029] The adjustment module determines the training risk value of the training sample based on the first predicted probability corresponding to the answer of the inferior sample, and is used to perform the following operations:

[0030] Answer C for the inferior sample d The corresponding first predicted probability is logarithmically processed to obtain a log-likelihood value;

[0031] Determine the negative value of the log-likelihood value as the poor choice sample answer C d Corresponding training risk value;

[0032] The training risk values ​​corresponding to at least two poorly selected sample answers are fused to obtain the training risk value of the training sample.

[0033] In one possible implementation, the adjustment module optimizes the sample loss value according to the training risk value to obtain the sample optimized loss value corresponding to the training sample, which is used to perform the following operations:

[0034] According to the training risk value, the training weight corresponding to the training sample is determined; there is an inverse relationship between the training risk value and the training weight;

[0035] The sample loss value is weighted according to the training weight to obtain the sample optimization loss value corresponding to the training sample.

[0036] In one possible implementation, the adjustment module determines the training weights corresponding to the training samples based on the training risk values, and performs the following operations:

[0037] The training risk value is weighted according to the weighted intensity parameter to obtain a weighted training risk value;

[0038] The negative of the weighted training risk value is determined as the risk value to be smoothed, and the smoothed risk value is subjected to exponential processing to obtain the training weight corresponding to the training sample.

[0039] In one possible implementation, the adjustment module generates a sample loss value corresponding to the training sample based on the preference label and the first predicted probabilities corresponding to the at least two sample answers, and performs the following operations:

[0040] According to the preference label, a preferred sample answer and a less preferred sample answer are determined from at least two sample answers; the quality of the answer corresponding to the preferred sample answer is better than the quality of the answer corresponding to the less preferred sample answer;

[0041] Obtaining a second predicted probability output by the reference model for the preferred sample answer and a third predicted probability output by the reference model for the inferior sample answer;

[0042] Generate a preferred log-likelihood ratio based on the first predicted probability and the second predicted probability corresponding to the preferred sample answer;

[0043] Generate a poor choice log likelihood ratio based on the first predicted probability and the third predicted probability corresponding to the poor choice sample answer;

[0044] According to the optimal log-likelihood ratio and the inferior log-likelihood ratio, the sample loss value corresponding to the training sample is generated.

[0045] In one possible implementation, the adjustment module generates a preferred log-likelihood ratio based on the first predicted probability and the second predicted probability corresponding to the preferred sample answer, and performs the following operations:

[0046] Perform ratio processing on the first predicted probability and the second predicted probability corresponding to the preferred sample answer to obtain a probability ratio;

[0047] The probability ratio is logarithmically processed to obtain the log likelihood ratio, and the log likelihood ratio is weighted to obtain the optimal log likelihood ratio.

[0048] In one possible implementation, the adjustment module generates a sample loss value corresponding to the training sample based on the preferred log-likelihood ratio and the inferior log-likelihood ratio, and performs the following operations:

[0049] Performing difference processing on the optimal log-likelihood ratio and the inferior log-likelihood ratio to obtain the log-likelihood ratio difference;

[0050] Normalize the log-likelihood ratio difference to obtain the preference probability, and perform logarithmic processing on the preference probability to obtain the initial loss value;

[0051] The negative of the initial loss value is determined as the sample loss value corresponding to the training sample.

[0052] In one possible implementation, the adjustment module adjusts parameters in the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model, which is used to perform the following operations:

[0053] At least two supervised training samples are input into the initial question answering model respectively; at least two supervised training samples include supervised training sample A b , b is a positive integer, and b is less than or equal to the total number of at least two supervised training samples;

[0054] In the initial question answering model, supervised training samples A are generated b The corresponding sample loss value;

[0055] Perform mean processing on the sample loss values ​​corresponding to at least two supervised training samples to obtain the sample mean loss value;

[0056] According to the sample mean loss value, the parameters in the initial question-answering model are adjusted to obtain the first question-answering model.

[0057] In one possible implementation, the adjustment module generates supervised training samples A in the initial question-answering model. b The corresponding sample loss value is used to perform the following operations:

[0058] In the initial question answering model, supervised training samples A are generated b The fourth prediction probability corresponding to the training label in; the fourth prediction probability refers to the output of the supervised training sample A for the sample problem b The probability of the training labels in ;

[0059] Perform logarithmic processing on the fourth predicted probability to obtain the initial loss value, and determine the negative of the initial loss value as the supervised training sample A b The corresponding sample loss value.

[0060] In one possible implementation, the adjustment module generates supervised training samples A in the initial question-answering model. b The corresponding sample loss value is used to perform the following operations:

[0061] In the initial question answering model, the supervised training sample A b Identify and process the sample questions in the , and get the estimated answers;

[0062] For the estimated answer and supervised training sample A b The training labels in are processed for loss, and the supervised training sample A is obtained b The corresponding sample loss value.

[0063] On the one hand, the present application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer program so that the computer device executes the method in the embodiment of the present application.

[0064] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the method in the embodiment of the present application.

[0065] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium; a processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method in the embodiment of the present application.

[0066] In an embodiment of the present application, a computer device obtains a training sample, which includes a sample question, at least two sample answers to the sample question, and a preference label for indicating a preference relationship between the at least two sample answers; the at least two sample answers are respectively determined as sample labels for the sample question, and at least two supervised training samples can be generated based on the sample question and the at least two sample labels; wherein, a supervised training sample includes a sample question carrying a sample label; according to the at least two supervised training samples, the parameters in the initial question-answering model are adjusted to obtain a first question-answering model, and the above-mentioned model adjustment is supervised fine-tuning; the training sample is input into the first question-answering model, and in the first question-answering model, first prediction probabilities corresponding to at least two sample answers are generated; wherein, a first prediction probability refers to the probability of outputting a sample answer for the sample question; according to the preference label and the first prediction probabilities corresponding to the at least two sample answers, the parameters in the first question-answering model are adjusted to obtain a second question-answering model, and the above-mentioned model adjustment is preference alignment adjustment. From the above, it can be seen that in the supervised fine-tuning stage, the embodiment of the present application uses all sample answers (including sample answers with low preference) as sample labels, that is, each sample answer is regarded as the correct answer. Therefore, the first question-answering model obtained by training can improve the generation probability and confidence of each sample answer (including sample answers with low preference), which can improve the output diversity and stability of the first question-answering model; since the generation probability and confidence of sample answers with low preference have been improved in the supervised fine-tuning stage, the suppression of sample answers with low preference in the preference alignment stage is smoother, avoiding the collapse of the predicted probability of sample answers with low preference due to model parameter adjustment, and thus avoiding the predicted probability from being excessively squeezed to a very small number of high-confidence sample answers (i.e., sample answers with high preference), so the model training stability and generation diversity can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0068] Figure 1 This is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0069] Figure 2 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 ;

[0070] Figure 3 This is a data processing scenario provided by the embodiment of the present application. Figure 1 ;

[0071] Figure 4 This is a data processing scenario provided by the embodiment of the present application. Figure 2 ;

[0072] Figure 5 This is a data processing scenario provided by the embodiment of the present application. Figure 3 ;

[0073] Figure 6 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 ;

[0074] Figure 7 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0075] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0077] See Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1 As shown, the system may include a business server 100 and a terminal device cluster. The terminal device cluster may include: terminal device 200a, terminal device 200b, terminal device 200c, ..., terminal device 200n. It can be understood that the above system may include one or more terminal devices, and this application does not limit the number of terminal devices.

[0078] Among them, there can be communication connections between terminal device clusters, for example, there is a communication connection between terminal device 200a and terminal device 200b, and there is a communication connection between terminal device 200a and terminal device 200c. At the same time, any terminal device in the terminal device cluster can have a communication connection with the business server 100, for example, there is a communication connection between terminal device 200a and business server 100. The above-mentioned communication connection is not limited to the connection method, and can be directly or indirectly connected through wired communication, directly or indirectly connected through wireless communication, or through other methods, and this application does not impose any restrictions on this.

[0079] It should be understood that Figure 1 Each terminal device in the terminal device cluster shown can be installed with an application client. When the application client runs in each terminal device, it can be respectively connected to the above Figure 1 The business server 100 shown performs data interaction, that is, the above-mentioned communication connection. Among them, the application client can be an application client with intelligent question-answering function such as video application, convenient life application, payment application, financial application, game application, shopping application, novel application, navigation application, browser, etc.

[0080] The application client can be a standalone client or an embedded sub-client integrated into a client (e.g., a video client, an education client, a multimedia client, etc.), which is not limited here. Taking a browser as an example, the business server 100 can be a collection of multiple servers including a backend server corresponding to the browser, a data processing server, etc. Therefore, each terminal device can transmit data with the business server 100 through the application client corresponding to the browser. For example, each terminal device can upload input questions (e.g., text questions or voice questions) to the business server 100 through the browser application client, and the business server 100 can generate output answers based on the input questions.

[0081] It is understandable that in the specific implementation of this application, related data such as user information (such as preference tags or input questions) is involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions.

[0082] To facilitate subsequent understanding and explanation, the embodiments of the present application can be Figure 1 A terminal device example is selected from the terminal device cluster shown, for example, terminal device 200a is used as an example. When a training sample is obtained in the application client and a model training instruction for the training sample is received, terminal device 200a can generate a model training request corresponding to the model training instruction in the application client, and send the model training request to the business server 100 through the application client. The embodiment of the present application does not limit the number of training samples, which can be one or more. It is understandable that if the number of training samples is multiple, the subsequent processing procedures of the multiple training samples are the same, so the embodiment of the present application does not limit the number of training samples.

[0083] A training sample includes a sample question, at least two sample answers to the sample question, and a preference label indicating a preference relationship between the at least two sample answers. For example, a training sample may be as follows: (sample question: How is winter in Region 1?; sample answer 1: Winter in Region 1 is warm and colorful; sample answer 2: Winter in Region 1 is quite comfortable?; preference label: answer 1 > answer 2). The preference label indicates that, in terms of human preference, sample answer 1 is superior to sample answer 2. This can also be understood as meaning that for the question "How is winter in Region 1?", "Winter in Region 1 is warm and colorful" is more popular than "Winter in Region 1 is quite comfortable."

[0084] When a model training request is received, the business server 100 obtains training samples and the initial question-answering model to be trained. The embodiment of the present application does not limit the manner in which the business server 100 obtains training samples, and can be set according to the actual application scenario. For example, the model training request can include training samples, and the model training request can include the storage address of the training samples, so the business server 100 can obtain the training samples based on the storage address. Similarly, the embodiment of the present application does not limit the manner in which the business server 100 obtains the initial question-answering model, and can be set according to the actual application scenario.

[0085] The service server 100 determines the at least two sample answers as sample labels for the sample question, and generates at least two supervised training samples based on the sample question and the at least two sample labels. For example, the service server 100 determines both sample answer 1 and sample answer 2 as sample labels for the sample question, resulting in two different sample labels. The sample question "How is winter in Region 1?" and the first sample label "Winter in Region 1 is warm and colorful" are combined into one supervised training sample, while the sample question "How is winter in Region 1?" and the second sample label "Winter in Region 1 is comfortable" are combined into another supervised training sample, resulting in two different supervised training samples.

[0086] Furthermore, based on at least two supervised training samples, the business server 100 adjusts the parameters in the initial question-answering model to obtain a first question-answering model. The above model adjustment is supervised fine-tuning. The initial question-answering model is an initialized large language model. The first question-answering model is a pre-trained model generated after the initial question-answering model is pre-trained. The embodiment of this application does not limit the specific implementation process of pre-training, and can be set according to the actual application scenario. Figure 2 The corresponding embodiments and Figure 6 The corresponding embodiments each describe a specific implementation process of pre-training.

[0087] After pre-training the initial question-answering model (i.e., supervised fine-tuning), the business server 100 can obtain a first question-answering model. Further, reinforcement learning is performed on the first question-answering model. In an embodiment of the present application, direct preference alignment training is performed on the first question-answering model. The business server 100 inputs a training sample comprising a sample question, at least two sample answers, and a sample label into the first question-answering model. In the first question-answering model, a first predicted probability corresponding to each of the at least two sample answers is generated. A first predicted probability refers to the probability of outputting a sample answer for the sample question. For example, a training sample (sample question: What is winter like in Region 1? Sample answer 1: Winter in Region 1 is warm and colorful; sample answer 2: Winter in Region 1 is quite comfortable? Preference label: Answer 1 > Answer 2) is used to illustrate this. In the first question-answering model, the business server 100 generates a first predicted probability corresponding to sample answer 1 for the sample question, and a first predicted probability corresponding to sample answer 2 for the sample question.

[0088] Based on the preference label and the first predicted probabilities corresponding to the at least two sample answers, the service server 100 adjusts the parameters of the first question-answering model to generate a second question-answering model. The second question-answering model is a large language model that has been trained to align with human preferences. This model is consistent with human preferences and can therefore accurately generate output answers that align with human preferences after receiving input questions from a mobile device. Specifically, the second question-answering model is used to generate output answers to input questions.

[0089] Human preference alignment refers to the process of aligning model behavior or output with human preferences or expectations during model training. This concept is particularly important in the field of artificial intelligence, especially in the training of large language models. By adjusting model parameters and behaviors to ensure that the generated output is more aligned with human needs and expectations, the human-computer interaction experience and the model's practicality can be improved.

[0090] Direct Preference Optimization (DPO): DPO is a reinforcement learning method used to optimize language model outputs to make them more consistent with human preferences. It compares multiple answers generated by the model and adjusts model parameters based on human annotated preferences. In DPO, multiple answers (e.g., two or more) are typically generated, and then human annotators rank their preferences. The model then adjusts its output probability distribution based on these preference rankings.

[0091] Supervised training is a traditional machine learning method in which the model optimizes performance by learning the mapping relationship between input and output. Each training sample has a clear label (i.e., the expected output). To distinguish it from the training samples of DPO, this application refers to the training samples in the supervised training phase as supervised training samples. In supervised training, the goal of the model is to minimize the difference between the predicted output and the true label (for example, through cross-entropy loss).

[0092] The difference between Descriptive Positive Algorithm (DPO) and supervised training: DPO is a reinforcement learning method that adjusts the model based on human preferences by comparing multiple generated responses. It does not rely on a single "correct" label, but rather on relative preferences. Supervised training relies on each example having a clear label, allowing the model to directly learn the mapping between input and output.

[0093] The business server 100 can return the second question-answering model to the terminal device 200a, so the terminal device 200a can input the question text (or question voice, collectively referred to as the input question in this application) into the second question-answering model, and generate the answer text (or answer voice, collectively referred to as the output answer in this application) to the question text through the second question-answering model. Alternatively, the terminal device 200a sends the input question to the business server 100, so the business server 100 can input the input question into the second question-answering model, and generate the output answer to the input question through the second question-answering model, and return the output answer to the terminal device 200a.

[0094] Optionally, if the terminal device 200a stores the above-mentioned initial question-answering model locally and the terminal device 200a has offline computing capabilities, the initial question-answering model can be pre-trained locally through at least two supervised training samples to obtain a first question-answering model. It can be understood that the process of the terminal device 200a training the initial question-answering model through at least two supervised training samples is the same as the process of the business server 100 training the initial question-answering model through at least two supervised training samples, so it will not be repeated here. Furthermore, the terminal device 200a can perform human preference alignment training on the first question-answering model locally through training samples to obtain a second question-answering model. It can be understood that the process of the terminal device 200a training the first question-answering model through training samples is the same as the process of the business server 100 training the first question-answering model through training samples, so it will not be repeated here.

[0095] From the above, it can be seen that in the supervised fine-tuning stage, the embodiment of the present application uses all sample answers (including sample answers with low preference) as sample labels, that is, each sample answer is regarded as the correct answer. Therefore, the first question-answering model obtained by training can improve the generation probability and confidence of each sample answer (including sample answers with low preference), which can improve the output diversity and stability of the first question-answering model; since the generation probability and confidence of sample answers with low preference have been improved in the supervised fine-tuning stage, the suppression of sample answers with low preference in the preference alignment stage is smoother, avoiding the collapse of the predicted probability of sample answers with low preference due to model parameter adjustment, and thus avoiding the predicted probability from being excessively squeezed to a very small number of high-confidence sample answers (i.e., sample answers with high preference), so the model training stability and generation diversity can be improved.

[0096] It is understood that the methods provided in the embodiments of the present application can be executed by computer devices, including but not limited to terminal devices or business servers. Among them, the business server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0097] Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. Among them, the terminal devices and the service servers can be connected directly or indirectly via wired or wireless means, which is not limited in this embodiment of the present application.

[0098] Further, see Figure 2 , Figure 2 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc. The embodiments of the present application can be applied to business scenarios such as content generation scenarios, content recommendation scenarios, content recognition scenarios, and content creation scenarios. Specific business scenarios will not be listed one by one here.

[0099] The data processing method can be performed by a business server (for example, Figure 1 The service server 100 shown in FIG. 1 may also be executed by a terminal device (for example, the above Figure 1The terminal device 200a shown in the figure can also be executed by the business server and the terminal device interactively. For ease of understanding, the embodiment of the present application takes the method executed by the business server as an example for explanation, that is, the business server executes the data processing method as a computer device. Figure 2 As shown, the data processing method may at least include the following steps S101 to S105.

[0100] Step S101 : obtaining a training sample; the training sample includes a sample question, at least two sample answers to the sample question, and a preference label for indicating a preference relationship between the at least two sample answers.

[0101] Specifically, the sample question is processed for answer generation at least twice through the initial question-answering model to obtain at least two different sample answers; wherein, one sample answer is obtained by performing an answer generation process on the sample question once through the initial question-answering model; the sample question and at least two sample answers are sent to the terminal device, so that the terminal object corresponding to the terminal device performs preference marking on the at least two sample answers for the sample question; when the terminal device responds to the confirmation operation of the preference marking for the at least two sample answers, the preference label sent by the terminal device for indicating the preference relationship between the at least two sample answers is obtained; the sample question, the at least two sample answers and the sample label are combined into a training sample.

[0102] Please also see Figure 3 , Figure 3 This is a data processing scenario provided by the embodiment of the present application. Figure 1 .like Figure 3 As shown, the business server takes the sample question as Question 1 and inputs Question 1 into the initial question-answering model 20b. The initial question-answering model 20b is an initialized large language model. The embodiment of the present application does not limit the model structure of the initial question-answering model 20b, and can be set according to the actual application scenario, such as the Generative Pre-trained Transformer (GPT, an advanced artificial intelligence language model), Large Language Model MetaAI (Llama), Large Language and Vision Assistant (LLaVA, an end-to-end large multimodal model).

[0103] Through the initial question-answer model 20b, the business server generates an answer to question 1 and obtains a sample answer. Figure 3The example is answer 1; subsequently, the business server re-inputs question 1 into the initial question-answering model 20b, and re-generates the answer to question 1 through the initial question-answering model 20b to obtain another sample answer. Figure 3 The example is answer 2. The two answer generation processes correspond to two different answers because the initial question-answering model 20b includes a temperature parameter, which is used to adjust the degree of random sampling from the generation model (in this application, the initial question-answering model). Therefore, each time "generation" occurs, the same prompt (in this application, the sample question) can produce different outputs, which in this application are sample answers.

[0104] In the same process, the business server re-inputs question 1 into the initial question-answering model 20b, and re-generates the answer to question 1 through the initial question-answering model 20b to obtain another sample answer. Figure 3 The example is answer 3. The business server re-inputs question 1 into the initial question-answering model 20b, and re-generates the answer to question 1 through the initial question-answering model 20b to obtain another sample answer. Figure 3 The example is answer 4. Figure 3 Give at least two sample answers as Figure 3 Answer 1, Answer 2, Answer 3, and Answer 4.

[0105] Please also see Figure 4 , Figure 4 This is a data processing scenario provided by the embodiment of the present application. Figure 2 .like Figure 4 As shown, the business server takes the sample question as question 1 and takes at least two sample answers as answer 1, answer 2, answer 3 and answer 4. The business server sends question 1, answer 1, answer 2, answer 3 and answer 4 to the terminal device 200a, and the terminal device 200a can display the preference marking page, and display the marking prompt information in the preference marking page, such as Figure 4 For example, “Please give preference marks to the following answers for question 1”. Therefore, the terminal object 20a corresponding to the terminal device 200a can give preference marks to the following answers through the preference mark page. Figure 4 The four answers in the example are labeled with preferences. A feasible implementation method of preference labeling is preference sorting, such as Figure 4 As shown, the terminal object 20a ranks the four answers by preference. Specifically, for question 1, answer 1 is better than answer 2, answer 2 is better than answer 3, and answer 3 is better than answer 4. Therefore, as shown in the preference marking page, the preference marking result is answer 1>answer 2>answer 3>answer 4.

[0106] Please see again Figure 4If the terminal object 20a triggers the confirmation control, the terminal device 200a responds to the confirmation operation of the preference marking for at least two sample answers and generates a preference label 40a for at least two sample answers. Figure 4 An example preference label 40a is as follows: Answer 1 > Answer 2 > Answer 3 > Answer 4. Further, the terminal device 200a returns the preference label 40a to the service server.

[0107] Another feasible preference marking implementation method is preference type, Figure 4 The example of the four sample answers is described as an example, prompting the terminal object 20a to mark the four sample answers as positive and negative respectively. The example terminal object 20a marks both answer 1 and answer 2 as positive sample answers. The positive sample answers can be understood as the preferred sample answers described in the embodiment of this application. Answers 3 and 4 are both marked as negative sample answers. The negative sample answers can be understood as the inferior sample answers described in the embodiment of this application. Based on the above positive and negative marking operations, the terminal device 200a can generate the following preference labels: positive sample answers (answer 1, answer 2), negative sample answers (answer 3, answer 4).

[0108] Therefore, the preference relationship described in the embodiment of the present application can be a preference ranking relationship corresponding to at least two sample answers, or a positive-negative relationship corresponding to at least two sample answers.

[0109] Step S102 : Determine at least two sample answers as sample labels of the sample question respectively, and generate at least two supervised training samples according to the sample question and the at least two sample labels; wherein one supervised training sample includes a sample question carrying one sample label.

[0110] For details, please refer to Figure 5 , Figure 5 This is a data processing scenario provided by the embodiment of the present application. Figure 3 .like Figure 5 As shown, the service server obtains a training sample set 40b, which includes one or more training samples. Figure 5 The training samples 401b, 402b, 403b and 404b are shown in the example. Figure 5The process of generating each training sample in this example is the same as the process of generating the training samples described in step S101 above. Each training sample includes a sample question and at least two sample answers to the sample question. For example, training sample 401b includes question 1 and at least two sample answers to question 1, training sample 402b includes question 2 and at least two sample answers to question 2, training sample 403b includes question 3 and at least two sample answers to question 3, and training sample 404b includes question 4 and at least two sample answers to question 4.

[0111] It is understandable that the processing process of the business server for each training sample is the same, so this step is based on Figure 5 The following description is made using the training sample 401b as an example. The processing procedures for the remaining training samples may refer to the processing procedures for the training sample 401b.

[0112] Figure 5 The example training sample 401b includes question 1 and four sample answers for question 1, namely answer 1, answer 2, answer 3 and answer 4. The business server determines all four sample answers as sample labels for question 1, specifically taking answer 1 as the first sample label for question 1, so it can generate a sample label ( Figure 5 The sample label is abbreviated as the label) of the first supervised training sample; the answer 2 is used as the second sample label of question 1, so the second supervised training sample containing question 1 and the sample label generated by answer 2 can be generated; the answer 3 is used as the third sample label of question 1, so the third supervised training sample containing question 1 and the sample label generated by answer 3 can be generated; the answer 4 is used as the fourth sample label of question 1, so the fourth supervised training sample containing question 1 and the sample label generated by answer 4 can be generated.

[0113] Step S103: Adjust the parameters of the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model.

[0114] Specifically, at least two supervised training samples are input into the initial question-answering model respectively; the at least two supervised training samples include supervised training sample A b , b is a positive integer, and b is less than or equal to the total number of at least two supervised training samples; in the initial question-answering model, supervised training samples A are generated b The corresponding sample loss value is obtained; the sample loss values ​​corresponding to at least two supervised training samples are averaged to obtain a sample mean loss value; according to the sample mean loss value, the parameters in the initial question-answering model are adjusted to obtain a first question-answering model.

[0115] Among them, in the initial question-answering model, the supervised training sample A is generated b The specific process of the corresponding sample loss value may include: in the initial question-answering model, generating supervised training sample A b The fourth prediction probability corresponding to the training label in; the fourth prediction probability refers to the output of the supervised training sample A for the sample problem b The probability of the training label in ; the fourth predicted probability is logarithmically processed to obtain the initial loss value, and the negative of the initial loss value is determined as the supervised training sample A b The corresponding sample loss value.

[0116] Please see again Figure 5 , the business server inputs the four supervised training samples corresponding to the training sample 401b into the initial question-answering model 20b, specifically Figure 5 The first supervised training sample of the example, i.e. (Question 1, Label: Answer 1), is input into the initial question-answering model 20b. In the initial question-answering model 20b, a sample loss value corresponding to the supervised training sample is generated. The embodiment of the present application does not limit the method for generating the sample loss value of the supervised training sample, and can be set according to the actual application scenario, such as the cross entropy loss function or the negative log-likelihood loss function.

[0117] For ease of understanding and description, Figure 5 The negative log-likelihood loss function 40d is used as an example for description. The negative log-likelihood loss (NLL) function is shown in the following formula (1):

[0118] -logp(y|x)(1)

[0119] In Formula (1), x represents the sample question in the supervised training sample, y represents the sample label in the supervised training sample, that is, the sample answer, and p(y|x) represents the predicted probability of output y for input x, that is, the fourth predicted probability mentioned above. The loss function shown in Formula (1) encourages the model to improve the accuracy of predicting the correct output or expected output, and optimizes the model by minimizing the negative log-likelihood loss.

[0120] Please see again Figure 5 , the business server will Figure 5 The first supervised training sample of the example, i.e. (question 1, label: answer 1) is input to the initial question-answering model 20b. In the initial question-answering model 20b, the fourth predicted probability corresponding to the training label (i.e. answer 1) in the first supervised training sample is generated. Figure 5Expressed as probability 1, it means that the probability of generating answer 1 for given question 1 is probability 1. The business server substitutes probability 1 into the above formula (1), that is, performs logarithmic processing on probability 1 to obtain the initial loss value corresponding to the first supervised training sample, and determines the negative of the initial loss value as the sample loss value corresponding to the first supervised training sample. Figure 5 Take the loss value 1 as an example.

[0121] Likewise, the business server will Figure 5 The second supervised training sample of the example, i.e. (question 1, label: answer 2), is input to the initial question-answering model 20b. In the initial question-answering model 20b, a fourth predicted probability corresponding to the training label (i.e. answer 2) in the second supervised training sample is generated. Figure 5 Expressed as probability 2, that is, the probability of generating answer 2 for given question 1 is probability 2. The business server substitutes probability 2 into the above formula (1), that is, performs logarithmic processing on probability 2 to obtain the initial loss value corresponding to the second supervised training sample, and determines the negative of the initial loss value as the sample loss value corresponding to the second supervised training sample. Figure 5 Take the loss value 2 as an example.

[0122] Likewise, the business server will Figure 5 The third supervised training sample of the example, i.e. (question 1, label: answer 3), is input to the initial question-answering model 20b. In the initial question-answering model 20b, a fourth predicted probability corresponding to the training label (i.e. answer 3) in the third supervised training sample is generated. Figure 5 Expressed as probability 3, that is, the probability of generating answer 3 for given question 1 is probability 3. The business server substitutes probability 3 into the above formula (1), that is, performs logarithmic processing on probability 3 to obtain the initial loss value corresponding to the third supervised training sample, and determines the negative of the initial loss value as the sample loss value corresponding to the third supervised training sample. Figure 5 Take the loss value 3 as an example.

[0123] Likewise, the business server will Figure 5 The fourth supervised training sample of the example, i.e. (question 1, label: answer 4), is input to the initial question-answering model 20b. In the initial question-answering model 20b, the fourth predicted probability corresponding to the training label (i.e. answer 4) in the fourth supervised training sample is generated. Figure 5 Expressed as probability 4, the probability of generating answer 4 for a given question 1 is probability 4. The business server substitutes probability 4 into the above formula (1), that is, performs logarithmic processing on probability 4 to obtain the initial loss value corresponding to the fourth supervised training sample, and determines the negative of the initial loss value as the sample loss value corresponding to the fourth supervised training sample. Figure 5 Take the loss value 4 as an example.

[0124] Furthermore, the business server Figure 5 The four sample loss values ​​(i.e., loss value 1, loss value 2, loss value 3, and loss value 4) of the example are averaged to obtain the sample average loss value 1 corresponding to the four supervised training samples, which can also be understood as the sample average loss value 1 of the training sample 401b in the supervised fine-tuning stage. In the same process, the business server can obtain the sample average loss values ​​corresponding to the remaining training samples in the training sample set 40b, such as Figure 5 As shown, the sample mean loss value of training sample 402b in the supervised fine-tuning stage is 2, the sample mean loss value of training sample 403b in the supervised fine-tuning stage is 3, and the sample mean loss value of training sample 404b in the supervised fine-tuning stage is 4.

[0125] The service server fuses sample mean loss values ​​1, 2, 3, and 4 to obtain a model loss value corresponding to initial question-answering model 20b. The fusion process includes, but is not limited to, summation and averaging. Based on the model loss values, the parameters of initial question-answering model 20b are adjusted to obtain first question-answering model 40c.

[0126] The embodiments of the present application relate to the fields of artificial intelligence and natural language processing, and specifically to a reinforcement learning preference optimization method based on dynamic optimization. The core concept is to propose a compression effect mitigation optimization mechanism to improve the alignment performance and generation quality of large language models in reinforcement learning fine-tuning methods such as direct preference optimization.

[0127] During preference fine-tuning, especially in reinforcement learning optimization methods such as Dependent Positive Order (DPO), large language models typically impose large bias weights when processing negative sample answers (non-preferred responses). This operation causes "weight bias" in the model's probability distribution, that is, the generation probability is excessively squeezed to a very small number of high-confidence responses (answers), causing the model output distribution to become extreme, manifesting problems such as decreased generalization ability, repeated output, and hallucinations.

[0128] To address these issues and mitigate the biased optimization mechanism, this step innovatively introduces rejected sample responses (i.e., suboptimal sample answers, also known as negative sample answers) into the supervised fine-tuning (SFT) phase. This improves the model's acceptance of low-probability responses before entering reinforcement learning, mitigating the resulting probability distribution distortion. This method is simple to implement, compatible with existing training processes, and significantly reduces the impact of compression effects, improving model training stability, generative diversity, and alignment with human preferences.

[0129] In step S104, the training sample is input into the first question-answering model, and in the first question-answering model, first prediction probabilities corresponding to at least two sample answers are generated; wherein a first prediction probability refers to the probability of outputting a sample answer for the sample question.

[0130] Specifically, in Direct Preference Optimization (DPO), when a negative gradient is applied to a low-probability response (i.e., an inferior sample answer) during model training, due to its very low original probability, the excessively strong gradient update can lead to an extreme concentration of the probability distribution (i.e., the compression effect), which in turn causes the following problems:

[0131] 1. Output confidence decreases: Not only the confidence of the inferior sample answer decreases, but the confidence of the optimal response (i.e., the optimal sample answer) may also decrease.

[0132] 2. Generate duplicate content: Model output tends to be extreme and unstable, lacking diversity.

[0133] The core innovation of the embodiment of the present application is to introduce inferior sample answers in the SFT stage, so that the model has a high acceptance of low-probability responses (i.e., inferior sample answers) when entering the DPO stage, thereby avoiding the probability collapse caused by excessively strong negative gradient updates in the DPO stage.

[0134] To sum up, after executing step S103, executing this step, that is, in the first question-answering model, the first prediction probability corresponding to at least two sample answers generated will be higher than the prediction probability of the inferior sample answer obtained by directly entering the DPO stage.

[0135] Step S105: Adjust the parameters in the first question-answering model according to the preference label and the first predicted probabilities corresponding to at least two sample answers to obtain a second question-answering model; the second question-answering model is used to generate an output answer to the input question.

[0136] Specifically, based on the preference label and the first predicted probabilities corresponding to at least two sample answers, a sample loss value corresponding to the training sample is generated; based on the sample loss value, the parameters in the first question-answering model are adjusted to obtain the second question-answering model.

[0137] Among them, the specific process of generating the sample loss value corresponding to the training sample based on the preference label and the first prediction probability corresponding to at least two sample answers can include: determining the preferred sample answer and the inferior sample answer among at least two sample answers based on the preference label; the answer quality corresponding to the preferred sample answer is better than the answer quality corresponding to the inferior sample answer; obtaining the second prediction probability output by the reference model for the preferred sample answer, and the third prediction probability output by the reference model for the inferior sample answer; generating a preferred log-likelihood ratio based on the first prediction probability and the second prediction probability corresponding to the preferred sample answer; generating an inferior log-likelihood ratio based on the first prediction probability and the third prediction probability corresponding to the inferior sample answer; generating a sample loss value corresponding to the training sample based on the preferred log-likelihood ratio and the inferior log-likelihood ratio.

[0138] Among them, the specific process of generating the preferred log-likelihood ratio based on the first predicted probability and the second predicted probability corresponding to the preferred sample answer may include: performing ratio processing on the first predicted probability and the second predicted probability corresponding to the preferred sample answer to obtain a probability ratio; performing logarithmic processing on the probability ratio to obtain a log-likelihood ratio; and performing weighted processing on the log-likelihood ratio to obtain a preferred log-likelihood ratio.

[0139] Among them, the specific process of generating the sample loss value corresponding to the training sample based on the preferred log-likelihood ratio and the inferior log-likelihood ratio may include: performing difference processing on the preferred log-likelihood ratio and the inferior log-likelihood ratio to obtain the log-likelihood ratio difference; normalizing the log-likelihood ratio difference to obtain the preference probability, performing logarithmic processing on the preference probability to obtain the initial loss value; and determining the negative of the initial loss value as the sample loss value corresponding to the training sample.

[0140] At least two sample answers can be divided into two categories, one is the preferred sample answer, which can also be called a positive sample answer or a preferred response, and the other is the inferior sample answer, which can also be called a negative sample answer or a inferior response.

[0141] The preference label is used to indicate the preference relationship between at least two sample answers. If the preference relationship is a preference ranking relationship, please refer to Figure 4 , preference label 40a is answer 1 > answer 2 > answer 3 > answer 4, indicating that for question 1, answer 1 is better than answer 2, answer 2 is better than answer 3, and answer 3 is better than answer 4. Based on preference label 40a, the business server can determine answer 1 and answer 2 as preferred sample answers, and answer 3 and answer 4 as inferior sample answers.

[0142] In practical applications, the total number of at least two sample answers can be an odd number F greater than 2, and the preference sorting is to put the higher preference in front and the lower preference in the back, such as Figure 4In the example preference tag 40a, the business server may determine the sample answers with sorting positions from 1 to (F-1) / 2 as preferred sample answers, and determine the sample answers with sorting positions from (F+1) / 2 to F as inferior sample answers.

[0143] If the total number of at least two sample answers is an even number G equal to or greater than 2, and the preference sorting is to put the higher preference in front and the lower preference in the back, such as Figure 4 For the example preference tag 40a, the business server may determine the sample answers with sorting positions from 1 to G / 2 as preferred sample answers, and determine the sample answers with sorting positions from G / 2+1 to G as inferior sample answers.

[0144] If the preference relationship is a positive-negative relationship, as described in step S101 above, the preference labels are as follows: positive sample answers (answer 1, answer 2), negative sample answers (answer 3, answer 4), then the business server can determine that answer 1 and answer 2 are positive sample answers, that is, preferred sample answers, and determine that answer 3 and answer 4 are negative sample answers, that is, inferior sample answers, based on the preference labels.

[0145] After determining the preferred sample answer and the inferior sample answer among at least two sample answers, the sample loss value corresponding to the training sample can be determined according to the following formula (2).

[0146]

[0147] In formula (2), x represents the sample problem, y + represents the preferred sample answer, π θ represents the first question-answering model, π ref Represents the reference model. The embodiment of the present application does not limit the model structure of the reference model, and can be set according to actual applications. It can be the same as the initial question-answer model or different from the initial question-answer model, the same as the first question-answer model or different from the first question-answer model. However, in the DPO stage, the model parameters of the reference model remain unchanged.

[0148] π in formula (2) θ (y + |x) represents the first predicted probability corresponding to the preferred sample answer, π ref (y + |x) represents the second predicted probability output by the reference model for the preferred sample answer. - represents the poor sample answer, π θ (y - |x) represents the first predicted probability corresponding to the answer of the inferior sample, π ref (y - |x) represents the third predicted probability output by the reference model for the answer to the inferior sample.

[0149] In formula (2) represents the probability ratio corresponding to the preferred sample answer, Represents the log-likelihood ratio corresponding to the preferred sample answer. This ratio measures the preference of the model to be trained (the first question-answering model in this application) relative to the reference model in generating the preferred sample answer. If the first question-answering model is more inclined to generate the preferred sample answer, this ratio will be larger. β is a hyperparameter that controls the contrast loss. It represents the contrast strength between the preferred response and the inferior response, and can also be called the preference adjustment parameter. represents the preferred log-likelihood ratio.

[0150] In formula (2) Represents the probability ratio corresponding to the poorly selected sample answer, represents the log-likelihood ratio corresponding to the poorly selected sample answer, represents the inferior choice log-likelihood ratio, Represents the difference in log-likelihood ratios. This expression encourages the first question-answering model to increase the probability of generating answers to preferred samples while reducing the probability of generating answers to inferior samples.

[0151] The σ in formula (2) represents the activation of the Sigmoid function, and the preference-adjusted log-likelihood ratio difference is mapped to the (0, 1) interval through the Sigmoid function, that is, the preference-adjusted log-likelihood ratio difference is converted into a probability value, which represents the degree of preference of the model for generating preferred sample answers relative to inferior sample answers. The embodiment of this application is referred to as the preference probability. Logσ represents the logarithmic loss function, which is a form of binary cross entropy loss. It measures the difference between the probability of the model output and the target label. The goal of this application is to make the first question-answering model more inclined to generate preferred sample answers. -logσ is to convert the log-likelihood ratio after maximizing the preference adjustment into a form of minimizing the loss. In the model optimization process, the loss function is usually minimized, so the negative sign is taken to make the optimization target consistent with increasing the probability of generating preferred sample answers.

[0152] If there are multiple training samples, the sample loss values ​​corresponding to the multiple training samples are averaged to obtain the model loss value corresponding to the first question-answering model. This process can be expressed by the following formula (3):

[0153]

[0154] Among them, E in formula (3) usually represents the expected value (Expectation), and formula (3) is a training sample set D (for example Figure 5 The expected value of the loss function of the training sample set 40b) in the example. In practical applications, the average loss of all training samples is usually used as the model loss value.

[0155] Because the pre-trained model will apply a large negative gradient to the inferior responses with low prediction probabilities during the preference alignment training process, this operation can easily lead to "weight bias" in the model's probability distribution, that is, the confidence of most responses generally decreases, and the prediction probability is forcibly concentrated on a very small number of high-confidence outputs.

[0156] This can lead to problems like:

[0157] 1. Significant weight bias damages model output quality

[0158] During DPO training, the model applies negative gradients to reduce the confidence in poorly selected responses. When these responses have extremely low probability in the current model, strong negative gradients further concentrate the model's output distribution, causing the confidence in other responses to decrease simultaneously. This "compression effect" can lead to duplicated generated content, rigid structures, and missing semantics, severely impacting the model's actual alignment effectiveness.

[0159] 2. Positive and negative samples come from the trough area of ​​the distribution, and the gradient update is unstable

[0160] DPO is typically trained on a fixed, off-policy sampled dataset. The optimal and inferior responses often do not come from the current model distribution. This bias causes the model to frequently update samples with very low confidence, amplifying the compression effect and exacerbating training instability and performance degradation.

[0161] 3. Existing mitigation solutions have limited effectiveness and lack universality and theoretical support

[0162] At present, some studies have attempted to alleviate the compression phenomenon by reducing the learning rate, adjusting the loss weight, adding regularization terms, etc. However, these methods generally rely on empirical adjustments, lack theoretical characterization of the essence of the phenomenon, and are difficult to generalize and apply in different model architectures and data scenarios.

[0163] Based on the above problems, an improved solution that can be directly integrated into the existing DPO framework is needed to alleviate the negative impact of the compression effect and improve the model alignment effect and generation quality. The embodiment of this application proposes a reinforcement learning preference optimization method based on dynamic optimization, which can effectively overcome the above defects and mainly solve the following technical problems:

[0164] 1. Alleviate the compression effect from the root: The embodiment of the present application introduces a "learning dynamics" analysis method to systematically reveal the cause and transmission path of the compression effect, clarifying that it is mainly caused by applying a negative gradient to extremely low-probability inferior selection responses, and accordingly designs a pre-mitigation mechanism.

[0165] 2. Optimize the SFT stage structure and reduce the risk of DPO shock: The embodiment of the present application actively introduces inferior responses to participate in training in the SFT (supervised fine-tuning) stage to improve their initial probability, thereby significantly reducing the gradient strength when negatively suppressing them in the DPO stage, effectively avoiding probability collapse.

[0166] The above method is concise and highly integrable. It does not require modifying the main structure of DPO training and can be implemented simply by adjusting the composition of training samples in the SFT phase. It has good engineering feasibility and compatibility with existing systems. In summary, the embodiments of the present application not only solve the performance degradation problem caused by the compression effect in the reinforcement learning process of existing preference optimization methods, but also significantly improve the quality of model alignment and output stability, and have broad practical value and promotion prospects.

[0167] From the above, it can be seen that in the supervised fine-tuning stage, the embodiment of the present application uses all sample answers (including sample answers with low preference) as sample labels, that is, each sample answer is regarded as the correct answer. Therefore, the first question-answering model obtained by training can improve the generation probability and confidence of each sample answer (including sample answers with low preference), which can improve the output diversity and stability of the first question-answering model; since the generation probability and confidence of sample answers with low preference have been improved in the supervised fine-tuning stage, the suppression of sample answers with low preference in the preference alignment stage is smoother, avoiding the collapse of the predicted probability of sample answers with low preference due to model parameter adjustment, and thus avoiding the predicted probability from being excessively squeezed to a very small number of high-confidence sample answers (i.e., sample answers with high preference), so the model training stability and generation diversity can be improved.

[0168] See Figure 6 , Figure 6 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 This method can be performed by a business server (for example, Figure 1 The service server 100 shown in FIG. 1 may also be executed by a terminal device (for example, the above Figure 1 The terminal device 200a shown in the figure can also be executed by the business server and the terminal device interactively. For ease of understanding, the embodiment of the present application takes the method executed by the business server as an example for explanation, that is, the business server executes the data processing method as a computer device. Figure 6 As shown, the process of the data processing method includes steps S201 to S208.

[0169] Step S201 : obtaining a training sample; the training sample includes a sample question, at least two sample answers to the sample question, and a preference label for indicating a preference relationship between the at least two sample answers.

[0170] Specifically, the initial question-answering model is used to generate answers to the sample question at least twice to obtain at least two different sample answers; wherein, one sample answer is obtained by performing an answer generation process on the sample question once through the initial question-answering model; the sample question and at least two sample answers are sent to the target device so that the target device inputs the sample question and at least two sample answers into the question-answering recognition model, which is a large language model but another model different from the initial question-answering model; for the sample question, the question-answering recognition model is used to perform preference labeling on the at least two sample answers, and one feasible way of preference labeling is to perform preference ranking on the at least two sample answers to obtain a preference label for indicating the preference ranking between the at least two sample answers. Another feasible way of preference labeling is to determine a positive sample answer (i.e., a preferred sample answer) and a negative sample answer (i.e., an inferior sample answer) in the at least two sample answers, obtain the preference label sent by the target device, and combine the sample question, at least two sample answers, and the sample label into a training sample.

[0171] It is understandable that if the business server itself stores the above-mentioned question-answer recognition model, the above-mentioned method can be executed locally.

[0172] Step S202 : Determine at least two sample answers as sample labels of the sample question respectively, and generate at least two supervised training samples according to the sample question and the at least two sample labels; wherein one supervised training sample includes a sample question carrying one sample label.

[0173] Specifically, at least two supervised training samples are input into the initial question-answering model respectively; the at least two supervised training samples include supervised training sample A b , b is a positive integer, and b is less than or equal to the total number of at least two supervised training samples; in the initial question-answering model, for the supervised training sample A b Identify the sample questions in and get the estimated answer; b The training labels in are processed for loss, and the supervised training sample A is obtained b The corresponding sample loss value is obtained; the sample loss values ​​corresponding to at least two supervised training samples are averaged to obtain the model loss value; according to the model loss value, the parameters in the initial question-answering model are adjusted to obtain the first question-answering model.

[0174] Step S203: Adjust the parameters of the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model.

[0175] Specifically, consider the model output probability distribution expressed as the following formula (4):

[0176] πθ (y|x)=Softmax(h θ (x)) (4)

[0177] Among them, θ in formula (4) represents the parameters of the model, h θ (x)∈R V Represents the model's internal representation of the input x (such as a sentence or a text). The internal representation is a vector, usually called logits, which is the original output of the model before the output layer and has not yet been converted into a probability distribution by the softmax function; R V represents a V-dimensional real vector space, where V is the size of the vocabulary.

[0178] Under the standard gradient descent (or ascent) update, we care about a training sample (x u ,y u ) for another input x o The output change can be expressed as follows using the learning dynamic differential formula (5):

[0179] Δlogπ θ (y|x o )≈-η·A(x o )·K(x o , x u )·G(x u ,y u )(5)

[0180] Where, Δlogπ in formula (5) θ (y|x o ) means that when input x o The change in the logarithmic probability of the model output y under the condition of . Η represents the learning rate, which represents the step size of the gradient descent (or ascent). A(x o ) means that when input x o The amount by which the model adjusts the output probability distribution under the given conditions. This is usually related to the gradient of the model’s logits. o , x u ) represents the input x o and training samples x u The similarity or correlation between them is usually related to the internal representation of the model (such as feature vectors). u ,y u ) indicates that in the training sample (x u ,y u ), the gradient of the model, which is usually related to the loss function and the parameters of the model.

[0181] During the gradient descent (or ascent) update process, the model parameters θ will be updated according to the gradient Adjust, where L is the loss function. For a given training sample (x u ,y u ),gradient It will reflect the error or bias of the model on this sample. When considering another input x o When the embodiment of the present application is concerned with the model in x o How is the output probability distribution affected by the training samples (x u ,y u ) can be affected by the gradient Propagate to the model parameters θ, which in turn affects x o The output probability distribution under .

[0182] Through differential expression, that is, the above formula (5), it can be understood that under the gradient descent (or ascent) update, a training sample (x u ,y u ) for another input x o The output change of this method depends on the learning rate, adjustment amplitude, similarity and gradient. The embodiment of this application accurately models the learning dynamic process and designs an optimization strategy to alleviate the compression effect based on the root cause of the compression effect. The core idea is to introduce the poor sample answer (y - ), providing more stable gradient updates for the subsequent DPO stage.

[0183] In Direct Preference Optimization (DPO), when the model is trained for low-probability responses (y - ) When applying a negative gradient, since its original probability is very low, an overly strong gradient update will lead to an extreme concentration of the probability distribution (i.e., the compression effect), which in turn causes the following problems:

[0184] 1. Output confidence decreases: Not only the confidence of the inferior response (i.e., the inferior sample answer) decreases, but also the confidence of the optimal response (y + The confidence level of the sample answer may also decrease.

[0185] 2. Generate duplicate content: Model output tends to be extreme and unstable, lacking diversity.

[0186] The core innovation of the embodiment of the present application is to introduce the inferior response (y - ), so that the model has a good acceptance of low-probability responses when entering the DPO stage, thereby avoiding the probability collapse caused by excessively strong negative gradient updates in the DPO stage.

[0187] In the SFT stage, the goal is to update the parameters θ in the initial question-answering model by minimizing the negative log-likelihood loss. The embodiment of the present application modifies the traditional SFT loss and adds the inferior response y- , so that the model not only responds to the optimal response y during optimization + Improve confidence and also respond to poor choices - Provide reasonable training goals.

[0188] The answer y for the inferior sample - The additional training enables the initial question-answering model to gradually increase the confidence of low-probability responses during the optimization process, avoiding the compression effect caused by excessive negative gradients during subsequent DPO training.

[0189] Positive and negative sample answers are co-trained, that is, the preferred responses and the inferior responses are jointly optimized in the same training process, which improves the diversity and stability of the output of the first question-answering model. This design of extended supervised sequence samples helps to increase the training of low-probability responses in the SFT stage, so that the model's ability to accept inferior sample answers is enhanced in advance.

[0190] In step S204, the training sample is input into the first question-answering model, and in the first question-answering model, first prediction probabilities corresponding to at least two sample answers are generated; wherein a first prediction probability refers to the probability of outputting a sample answer for the sample question.

[0191] Specifically, when the first question-answering model enters the DPO stage, it continues to use the DPO optimization framework and uses the fine-tuned model from the SFT stage (i.e., the first question-answering model) to perform preference optimization. One of the innovations of the embodiment of the present application is the introduction of inferior sample answers in the SFT stage. Through preliminary training, the first question-answering model can suppress inferior sample answers (i.e., negative sample answers) more smoothly when entering the DPO stage, avoiding excessive compression.

[0192] In the DPO phase, we usually respond to low probability - (ie, negative sample answer) applies a negative gradient, but if y - The probability in the original distribution of the model is extremely low (i.e., π θ (y - ) is very small), negative gradients can lead to a compression effect. This compression effect concentrates the probability of the entire output distribution on a few high-probability responses, causing the model's output to become sharp and repetitive. By comparing the gradient updates of the model with and without the introduction of inferior responses, the method provided in the embodiments of this application can effectively reduce the impact of the compression effect.

[0193] Step S205 : Generate a sample loss value corresponding to the training sample based on the preference label and the first prediction probabilities corresponding to the at least two sample answers.

[0194] For the specific implementation process of step S205, please refer to the above Figure 2Step S105 in the corresponding embodiment is not described in detail here.

[0195] Step S206: According to the preference label, obtain the inferior sample answer from at least two sample answers, and determine the training risk value of the training sample according to the first predicted probability corresponding to the inferior sample answer; the training risk value is used to indicate the disturbance intensity of the training sample on the model output distribution of the first question-answering model.

[0196] Specifically, the first predicted probability corresponding to the answer to the inferior sample is logarithmically processed to obtain a log-likelihood value; the negative of the log-likelihood value is determined as the training risk value of the training sample; there is an inverse relationship between the training risk value and the first predicted probability corresponding to the answer to the inferior sample.

[0197] Specifically, the number of the inferior sample answers is at least two, and the at least two inferior sample answers include the inferior sample answer C d , d is a positive integer, and d is less than or equal to the total number of at least two inferior sample answers; for inferior sample answers C d The corresponding first predicted probability is processed logarithmically to obtain the log-likelihood value; the negative of the log-likelihood value is determined as the poor sample answer C d Corresponding training risk value; fusing the training risk values ​​corresponding to at least two inferior sample answers to obtain the training risk value of the training sample.

[0198] Although the poor sample answer y - Even though they are used as training labels, there is still an uneven "influence" between samples during the DPO phase. That is, some training samples bring a higher risk of compression when updated, while others cause almost no distribution distortion. Dynamically identifying "high-risk samples" and reducing their weight during DPO training, while prioritizing "safe samples," can further stabilize the optimization process.

[0199] Based on the above, the embodiment of the present application introduces a training risk value, which can also be called a gradient influence score, to measure the disturbance intensity of the training sample on the model output distribution of the first question-answering model, and adjusts the weight of each training sample during training accordingly, to achieve a training strategy of "light compression first, heavy compression suppression".

[0200] For each training sample (x, y + ,y - ), the embodiment of the present application measures the training risk value of the possible compression effect caused by the training sample through the following formula (6):

[0201] Score(x,y - )=-logπ θ (y - ∣x) (6)

[0202] Among them, π in formula (6) θ (y - |x) represents the first predicted probability corresponding to the answer of the inferior sample.

[0203] The training risk value (also known as the gradient influence) reflects the model's current response to the poorly selected sample y - The "confidence level is low" if π θ (y - ∣x) is very small, indicating that the model does not "believe" the answer y of the inferior sample - , then it is easier to cause compression effect when applying negative gradient to it in DPO, which means that the lower the prediction probability (the larger the negative log probability), the higher the compression risk, so the greater the training risk value.

[0204] There may be one inferior sample answer in a training sample. In the case of only one inferior sample answer, the training risk value corresponding to the training sample can be determined by formula (6); there may be multiple inferior sample answers in a training sample, as shown above. Figure 4 as well as Figure 5 Answers 3 and 4 in the example are both poorly selected sample answers. In this scenario, the training risk value corresponding to each poorly selected sample answer can be determined using formula (6), and then the training risk values ​​corresponding to multiple poorly selected sample answers can be fused to obtain the training risk value of the training sample. The fusion processing includes but is not limited to mean processing, sum processing, and weighted sum processing.

[0205] Step S207: Optimize the sample loss value according to the training risk value to obtain a sample optimized loss value corresponding to the training sample; the sample optimized loss value is less than the sample loss value.

[0206] Specifically, the training weight corresponding to the training sample is determined according to the training risk value; there is an inverse relationship between the training risk value and the training weight; the sample loss value is weighted according to the training weight to obtain the sample optimization loss value corresponding to the training sample.

[0207] Among them, the specific process of determining the training weight corresponding to the training sample based on the training risk value may include: weighting the training risk value according to the weighted intensity parameter to obtain the weighted training risk value; determining the negative of the weighted training risk value as the risk value to be smoothed, performing exponential processing on the risk value to be smoothed, and obtaining the training weight corresponding to the training sample.

[0208] The embodiment of the present application does not limit the method of determining the training weight corresponding to the training sample based on the training risk value. It can be set according to the actual application scenario, as long as the condition that there is an inverse relationship between the training risk value and the training weight is met.

[0209] For ease of understanding and description, the embodiment of the present application adopts a smooth weighting function to adjust the training weight of the training sample during training, as shown in formula (7):

[0210] ω(x,y - )=exp(-α*score(x,y - )) (7)

[0211] Wherein, α>0 in formula (7) represents a hyperparameter controlling the weighted intensity, and exp represents a natural exponential function, i.e., an exponential function with the real number e (e≈2.71828) as its base. Throughout this application, the base of log is the real number e.

[0212] When π θ (y - |x) is very low, the training risk value is large, but the training weight will be pressed even smaller; in this embodiment of the application, the poor sample answer y - has been trained in advance (in the SFT stage), at this time, π θ (y - |x) is larger, so the training risk value is relatively small, and the training weight will increase.

[0213] In the scenario where the training sample only includes one inferior sample answer, combining formula (6) and formula (7), the following formula (8) can be derived:

[0214]

[0215] Another feasible way to determine the training weight corresponding to the training sample according to the training risk value is as shown in the following formula (9):

[0216]

[0217] In step S205, the business server generates a sample loss value corresponding to the training sample. For details, please refer to the above Figure 2 In the corresponding embodiment, the description of step S105 is combined with the above formula (2), and the sample optimization loss value corresponding to the training sample can be generated by the following formula (10):

[0218] L DPO _ weighted =ω(x,y - ) * L DPO (θ) (10)

[0219] In step S208, the parameters in the first question-answering model are adjusted according to the sample optimization loss value to obtain a question-answering model; the second question-answering model is used to generate an output answer to the input question.

[0220] Specifically, it can be understood that the principle of adjusting the parameters in the first question-answering model according to the sample loss value is the same as the principle of adjusting the parameters in the first question-answering model according to the sample optimization loss value, so it will not be repeated here.

[0221] Figure 6 The corresponding embodiment can be summarized as follows:

[0222] 1. Data preparation:

[0223] According to the training samples, supervised training samples are constructed. The training labels of supervised training samples include the preferred response (i.e., the preferred sample answer) y + and inferior response (ie inferior sample answer) y - .

[0224] 2. SFT stage training:

[0225] The loss function is minimized using supervised training samples, and the initial question-answering model is optimized through gradient updates.

[0226] 3. DPO stage training:

[0227] Preference optimization is performed on the first question-answering model after SFT fine-tuning, and DPO loss function is used for training.

[0228] 4. Dynamic sample reweighting:

[0229] Dynamically identify high-risk training samples and reduce their training weights during training to further stabilize the optimization process.

[0230] 5. Output generation:

[0231] The final trained second question-answering model is used for inference or generation to output responses that meet human preferences.

[0232] The embodiment of the present application uses an innovative design to introduce a training mechanism for inferior response in the SFT stage, thereby alleviating the compression effect in the DPO process from the source and improving the model generation quality and alignment performance.

[0233] The following briefly describes the reasoning process of the second question-answering model. The question text is obtained, and the business server inputs the question text into the second question-answering model. The second question-answering model extracts features from the question text to obtain question features.

[0234] Specifically, the second question-answering model has a retrieval-augmented generation function, which can generate answer text for question text through the retrieval-augmented generation function.

[0235] Using the second question-answering model, the business server extracts features from multiple search texts to obtain multiple search features. A search feature is obtained by extracting features from a search text using the question-answering model. The business server matches each search feature with the question feature to obtain a matching probability for each search feature. Based on the multiple matching probabilities, the business server determines a first search feature from the multiple search features. The matching probability corresponding to the first search feature is greater than or equal to the matching probability corresponding to the second search feature. The second search feature is a search feature other than the first search feature from the multiple search features.

[0236] The service server obtains the search text represented by the first search feature from among the multiple search texts, and generates an answer text for replying to the question text according to the search text represented by the first search feature.

[0237] Specifically, the search text represented by the first search feature is summarized to obtain a summary answer; the search text represented by the first search feature is abstracted to obtain a detailed answer; the search text represented by the first search feature is classified to obtain a list answer; the source platform that provides the search text represented by the first search feature is determined, and based on the summary answer, detailed answer, list answer and source platform, an answer text for replying to the question text is generated.

[0238] This application provides a reinforcement learning preference optimization method based on dynamic optimization, especially the introduction of a preference mitigation mechanism and a dynamic sample reweighting strategy, which has significant beneficial effects, including the following:

[0239] 1. Effectively alleviate compression effects and improve model generation quality

[0240] This application introduces inferior response (y - ) training effectively improves the first question-answering model's ability to accept low-probability responses, reducing the "bias effect" caused by applying excessively strong negative gradients to inferior responses during the DPO phase. This mechanism avoids the problem of extreme model output and lack of diversity, significantly improving the quality of generated responses.

[0241] Specific performance: During the SFT training process, the initial question-answering model not only learns the preferred response (y +), and by optimizing the inferior responses, the confidence of the model output is more balanced, avoiding excessive concentration on a few responses, and the generated content is more diverse and natural.

[0242] 2. Improve training stability and reduce instability during training

[0243] By introducing a dynamic sample reweighting mechanism, this application dynamically adjusts the training weight of each sample according to its gradient influence (i.e., training risk value) during the DPO training process, so that during the optimization process, the first question-answering model prioritizes reinforcement learning of training samples with lower "risk", thereby avoiding the instability of gradient updates caused by high-risk samples.

[0244] Specific performance: When training the first question-answering model, lower weights are applied to "high-influence" samples (i.e., samples that easily cause compression effects), making the training process smoother and avoiding performance fluctuations caused by applying excessively large negative gradients to extreme samples.

[0245] 3. Wide range of applications, compatible with various reinforcement learning optimization methods

[0246] The solution of this application can not only optimize the traditional DPO (direct preference optimization) method, but also be compatible with other reinforcement learning preference optimization methods such as RLHF and PPO (full name Proximal Policy Optimization, a policy optimization algorithm, which belongs to the policy gradient method in reinforcement learning. It maximizes cumulative rewards by optimizing strategies while avoiding instability caused by excessive strategy updates), and has strong versatility and adaptability.

[0247] Specific performance: Through in-depth analysis of learning dynamics, this application provides a general improvement strategy for various reinforcement learning optimization methods, further broadening their application scenarios. It is not limited to preference alignment of large language models, but can also be extended to other fields that require model alignment, such as recommendation systems, personalized dialogue generation, etc.

[0248] In summary, this application significantly improves the performance of large language models in preference optimization tasks through an innovative compression effect mitigation mechanism and dynamic sample reweighting strategy, improves generation quality, enhances training stability, and optimizes the model's alignment with human preferences. This solution is not only simple and efficient, but also has wide applicability and strong versatility.

[0249] Further, see Figure 7 , Figure 7 1 is a structural diagram of a data processing device provided in an embodiment of the present application. The above-mentioned data processing device 1 can be used to execute the corresponding steps in the method provided in an embodiment of the present application. Figure 7As shown, the data processing device 1 may include: an acquisition module 11 , a generation module 12 and an adjustment module 13 .

[0250] An acquisition module 11 is configured to acquire a training sample; the training sample includes a sample question, at least two sample answers to the sample question, and a preference label indicating a preference relationship between the at least two sample answers;

[0251] A generating module 12 is configured to determine the at least two sample answers as sample labels of the sample question, and generate at least two supervised training samples based on the sample question and the at least two sample labels; wherein one supervised training sample includes a sample question carrying one sample label;

[0252] An adjustment module 13 is configured to adjust parameters in the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model;

[0253] The generation module 12 is further configured to input the training sample into the first question-answering model, and generate first predicted probabilities corresponding to at least two sample answers in the first question-answering model; wherein a first predicted probability refers to the probability of outputting a sample answer for the sample question;

[0254] The adjustment module 13 is also used to adjust the parameters in the first question-answering model according to the preference label and the first predicted probabilities corresponding to at least two sample answers to obtain a second question-answering model; the second question-answering model is used to generate an output answer to the input question.

[0255] In one possible implementation, the adjustment module 13 adjusts the parameters in the first question-answering model based on the preference label and the first predicted probabilities corresponding to the at least two sample answers, to obtain a second question-answering model for performing the following operations:

[0256] Generate a sample loss value corresponding to the training sample based on the preference label and the first predicted probabilities corresponding to the at least two sample answers;

[0257] Obtaining a poorly selected sample answer from at least two sample answers based on the preference label, and determining a training risk value for the training sample based on a first predicted probability corresponding to the poorly selected sample answer; the training risk value is used to indicate the intensity of the disturbance of the training sample on the model output distribution of the first question-answering model;

[0258] According to the training risk value, the sample loss value is optimized to obtain the sample optimization loss value corresponding to the training sample; the sample optimization loss value is less than the sample loss value;

[0259] According to the sample optimization loss value, the parameters in the first question-answering model are adjusted to obtain the question-answering model.

[0260] In a possible implementation, the adjustment module 13 determines the training risk value of the training sample according to the first predicted probability corresponding to the answer of the inferior sample, and is configured to perform the following operations:

[0261] Perform logarithmic processing on the first predicted probability corresponding to the answer of the inferior sample to obtain the log-likelihood value;

[0262] The negative of the log-likelihood value is determined as the training risk value of the training sample; there is an inverse relationship between the training risk value and the first predicted probability corresponding to the answer of the inferior sample.

[0263] In a possible implementation, the number of the inferior sample answers is at least two, and the at least two inferior sample answers include the inferior sample answer C d , d is a positive integer, and d is less than or equal to the total number of at least two inferior sample answers;

[0264] The adjustment module 13 determines the training risk value of the training sample based on the first predicted probability corresponding to the answer of the inferior sample, and is used to perform the following operations:

[0265] Answer C for the inferior sample d The corresponding first predicted probability is logarithmically processed to obtain a log-likelihood value;

[0266] Determine the negative value of the log-likelihood value as the poor choice sample answer C d Corresponding training risk value;

[0267] The training risk values ​​corresponding to at least two poorly selected sample answers are fused to obtain the training risk value of the training sample.

[0268] In a possible implementation, the adjustment module 13 optimizes the sample loss value according to the training risk value to obtain the sample optimized loss value corresponding to the training sample, which is used to perform the following operations:

[0269] According to the training risk value, the training weight corresponding to the training sample is determined; there is an inverse relationship between the training risk value and the training weight;

[0270] The sample loss value is weighted according to the training weight to obtain the sample optimization loss value corresponding to the training sample.

[0271] In a possible implementation, the adjustment module 13 determines the training weight corresponding to the training sample according to the training risk value, and is used to perform the following operations:

[0272] The training risk value is weighted according to the weighted intensity parameter to obtain a weighted training risk value;

[0273] The negative of the weighted training risk value is determined as the risk value to be smoothed, and the smoothed risk value is subjected to exponential processing to obtain the training weight corresponding to the training sample.

[0274] In one possible implementation, the adjustment module 13 generates a sample loss value corresponding to the training sample based on the preference label and the first predicted probabilities corresponding to the at least two sample answers, and performs the following operations:

[0275] According to the preference label, a preferred sample answer and a less preferred sample answer are determined from at least two sample answers; the quality of the answer corresponding to the preferred sample answer is better than the quality of the answer corresponding to the less preferred sample answer;

[0276] Obtaining a second predicted probability output by the reference model for the preferred sample answer and a third predicted probability output by the reference model for the inferior sample answer;

[0277] Generate a preferred log-likelihood ratio based on the first predicted probability and the second predicted probability corresponding to the preferred sample answer;

[0278] Generate a poor choice log likelihood ratio based on the first predicted probability and the third predicted probability corresponding to the poor choice sample answer;

[0279] According to the optimal log-likelihood ratio and the inferior log-likelihood ratio, the sample loss value corresponding to the training sample is generated.

[0280] In a possible implementation, the adjustment module 13 generates a preferred log-likelihood ratio based on the first predicted probability and the second predicted probability corresponding to the preferred sample answer, and performs the following operations:

[0281] Perform ratio processing on the first predicted probability and the second predicted probability corresponding to the preferred sample answer to obtain a probability ratio;

[0282] The probability ratio is logarithmically processed to obtain the log likelihood ratio, and the log likelihood ratio is weighted to obtain the optimal log likelihood ratio.

[0283] In a possible implementation, the adjustment module 13 generates a sample loss value corresponding to the training sample according to the preferred log-likelihood ratio and the inferior log-likelihood ratio, and performs the following operations:

[0284] Performing difference processing on the optimal log-likelihood ratio and the inferior log-likelihood ratio to obtain the log-likelihood ratio difference;

[0285] Normalize the log-likelihood ratio difference to obtain the preference probability, and perform logarithmic processing on the preference probability to obtain the initial loss value;

[0286] The negative of the initial loss value is determined as the sample loss value corresponding to the training sample.

[0287] In one possible implementation, the adjustment module 13 adjusts the parameters of the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model, which is used to perform the following operations:

[0288] At least two supervised training samples are input into the initial question answering model respectively; at least two supervised training samples include supervised training sample A b , b is a positive integer, and b is less than or equal to the total number of at least two supervised training samples;

[0289] In the initial question answering model, supervised training samples A are generated b The corresponding sample loss value;

[0290] Perform mean processing on the sample loss values ​​corresponding to at least two supervised training samples to obtain the sample mean loss value;

[0291] According to the sample mean loss value, the parameters in the initial question-answering model are adjusted to obtain the first question-answering model.

[0292] In a possible implementation, the adjustment module 13 generates supervised training samples A in the initial question-answering model. b The corresponding sample loss value is used to perform the following operations:

[0293] In the initial question answering model, supervised training samples A are generated b The fourth prediction probability corresponding to the training label in; the fourth prediction probability refers to the output of the supervised training sample A for the sample problem b The probability of the training labels in ;

[0294] Perform logarithmic processing on the fourth predicted probability to obtain the initial loss value, and determine the negative of the initial loss value as the supervised training sample A b The corresponding sample loss value.

[0295] In a possible implementation, the adjustment module 13 generates supervised training samples A in the initial question-answering model. b The corresponding sample loss value is used to perform the following operations:

[0296] In the initial question answering model, the supervised training sample A b Identify and process the sample questions in the , and get the estimated answers;

[0297] For the estimated answer and supervised training sample A b The training labels in are processed for loss, and the supervised training sample A is obtained b The corresponding sample loss value.

[0298] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0299] From the above, it can be seen that in the supervised fine-tuning stage, the embodiment of the present application uses all sample answers (including sample answers with low preference) as sample labels, that is, each sample answer is regarded as the correct answer. Therefore, the first question-answering model obtained by training can improve the generation probability and confidence of each sample answer (including sample answers with low preference), which can improve the output diversity and stability of the first question-answering model; since the generation probability and confidence of sample answers with low preference have been improved in the supervised fine-tuning stage, the suppression of sample answers with low preference in the preference alignment stage is smoother, avoiding the collapse of the predicted probability of sample answers with low preference due to model parameter adjustment, and thus avoiding the predicted probability from being excessively squeezed to a very small number of high-confidence sample answers (i.e., sample answers with high preference), so the model training stability and generation diversity can be improved.

[0300] Further, see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be Figure 1 At least one of the terminal device or service server shown. Figure 8 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to implement connection and communication between these components.

[0301] In some embodiments, the user interface 1003 may include a display screen and a keyboard, and the network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may also optionally be at least one storage device located remotely from the processor 1001.

[0302] like Figure 8As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0303] exist Figure 8 In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an interface for user input; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0304] Obtaining a training sample; the training sample includes a sample question, at least two sample answers to the sample question, and a preference label for indicating a preference relationship between the at least two sample answers;

[0305] Determine at least two sample answers as sample labels of the sample question, and generate at least two supervised training samples based on the sample question and the at least two sample labels; wherein one supervised training sample includes a sample question carrying one sample label;

[0306] Adjusting parameters of the initial question-answering model based on at least two supervised training samples to obtain a first question-answering model;

[0307] Inputting the training sample into the first question-answering model, generating first predicted probabilities corresponding to at least two sample answers in the first question-answering model; wherein a first predicted probability refers to the probability of outputting a sample answer for the sample question;

[0308] According to the preference label and the first predicted probabilities corresponding to at least two sample answers, the parameters in the first question-answering model are adjusted to obtain a second question-answering model; the second question-answering model is used to generate an output answer to the input question.

[0309] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the description of the data processing method or device in the above embodiments, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

[0310] The present application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the data processing methods or apparatuses described in the preceding embodiments, which are not described in detail here. Furthermore, the description of the beneficial effects of the same methods is not described in detail here.

[0311] The computer-readable storage medium may be the data processing apparatus provided in any of the aforementioned embodiments or an internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., provided on the computer device.

[0312] Furthermore, the computer-readable storage medium may include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0313] The present application also provides a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, enabling the computer device to perform the data processing methods or apparatuses described in the preceding embodiments, which are not further detailed here. Furthermore, the beneficial effects of the same methods are not further detailed here.

[0314] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0315] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0316] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Get training samples; The training sample includes a sample question, at least two sample answers to the sample question, and a preference label for indicating a preference relationship between the at least two sample answers; Determining the at least two sample answers as sample labels for the sample question, respectively, and generating at least two supervised training samples based on the sample question and the at least two sample labels; wherein one supervised training sample includes the sample question carrying one sample label; Adjusting parameters in the initial question-answering model according to the at least two supervised training samples to obtain a first question-answering model; Inputting the training sample into the first question-answering model, and generating, in the first question-answering model, first predicted probabilities corresponding to the at least two sample answers, respectively; wherein a first predicted probability refers to the probability of outputting a sample answer for the sample question; According to the preference label and the first predicted probabilities corresponding to the at least two sample answers, the parameters in the first question-answering model are adjusted to obtain a second question-answering model; the second question-answering model is used to generate an output answer to the input question.

2. The method according to claim 1, characterized in that The adjusting the parameters in the first question-answering model according to the preference label and the first predicted probabilities respectively corresponding to the at least two sample answers to obtain a second question-answering model includes: generating a sample loss value corresponding to the training sample according to the preference label and the first predicted probabilities respectively corresponding to the at least two sample answers; According to the preference label, obtaining a poorly selected sample answer from the at least two sample answers, and determining a training risk value of the training sample according to a first predicted probability corresponding to the poorly selected sample answer; the training risk value is used to indicate the intensity of disturbance of the training sample on the model output distribution of the first question-answering model; Optimizing the sample loss value according to the training risk value to obtain a sample optimization loss value corresponding to the training sample; the sample optimization loss value is less than the sample loss value; According to the sample optimization loss value, the parameters in the first question-answering model are adjusted to obtain a question-answering model.

3. The method according to claim 2, characterized in that The determining of the training risk value of the training sample according to the first predicted probability corresponding to the answer to the inferior sample includes: Performing logarithmic processing on the first predicted probability corresponding to the answer of the inferior sample to obtain a log-likelihood value; The negative of the log-likelihood value is determined as the training risk value of the training sample; there is an inverse relationship between the training risk value and the first predicted probability corresponding to the inferior sample answer.

4. The method according to claim 2, characterized in that The number of the inferior sample answers is at least two, and the at least two inferior sample answers include the inferior sample answer C d , d is a positive integer, and d is less than or equal to the total number of the at least two inferior sample answers; The determining of the training risk value of the training sample according to the first predicted probability corresponding to the answer to the inferior sample includes: Answer C to the inferior sample d The corresponding first predicted probability is logarithmically processed to obtain a log-likelihood value; The negative of the log-likelihood value is determined as the inferior sample answer C d Corresponding training risk value; The training risk values ​​corresponding to the at least two inferior sample answers are fused to obtain the training risk value of the training sample.

5. The method according to claim 2, characterized in that The optimizing process of the sample loss value according to the training risk value to obtain the sample optimized loss value corresponding to the training sample includes: Determining a training weight corresponding to the training sample according to the training risk value; there is an inverse relationship between the training risk value and the training weight; The sample loss value is weighted according to the training weight to obtain a sample optimization loss value corresponding to the training sample.

6. The method according to claim 5, characterized in that The determining, according to the training risk value, a training weight corresponding to the training sample includes: weighting the training risk value according to the weighted intensity parameter to obtain a weighted training risk value; The negative of the weighted training risk value is determined as the risk value to be smoothed, and the risk value to be smoothed is subjected to exponential processing to obtain a training weight corresponding to the training sample.

7. The method according to claim 2, characterized in that Generating a sample loss value corresponding to the training sample according to the preference label and the first predicted probabilities respectively corresponding to the at least two sample answers includes: Determining a preferred sample answer and a less preferred sample answer from the at least two sample answers according to the preference label; the quality of the answer corresponding to the preferred sample answer is better than the quality of the answer corresponding to the less preferred sample answer; Obtaining a second predicted probability output by the reference model for the preferred sample answer and a third predicted probability output by the reference model for the inferior sample answer; Generating a preferred log-likelihood ratio according to the first predicted probability and the second predicted probability corresponding to the preferred sample answer; Generating a poor choice log likelihood ratio according to the first predicted probability corresponding to the poor choice sample answer and the third predicted probability; A sample loss value corresponding to the training sample is generated according to the preferred log-likelihood ratio and the inferior log-likelihood ratio.

8. The method according to claim 7, characterized in that Generating a preferred log-likelihood ratio according to the first predicted probability and the second predicted probability corresponding to the preferred sample answer includes: Performing ratio processing on the first predicted probability and the second predicted probability corresponding to the preferred sample answer to obtain a probability ratio; Performing logarithmic processing on the probability ratio to obtain a logarithmic likelihood ratio, and performing weighted processing on the logarithmic likelihood ratio to obtain an optimal logarithmic likelihood ratio.

9. The method according to claim 7, characterized in that Generating a sample loss value corresponding to the training sample according to the preferred log-likelihood ratio and the inferior log-likelihood ratio includes: performing a difference processing on the preferred log-likelihood ratio and the inferior log-likelihood ratio to obtain a log-likelihood ratio difference; Normalizing the log-likelihood ratio difference to obtain a preference probability, and performing logarithmic processing on the preference probability to obtain an initial loss value; The negative of the initial loss value is determined as the sample loss value corresponding to the training sample.

10. The method according to claim 1, characterized in that The step of adjusting parameters in the initial question-answering model based on the at least two supervised training samples to obtain a first question-answering model includes: The at least two supervised training samples are respectively input into the initial question answering model; the at least two supervised training samples include supervised training sample A b , b is a positive integer, and b is less than or equal to the total number of the at least two supervised training samples; In the initial question-answering model, the supervised training sample A is generated b The corresponding sample loss value; Performing mean processing on the sample loss values ​​corresponding to the at least two supervised training samples to obtain a sample mean loss value; According to the sample mean loss value, the parameters in the initial question-answering model are adjusted to obtain a first question-answering model.

11. The method according to claim 10, characterized in that In the initial question-answering model, the supervised training sample A is generated b The corresponding sample loss values ​​include: In the initial question-answering model, the supervised training sample A is generated b The fourth prediction probability corresponding to the training label in; the fourth prediction probability refers to the output of the supervised training sample A for the sample problem b The probability of the training labels in ; Perform logarithmic processing on the fourth predicted probability to obtain an initial loss value, and determine the negative of the initial loss value as the supervised training sample A b The corresponding sample loss value.

12. The method according to claim 10, characterized in that In the initial question-answering model, the supervised training sample A is generated b The corresponding sample loss values ​​include: In the initial question-answering model, the supervised training sample A b Identify and process the sample questions in to obtain estimated answers; For the estimated answer and the supervised training sample A b The training labels in are processed for loss to obtain the supervised training sample A b The corresponding sample loss value.

13. A data processing device, characterized in that: include: Acquisition module, used to obtain training samples; The training sample includes a sample question, at least two sample answers to the sample question, and a preference label for indicating a preference relationship between the at least two sample answers; a generating module, configured to determine the at least two sample answers as sample labels for the sample question, and generate at least two supervised training samples based on the sample question and the at least two sample labels; wherein one supervised training sample includes the sample question carrying one sample label; an adjustment module, configured to adjust parameters in the initial question-answering model based on the at least two supervised training samples to obtain a first question-answering model; The generation module is further configured to input the training sample into the first question-answering model, and generate, in the first question-answering model, first predicted probabilities corresponding to the at least two sample answers, respectively; wherein a first predicted probability refers to a probability of outputting a sample answer for the sample question; The adjustment module is also used to adjust the parameters in the first question-answering model according to the preference label and the first predicted probabilities corresponding to the at least two sample answers to obtain a second question-answering model; the second question-answering model is used to generate an output answer to the input question.

14. A computer device, characterized in that: include: processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product comprises a computer program stored in a computer-readable storage medium. The computer program is suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 12.

Images