Text processing model processing method and device, text processing method and device, computer equipment and readable storage medium

Optimizing large language models through multi-objective preference evaluation and iterative training solves the problem of harmful content and distribution deviation in text processing, and improves the accuracy and stability of text processing.

CN118939765AActive Publication Date: 2024-11-12CHINA CONSTRUCTION BANK +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410958502.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-12
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

Existing large language models are prone to output harmful or unhelpful content in text processing, and their learning distribution deviates from the actual data distribution over time, resulting in a decrease in text processing accuracy.

Method used

By acquiring multiple input and output groups, using the pre-trained vector reward model for multi-objective preference evaluation, adjusting the loss value of the text processing strategy model, and using iterative training to reduce the deviation from the actual data distribution.

Benefits of technology

Improve the accuracy and stability of text processing, ensure that model output is more in line with human preferences, and reduce model distribution offset over time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939765A_ABST
    Figure CN118939765A_ABST
Patent Text Reader

Abstract

The invention relates to the field of artificial intelligence, and particularly discloses a processing method of a text processing model, a text processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that multiple pieces of reply content information output by a to-be-trained text processing strategy model for prompt information are acquired, and multiple input and output groups are obtained; inputting each input and output group into a pre-trained vector reward model, and outputting preference evaluation sorting information corresponding to each input and output group; determining a loss value of the to-be-trained text processing strategy model according to the preference evaluation sorting information corresponding to each input and output group; and adjusting the to-be-trained text processing strategy model according to the loss value to obtain an adjusted text processing strategy model, and returning to the step of obtaining multiple pieces of reply content information output by the to-be-trained text processing strategy model for the prompt information until a training ending condition is met. By adopting the method, the accuracy of text processing can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text processing model processing method, a text processing method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of generative large language models, the output of large models after fine-tuning instructions can show powerful capabilities. However, without constraints, large models are prone to output harmful and unhelpful answers. Therefore, nowadays, people are paying attention to how to train these fine-tuned large language models so that they can interact with humans in a more humane, ethical, and safe way and provide output that is beneficial to society.

[0003] The reinforcement learning algorithm based on human feedback mechanism (Reinforcement Learning from HumanFeedback, RLHF) first provides heuristic ideas. The core of this method is to integrate human value preferences with the generation ability of the model. Specifically, human annotators first rank the content generated by the large model by preference. These data are not only an evaluation of the content generated by the model, but also reflect whether the content generated by the model is in line with human values ​​and expectations. Then, the sorting data is used to train the reward model to score the output content of the large model, and then the large unsupervised language model is fine-tuned through reinforcement learning to maximize the evaluation reward and avoid being too far away from the original model.

[0004] However, in the related art, a large language model can be used for text processing. During the operation of the large language model, the deviation between its learned distribution and the actual data distribution becomes increasingly greater over time, resulting in a decrease in the accuracy of the model output, thereby resulting in a decrease in the accuracy of text processing.

[0005] Therefore, there is a problem of low accuracy in text processing in the related art. Summary of the invention

[0006] Based on this, it is necessary to provide a processing method, text processing model, device, computer equipment, computer-readable storage medium and computer program product for a text processing model that can improve the accuracy of text processing in response to the above technical problems.

[0007] In a first aspect, the present application provides a method for processing a text processing model, comprising:

[0008] Acquire multiple reply content information output by the text processing strategy model to be trained for the prompt information, and obtain multiple input and output groups corresponding to the prompt information; the multiple input and output groups correspond to the prompt information and different reply content information respectively; the text processing strategy model is a large language model whose model parameters are to be adjusted;

[0009] Inputting each of the input-output groups into a pre-trained vector reward model, and outputting preference evaluation ranking information corresponding to each of the input-output groups; the preference evaluation ranking information is determined by performing a multi-objective preference evaluation on the input-output groups by the pre-trained vector reward model using at least two evaluation attributes;

[0010] Determining the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each of the input and output groups;

[0011] The text processing strategy model to be trained is adjusted according to the loss value to obtain the adjusted text processing strategy model, and the adjusted text processing strategy model is used as the text processing strategy model to be trained, and the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information is returned until the text processing strategy model to be trained meets the training end condition.

[0012] In one embodiment, determining the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each of the input and output groups includes:

[0013] Obtaining the initial loss function corresponding to the text processing strategy model to be trained under the direct preference optimization method;

[0014] The loss value of the text processing strategy model to be trained is determined through the initial loss function according to the information divergence of the text processing strategy model to be trained and the text processing reference model; the text processing reference model is a large language model with fixed model parameters.

[0015] In one embodiment, the method further comprises:

[0016] Determine the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each of the input-output groups; the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group;

[0017] According to the first input-output group and the second input-output group, the information divergence of the text processing strategy model to be trained and the text processing reference model is determined.

[0018] In one embodiment, determining the information divergence of the text processing strategy model to be trained and the text processing reference model according to the first input-output group and the second input-output group includes:

[0019] Obtain the strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and obtain the reference cumulative probability that the text processing reference model outputs the second reply content information when the prompt information is input; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group;

[0020] According to the information divergence of the strategy cumulative probability and the reference cumulative probability, the information divergence of the text processing strategy model to be trained and the text processing reference model is determined.

[0021] In one embodiment, determining the loss value of the text processing strategy model to be trained according to the information divergence between the text processing strategy model to be trained and the text processing reference model through the initial loss function includes:

[0022] According to the information divergence, the initial loss function is adjusted to obtain a target loss function;

[0023] According to the target loss function, a loss value of the text processing strategy model to be trained is determined.

[0024] In one embodiment, determining the loss value of the text processing strategy model to be trained according to the target loss function includes:

[0025] Obtain the first strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and the second strategy cumulative probability that the text processing strategy model to be trained outputs the second reply content information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group;

[0026] Obtaining a first reference cumulative probability that the text processing reference model outputs first reply content information when the prompt information is input, and a second reference cumulative probability that the text processing reference model outputs second reply content information;

[0027] The first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability are calculated by using the target loss function to obtain a loss value of the text processing strategy model to be trained.

[0028] In one embodiment, inputting each of the input-output groups into a pre-trained vector reward model and outputting preference evaluation ranking information corresponding to each of the input-output groups includes:

[0029] Inputting each of the input-output groups into the pre-trained vector reward model, and outputting a plurality of preference evaluation information corresponding to each of the input-output groups; the plurality of preference evaluation information corresponding to each of the input-output groups includes preference evaluation information obtained by performing multi-objective preference evaluation on the input-output groups by the pre-trained vector reward model using at least two of the evaluation attributes;

[0030] According to the plurality of preference evaluation information corresponding to each of the input-output groups, the preference evaluation information corresponding to each of the input-output groups is sorted under each of the evaluation attributes to obtain the preference evaluation sorting information of each of the input-output groups under each of the evaluation attributes.

[0031] In a second aspect, the present application provides a text processing method, comprising:

[0032] Get the text information to be responded to;

[0033] The text information is input into a pre-trained text processing strategy model, and the target reply content information is output; the pre-trained text processing strategy model is obtained by using the processing method of the text processing model described above.

[0034] In a third aspect, the present application also provides a processing device for a text processing model, including:

[0035] An acquisition module is used to acquire multiple reply content information output by the text processing strategy model to be trained for the prompt information, and obtain multiple input and output groups corresponding to the prompt information; the multiple input and output groups correspond to the prompt information and different reply content information respectively; the text processing strategy model is a large language model whose model parameters are to be adjusted;

[0036] A sorting module, used to input each of the input-output groups into a pre-trained vector reward model, and output preference evaluation sorting information corresponding to each of the input-output groups; the preference evaluation sorting information is determined by performing a multi-objective preference evaluation on the input-output groups by the pre-trained vector reward model using at least two evaluation attributes;

[0037] A determination module, used to determine the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each of the input and output groups;

[0038] An adjustment module is used to adjust the text processing strategy model to be trained according to the loss value to obtain the adjusted text processing strategy model, and use the adjusted text processing strategy model as the text processing strategy model to be trained, and return to the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, until the text processing strategy model to be trained meets the training end condition.

[0039] In one embodiment, the determining module includes:

[0040] A function acquisition unit, used to acquire the initial loss function corresponding to the text processing strategy model to be trained under the direct preference optimization method;

[0041] A loss value determination unit is used to determine the loss value of the text processing strategy model to be trained according to the information divergence of the text processing strategy model to be trained and the text processing reference model through the initial loss function; the text processing reference model is a large language model with fixed model parameters.

[0042] In one embodiment, the device further comprises:

[0043] A screening module, configured to determine the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each of the input-output groups; the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group;

[0044] The divergence determination module is used to determine the information divergence of the text processing strategy model to be trained and the text processing reference model according to the first input-output group and the second input-output group.

[0045] In one embodiment, the divergence determination module includes:

[0046] A probability determination unit, used for obtaining the strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information under the input of the prompt information, and obtaining the reference cumulative probability that the text processing reference model outputs the second reply content information under the input of the prompt information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group;

[0047] The divergence determination unit is used to determine the information divergence of the text processing strategy model to be trained and the text processing reference model according to the information divergence of the strategy cumulative probability and the reference cumulative probability.

[0048] In one embodiment, the loss value determination unit is specifically used to:

[0049] According to the information divergence, the initial loss function is adjusted to obtain a target loss function;

[0050] According to the target loss function, a loss value of the text processing strategy model to be trained is determined.

[0051] In one embodiment, the loss value determination unit is specifically used to:

[0052] Obtain the first strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and the second strategy cumulative probability that the text processing strategy model to be trained outputs the second reply content information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group;

[0053] Obtaining a first reference cumulative probability that the text processing reference model outputs first reply content information when the prompt information is input, and a second reference cumulative probability that the text processing reference model outputs second reply content information;

[0054] The first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability are calculated by using the target loss function to obtain a loss value of the text processing strategy model to be trained.

[0055] In one embodiment, the sorting module includes:

[0056] An evaluation unit, used to input each of the input-output groups into the pre-trained vector reward model, and output a plurality of preference evaluation information corresponding to each of the input-output groups; the plurality of preference evaluation information corresponding to each of the input-output groups includes preference evaluation information obtained by the pre-trained vector reward model using at least two of the evaluation attributes to perform multi-objective preference evaluation on the input-output groups;

[0057] A sorting unit is used to sort the preference evaluation information corresponding to each input-output group under each evaluation attribute according to the multiple preference evaluation information corresponding to each input-output group, so as to obtain the preference evaluation sorting information of each input-output group under each evaluation attribute.

[0058] In a fourth aspect, the present application further provides a text processing device, comprising:

[0059] A text acquisition module is used to acquire text information to be responded to;

[0060] An output module is used to input the text information into a pre-trained text processing strategy model and output target reply content information; the pre-trained text processing strategy model is obtained by using the processing method of the text processing model described above.

[0061] In a fifth aspect, the present application further provides an electronic device, wherein the electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0062] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0063] In a seventh aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0064] The processing method, apparatus, computer device, computer-readable storage medium and computer program product of the above-mentioned text processing model obtain multiple input-output groups corresponding to the prompt information by obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information; the multiple input-output groups correspond to the prompt information and the different reply content information respectively; the text processing strategy model is a large language model whose model parameters are to be adjusted; each input-output group is input into a pre-trained vector reward model, and the preference evaluation ranking information corresponding to each input-output group is output; the preference evaluation ranking information is obtained by using at least two evaluation attributes of the pre-trained vector reward model to perform multi-objective preference evaluation on the input-output group; according to the preference evaluation ranking information corresponding to each input-output group, the loss value of the text processing strategy model to be trained is determined; the text processing strategy model to be trained is adjusted according to the loss value to obtain the adjusted text processing strategy model, and the adjusted text processing strategy model is used as the text processing strategy model to be trained, and the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information is returned until the text processing strategy model to be trained meets the training end condition.

[0065] In this way, by using at least two evaluation attributes through the pre-trained vector reward model, a multi-target preference evaluation is performed on each input and output group corresponding to the prompt information, and the preference evaluation ranking information corresponding to each input and output group is obtained, so that the vector reward model can be used to reflect diverse preferences, and the diverse preferences are reflected more accurately. At the same time, it also alleviates the possible conflicts between multi-target preferences to a certain extent, which is more in line with the actual preference situation, so that the trained text processing strategy model can process text information more accurately. At the same time, compared with the previous technology that only uses a single stage to align the training model, this application uses iterative training based on the preference evaluation ranking information output by the pre-trained vector reward model to continuously iterate and optimize the model, dynamically improve the ability of the large language model itself, and reduce its deviation from the actual changing data distribution, thereby alleviating the model distribution deviation caused by the passage of time, so that the text processing ability of the trained text processing strategy model is further improved, and the technical effect of improving the accuracy of text processing is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A flowchart of a method for processing a text processing model in an embodiment;

[0068] Figure 2 A flowchart of a processing method of a text processing model in an embodiment;

[0069] Figure 3 is a flowchart of a processing method of a text processing model in another embodiment;

[0070] Figure 4 A flowchart of a text processing method in an embodiment;

[0071] Figure 5 is a structural block diagram of a processing device of a text processing model in one embodiment;

[0072] Figure 6 is a structural block diagram of a text processing device in one embodiment;

[0073] FIG. 7 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, groups, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0075] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0076] In one embodiment, Figure 1 As shown, a processing method for a text processing model is provided. This embodiment uses the method applied to a computer device as an example for illustration. It can be understood that the computer device can be a terminal, a server, or a system including a terminal and a server. In this embodiment, the method includes the following steps:

[0077] Step S110, obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, and obtaining multiple input and output groups corresponding to the prompt information.

[0078] Among them, multiple input and output groups correspond to prompt information and different reply content information respectively.

[0079] Among them, the text processing strategy model is a large language model whose model parameters are to be adjusted.

[0080] In practical applications, the text processing strategy model may be a strategy model for processing text information.

[0081] Each input-output group includes a prompt message and a matching reply content message.

[0082] Among them, for multiple input-output groups corresponding to the same prompt information, each input-output group includes the same prompt information and different reply content information.

[0083] Among them, the prompt information and the reply content information can be text information.

[0084] In a specific implementation, the computer device can obtain multiple reply content information output by the text processing strategy model to be trained for the prompt information, and form multiple input and output groups corresponding to the prompt information according to the prompt information and the corresponding reply content information. The multiple input and output groups corresponding to the same prompt information correspond to the prompt information and different reply content information respectively.

[0085] Step S120, input each input-output group into the pre-trained vector reward model, and output the preference evaluation ranking information corresponding to each input-output group.

[0086] The preference evaluation ranking information is obtained by using a pre-trained vector reward model using at least two evaluation attributes to perform a multi-objective preference evaluation on the input and output groups.

[0087] The vector reward model may be a reward model that reflects the diverse preferences of multiple users.

[0088] Among them, the pre-trained vector reward model is obtained by training with the perspective of multi-target vector representation.

[0089] The preference evaluation ranking information is used to represent the ranking information of the degree of human preference corresponding to the response content information in the input and output groups.

[0090] The evaluation attribute may be an evaluation dimension used when the pre-trained vector reward performs multi-objective preference evaluation on the input-output group. The evaluation attribute may include, but is not limited to, attributes such as helpfulness, correctness, and redundancy.

[0091] In a specific implementation, a computer device may input each input-output group into a pre-trained vector reward model, and use at least two evaluation attributes through the pre-trained vector reward model to perform a multi-objective preference evaluation on the input-output group to obtain preference evaluation information corresponding to each input-output group, and sort each input-output group according to the preference evaluation information corresponding to each input-output group to obtain preference evaluation sorting information corresponding to each input-output group.

[0092] The preference evaluation information can be used to characterize the degree of human preference corresponding to the input-output group. Further, the preference evaluation information can be used to characterize the degree of human preference for the reply content information in the input-output group.

[0093] Step S130, determining the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each input and output group.

[0094] In a specific implementation, the computer device can determine the loss value of the text processing strategy model to be trained based on the preference evaluation sorting information corresponding to each input and output group.

[0095] In some embodiments, the computer device may determine the loss value of the text processing strategy model to be trained by using the direct preference optimization (DPO) method based on the preference evaluation ranking information corresponding to each input and output group. Specifically, the computer device may screen out input and output groups with higher preference evaluation rankings and input and output groups with lower preference evaluation rankings based on the preference evaluation ranking information corresponding to each input and output group, and use the screened sample pairs to determine the loss value of the text processing strategy model to be trained by using the direct preference optimization method. Among them, the reply content information in the input and output groups with higher preference evaluation rankings is more in line with human preferences.

[0096] Step S140, adjust the text processing strategy model to be trained according to the loss value to obtain the adjusted text processing strategy model, and use the adjusted text processing strategy model as the text processing strategy model to be trained, and return to the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, until the text processing strategy model to be trained meets the training end condition.

[0097] The training end condition may be that the loss value of the text processing strategy model to be trained is less than a preset threshold, or the number of model iterations meets a preset number.

[0098] In a specific implementation, the computer device can adjust the text processing strategy model to be trained according to the loss value to obtain the adjusted text processing strategy model, and use the adjusted text processing strategy model as the text processing strategy model to be trained, and return to the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, until the loss value meets the training end condition of the text processing strategy model to be trained, thereby realizing iterative optimization of the text processing strategy model.

[0099] In the processing method of the above-mentioned text processing model, multiple input-output groups corresponding to the prompt information are obtained by obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information; the multiple input-output groups correspond to the prompt information and different reply content information respectively; the text processing strategy model is a large language model whose model parameters are to be adjusted; each input-output group is input into a pre-trained vector reward model, and the preference evaluation ranking information corresponding to each input-output group is output; the preference evaluation ranking information is obtained by using at least two evaluation attributes of the pre-trained vector reward model to perform multi-objective preference evaluation on the input-output group; according to the preference evaluation ranking information corresponding to each input-output group, the loss value of the text processing strategy model to be trained is determined; the text processing strategy model to be trained is adjusted according to the loss value to obtain the adjusted text processing strategy model, and the adjusted text processing strategy model is used as the text processing strategy model to be trained, and the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information is returned until the text processing strategy model to be trained meets the training end condition.

[0100] In this way, by using at least two evaluation attributes through the pre-trained vector reward model, a multi-target preference evaluation is performed on each input and output group corresponding to the prompt information, and the preference evaluation ranking information corresponding to each input and output group is obtained, so that the vector reward model can be used to reflect diverse preferences, and the diverse preferences are reflected more accurately. At the same time, it also alleviates the possible conflicts between multi-target preferences to a certain extent, which is more in line with the actual preference situation, so that the trained text processing strategy model can process text information more accurately. At the same time, compared with the previous technology that only uses a single stage to align the training model, this application uses iterative training based on the preference evaluation ranking information output by the pre-trained vector reward model to continuously iterate and optimize the model, dynamically improve the ability of the large language model itself, and reduce its deviation from the actual changing data distribution, thereby alleviating the model distribution deviation caused by the passage of time, so that the text processing ability of the trained text processing strategy model is further improved, and the technical effect of improving the accuracy of text processing is achieved.

[0101] In one embodiment, the loss value of the text processing strategy model to be trained is determined according to the preference evaluation ranking information corresponding to each input and output group, including: obtaining the corresponding initial loss function of the text processing strategy model to be trained under the direct preference optimization method; through the initial loss function, according to the information divergence between the text processing strategy model to be trained and the text processing reference model, determining the loss value of the text processing strategy model to be trained.

[0102] The information divergence may be KL divergence (Kullback–Leibler divergence, referred to as KLD), also known as relative entropy.

[0103] The text processing reference model is a large language model with fixed model parameters. In practical applications, the text processing reference model can be a reference model for processing text information.

[0104] In a specific implementation, when a computer device determines the loss value of a text processing strategy model to be trained according to preference evaluation ranking information corresponding to each input and output group, the computer device can obtain the corresponding initial loss function of the text processing strategy model to be trained under the direct preference optimization method (DPO), and obtain the information divergence between the text processing strategy model to be trained and the text processing reference model; through the initial loss function, the loss value of the text processing strategy model to be trained is determined according to the information divergence between the text processing strategy model to be trained and the text processing reference model.

[0105] Among them, reinforcement learning is used to fine-tune the large language model, that is, DPO (Direct Preference Optimization) is adopted. Two models are used in the optimization process, namely the reference model and the policy model. When the model training starts, the reference model and the policy model are the same model, which is the large language model. During the training process, the reference model will not update the model parameters, and the reference model is used as a comparison object.

[0106] Among them, through the initial loss function under the direct preference optimization method, the information divergence of the text processing strategy model to be trained and the text processing reference model is used to determine the loss value of the text processing strategy model to be trained, so as to update and iterate the text processing strategy model to be trained, thereby achieving the beneficial effect of alleviating the problem of the model being increasingly distant from the ever-changing data distribution in actual operation through continuous DPO optimization.

[0107] The technical solution of this embodiment obtains the corresponding initial loss function of the text processing strategy model to be trained under the direct preference optimization method; through the initial loss function, the loss value of the text processing strategy model to be trained is determined according to the information divergence of the text processing strategy model to be trained and the text processing reference model; wherein the text processing reference model is a large language model with fixed model parameters. In this way, through the direct preference optimization method, the information divergence of the text processing strategy model to be trained and the text processing reference model is used to determine the loss value of the text processing strategy model to be trained, so that in the process of iterative model training, the exploration and experience can be balanced, and the optimized model will not be too far away from the initial model, thereby achieving the beneficial effect of effectively improving the stability of model training.

[0108] In one embodiment, the method also includes: determining the first input-output group and the second input-output group corresponding to the prompt information based on the preference evaluation ranking information corresponding to each input-output group; and determining the information divergence of the text processing strategy model to be trained and the text processing reference model based on the first input-output group and the second input-output group.

[0109] Among them, the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group.

[0110] The preference level of the reply content information in the first input-output group is higher than the preference level of the reply content information in the second input-output group.

[0111] In a specific implementation, when a computer device determines the loss value of a text processing strategy model to be trained according to preference evaluation ranking information corresponding to each input-output group, the computer device may determine the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each input-output group; the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group; in this way, the loss value of the text processing strategy model to be trained may be determined according to the first input-output group and the second input-output group.

[0112] There may be at least one prompt message, and each prompt message may correspond to multiple reply content messages, such as the prompt message is , the corresponding multiple reply content information can be , then prompt information The corresponding multiple input and output groups can be , , and .

[0113] In the process of determining the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each input-output group, wherein the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group, the computer device may generate a plurality of sample pairs corresponding to each prompt information according to the preference evaluation ranking information of each input-output group corresponding to each prompt information; each sample pair includes the first input-output group and the second input-output group; the first input-output group and the second input-output group in the same sample pair correspond to the same prompt information and different reply content information for the prompt information. The plurality of sample pairs corresponding to each prompt information may form a sample set.

[0114] Exemplarily, a computer device may sort multiple input-output groups corresponding to the same prompt information in descending order of preference evaluation to obtain a preference queue; randomly select two input-output groups from the multiple input-output groups corresponding to the same prompt information as preset input-output groups; query the preference evaluation order of the preset input-output groups in the preference queue; use the preset input-output group with a larger preference evaluation order as the first input-output group of a sample pair, and use the preset input-output group with a smaller preference evaluation order as the second input-output group of the same sample pair.

[0115] For example, as an optional example, for the prompt information , the corresponding multiple input and output groups can be , , and Sort by preference evaluation order to get the preference queue ( > > > ), then the preference queue can form 6 sample pairs, namely [ , ]、[ , ]、[ , ]、[ , ]、[ , ]as well as[ , ]. In this example, in each sample pair, the first input-output group is located at the leading position, and the second input-output group is located at the trailing position.

[0116] It is understandable that the computer device can use multiple samples corresponding to each prompt information to train the policy model to be trained, and the first input-output group and the second input-output group in each sample pair are used as a set of training data for the policy model to be trained. The policy model to be trained is trained in this round through multiple sets of training data corresponding to each prompt information. After the end of this round of training, if the preset training end condition is not met, the adjusted policy model is obtained to output new reply content information for each prompt information, and the policy model training is performed again. That is, the adjusted policy model is used as the policy model to be trained, and the step of obtaining multiple reply content information output by the policy model to be trained for the prompt information is returned until the preset training end condition is met.

[0117] In practical applications, preference evaluation ranking information of multiple input and output groups corresponding to prompt information can also be used to perform rejection selection, select high-quality answer content information with a high preference evaluation ranking, and directly use the input and output groups corresponding to these high-quality answer content information to fine-tune and update the training model, and then reflux the model to generate answer content information, select high-quality answer content information, and iterate in this way. In this way, the amount of training data used in the training process can be reduced, and the pre-processing efficiency of the training data can be improved, thereby achieving the technical effect of improving the efficiency of model training.

[0118] Exemplarily, the computer device may utilize the preference evaluation ranking information of multiple input-output groups corresponding to the prompt information, and select input-output groups that meet a preset number in a descending order of preference evaluation ranking to form sample pairs corresponding to the prompt information.

[0119] In this way, each sample pair includes a first input-output group and a second input-output group, and the first input-output group and the second input-output group respectively include different reply content information for the same prompt information, and the reply content information in the first input-output group is more in line with human preferences than the reply content information in the second input-output group. In the process of training the text processing strategy model to be trained, comparative training is performed based on the first input-output group and the second input-output group, so that the reply result output by the trained text processing strategy model is closer to the reply content information in the first input-output group, thereby obtaining a reply result that is more in line with human preferences and achieving the beneficial effect of improving text processing capabilities.

[0120] In some embodiments, the computer device may determine the information divergence of the text processing strategy model to be trained and the text processing reference model based on the first input-output group and the second input-output group in multiple samples corresponding to each prompt information.

[0121] The technical solution of this embodiment determines the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each input-output group; the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group; based on the first input-output group and the second input-output group, the information divergence of the text processing strategy model to be trained and the text processing reference model is determined.

[0122] In this way, based on the first input-output group and the second input-output group that form a contrast with different preference evaluation rankings, the information divergence of the text processing strategy model to be trained and the text processing reference model can be accurately determined, so that model training can be performed based on the information divergence, so that the output reply result of the trained text processing strategy model is closer to the reply content information in the first input-output group, thereby obtaining a reply result that is more in line with human preferences and achieving the beneficial effect of improving text processing capabilities.

[0123] In one embodiment, the information divergence of the text processing strategy model to be trained and the text processing reference model is determined based on the first input-output group and the second input-output group, including: obtaining the strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information under the input prompt information, and obtaining the reference cumulative probability that the text processing reference model outputs the second reply content information under the input prompt information; according to the information divergence of the strategy cumulative probability and the reference cumulative probability, the information divergence of the text processing strategy model to be trained and the text processing reference model is determined.

[0124] Among them, the first reply content information is the reply content information in the first input and output group; the second reply content information is the reply content information in the second input and output group.

[0125] The strategy cumulative probability may be the cumulative probability of the text processing strategy model to be trained outputting the first reply content information under the input prompt information.

[0126] Among them, the reference cumulative probability can be the cumulative probability of the text processing reference model outputting the second reply content information under the input prompt information.

[0127] In a specific implementation, when a computer device determines the information divergence of a text processing strategy model to be trained and a text processing reference model based on a first input-output group and a second input-output group, the computer device obtains a strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information under input prompt information, and obtains a reference cumulative probability that the text processing reference model outputs the second reply content information under input prompt information, so as to determine the information divergence of the text processing strategy model to be trained and the text processing reference model based on the information divergence of the strategy cumulative probability and the reference cumulative probability.

[0128] The technical solution of this embodiment is to obtain the strategy cumulative probability of the text processing strategy model to be trained outputting the first reply content information under the input prompt information, and to obtain the reference cumulative probability of the text processing reference model outputting the second reply content information under the input prompt information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group; according to the information divergence of the strategy cumulative probability and the reference cumulative probability, determine the information divergence of the text processing strategy model to be trained and the text processing reference model. In this way, by calculating the information divergence between the probability distributions of the reply content information with different preference levels output by the strategy model and the reference model under given input prompt information, the information divergence can be used to measure the degree of difference between the reply generated by the strategy model and the reply generated by the reference model, thereby guiding model training and optimization. Based on this difference, a suitable direct preference optimization loss function can be designed to optimize the parameters of the model, so as to achieve the beneficial effect of making the model generate replies that are more in line with human preferences.

[0129] In one embodiment, the loss value of the text processing strategy model to be trained is determined through an initial loss function according to the information divergence between the text processing strategy model to be trained and the text processing reference model, including: adjusting the initial loss function according to the information divergence to obtain a target loss function; and determining the loss value of the text processing strategy model to be trained according to the target loss function.

[0130] In a specific implementation, when a computer device determines the loss value of a text processing strategy model to be trained through an initial loss function according to the information divergence between the text processing strategy model to be trained and a text processing reference model, the computer device can adjust the initial loss function according to the information divergence to obtain a target loss function; and determine the loss value of the text processing strategy model to be trained according to the target loss function.

[0131] The technical solution of this embodiment adjusts the initial loss function according to the information divergence to obtain the target loss function; according to the target loss function, the loss value of the text processing strategy model to be trained is determined. In this way, on the basis of the corresponding initial loss function of the text processing strategy model under the direct preference optimization method, the information divergence of the text processing strategy model to be trained and the text processing reference model is added to generate the target loss function, which can better control the distance between the optimized model and the initial model to avoid over-optimization causing the model to deviate from the original setting; by adding the information divergence term to the initial loss function, the model can be prompted to better balance the relationship between exploring new replies and maintaining an understanding of human preferences during the optimization process. Such an adjustment helps to maintain appropriate diversity and stability during the model optimization process, while avoiding excessive pursuit of performance indicators and ignoring the overall performance of the model.

[0132] In one embodiment, the loss value of the text processing strategy model to be trained is determined according to the target loss function, including: obtaining the first strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information under the input prompt information, and the second strategy cumulative probability that the text processing strategy model to be trained outputs the second reply content information; obtaining the first reference cumulative probability that the text processing reference model outputs the first reply content information under the input prompt information, and the second reference cumulative probability that the text processing reference model outputs the second reply content information; through the target loss function, the first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability are calculated to obtain the loss value of the text processing strategy model to be trained.

[0133] The first strategy cumulative probability may be the cumulative probability that the text processing strategy model to be trained outputs the first reply content information under the input prompt information.

[0134] The second strategy cumulative probability may be the cumulative probability that the text processing strategy model to be trained outputs the second reply content information under the input prompt information.

[0135] The first reference cumulative probability may be the cumulative probability of the text processing reference model outputting the first reply content information under the input prompt information.

[0136] The second reference cumulative probability may be the cumulative probability of the text processing reference model outputting the second reply content information under the input prompt information.

[0137] In a specific implementation, after obtaining the first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability, the computer device can calculate the first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability through the target loss function to obtain the loss value of the text processing strategy model to be trained.

[0138] As mentioned above, DPO (Direct Preference Optimization) is adopted, and two models, namely, the reference model and the policy model, are used in the optimization process. When the model training starts, the reference model and the policy model are the same model, which is a large language model. The reference model will not update the model parameters during the training process. The reference model is used as a comparison object. The parameters of the policy model are adjusted by the gradient descent method during the training process. The probability that the policy model and the reference model output the first reply content information and the second reply content information respectively is calculated to determine the loss value of the large language model, and then the parameters of the large language model are optimized according to the loss value to obtain a model that can accurately reflect human preferences.

[0139] Among them, the corresponding initial loss function of the policy model to be trained under the direct preference optimization method can be shown as follows:

[0140]

[0141] Among them, DPO is the direct preference optimization model; is the initial loss function; For the strategy model; is the reference model; Prompt information for input; The content of the first reply; The content information of the second reply; is the Sigmoid function; is the parameter factor; is the sample set; is the expected value; Prompt message for a given input , the cumulative probability that the current strategy model to be trained generates good reply content information, that is, the cumulative probability that the current strategy model to be trained generates the first reply content information with a higher preference evaluation ranking, as the first strategy cumulative probability; Prompt message for a given input , the cumulative probability of the original reference model generating good reply content information, that is, the cumulative probability of the original reference model generating the first reply content information with a higher preference evaluation ranking, as the first reference cumulative probability; Prompt message for a given input , the cumulative probability that the current strategy model to be trained generates poor reply content information, that is, the cumulative probability that the current strategy model to be trained generates the second reply content information with a lower preference evaluation ranking, as the second strategy cumulative probability; Prompt message for a given input , the cumulative probability that the original reference model generates poor reply content information, that is, the cumulative probability that the original reference model generates the second reply content information with a lower preference evaluation ranking, is used as the second reference cumulative probability.

[0142] Specifically, in the loss function is the sigmoid function , is a parameter, generally between 0.1 and 0.5. It is the information about the better response content for the input prompt information. It is the response content information with poor feedback for the input prompt information. Prompt message for a given input , the cumulative probability that the original reference model generates good reply content information; Prompt message for a given input , the current policy model to be trained generates the cumulative probability of bad response content information, then the left half of the loss function is the cumulative probability difference between good response content information and no training, and the right half of the loss function represents the cumulative probability difference between bad response content information and no training. If the cumulative probability difference becomes larger, it means that the left side of the loss function becomes larger and the right side becomes smaller. Ideally, the probability of good response content information increases and the probability of bad response content information decreases; if the left side of the loss function becomes smaller and the right side is even smaller, the probability of good response content information decreases, but the probability of bad response content information decreases more, and it still tends to be good response content information when generating; if the left side of the loss function becomes larger and the right side is only a little larger, the probability of good response content information decreases, but the probability of bad response content information decreases more, and it still tends to be good response content information when generating. Therefore, the loss function is more inclined to generate good response content information.

[0143] Furthermore, the strategy cumulative probability that the strategy model to be trained outputs the first answer content information with a higher preference evaluation ranking under the input prompt information can be referred to as ; The reference cumulative probability that the reference model outputs the second answer content information with a lower preference evaluation ranking under the input prompt information can be referred to as The KL divergence between the strategy cumulative probability and the reference cumulative probability can be ; is the regularization hyperparameter of the KL divergence.

[0144] In this way, the KL divergence of the strategy cumulative probability and the reference cumulative probability can be used as the KL divergence of the strategy model to be trained and the reference model. The KL divergence of the strategy model to be trained and the reference model can be Therefore, the computer device adjusts the initial loss function according to the KL divergence, and the obtained target loss function is as follows:

[0145]

[0146] in, is the target loss function.

[0147] In this way, the computer device can determine the loss value of the policy model to be trained based on the target loss function.

[0148] Specifically, the computer device can obtain the first strategy cumulative probability that the strategy model to be trained outputs the first reply content information under the input prompt information, and the second strategy cumulative probability that the strategy model outputs the second reply content information; obtain the first reference cumulative probability that the reference model outputs the first reply content information under the input prompt information, and the second reference cumulative probability that the reference model outputs the second reply content information; substitute the first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability into the target loss function for calculation to obtain the loss value of the strategy model to be trained.

[0149] The technical solution of this embodiment is to obtain the first strategy cumulative probability of the text processing strategy model to be trained outputting the first reply content information under the input prompt information, and the second strategy cumulative probability of the second reply content information; the first reply content information is the reply content information in the first input and output group; the second reply content information is the reply content information in the second input and output group; obtain the first reference cumulative probability of the text processing reference model outputting the first reply content information under the input prompt information, and the second reference cumulative probability of the second reply content information; calculate the first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability through the target loss function to obtain the loss value of the text processing strategy model to be trained. In this way, the reference model will not update the model parameters during the training process, and the reference model is used as a comparison object. During the training process, the model parameters of the strategy model are adjusted, and the cumulative probabilities of the strategy model and the reference model outputting the first reply content information and the second reply content information are calculated respectively to determine the loss value of the strategy model, so as to optimize the parameters of the strategy model according to the loss value, so as to obtain a strategy model that can accurately reflect human preferences, and achieve the beneficial effect of outputting more accurate reply content information that conforms to human preferences based on the input text information.

[0150] In one embodiment, each input-output group is input into a pre-trained vector reward model, and preference evaluation ranking information corresponding to each input-output group is output, including: each input-output group is input into a pre-trained vector reward model, and a plurality of preference evaluation information corresponding to each input-output group is output; the plurality of preference evaluation information corresponding to each input-output group includes preference evaluation information obtained by performing multi-objective preference evaluation on the input-output group by the pre-trained vector reward model using at least two evaluation attributes; and according to the plurality of preference evaluation information corresponding to each input-output group, the preference evaluation information corresponding to each input-output group is sorted under each evaluation attribute to obtain preference evaluation ranking information of each input-output group under each evaluation attribute.

[0151] In a specific implementation, when the computer device inputs each input-output group into a pre-trained vector reward model and outputs the preference evaluation ranking information corresponding to each input-output group, the computer device may input each input-output group into a pre-trained vector reward model and output multiple preference evaluation information corresponding to each input-output group, and the multiple preference evaluation information corresponding to each input-output group includes the preference evaluation information obtained by performing multi-objective preference evaluation on the input-output group using at least two evaluation attributes by the pre-trained vector reward model. In this way, the computer device may sort the preference evaluation information corresponding to each input-output group under each evaluation attribute according to the multiple preference evaluation information corresponding to each input-output group, and obtain the preference evaluation ranking information of each input-output group under each evaluation attribute.

[0152] Therefore, in the process of training the text processing strategy model using the preference evaluation ranking information corresponding to each input and output group, the preference evaluation ranking information of each input and output group under each evaluation attribute can be used to train the text processing strategy model, so that the diverse preferences of the pre-trained text processing strategy model are reflected more accurately, while also alleviating the conflict between multi-target preferences to a certain extent, and being more in line with actual preference situations.

[0153] The technical solution of this embodiment is to input each input-output group into a pre-trained vector reward model, and output multiple preference evaluation information corresponding to each input-output group; the multiple preference evaluation information corresponding to each input-output group includes the preference evaluation information obtained by performing multi-objective preference evaluation on the input-output group by the pre-trained vector reward model using at least two evaluation attributes; according to the multiple preference evaluation information corresponding to each input-output group, the preference evaluation information corresponding to each input-output group is sorted under each evaluation attribute to obtain the preference evaluation sorting information of each input-output group under each evaluation attribute.

[0154] In this way, a multi-objective preference evaluation is performed on the input-output group under multiple evaluation attributes through the pre-trained vector reward model, so that the preference evaluation ranking information of each input-output group under each evaluation attribute can be obtained, so that the evaluation results can reflect the diverse preferences more accurately, and achieve a technical effect that can more fully represent the diversity of multi-user preferences and comprehensively consider the different preferences of different users and possible conflicts between different preferences.

[0155] In some embodiments, to facilitate understanding by those skilled in the art, Figure 2 This paper provides a process framework diagram of the processing method of a text processing model, which can achieve more fine-grained RLHF training. It specifically includes the following three parts:

[0156] A-1. Multi-objective vector reward model training

[0157] For the training of the vector reward model, we first train the reward model from the perspective of multi-target vector representation. In the process of fine-tuning the large language model according to RLHF, the quality of the reward model as a preference proxy is particularly critical. If the reward model cannot well reflect human preferences or properly evaluate the language model, the subsequent PPO (Proximal Policy Optimization) process will not learn preference knowledge, and may even cause the model effect to deteriorate.

[0158] Therefore, in this embodiment, a reward model represented by a multi-objective vector is used. First, the prompt (prompt information) space and the reply (reply content information) space are defined as X, Y, is the unit circle under the L2 norm in regularization. Represents a policy model.

[0159] For prompt , the reply is In the case of K targets, the reward function is defined as follows:

[0160]

[0161] Each It is a rating of a single evaluation attribute, such as helpfulness, correctness, redundancy, etc. express Distribution of, using regression loss to train multi-objective reward models :

[0162]

[0163] The trained reward model can be used for any Score from the perspective of K evaluation attributes to achieve preference evaluation.

[0164] In some embodiments, when calculating the loss function of the reward model, a threshold m may be further set to represent the two responses. and There is not only a quality gap between them, but the quality gap must also be greater than a preset threshold m.

[0165] In other embodiments, A-1 models the problem of diversified preferences based on a vector reward model, and may also integrate diversified preferences in other nonlinear ways, such as using an absolute value method.

[0166] In this way, compared with the existing technology that only gives an overall score to the entire response based on different attributes, preferences in different directions are sent into the same space, and the vector reward model is used to reflect diverse preferences and alleviate possible conflicts between preferences, which is more in line with actual preferences.

[0167] A-2. Evaluate and rank responses based on reward model

[0168] This module is used to perform multi-objective evaluation of the responses generated by the policy model.

[0169] First from The model generates a series of response content information based on the prompt information, and then uses the A-1 pre-trained reward model to score from multiple dimensions (i.e., from multiple evaluation attributes). In order to express preferences in a more fine-grained manner, each evaluation attribute in the multi-objective reward space is regarded as a direction in the space, i.e., a unit vector , then the conditional preference reward R(x,v,y) is expressed as:

[0170]

[0171] In order to incorporate user preferences into the large language model, when generating responses When the attribute Add prompt , so reply .

[0172] A-3. Iterative DPO Optimization

[0173] The A2 module gives the preference evaluation ranking information of the reply content information output by the previous round of strategy model, and combines the DPO algorithm to iteratively update the strategy model. The specific steps refer to the embodiment of the processing method of the above-mentioned text processing model, which will not be repeated here.

[0174] The processing method of the above text processing model comprehensively considers the different preferences of different users and the possible conflicts between different preferences, and thus proposes a vector-based multi-objective reward to model the diverse preferences of multiple users, so as to generate more diverse responses, and at the same time express the diverse preferences in vector form to alleviate the possible conflicts between the preferences. At the same time, the vector reward model and the model optimized after each round of alignment are used for multiple rounds of alignment to dynamically improve the capabilities of the large language model itself and reduce its deviation from the actual changing data distribution. Finally, the DPO method is used in the iterative alignment process to achieve better results in terms of the stability of model training.

[0175] In one embodiment, Figure 3 As shown, another processing method of the text processing model is provided, comprising the following steps:

[0176] Step S302, obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, and obtaining multiple input and output groups corresponding to the prompt information.

[0177] Step S304, input each input-output group into the pre-trained vector reward model, and output the preference evaluation ranking information corresponding to each input-output group.

[0178] Step S306 , determining the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each input-output group.

[0179] Step S308, determining the information divergence of the text processing strategy model to be trained and the text processing reference model according to the first input-output group and the second input-output group.

[0180] Step S310, obtaining the corresponding initial loss function of the text processing strategy model to be trained under the direct preference optimization method.

[0181] Step S312: adjusting the initial loss function according to the information divergence to obtain the target loss function.

[0182] Step S314, obtaining the first strategy cumulative probability of the text processing strategy model to be trained outputting the first reply content information under the input prompt information, and the second strategy cumulative probability of outputting the second reply content information.

[0183] Step S316, obtaining a first reference cumulative probability of the text processing reference model outputting the first reply content information under the input prompt information, and a second reference cumulative probability of outputting the second reply content information.

[0184] Step S318, calculating the first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability through the target loss function to obtain the loss value of the text processing strategy model to be trained.

[0185] Step S320, adjust the text processing strategy model to be trained according to the loss value to obtain the adjusted text processing strategy model, and use the adjusted text processing strategy model as the text processing strategy model to be trained, and return to the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, until the text processing strategy model to be trained meets the training end condition.

[0186] It should be noted that the specific limitations of the above steps can be found in the specific limitations of the processing method of a text processing model mentioned above, which will not be repeated here.

[0187] In one embodiment, Figure 4 As shown, a text processing method is provided. This embodiment uses the method applied to a computer device as an example for illustration, and includes the following steps:

[0188] Step S410, obtaining text information to be responded to.

[0189] Step S420, input the text information into the pre-trained text processing strategy model, and output the target reply content information.

[0190] Among them, the pre-trained text processing strategy model is obtained by adopting the processing method of the above-mentioned text processing model.

[0191] In a specific implementation, a computer device may obtain text information to be responded to, input the text information into a pre-trained text processing strategy model, and output target reply content information for responding to the text information.

[0192] The above-mentioned text processing method obtains text information to be responded to; inputs the text information into a pre-trained text processing strategy model, and outputs target reply content information; the pre-trained text processing strategy model is obtained by using the processing method of the above-mentioned text processing model. In this way, the pre-trained text processing strategy model is obtained by iteratively updating the large language model by combining the vector reward model with multiple evaluation attributes to perform multi-objective preference evaluation on sample data. The vector reward model can reflect diverse preferences, and the reflection of diverse preferences is more accurate. At the same time, it also alleviates the possible conflicts between multi-objective preferences to a certain extent, which is more in line with the actual preference situation, so that the pre-trained text processing strategy model can process text information more accurately. In addition, based on the preference evaluation ranking information output by the pre-trained vector reward model, iterative training is used to continuously iterate and optimize the model, dynamically improve the ability of the large language model itself, reduce its deviation from the actual changing data distribution, thereby alleviating the model distribution deviation phenomenon caused by the passage of time, so that the text processing ability of the pre-trained text processing strategy model is further improved, and the technical effect of improving the accuracy of text processing is achieved.

[0193] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0194] Based on the same inventive concept, the embodiment of the present application also provides a text processing model processing device for implementing the text processing model processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the processing device for one or more text processing models provided below can refer to the limitations of the text processing model processing method above, and will not be repeated here.

[0195] In an exemplary embodiment, Figure 5 As shown, a processing device for a text processing model is provided, comprising: an acquisition module 510, a sorting module 520, a determination module 530 and an adjustment module 540, wherein:

[0196] The acquisition module 510 is used to obtain multiple reply content information output by the text processing strategy model to be trained for the prompt information, and obtain multiple input-output groups corresponding to the prompt information; the multiple input-output groups correspond to the prompt information and different reply content information respectively; the text processing strategy model is a large language model whose model parameters are to be adjusted.

[0197] The sorting module 520 is used to input each of the input-output groups into a pre-trained vector reward model, and output preference evaluation sorting information corresponding to each of the input-output groups; the preference evaluation sorting information is obtained by performing a multi-objective preference evaluation on the input-output groups using at least two evaluation attributes of the pre-trained vector reward model.

[0198] The determination module 530 is used to determine the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each of the input and output groups.

[0199] The adjustment module 540 is used to adjust the text processing strategy model to be trained according to the loss value to obtain the adjusted text processing strategy model, and use the adjusted text processing strategy model as the text processing strategy model to be trained, and return to the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, until the text processing strategy model to be trained meets the training end condition.

[0200] In one embodiment, the determination module 530 includes: a function acquisition unit, used to obtain the corresponding initial loss function of the text processing strategy model to be trained under the direct preference optimization method; a loss value determination unit, used to determine the loss value of the text processing strategy model to be trained according to the information divergence between the text processing strategy model to be trained and the text processing reference model through the initial loss function; the text processing reference model is a large language model with fixed model parameters.

[0201] In one embodiment, the device also includes: a screening module, used to determine the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each of the input-output groups; the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group; a divergence determination module, used to determine the information divergence of the text processing strategy model to be trained and the text processing reference model according to the first input-output group and the second input-output group.

[0202] In one embodiment, the divergence determination module includes: a probability determination unit, used to obtain the strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and to obtain the reference cumulative probability that the text processing reference model outputs the second reply content information when the prompt information is input; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group; a divergence determination unit, used to determine the information divergence of the text processing strategy model to be trained and the text processing reference model according to the information divergence of the strategy cumulative probability and the reference cumulative probability.

[0203] In one of the embodiments, the loss value determination unit is specifically used to: adjust the initial loss function according to the information divergence to obtain a target loss function; and determine the loss value of the text processing strategy model to be trained according to the target loss function.

[0204] In one embodiment, the loss value determination unit is specifically used to: obtain the first strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and the second strategy cumulative probability that the text processing strategy model to be trained outputs the second reply content information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group; obtain the first reference cumulative probability that the text processing reference model outputs the first reply content information when the prompt information is input, and the second reference cumulative probability that the text processing reference model outputs the second reply content information; through the target loss function, the first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability are calculated to obtain the loss value of the text processing strategy model to be trained.

[0205] In one embodiment, the sorting module 520 includes: an evaluation unit, used to input each of the input-output groups into the pre-trained vector reward model, and output a plurality of preference evaluation information corresponding to each of the input-output groups; the plurality of preference evaluation information corresponding to each of the input-output groups includes preference evaluation information obtained by performing multi-objective preference evaluation on the input-output groups by the pre-trained vector reward model using at least two of the evaluation attributes; and a sorting unit, used to sort the preference evaluation information corresponding to each of the input-output groups under each of the evaluation attributes according to the plurality of preference evaluation information corresponding to each of the input-output groups, and obtain preference evaluation sorting information of each of the input-output groups under each of the evaluation attributes.

[0206] Each module in the processing device of the above text processing model can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0207] Based on the same inventive concept, the embodiment of the present application also provides a text processing device for implementing the above-mentioned text processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more text processing device embodiments provided below can refer to the limitations on the text processing method above, and will not be repeated here.

[0208] In an exemplary embodiment, Figure 6 As shown, a text processing device is provided, including: a text acquisition module 610 and an output module 620, wherein:

[0209] The text acquisition module 610 is used to acquire text information to be responded to.

[0210] The output module 620 is used to input the text information into a pre-trained text processing strategy model and output the target reply content information; the pre-trained text processing strategy model is obtained by using the processing method of the text processing model described above.

[0211] Each module in the above-mentioned text processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0212] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store initial loss function data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a processing method and / or a text processing method of a text processing model is implemented.

[0213] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0214] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0215] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0216] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0217] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0218] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0219] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A processing method for a text processing model, characterized in that: The method comprises: Acquire multiple reply content information output by the text processing strategy model to be trained for the prompt information, and obtain multiple input and output groups corresponding to the prompt information; the multiple input and output groups correspond to the prompt information and different reply content information respectively; the text processing strategy model is a large language model whose model parameters are to be adjusted; Inputting each of the input-output groups into a pre-trained vector reward model, and outputting preference evaluation ranking information corresponding to each of the input-output groups; the preference evaluation ranking information is determined by performing a multi-objective preference evaluation on the input-output groups by the pre-trained vector reward model using at least two evaluation attributes; Determining the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each of the input and output groups; The text processing strategy model to be trained is adjusted according to the loss value to obtain the adjusted text processing strategy model, and the adjusted text processing strategy model is used as the text processing strategy model to be trained, and the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information is returned until the text processing strategy model to be trained meets the training end condition.

2. The method according to claim 1, characterized in that Determining the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each of the input and output groups includes: Obtaining the initial loss function corresponding to the text processing strategy model to be trained under the direct preference optimization method; The loss value of the text processing strategy model to be trained is determined through the initial loss function according to the information divergence of the text processing strategy model to be trained and the text processing reference model; the text processing reference model is a large language model with fixed model parameters.

3. The method according to claim 2, characterized in that The method further comprises: Determine the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each of the input-output groups; the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group; According to the first input-output group and the second input-output group, the information divergence of the text processing strategy model to be trained and the text processing reference model is determined.

4. The method according to claim 3, characterized in that: The step of determining the information divergence of the text processing strategy model to be trained and the text processing reference model according to the first input-output group and the second input-output group includes: Obtain the strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and obtain the reference cumulative probability that the text processing reference model outputs the second reply content information when the prompt information is input; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group; According to the information divergence of the strategy cumulative probability and the reference cumulative probability, the information divergence of the text processing strategy model to be trained and the text processing reference model is determined.

5. The method according to claim 3, characterized in that: Determining the loss value of the text processing strategy model to be trained by the initial loss function according to the information divergence of the text processing strategy model to be trained and the text processing reference model includes: According to the information divergence, the initial loss function is adjusted to obtain a target loss function; According to the target loss function, a loss value of the text processing strategy model to be trained is determined.

6. The method according to claim 5, characterized in that Determining the loss value of the text processing strategy model to be trained according to the target loss function includes: Obtain the first strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and the second strategy cumulative probability that the text processing strategy model to be trained outputs the second reply content information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group; Obtaining a first reference cumulative probability that the text processing reference model outputs the first reply content information when the prompt information is input, and a second reference cumulative probability that the text processing reference model outputs the second reply content information; The first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability are calculated by using the target loss function to obtain a loss value of the text processing strategy model to be trained.

7. The method according to any one of claims 1 to 6, characterized in that: Inputting each of the input-output groups into a pre-trained vector reward model and outputting preference evaluation ranking information corresponding to each of the input-output groups includes: Inputting each of the input-output groups into the pre-trained vector reward model, and outputting a plurality of preference evaluation information corresponding to each of the input-output groups; the plurality of preference evaluation information corresponding to each of the input-output groups includes preference evaluation information obtained by performing multi-objective preference evaluation on the input-output groups by the pre-trained vector reward model using at least two of the evaluation attributes; According to the plurality of preference evaluation information corresponding to each of the input-output groups, the preference evaluation information corresponding to each of the input-output groups is sorted under each of the evaluation attributes to obtain the preference evaluation sorting information of each of the input-output groups under each of the evaluation attributes.

8. A text processing method, characterized in that: The method comprises: Get the text information to be responded to; The text information is input into a pre-trained text processing strategy model, and the target reply content information is output; the pre-trained text processing strategy model is obtained by using the processing method of the text processing model described in any one of claims 1 to 7.

9. A processing device for a text processing model, characterized in that: The device comprises: An acquisition module is used to acquire multiple reply content information output by the text processing strategy model to be trained for the prompt information, and obtain multiple input and output groups corresponding to the prompt information; the multiple input and output groups correspond to the prompt information and different reply content information respectively; the text processing strategy model is a large language model whose model parameters are to be adjusted; A sorting module, used to input each of the input-output groups into a pre-trained vector reward model, and output preference evaluation sorting information corresponding to each of the input-output groups; the preference evaluation sorting information is determined by performing a multi-objective preference evaluation on the input-output groups by the pre-trained vector reward model using at least two evaluation attributes; A determination module, used to determine the loss value of the text processing strategy model to be trained according to the preference evaluation ranking information corresponding to each of the input and output groups; An adjustment module is used to adjust the text processing strategy model to be trained according to the loss value to obtain the adjusted text processing strategy model, and use the adjusted text processing strategy model as the text processing strategy model to be trained, and return to the step of obtaining multiple reply content information output by the text processing strategy model to be trained for the prompt information, until the text processing strategy model to be trained meets the training end condition.

10. The device according to claim 9, characterized in that The determination module comprises: A function acquisition unit, used to acquire the initial loss function corresponding to the text processing strategy model to be trained under the direct preference optimization method; A loss value determination unit is used to determine the loss value of the text processing strategy model to be trained through the initial loss function according to the information divergence of the text processing strategy model to be trained and the text processing reference model; the text processing reference model is a large language model with fixed model parameters.

11. The device according to claim 10, characterized in that The device also includes: A screening module, configured to determine the first input-output group and the second input-output group corresponding to the prompt information according to the preference evaluation ranking information corresponding to each of the input-output groups; the preference evaluation ranking of the first input-output group is higher than the preference evaluation ranking of the second input-output group; The divergence determination module is used to determine the information divergence of the text processing strategy model to be trained and the text processing reference model according to the first input-output group and the second input-output group.

12. The device according to claim 11, characterized in that The divergence determination module comprises: A probability determination unit, used for obtaining the strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information under the input of the prompt information, and obtaining the reference cumulative probability that the text processing reference model outputs the second reply content information under the input of the prompt information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group; The divergence determination unit is used to determine the information divergence of the text processing strategy model to be trained and the text processing reference model according to the information divergence of the strategy cumulative probability and the reference cumulative probability.

13. The device according to claim 11, characterized in that The loss value determination unit is specifically used to: According to the information divergence, the initial loss function is adjusted to obtain a target loss function; According to the target loss function, a loss value of the text processing strategy model to be trained is determined.

14. The device according to claim 13, characterized in that The loss value determination unit is specifically used to: Obtain the first strategy cumulative probability that the text processing strategy model to be trained outputs the first reply content information when the prompt information is input, and the second strategy cumulative probability that the text processing strategy model to be trained outputs the second reply content information; the first reply content information is the reply content information in the first input-output group; the second reply content information is the reply content information in the second input-output group; Obtaining a first reference cumulative probability that the text processing reference model outputs the first reply content information when the prompt information is input, and a second reference cumulative probability that the text processing reference model outputs the second reply content information; The first strategy cumulative probability, the second strategy cumulative probability, the first reference cumulative probability and the second reference cumulative probability are calculated by using the target loss function to obtain a loss value of the text processing strategy model to be trained.

15. The device according to any one of claims 9 to 14, characterized in that: The sorting module comprises: An evaluation unit, used to input each of the input-output groups into the pre-trained vector reward model, and output a plurality of preference evaluation information corresponding to each of the input-output groups; the plurality of preference evaluation information corresponding to each of the input-output groups includes preference evaluation information obtained by the pre-trained vector reward model using at least two of the evaluation attributes to perform multi-objective preference evaluation on the input-output groups; A sorting unit is used to sort the preference evaluation information corresponding to each input-output group under each evaluation attribute according to the multiple preference evaluation information corresponding to each input-output group, so as to obtain the preference evaluation sorting information of each input-output group under each evaluation attribute.

16. A text processing device, characterized in that: The device comprises: A text acquisition module is used to acquire text information to be responded to; An output module is used to input the text information into a pre-trained text processing strategy model and output target reply content information; the pre-trained text processing strategy model is obtained by using the processing method of the text processing model described in any one of claims 1 to 7.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Data processing method and server based on man-machine interaction model or large model

    CN116757270A

  • Traditional Chinese medicine large model based on reinforcement learning and preference alignment method

    CN118230908A