Model training method and device, electronic equipment and storage medium

Abstract

The invention discloses a model training method and device, electronic equipment and a computer storage medium. The model training method comprises the following steps: performing supervision fine tuning on an initial baseline model based on a first training sample set in a preset training sample set to obtain a supervision fine tuning model and a preference data sample; performing optimization training on the initial reward model based on the preference data sample to obtain a target reward model, and performing optimization training on the supervision fine tuning model based on the preference data sample to obtain a preference optimization model; generating a first reward signal based on the preference optimization model and the first training sample set, and generating a second reward signal based on the preference optimization model, the target reward model and a second training sample set in the preset training sample set; and performing optimization training on the preference optimization model based on the first reward signal, the second reward signal and the second training sample set to obtain a target object generation type pre-training model. By adopting the method, the generalization ability of the model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular, to a model training method, apparatus, electronic device, and storage medium. Background Art

[0002] Background Art: With the development of on-demand food delivery (OFD) platforms, related platforms need to simultaneously handle multiple tasks such as user intent understanding, product attribute prediction, title rewriting, and relevance matching, posing higher requirements for the semantic understanding and generation capabilities of models.

[0003] However, due to the complex data structures of objects such as products and stores, diverse tasks, and scarce labeled data in the OFD scenario, traditional models are difficult to directly adapt to, and it is difficult to meet the platform's requirements for low-resource and highly adaptable semantic models. Therefore, there is an urgent need for an object generative pre-training model with efficient training capabilities to improve the training efficiency and generalization ability of the model. Summary of the Invention

[0004] Embodiments of the present application provide a model training method, apparatus, electronic device, and computer storage medium, which can improve the training efficiency and generalization ability of the model. The above technical solutions are as follows: In a first aspect, an embodiment of the present application provides a model training method, including: Obtain an initial baseline model, an initial reward model, and a preset training sample set, where the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; Perform supervised fine-tuning on the initial baseline model based on a first training sample set in the preset training sample set to obtain a supervised fine-tuned model and preference data samples; Perform optimization training on the initial reward model based on the preference data samples to obtain a target reward model, and perform optimization training on the supervised fine-tuned model based on the preference data samples to obtain a preference-optimized model; Generate a first reward signal based on the preference-optimized model and the first training sample set, and generate a second reward signal based on the preference-optimized model, the target reward model, and a second training sample set in the preset training sample set; Perform optimization training on the preference-optimized model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

[0005] In a possible implementation manner, the performing supervised fine-tuning on the initial baseline model based on a first training sample set in the preset training sample set to obtain a supervised fine-tuned model and preference data samples includes: Based on the first training sample set in the above preset training sample set, the above initial baseline model is updated with parameters in multiple rounds using the supervised learning method to obtain a supervised fine-tuning model; During the process of updating the parameters of the above initial baseline model in multiple rounds, the above first training sample set is sampled to obtain input samples and multiple candidate responses corresponding to each of the above input samples; Based on a preset metric, multiple candidate responses corresponding to each of the above input samples are screened to obtain a first response and a second response corresponding to each of the above input samples, so as to construct paired preference data samples, where the above first response is better than the above second response.

[0006] In a possible implementation manner, the above-mentioned optimizing and training the above initial reward model based on the above preference data samples to obtain a target reward model includes: Inputting each of the above input samples included in the above preference data samples, and the above first response and the above second response corresponding to each of the above input samples into the above initial reward model; Performing a scoring value generation process on the above first response and the above second response corresponding to each of the above input samples through the above initial reward model to obtain a first response scoring value corresponding to the above first response and a second response scoring value corresponding to the above second response; Construct a scoring loss function according to the above first response scoring value and the above second response scoring value; Based on the above scoring loss function, iteratively optimize the parameters of the above initial reward model to obtain a target reward model.

[0007] In a possible implementation manner, the above-mentioned optimizing and training the above supervised fine-tuning model based on the above preference data samples to obtain a preference optimization model includes: Input each of the above preference data samples into the above supervised fine-tuning model; Calculate the generation probability information of the above first response and the above second response in each of the above preference data samples by the above supervised fine-tuning model; Construct a contrast loss function based on the above generation probability information; Based on the above contrast loss function, iteratively update the parameters of the above supervised fine-tuning model to obtain a preference optimization model.

[0008] In a possible implementation manner, the above-mentioned generating a first reward signal based on the above preference optimization model and the above first training sample set includes: For each first training sample data in the above first training sample set, determine a first predicted response corresponding to the above first training sample data through the above preference optimization model; Compare the above first predicted response with the reference output of the above first training sample data, and calculate a difference metric between the above first predicted response and the above reference output; Generate a first reward signal according to the above difference metric.

[0009] In a possible implementation, the above generating a second reward signal based on the above preference optimization model, the above target reward model, and a second training sample set in the above preset training sample set includes: For each second training sample data in the second training sample set in the above preset training sample set, determine a second predicted response corresponding to the second training sample data through the above preference optimization model; Input the above second training sample data and the corresponding second predicted response into the above target reward model to perform reward signal generation processing through the above target reward model, and generate a second reward signal.

[0010] In a possible implementation, the above optimizing and training the above preference optimization model based on the above first reward signal, the above second reward signal, and the above second training sample set to obtain a target object generative pre-training model includes: Perform reinforcement learning on the above preference optimization model based on the above first reward signal to obtain a reinforced object generative pre-training model; Perform unsupervised training on the above reinforced object generative pre-training model based on the above second reward signal and the above second training sample set to obtain a target object generative pre-training model.

[0011] In a possible implementation, the above performing unsupervised training on the above reinforced object generative pre-training model based on the above second reward signal and the above second training sample set to obtain a target object generative pre-training model includes: For each second training sample data in the above second training sample set, use the above reinforced object generative pre-training model to generate object response information corresponding to the second training sample data; Input the above second training sample data and the corresponding above object response information into the target reward model to obtain a reward score for reinforcement training; Construct a reinforcement learning objective function with the reward score as the return signal; Optimize the parameters of the above reinforced object generative pre-training model based on the above reinforcement learning objective function to obtain a trained target object generative pre-training model.

[0012] In a second aspect, an embodiment of the present application provides a model training method, including: Obtain a target object generative pre-training model and a task training data set corresponding to a preset sub-task; Use the above-mentioned target object generative pre-training model as the initial base model, and perform fine-tuning training using the above-mentioned task training data set to obtain a target task model for the preset subtask; Among them, the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by the model training method provided in the first aspect.

[0013] In a third aspect, an embodiment of the present application provides a model interaction method, and the above method includes: Obtain object request data; Input the above object request data into the target object generative pre-training model, and analyze and process the above object request data through the above target object generative pre-training model to generate corresponding object response data; Output the above object response data; Among them, the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by the model training method provided in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a model training device, including: A first acquisition module, configured to acquire an initial baseline model, an initial reward model, and a preset training sample set, where the above preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; A fine-tuning module, configured to perform supervised fine-tuning on the above initial baseline model based on the first training sample set in the above preset training sample set to obtain a supervised fine-tuning model and preference data samples; A first training module, configured to perform optimization training on the above initial reward model based on the above preference data samples to obtain a target reward model, and perform optimization training on the above supervised fine-tuning model based on the above preference data samples to obtain a preference optimization model; A generation module, configured to generate a first reward signal based on the above preference optimization model and the above first training sample set, and generate a second reward signal based on the above preference optimization model, the above target reward model, and the second training sample set in the above preset training sample set; A second training module, configured to perform optimization training on the above preference optimization model based on the above first reward signal, the above second reward signal, and the above second training sample set to obtain a target object generative pre-training model.

[0015] In a fifth aspect, an embodiment of the present application provides a model training device, including: A second acquisition module, configured to acquire a target object generative pre-training model and a task training data set corresponding to a preset subtask; The third training module is used to use the above-mentioned target object generative pre-training model as the initialized basic model, and perform fine-tuning training using the above-mentioned task training dataset to obtain a target task model for the preset subtask; Among them, the above-mentioned target object generative pre-training model is a model generated by the model training method provided in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a model interaction device, including: The third acquisition module is used to acquire object request data; The input module is used to input the above-mentioned object request data into the target object generative pre-training model, and use the above-mentioned target object generative pre-training model to analyze and process the above-mentioned object request data to generate corresponding object response data; The output module is used to output the above-mentioned object response data; Among them, the above-mentioned target object generative pre-training model is a model generated by the model training method provided in the first aspect.

[0017] In a seventh aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory; among them, the above-mentioned memory stores a computer program, and when the processor executes the computer program, it implements the method steps provided in any one of the first aspect, the second aspect, and the third aspect of the embodiment of the present application.

[0018] In an eighth aspect, an embodiment of the present application provides a computer storage medium, and the above-mentioned computer storage medium stores multiple instructions, and the above-mentioned instructions are suitable for being loaded and executed by a processor to implement the method steps provided in any one of the first aspect, the second aspect, and the third aspect of the embodiment of the present application.

[0019] In the embodiments of the present application, an initial baseline model, an initial reward model, and a preset training sample set are obtained. Among them, the above-mentioned preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; the initial baseline model is supervised and fine-tuned based on the first training sample set in the above-mentioned preset training sample set to obtain a supervised fine-tuned model and preference data samples; the initial reward model is optimized and trained based on the above-mentioned preference data samples to obtain a target reward model, and the supervised fine-tuned model is optimized and trained based on the above-mentioned preference data samples to obtain a preference-optimized model; a first reward signal is generated based on the above-mentioned preference-optimized model and the above-mentioned first training sample set, and a second reward signal is generated based on the above-mentioned preference-optimized model, the above-mentioned target reward model, and the second training sample set in the above-mentioned preset training sample set; the preference-optimized model is optimized and trained based on the above-mentioned first reward signal, the above-mentioned second reward signal, and the above-mentioned second training sample set to obtain a target object generative pre-training model. On the one hand, fine-tuning the initial baseline model based on supervised samples can improve the basic capabilities of the model for various tasks; on the other hand, by constructing preference data samples to jointly train the reward model and the policy model, the model behavior and human preferences are effectively aligned. At the same time, two types of reward signals are introduced, which respectively come from the reward model and the preference-optimized model, and are fused in the unsupervised data training stage, effectively improving the generalization ability of the model and significantly reducing the dependence on high-quality manually annotated data. The finally obtained target object generative pre-training model has better object understanding, policy generation, and interaction response capabilities, and can better adapt to the intelligent requirements of multiple scenarios in the on-demand food delivery platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 It is a schematic diagram of the system architecture of a model training method provided by an exemplary embodiment of the present application; Figure 2 It is a schematic flowchart of a model training method provided by an exemplary embodiment of the present application; Figure 3 It is a schematic flowchart of another model training method provided by an exemplary embodiment of the present application; Figure 4 It is a schematic flowchart of another model training method provided by an exemplary embodiment of the present application; Figure 5Schematic diagram of another model training method provided by an exemplary embodiment of the present application; Figure 6 Schematic diagram of another model training method provided by an exemplary embodiment of the present application; Figure 7 Overall schematic diagram of a training and optimization method for a target object generative pre-training model provided by an exemplary embodiment of the present application; Figure 8 Schematic diagram of a performance comparison result provided by an exemplary embodiment of the present application; Figure 9 Schematic diagram of an ablation experiment result provided by an exemplary embodiment of the present application; Figure 10 Schematic diagram of an experimental result provided by an exemplary embodiment of the present application; Figure 11 Schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application; Figure 12 Schematic diagram of the structure of another model training device provided by an exemplary embodiment of the present application; Figure 13 Schematic diagram of the structure of a model interaction device provided by an exemplary embodiment of the present application; Figure 14 Schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Detailed implementation manners

[0022] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

[0023] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes, and cannot be construed as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific situations. In addition, in the description of the present application, unless otherwise specified, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0024] Please refer to Figure 1 , Figure 1 Exemplary system architecture diagram of a model training method provided by an embodiment of the present application.

[0025] As Figure 1 shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links. For example, the wired communication links include optical fibers, twisted pairs, or coaxial cables, and the wireless communication links include Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0026] The terminal 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103. Alternatively, the terminal 101 can interact with the server 103 through the network 102, and then receive messages or data sent by other users to the server 103. The terminal 101 can be hardware or software. When the terminal 101 is hardware, it can be various electronic devices, including but not limited to tablet computers, laptop portable computers, and desktop computers, etc. When the terminal 101 is software, it can be installed in the above-listed electronic devices, which can be implemented as multiple software or software modules (for example, used to provide distributed services), or can be implemented as a single software or software module, and no specific limitation is made here.

[0027] In the embodiment of the present application, the terminal 101 first obtains an initial baseline model, an initial reward model, and a preset training sample set. Among them, the above preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks. The preset training sample set may include a first training sample set and a second training sample set. After the sample set and the initial model architecture are ready, the terminal 101 can then perform supervised fine-tuning on the initial baseline model based on the above first training sample set to obtain a supervised fine-tuned model and preference data samples; optimize and train the initial reward model based on the above preference data samples to obtain a target reward model, and optimize and train the supervised fine-tuned model based on the above preference data samples to obtain a preference optimized model; subsequently, generate a first reward signal based on the above preference optimized model and the above first training sample set, and generate a second reward signal based on the above preference optimized model, the above target reward model, and the above second training sample set; and then optimize and train the preference optimized model based on the above first reward signal, the above second reward signal, and the above second training sample set to obtain a target object generative pre-training model.

[0028] Server 103 may be a server that provides various services. It should be noted that server 103 can be hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When server 103 is software, it can be implemented as multiple software or software modules (such as those used to provide distributed services) or as a single software or software module, and no specific limitation is made here.

[0029] Alternatively, the system architecture may not include server 103. In other words, server 103 can be an optional device in the embodiments of this application. That is, the method provided in the embodiments of this application can be applied to a system structure that only includes terminal 101, and no limitation is made in the embodiments of this application.

[0030] It should be understood that Figure 1 the numbers of terminals, networks, and servers in

[0031] Please refer to Figure 2 , Figure 2 , which is a schematic flowchart of a model training method provided in the embodiments of this application. The execution subject of the embodiments of this application can be a terminal that executes the model training method, a processor in the terminal that executes the model training method, or a training service of a feature extraction model in the terminal that executes the model training method. For ease of description, the following takes the execution subject as the processor in the terminal as an example to introduce the specific execution process of the model training method.

[0032] S201: Obtain an initial baseline model, an initial reward model, and a preset training sample set, where the above preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks.

[0033] Among them, the initial baseline model can be a large language model that has completed general language pre-training and can be used as the starting model for the training process. The initial baseline model can have basic language understanding and generation capabilities but has not been fine-tuned for specific tasks. For example, the initial baseline model can be an open-source large language model, such as Tongyi Qianwen large model Qwen2.5-7B in the related art. In addition, the initial reward model is an initial reward scorer and can be used to estimate the quality or preference degree of the model generation result. The initial reward model can be inherited from the initial baseline model and has not been trained on preference data.

[0034] Optionally, the object response training task may be a training task that takes the relevant information of a certain type of object as the input target and generates response information for that object. Among them, the above object may be any one of the following types: e-commerce objects (such as products, stores, evaluations, categories, activities, etc.), social objects (such as social dynamics, topics, tags, follow relationships, etc.), medical objects (such as medical records, inspection sheets, clinical notes, etc.), educational objects (such as test questions, knowledge points, courses, answer sheets), and transportation objects (travel routes, locations, stations, etc.), etc., which may specifically depend on the relevant platform or scenario type.

[0035] For example, when the object is a test question, the object response training task may be to generate the corresponding knowledge point label or reference answer for the question according to the question content. When the object is a travel route, the object response training task may be to generate travel suggestions or congestion prediction results according to the route information.

[0036] Exemplarily, the present application will be described below by taking the object as a commodity as an example. In the case where the object is a commodity, the response training tasks of multiple objects can at least include one or more of the following types of tasks: category prediction type, title generation type, information extraction type, product matching type, search relevance type, and query annotation type. Specifically, the object response training task of the category prediction type can be used to predict the category label of the Standard Product Unit (SPU) to which the commodity belongs according to the basic attribute information of the commodity. For example, for the commodity "Local Flammulina velutipes about 150g; Color: white", its SPU can be predicted by the model to be "White Flammulina velutipes". The object response training task of the title generation type can be used to generate a more suitable new title according to the original title, brand, flavor, specifications and other attributes of the commodity for display. For example, given the original title "XX Brand Night Use Sanitary Napkin 35cm 8 pieces / pack", the generated new title can be "XX Brand Night Use Cotton Soft and Delicate Sanitary Napkin 350mm 8 pieces", which helps to improve the standardization and user readability of the commodity title. The object response training task of the information extraction type can be used to extract structured attribute information from the commodity title or commodity attributes (such as brand, series, origin, etc.). For example, for the title "XXX Brand Vita Moisturizing Cream", the fields extracted by the model can be "Net content: 150; Packaging method: bottled; Packaging specification: 150ml * 1 bottle; Brand: XXX; Quantity: 1 bottle; Category: cream". The object response training task of the product matching type can be used to select a suitable Child Standard Product Unit (CSPU) for the commodity, that is, a finer-grained product identifier. For example, for the commodity "Frozen Terebralia palustris meat about 500 - 550 grams; Net content: 500 grams", the determined Child Standard Product Unit by the model is "Chilled Terebralia palustris". The object response training task of the search relevance type can be used to evaluate the relevance between the user query and the commodity title in the context information corresponding to the commodity. For example, for the query content "Stomach-nourishing porridge" and the commodity title "Stomach-nourishing purple sweet potato porridge [containing purple sweet potato dices]", the model can determine the relevance as "strongly relevant". The object response training task of the query annotation type can be used to assign part-of-speech tags to the user search query. For example, for the user query content "Men's upper wear", the tagged result is "Upper wear: commodity type, Attribute: men's clothing". The data of the above various types of tasks can be respectively organized into corresponding training sample groups and uniformly constitute the above-mentioned preset training sample set for subsequent model training to improve the comprehensive modeling ability of the model in the commodity response generation scenario.

[0037] Optionally, the preset training sample set may include a first training sample set and a second training sample set. Among them, the first training sample set may be a pre-annotated set of annotation data, which can be used for subsequent supervised training. Specifically, the first training sample set may contain multiple training sample groups related to commodity response tasks, and each training sample group may correspond to a specific object response training task. That is to say, the first training sample set may include multiple training sample groups corresponding to multiple object response training tasks. Each training sample group in the first training sample set includes multiple first training sample data, as well as the reference output corresponding to each first training sample data, that is, the true label. In addition, the second training sample set may be a set of commodity-related input data that does not include reference outputs (labels). Specifically, the second training sample set may contain multiple training sample groups related to commodity response tasks, and each training sample group may correspond to a specific object response training task. That is to say, the second training sample set may include multiple training sample groups corresponding to multiple object response training tasks. Each training sample group in the second training sample set includes multiple second training sample data.

[0038] S202: Perform supervised fine-tuning on the above initial baseline model based on the first training sample set in the above preset training sample set to obtain a supervised fine-tuning model and preference data samples.

[0039] In one embodiment, fine-tuning methods such as Low-Rank Adaptation of Large Language Models (LoRA) or full-scale fine-tuning can be used to perform supervised fine-tuning on the above initial baseline model to obtain a Supervised Fine-Tuning (SFT) model.

[0040] It should be noted that during this supervised fine-tuning process, an iterative training method is used to continuously optimize the model parameters, and multiple training epochs can be performed on the complete first training sample set. Each Epoch represents that the model fully learns the entire first training sample set once. Through multiple rounds of training, the model can continuously adjust its internal representation and parameters to more fully learn the general laws and fine-grained features between various tasks, thereby improving its generalization ability for unseen samples. After training is completed, the obtained supervised fine-tuning model has multi-task response capabilities.

[0041] In one embodiment, the preference data sample may refer to a set of one or more sets of output responses constructed based on the same input data, and a data set with the relative preference relationship of "select" and "reject" clearly marked between these responses, which is used to train the model to learn human preferences. Optionally, the preference data sample can be obtained by expanding and constructing the error cases of the supervised fine-tuning model in the object response training task. The finally generated pairwise preference data can be used as the training data input in the subsequent DPO (Direct Preference Optimization) stage, that is, the preference data sample, which can be used to guide the model to be more inclined to generate high-quality responses under the same input conditions, thereby improving the response quality and preference consistency.

[0042] S203: Optimize and train the above initial reward model based on the above preference data sample to obtain a target reward model, and optimize and train the above supervised fine-tuning model based on the above preference data sample to obtain a preference optimization model.

[0043] Among them, the input of the target reward model can be the input sample and the corresponding response, and the output is a score value that can measure the quality of the response. The target reward model can reflect the relative quality of the response, and can be used to evaluate the model generation result in the scenario without reference output, and provide a reward signal for the policy model in the subsequent reinforcement learning stage.

[0044] Optionally, the goal of the preference optimization model can be to learn the ability to select high-quality responses and reject low-quality responses. The preference optimization model tends to generate responses that conform to human preferences under the same input. Compared with the supervised fine-tuning model, the preference optimization model pays more attention to the relative difference in output quality, and can be used to initialize the reinforcement policy model in the subsequent stage.

[0045] In one embodiment, the preference data sample includes multiple pairwise preference data. Each pair of preference data can be composed of the same input, a preferred response and a non-preferred response, which respectively represent the "select" and "reject" relationships in human preferences. Optionally, the input sample and the corresponding responses (including the preferred response and the non-preferred response) can be used as the input to train the reward model to predict the score difference between the two responses, and minimize the expected ranking error, so that the score of the preferred response is higher than that of the non-preferred response. After the training is completed, a target reward model that can quantify the response quality is obtained.

[0046] In one embodiment, direct preference optimization training can be performed on the supervised fine-tuning model based on the same preference data sample. This process uses the supervised fine-tuning model as the initial policy model, and by introducing the DPO loss function, adjusts the model output distribution to make the model more inclined to generate preferred responses and suppress the generation of non-preferred responses under the same input conditions. After the optimization is completed, a preference optimization model that can better conform to the human preference orientation is obtained.

[0047] It should be noted that the two optimization processes of optimizing and training the above initial reward model based on the above preference data sample to obtain the target reward model, and optimizing and training the above supervised fine-tuning model based on the above preference data sample to obtain the preference optimization model can be executed in parallel or sequentially, respectively producing the target reward model and the preference optimization model, providing a scoring ability and a policy basis for the subsequent reinforcement learning training stage.

[0048] S204: Generate a first reward signal based on the above preference optimization model and the above first training sample set, and generate a second reward signal based on the above preference optimization model, the above target reward model, and the second training sample set in the above preset training sample set.

[0049] Optionally, the first reward signal may refer to a score-based reward signal calculated by generating a predicted response based on the preference optimization model on the first training sample set, comparing the predicted response with the known reference output, and using a difference metric. The difference metric can be an edit distance or an automatic text generation quality evaluation metric (Recall-Oriented Understudy for Gisting Evaluation, ROUGE), etc. The first reward signal is a rule-based and task-interpretable reward form.

[0050] Optionally, the second reward signal may refer to generating a predicted response based on the preference optimization model on the second training sample set (without a reference output), and inputting both the input and the response into the target reward model for scoring to obtain a numerical reward representing the generation quality.

[0051] In one embodiment, on the one hand, the preference optimization model is used to generate corresponding predicted responses for each input data in the first training sample set. Since the first training sample set contains a reference output, the generated responses can be compared with the reference output one by one, and a difference metric such as the edit distance is used to calculate the degree of difference between the response and the reference output, further generating a first reward signal for reinforcement learning. The first reward signal can be used to quantify the closeness between the model generation result and the standard answer. On the other hand, the preference optimization model is used to generate responses for the input data in the second training sample set. Since the second training sample set is an unsupervised sample and lacks a reference output, the input data and the generated responses are input into the target reward model together, and the target reward model outputs a scoring result as the second reward signal. This second reward signal can be used to measure the preference matching degree of the model generation output in dimensions such as actual semantic quality, fluency, and integrity.

[0052] The above first reward signal and second reward signal are respectively used for different stages of the reinforcement learning phase to guide the policy model to optimize its generated policy, ensuring that the form of the generated result meets the expectations and improving the consistency between the response quality to the open task input and human preferences.

[0053] S205: Optimize and train the above preference optimization model based on the above first reward signal, the above second reward signal, and the above second training sample set to obtain a target object generative pre-training model.

[0054] In some embodiments, the preference optimization model can first be constructed as a Proximal Policy Optimization (PPO) model, and the first stage of training is performed on the PPO model based on the first reward signal to achieve preliminary reinforcement learning objective optimization. Further, an additional round of training can be performed on the PPO model based on the second reward signal and the second training sample set, so as to further improve the model performance on the basis of the preliminary optimization to obtain a target object generative pre-training model.

[0055] It can be understood that the above target object generative pre-training model can be used as a basic model for supervised fine-tuning, preference modeling, or reinforcement learning optimization of downstream subtasks, improving the accuracy and diversity of object response generation.

[0056] In the embodiments of the present application, an initial baseline model, an initial reward model, and a preset training sample set are obtained. Among them, the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; the initial baseline model is supervised and fine-tuned based on the first training sample set in the preset training sample set to obtain a supervised fine-tuned model and preference data samples; the initial reward model is optimized and trained based on the preference data samples to obtain a target reward model, and the supervised fine-tuned model is optimized and trained based on the preference data samples to obtain a preference-optimized model; a first reward signal is generated based on the preference-optimized model and the first training sample set, and a second reward signal is generated based on the preference-optimized model, the target reward model, and the second training sample set in the preset training sample set; the preference-optimized model is optimized and trained based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model. On the one hand, fine-tuning the initial baseline model based on supervised samples can improve the basic capabilities of the model for various tasks; on the other hand, by constructing preference data samples to jointly train the reward model and the policy model, the model behavior and human preferences are effectively aligned. At the same time, two types of reward signals are introduced, which are respectively derived from the reward model and the preference-optimized model, and are fused in the unsupervised data training stage, effectively improving the generalization ability of the model and significantly reducing the dependence on high-quality manually annotated data. The finally obtained target object generative pre-training model has better object understanding, policy generation, and interaction response capabilities, and can better adapt to the multi-scenario intelligent requirements in relevant platforms.

[0057] In one embodiment, as Figure 3 shown, another model training method is provided, including the following steps: S301: Obtain an initial baseline model, an initial reward model, and a preset training sample set, where the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks.

[0058] Specifically, S301 is the same as S201 and will not be elaborated here.

[0059] S302: Based on the first training sample set in the preset training sample set, the initial baseline model is updated with parameters in multiple rounds in a supervised learning manner to obtain a supervised fine-tuned model.

[0060] In one embodiment, first, a first training sample set in a preset training sample set is selected. The first training sample set can be a supervised data set with manually annotated output labels and cover multiple object response training tasks (such as: product title generation, attribute extraction, category prediction, query annotation, product matching, and search relevance judgment, etc.). Each first training sample includes an input sample, a prompt word, and a corresponding expected output result (i.e., the true label). Next, the initial baseline model is used as the initial state of the pre-trained language model, and supervised learning is performed based on the above first training sample set. The process of supervised learning can include training for multiple epochs. In each epoch, the model outputs a prediction result through forward inference, calculates the loss between the prediction result and the true label (such as cross-entropy loss), and then updates the model parameters through the backpropagation algorithm. This process is iterated until the model converges on the validation set.

[0061] Through the above multi-round parameter update process, the model forms a preliminary representation learning ability and task response ability on multiple product response tasks, and finally obtains a supervised fine-tuning model. This supervised fine-tuning model not only has a general understanding ability for various product information processing tasks, but also provides a robust initialization basis for the subsequent preference training and reinforcement learning stages.

[0062] S303: During the process of performing multi-round parameter updates on the above initial baseline model, sample the above first training sample set to obtain input samples and multiple candidate responses corresponding to each of the above input samples.

[0063] Optionally, during the supervised fine-tuning training process, samples can be sampled from the first training sample set according to task categories or data distributions, and multiple candidate responses are generated based on the currently trained supervised fine-tuning model using a sampling strategy with a higher temperature. This sampling strategy helps to generate response samples containing diverse expressions or potential errors, facilitating the construction of preference data required for subsequent training.

[0064] S304: Screen the multiple candidate responses corresponding to each of the above input samples based on a preset metric to obtain a first response and a second response corresponding to each of the above input samples, so as to construct paired preference data samples, where the above first response is better than the above second response.

[0065] Furthermore, based on a preset metric (such as edit distance, ROUGE, etc.), or combined with manual scoring rules, screen the multiple candidate responses corresponding to each input sample, select the response with relatively better semantic, structural, or content performance as the "first response", and select the response with language defects or information missing as the "second response", thereby constructing paired samples with a clear "select - reject" preference relationship.

[0066] In the embodiments of the present application, the constructed preference data samples can be used to train the reward model and optimize the policy model, enabling the model to directly learn human preferences for high-quality responses, thereby achieving more stable and intuitive behavior adjustment in the subsequent DPO stage, effectively improving the accuracy, naturalness, and adaptability of the generated content.

[0067] S305: Input each of the above input samples included in the above preference data sample, and the above first response and the above second response corresponding to each of the above input samples into the above initial reward model.

[0068] Optionally, input the constructed preference data sample (input + first response + second response) into the initial reward model, and this initial reward model will be used to learn the ability of "preference judgment".

[0069] S306: Perform a scoring value generation process on the above first response and the above second response corresponding to each of the above input samples through the above initial reward model to obtain the first response scoring value corresponding to the above first response and the second response scoring value corresponding to the above second response.

[0070] In one embodiment, forward inference operations can be respectively performed on the first response and the second response in the constructed pairwise preference data based on the initial reward model, so as to generate a scoring value for characterizing the degree of response preference. Specifically, splice the input sample and the first response to form a complete input sequence, and input it into the initial reward model to generate the first response scoring value; similarly, generate the corresponding second response scoring value based on the second response. The first response scoring value and the second response scoring value can be used to construct a scoring loss function subsequently to guide the reward model to learn the preference ranking relationship of response quality.

[0071] S307: Construct a scoring loss function according to the above first response scoring value and the above second response scoring value.

[0072] Optionally, a pairwise learning loss function can be constructed using a ranking loss, and the goal is to encourage the model to give a higher score to the first response. Specifically, the scoring loss function can be represented by the following formula (1): (1); Where represents the parameters of the reward model, represents the scoring loss function, K represents the number of preference data samples in the preference dataset (the number of pairwise samples is K / 2 pairs). D represents the preference data sample set, which can contain multiple groups of (x, y w , y l ) triples; represents a triple sampled from the preference data sample set D: input sample x, preferred response (first response) , non-preferred response (second response) , represents the first response score value, represents the second response score value. σ(·) represents the Sigmoid function, and E(·) represents the mathematical expectation.

[0073] S308: Based on the above scoring loss function, iteratively optimize the parameters of the above initial reward model to obtain the target reward model.

[0074] Specifically, based on the above scoring loss function, use optimization algorithms such as gradient descent to update the parameters of the initial reward model, and finally train to obtain a target reward model that can stably distinguish between good and bad preferences.

[0075] In the embodiments of the present application, by inputting the constructed preference data samples into the initial reward model, generating score values based on paired responses, further constructing a ranking loss function and performing parameter optimization operations, the reward model can learn the preference ranking relationship between responses, so as to provide a more accurate and stable reward signal for policy optimization in the subsequent reinforcement learning stage. This method not only improves the accuracy of the model's preference judgment for responses, but also has good scalability and generality.

[0076] S309: Input each of the above preference data samples into the above supervised fine-tuning model.

[0077] Among them, inputting the above preference data samples into the previously obtained supervised fine-tuning model can be used to further guide the model to understand preference differences, so as to perform behavior adjustment.

[0078] S310: Calculate the generation probability information of the above supervised fine-tuning model for the above first response and the above second response in each of the above preference data samples respectively.

[0079] Specifically, the supervised fine-tuning model makes predictions on the first response and the second response respectively based on the current parameter state, and calculates the conditional probability information of the supervised fine-tuning model for generating the first response and the second response, which is used to reflect the generation preference of the model for the two candidate outputs.

[0080] S311: Construct a contrast loss function based on the above generation probability information.

[0081] Furthermore, a contrastive loss function can be constructed based on this generation probability information. For example, using the contrastive log-probability difference, this function encourages the model to increase the generation probability of the first response and suppress the generation probability of the second response. This contrastive loss function measures the model's ability to distinguish between two responses under the same input conditions. The optimization goal is to maximize the difference between the generation probability of the preferred response and the generation probability of the second response. By minimizing this loss function, the parameters of the supervised fine-tuning model can be updated, further improving the model's ability to fit human preferences.

[0082] Among them, the DPO loss function can be constructed by the following formula (2): (2); in, represents the DPO loss function, represents the initial supervised fine-tuning model, which can be used as a reference model. Represents the target to be trained, x represents the input sample in the preferred data sample, Indicates the preferred response chosen by humans, i.e. the first response, represents the second choice response, i.e. the second response. D represents the preference data sample set, which can contain multiple groups (x, y w ,y l ) triples, representing the preferred data samples. β can represent the scaling factor, which is used to control the contrast intensity; σ(·) represents the Sigmoid function, and E(·) represents the mathematical expectation. The goal of the DPO loss function is: if the current model to be trained Compared to the reference model More inclined to generate better response y w , rather than the poor response y l , then the loss is smaller. Conversely, if the model is biased towards poor responses, the loss increases, thereby guiding the model to optimize more in line with the preference.

[0083] S312: Iteratively update the parameters of the supervised fine-tuning model based on the contrast loss function to obtain a preference optimization model.

[0084] Optionally, based on the constructed contrastive loss function, the parameters of the supervised fine-tuning model can be optimized through back-propagation, and ultimately a preference optimization model that can reflect human preference judgments can be obtained.

[0085] S313: Generate a first reward signal based on the preference optimization model and the first training sample set, and generate a second reward signal based on the preference optimization model, the target reward model, and the second training sample set in the preset training sample set.

[0086] Specifically, S313 is consistent with S204 and will not be repeated here.

[0087] S314: Optimize and train the preference optimization model based on the above first reward signal, the above second reward signal, and the above second training sample set to obtain a target object generative pre-training model.

[0088] Specifically, S314 is the same as S205 and will not be elaborated here.

[0089] In the embodiments of the present application, by comparing and modeling the generation probabilities of the first response and the second response, the model is guided to preferentially generate outputs that users prefer under the same input conditions, thereby achieving fine-tuning at the behavioral level. As a result, not only the generation tendency of the model for high-quality responses is strengthened, but also its ability to suppress low-quality responses is significantly enhanced, which helps to improve the naturalness, consistency, and controllability of the overall response. The finally obtained preference optimization model is more in line with the preference selection criteria of real users and provides better strategy guidance for downstream generation tasks.

[0090] In one embodiment, as Figure 4 shown, another model training method is provided, including the following steps: S401: Obtain an initial baseline model, an initial reward model, and a preset training sample set, where the above preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks.

[0091] Specifically, S401 is the same as S201 and will not be elaborated here.

[0092] S402: Supervise and fine-tune the above initial baseline model based on the first training sample set in the above preset training sample set to obtain a supervised fine-tuned model and preference data samples.

[0093] Specifically, S403 is the same as S202 and will not be elaborated here.

[0094] S403: Optimize and train the above initial reward model based on the above preference data samples to obtain a target reward model, and optimize and train the above supervised fine-tuned model based on the above preference data samples to obtain a preference optimization model.

[0095] Specifically, S403 is the same as S203 and will not be elaborated here.

[0096] S404: For each first training sample data in the above first training sample set, determine a first predicted response corresponding to the above first training sample data through the above preference optimization model.

[0097] In one embodiment, each prompt word in the first training sample data can be input into the preference optimization model to generate a corresponding natural language prediction response, that is, the first prediction response, as the input basis for subsequent evaluation of the model performance and generation of rewards.

[0098] S405: Compare the first prediction response with the reference output of the first training sample data, and calculate the difference metric between the first prediction response and the reference output.

[0099] In one embodiment, the first prediction response generated by the preference optimization model can be compared character by character with the corresponding reference output, and the edit distance algorithm in related technologies can be used to calculate the minimum number of operations, and further obtain a standardized difference ratio as the difference metric between the first prediction response and the reference output.

[0100] Specifically, the ratio of the edit distance to the character length of the reference output can be used as the normalized edit difference rate to measure the similarity between the prediction response and the reference output (i.e., the difference metric).

[0101] S406: Generate a first reward signal according to the difference metric.

[0102] Specifically, the difference metric can be used as an input and converted through a preset reward mapping function to generate a corresponding first reward signal. The preset reward mapping function can be determined according to actual needs, and the present application does not make specific limitations on this.

[0103] S407: For each second training sample data in the second training sample set included in the preset training sample set, determine the second prediction response corresponding to the second training sample data through the preference optimization model.

[0104] Among them, the second training sample data can include input samples and prompt words.

[0105] Optionally, each second training sample data in the second training sample set can be sequentially input into the preference optimization model to output its corresponding natural language response, that is, the second prediction response.

[0106] S408: Input the second training sample data and the corresponding second prediction response into the target reward model to perform reward signal generation processing through the target reward model to generate a second reward signal.

[0107] In one embodiment, each second training sample data and the corresponding second predicted response can be jointly input into the optimized target reward model. The target reward model performs semantic matching or correlation scoring on each group of input samples and the second predicted response, and thus outputs the corresponding reward value. This reward value serves as the second reward signal in the second stage, and is used to further optimize the preference optimization model in reinforcement learning policy optimization, so as to improve its generation ability and response quality under unsupervised input conditions.

[0108] S409: Optimally train the preference optimization model based on the above first reward signal, the above second reward signal, and the above second training sample set, to obtain a target object generative pre-training model.

[0109] Specifically, S409 is the same as S205, and details are not described herein again.

[0110] Through the above model training method in the embodiments of the present application, the advantages of supervised learning and reinforcement learning can be fully integrated, high-quality preliminary modeling can be achieved based on the labeled first training sample set, then a refined preference learning path can be constructed by using paired preference data and a reward model, and further, more actual scenario data can be introduced by combining the unsupervised second training sample set. Through the joint optimization of the two-stage reward signal, the generalization ability and response quality of the object generative pre-training model in multiple tasks and multiple scenarios can be effectively improved. This method not only improves the model's understanding and generation ability of complex commodity data, but also significantly reduces the dependence on large-scale manually labeled data, and has stronger scalability and deployment adaptability.

[0111] In one embodiment, as Figure 5 shown, another model training method is provided, including the following steps: S501: Obtain an initial baseline model, an initial reward model, and a preset training sample set, where the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks.

[0112] Specifically, S501 is the same as S201, and details are not described herein again.

[0113] S502: Obtain an initial baseline model, an initial reward model, and a preset training sample set, where the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks.

[0114] Specifically, S501 is the same as S201, and details are not described herein again.

[0115] S503: Optimally train the initial reward model based on the above preference data samples to obtain a target reward model, and optimally train the supervised fine-tuning model based on the above preference data samples to obtain a preference optimization model.

[0116] Specifically, S503 is the same as S203, and details are not repeated here.

[0117] S504: Based on the above first reward signal, perform reinforcement learning on the above preference optimization model to obtain a reinforced object generative pre-training model.

[0118] In one embodiment, based on the first reward signal, the proximal policy optimization algorithm in reinforcement learning can be used to further train and optimize the preference optimization model to obtain a reinforced object generative pre-training model. Among them, in the reinforcement learning process, with this first reward signal as feedback, the policy model parameters are updated through policy gradients to iteratively optimize the model behavior.

[0119] Optionally, in the reinforcement learning training process, the following formula (3) can be used as the loss function to optimize the policy model parameters: (3); Where represents the loss function, represents the parameters of the current preference optimization model, represents the expectation under the data distribution generated by the current model, represents the language modeling constraint term, which is used to maintain the rationality and smoothness of the generated language and prevent semantic collapse. Specifically, represents the input sample pair, where x is the input for the object response training task (such as product information and related prompt words, etc.), and y is the response text generated by the current model (such as product title, etc.). represents the data distribution sampled from the current model (reinforced object generative pre-training model), represents the score of the reward model for the generated response y, which is used to measure the quality, relevance or preference satisfaction degree of the response. represents the probability that the current model generates the response y under the input x, represents the probability that the preference optimization model generates y under the same input x, as a reference baseline. represents the original language modeling corpus data distribution, the probability that the reinforcement policy model generates a complete sentence, represents the probability that the reinforced object generative pre-training model generates a complete sentence, and γ represents the regularization strength parameter, which is used to control the strength of considering the language modeling task objective.

[0120] S505: Based on the above second reward signal and the above second training sample set, perform unsupervised training on the above reinforced object generative pre-training model to obtain a target object generative pre-training model.

[0121] In one embodiment, the unsupervised training of the reinforcement object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model includes: for each second training sample data in the second training sample set, using the reinforcement object generative pre-training model to generate object response information corresponding to the second training sample data; inputting the second training sample data and the corresponding object response information into a target reward model to obtain a reward score for reinforcement training; constructing a reinforcement learning objective function with the reward score as the return signal; and optimizing the parameters of the reinforcement object generative pre-training model based on the reinforcement learning objective function to obtain a trained target object generative pre-training model.

[0122] Specifically, first, each second training sample data in the second training sample set is sequentially input into the already trained reinforcement object generative pre-training model to generate corresponding natural language object response information. Subsequently, the second training sample data and the corresponding object response information are jointly input into the target reward model. The target reward model can comprehensively evaluate the accuracy, relevance, readability, etc. of the response content based on its internal policy and output a corresponding reward score as the return signal for this generation behavior. Further, a reinforcement learning objective function is constructed with this reward score as the core, such as an expected return maximization objective based on the PPO policy gradient algorithm. The system encourages policy exploration by minimizing the policy loss function of the negative return and combining means such as entropy regularization, and continuously optimizes the parameters of the reinforcement object generative pre-training model in each round of training. Thus, a trained target object generative pre-training model is finally obtained. This model can still generate object response texts that are more in line with human preferences and real relevant logic under the condition of lacking explicit supervision labels, further improving its practicality and stability in complex scenarios.

[0123] In the embodiments of the present application, by introducing a two-stage reinforcement learning mechanism (i.e., using the first reward signal for supervised reinforcement and using the second reward signal for unsupervised optimization), the human preferences and semantic information contained in the training data are fully exploited, enabling the generative model to be closer to the user's intention, application scenario, and natural language expression habits when generating object responses. Especially in the unsupervised data scenario, by scoring and feedback the generation results through the target reward model, the policy model is guided to continuously adjust the generation strategy, significantly improving the model's response ability and generalization ability to complex queries and fuzzy requirements.

[0124] Furthermore, the generated target object generative pre-training model can be used in subsequent various object-related tasks to support various application scenarios such as object information structuring, search recommendation optimization, intelligent classification, and question-answer alignment, further improving the intelligent level and processing efficiency in object content processing in scenarios such as e-commerce platforms or distribution systems. Optionally, the target object generative pre-training model can be used in the OFD field.

[0125] Please refer to Figure 6 , Figure 6 FIG. 3 is a schematic flowchart of another model training method provided by an embodiment of the present application. The execution subject of the embodiment of the present application can be a terminal that executes the model training method, a processor in the terminal that executes the model training method, or a training service of a feature extraction model in the terminal that executes the model training method. For ease of description, the following takes the execution subject as the processor in the terminal as an example to introduce the specific execution process of the model training method.

[0126] S601: Obtain a target object generative pre-training model and a task training data set corresponding to a preset subtask.

[0127] S601: Use the above target object generative pre-training model as an initial basic model, and perform fine-tuning training using the above task training data set to obtain a target task model for the preset subtask; wherein, the above target object generative pre-training model is a target object generative pre-training model generated by any one of the model training methods provided by the embodiments of the present application.

[0128] Optionally, when the object is a commodity, the preset subtask can be a preset on-demand food delivery subtask.

[0129] In a specific embodiment, first, obtain a pre-trained target object generative pre-training model. The above model is trained based on large-scale text data such as commodity titles, attributes, user queries, and comments, and has good commodity semantic understanding and text generation capabilities. Subsequently, for the "commodity title generation" subtask in the OFD scenario, construct a task training data set, and the above data set includes multiple input commodity attribute and target title pairs. Use the above target object generative pre-training model as an initial model, and perform fine-tuning training on it with the above task training data set. During the fine-tuning process, by minimizing the generation loss function of the target title, guide the model to learn to generate a title expression that conforms to semantics and e-commerce style from commodity attributes, and finally obtain a target task model suitable for the title generation subtask. This model can be directly deployed in real scenarios such as commodity new arrivals and title optimization to achieve automatic generation of high-quality commodity titles.

[0130] It can be understood that since the target object generative pre-training model has been trained on a large amount of task data in the OFD field and has good general semantic modeling and task transfer capabilities, when performing task-specific fine-tuning, the amount of supervised data required is significantly lower than that required for direct training from the initial baseline model. Further, thanks to its pre-training ability with a task-aligned structure, the target object generative pre-training model can converge with only a few rounds of fine-tuning, significantly reducing the computational resource overhead and time cost of model training.

[0131] In the embodiments of the present application, by means of the above-mentioned target object generative pre-training model, without the need to re-perform large-scale training for each downstream task, through a small amount of fine-tuning or direct call, it is possible to achieve rapid adaptation and high-quality response to multiple object response tasks, thereby significantly reducing the training cost, improving the system generalization ability and actual deployment efficiency.

[0132] It is worth noting that the target object generative pre-training model can be applied to multiple scenarios in the commercial (Business, B) and consumer (Consumer, C) ends of the relevant platform, and shows strong generalization and adaptation capabilities in different tasks. Specifically, in the B-side scenario, the target object generative pre-training model can be used for the following tasks: retail commodity category prediction, click-through rate per view (CPV) prediction for retail commodities, retail commodity title rewriting, prediction of the standard product unit or standard product subunit to which the retail commodity belongs, etc., to improve the effect of commodity clustering and standardization processing. In addition, in the C-side scenario, the above-mentioned target object generative pre-training model can also be deployed in the following tasks: query-commodity (food) relevance evaluation, query-commodity (retail) relevance evaluation, query-store relevance evaluation, etc. Through the specific deployment of the above tasks, the target object generative pre-training model provided by the present application can not only cover B-side data-driven tasks, but also adapt to the intelligent processing requirements of the C-side for user intention understanding, further reflecting the wide applicability of the model in the OFD field.

[0133] [[ID=⑨]]The following describes the present application with reference to 7 Figure 7 A schematic diagram of the overall process of training and optimizing a target object generative pre-training model is provided, covering two major steps: multi-task post-training and specific task fine-tuning. Among them, the multi-task post-training step can specifically include three stages: supervised fine-tuning, direct preference optimization, and proximal policy optimization, reflecting a layer-by-layer optimization path from general ability construction to human preference alignment and then to scenario adaptation. The overall data preparation required includes multi-task training data, which contains sample data corresponding to object response training tasks of category prediction type, title generation type, information extraction type, product matching type, search relevance type, and query annotation type.

[0134] In one embodiment, as Figure 7 , in the supervised fine-tuning stage, based on the above multi-task training data, the initial baseline model is optimized for multiple rounds of parameters to obtain a supervised fine-tuning model with general generation ability. In the direct preference optimization stage, preference data is introduced. By constructing paired preference data samples, combined with a reward model, different response qualities are scored, and training optimization is carried out through a contrastive loss function, so as to make the model output results closer to human expectations in terms of style, format, content rationality, etc. The reward model can generate scoring signals based on techniques such as edit distance and semantic matching degree to help the model strengthen the learning of high-quality outputs. In the proximal policy optimization stage, under the condition of unsupervised data, reinforcement learning techniques such as the PPO algorithm are further introduced, and the output of the trained reward model is used as the reward signal to guide the model to optimize its response strategy. In each round of optimization, the change range of policy update is controlled to avoid the occurrence of policy instability problems, thereby improving the generalization ability and robustness of the model in the open input scenario. After completing the above multi-task training steps, the specific task fine-tuning step can be entered. For a specific OFD object response sub-task (such as title generation, category matching, etc.), the optimized basic model is further lightly fine-tuned using the supervised data set of this task. At this stage, since the model already has strong general semantic understanding and preference alignment capabilities, only a small amount of specific task data and low computing resources are required to quickly complete the fine-tuning, significantly improving the performance and usability on this task. Thus, Figure 7 the shown process effectively combines multi-source heterogeneous data, supervised learning, and reinforcement learning techniques. While constructing general generation ability, human preferences are gradually introduced, and finally a high-quality and strongly adaptable target object generative pre-training model is achieved.

[0135] In one embodiment, to verify the generalization ability of the target object generative pre-training model on new specific tasks in the OFD field, supervised fine-tuning can be performed on the preset Hypernym-Hyponym Classification (HHC) data set and Title-Query Alignment (TQA) data set respectively based on the trained target object generative pre-training model to achieve adaptation to specific tasks. Among them, the HHC data set can be used to judge whether there is a hypernym-hyponym relationship between two commodity-related words, and the TQA data set can be used to judge whether the given commodity title is semantically matched with the user query. Both of the above data sets belong to the actual tasks in the OFD field, but are not included in the original multi-task training set of the target object generative pre-training model (i.e., the above preset training sample set).

[0136] Specifically, the performance comparison between the generative pre-trained model for the target object and the initial baseline model can be carried out on 12 preset online-to-offline (O2O) object response-related tasks, and the comparison metric is the accuracy rate (in %) on the test sets of each task. The tasks involved cover six major types of tasks, namely category prediction type, title generation type, information extraction type, product matching type, search relevance type, and query annotation type, which may include 12 specific tasks: commodity standardized product unit prediction task, commodity standardized product subunit prediction task, commodity category prediction task, commodity category prediction task under multi-turn conversation, commodity category prediction task based on the store, commodity attribute extraction task, query category prediction task, query-product (food) relevance evaluation task, query-product relevance evaluation task, query word part-of-speech tagging task, query-store relevance evaluation task, and commodity title generation task. The comparison results can show that the initial baseline model shows a certain accuracy level in multiple typical O2O commodity response-related tasks. In contrast, the generative pre-trained model for the target object provided by the embodiment of the present application shows overall better performance on each task under the same data and training settings, and the average accuracy rate has been improved. Among them, the improvement amplitude on individual tasks is particularly significant. For example, the accuracy rate of the query word part-of-speech tagging task can be increased by more than 4%. The comprehensive comparison results show that the generative pre-trained model for the target object, which is pre-trained by fusing multi-task corpora and optimized by combining reinforcement learning strategies, can obtain higher accuracy rates in most tasks as the initialization basis for downstream tasks, thus effectively improving the overall performance of the commodity response system in actual O2O scenarios.

[0137] In addition, to further verify the performance advantages of the target object generative pre-training model provided by this application under multiple mainstream basic models, three large language models in related technologies can also be selected as the initialization basic models, and under the same training data and SFT strategies, four typical representative tasks are compared and evaluated, including the product title generation task, the product sub-standard unit prediction task, the product category prediction task, and the product category prediction task under multi-round conversations. The comparative evaluation can show that although some large language models in related technologies have advantages in individual tasks, the overall average accuracy still needs to be improved. In contrast, the target object generative pre-training model provided by the embodiments of this application shows higher consistency and accuracy under the same training strategy and evaluation conditions, and achieves a relatively high average accuracy level in the above four tasks. It can be seen that the target object generative pre-training model provided by this application not only has good generalization ability in 12 multi-task training tasks, but also shows high task adaptability and overall performance advantages under the comparison of multiple mainstream basic models. This model has strong migration ability and is applicable to complex and diverse actual product response generation tasks in the OFD field, effectively improving the comprehensive performance of the system in terms of accuracy, robustness, and response quality.

[0138] Furthermore, relevant algorithm modules or functional components used in the entire model optimization process can be removed or replaced one by one for testing to evaluate the actual contribution of each component to the final model performance. That is to say, ablation experiments are carried out on the post-training steps of multi-tasks. The experimental results can show that, with the performance improvement of the supervised fine-tuning model as the benchmark (that is, the relative performance improvement corresponding to the supervised fine-tuning model is 0), after adding the direct preference optimization stage, compared with the supervised fine-tuning model, the average accuracy increases by more than 1%, which indicates that the direct preference optimization method can better optimize the alignment of the model behavior with human preferences. Further, when proximal policy optimization based on the edit distance is introduced on the basis of direct preference optimization, the performance is further improved, and when proximal policy optimization based on the reward model is introduced, the improvement amplitude is further increased, which shows that both reward signals can effectively improve the accuracy and practicality of the model generation results. More significantly, when both the edit distance reward and the reward model are introduced as joint reward signals for proximal policy optimization, the average performance of the final model is significantly improved, with the largest improvement compared to the benchmark model, indicating the importance of the diversity and complementarity of reward signals in the post-training process. It can be seen that this ablation experiment fully verifies the independent and collaborative value of each component in the entire model optimization process, especially in the reinforcement learning stage, by introducing multi-dimensional reward signals, the generalization ability of the model and its adaptability to complex response tasks can be further improved.

[0139] Furthermore, referring to Figure 8 , Figure 8The performance comparison results after supervised fine-tuning using different base models on two newly introduced OFD-specific tasks are shown, further verifying the generalization ability and training efficiency of the target object generative pre-training model proposed in this application when migrated to new specific tasks. Figure 8 In Figure 8 , the accuracy performance within five consecutive supervised fine-tuning training rounds was compared when using the initial baseline model and the target object generative pre-training model as the base models in the HHC task and the TQA task respectively. It can be seen from the results that the target task models based on the target object generative pre-training model are significantly superior to the target task models based on the initial baseline model in both tasks. Specifically, whether in the initial training rounds or in the subsequent iterative process, the fine-tuning models based on the target object generative pre-training model are significantly superior to the similar models based on the traditional initial baseline model. In the initial stage of training, the fine-tuning model based on the target object generative pre-training model showed a higher accuracy, reflecting its good initialization ability and task migration performance. As the number of training rounds increased, the advantages of the fine-tuning model based on the target object generative pre-training model continued to be maintained, and finally it was still able to maintain a stable and high performance after multiple training cycles. This result indicates that the target object generative pre-training model provided in the embodiments of this application not only has good generality, but also can show better performance during the migration of specific tasks, providing a basis for the efficient deployment of complex commodity response tasks in the OFD field.

[0140] To further verify the generalization ability and training efficiency of the target object generative pre-training model as a domain base model in the fine-tuning stage, ablation experiments can be carried out on two tasks of commodity category prediction and commodity standardized product subunit prediction, evaluate the performance under different training rounds and training data ratios, and the experimental results are as Figure 9 shown. As Figure 9, the horizontal axis represents the number of training rounds, and the vertical axis represents the accuracy. Line a represents the change in accuracy of the target task model based on the initial baseline model in the product category prediction task during the 1st to 5th training rounds; Line b represents the change in accuracy of the target task model based on the target object generative pre-training model in the product category prediction task during the 1st to 5th training rounds. It can be seen that in the product category prediction task, compared with the accuracy of the target task model based on the initial baseline model, which always remains at a relatively low level, the target task model provided by the embodiments of the present application based on the target object generative pre-training model demonstrates better performance in the initial training stage, gradually improves in subsequent rounds, and shows a faster convergence trend. This indicates that the target task model based on the target object generative pre-training model has stronger initial generalization ability and faster convergence speed in the product category prediction task, and can achieve better performance at an early stage. Line c represents the change in accuracy of the target task model based on the initial baseline model in the product standardized product subunit prediction task during the 1st to 5th training rounds; Line d represents the change in accuracy of the target task model based on the target object generative pre-training model in the product standardized product subunit prediction task during the 1st to 5th training rounds. It can be seen that in the product standardized product subunit prediction task, the accuracy of the target task model based on the initial baseline model is stable but has limited room for improvement, while the target task model based on the target object generative pre-training model continuously achieves small improvements in multiple training rounds and finally reaches a higher accuracy level. This shows that the target task model based on the target object generative pre-training model not only has strong expressive ability in the initial training stage but also can further improve its performance after multiple rounds of training, verifying its superiority and higher convergence efficiency in the product standardized product subunit prediction task.

[0141] Furthermore, to evaluate the performance stability and generalization ability of the target task model based on the target object generative pre-training model under different training data volumes, two tasks, namely product category prediction and product standardized product subunit prediction, can be selected respectively to conduct an accuracy comparison experiment based on the change in the proportion of the training set. Set the proportions of the training set to 20%, 40%, 60%, 80%, and 100% respectively, and record the accuracy performance of the model on the validation set at each proportion. At the same time, set the target task model based on the initial baseline model for comparison. The experimental results are as Figure 10 shown, as Figure 10, the horizontal axis is the proportion of the training set, and the vertical axis is the accuracy rate. Line e represents the change in the accuracy rate of the target task model based on the initial baseline model in the commodity category prediction task, and line f represents the change in the accuracy rate of the target task model based on the initial baseline model in the commodity standardized product subunit prediction task; line g represents the change in the accuracy rate of the target task model based on the target object generative pre-training model in the commodity category prediction task, and line h represents the change in the accuracy rate of the target task model based on the target object generative pre-training model in the commodity standardized product subunit prediction task. The comparison between line e and line g shows that in the commodity category prediction task, even when only a small amount of training data is used, the accuracy rate of the target task model based on the target object generative pre-training model is significantly better than that of the target task model based on the initial baseline model, and the overall trend shows good stability and adaptability as the data volume increases. That is to say, the target task model of the target object generative pre-training model has strong low-resource learning ability and can still provide better prediction performance when the training data is limited. The comparison between line f and line h shows that in the commodity standardized product subunit prediction task, the target task model of the target object generative pre-training model also shows better results. At different training set proportions, the accuracy rate always remains at a high level and shows a steady increase as the data volume increases, further reflecting the excellent expression ability and convergence efficiency of the target task model of the target object generative pre-training model.

[0142] It can be seen that the target task model based on the target object generative pre-training model shows strong training data adaptation ability in both tasks. Especially in the case of low resources (such as 40% training set), it can still stably outperform the target task model based on the initial baseline model, verifying its high generalization ability and data utilization efficiency as a domain-based model in downstream tasks.

[0143] Furthermore, the embodiment of the present application also provides a model interaction method, which at least includes: obtaining object request data; inputting the above object request data into the target object generative pre-training model, analyzing and processing the above object request data through the above target object generative pre-training model to generate corresponding object response data; outputting the above object response data; wherein, the above target object generative pre-training model is the target object generative pre-training model generated by any one of the model training methods provided by the present application.

[0144] Among them, the object request data can be input data received from relevant platform systems or user terminals for generating responses. Specifically, the object request data can be a task instruction in the form of natural language. For example: Please generate an attractive title for the product "Honey Pomelo Tea"; the object request data can also include structured product information, such as product name, category, attribute description, etc.

[0145] Optionally, the object request data can be used to execute a specific task corresponding to any one of multiple object response task types. When the object is a product, the multiple object response task types include but are not limited to: category prediction type, title generation type, information extraction type, product matching type, search relevance type, and query annotation type.

[0146] In one embodiment, the target object generative pre-trained model is generated by any one of the model training methods provided in this application and can have the ability to understand and generate multiple object task responses. Input the above object request data into the target object generative pre-trained model, so that the target object generative pre-trained model automatically identifies the corresponding task type based on the learned semantic representation ability and task characteristics, and generates corresponding object response data based on the internally encoded task representation and the trained domain knowledge. Among them, the object response data can be presented in the form of natural language text or in a structured data format, such as category labels, category codes, matching scores, etc., to adapt to different scenario requirements. Furthermore, the generated object response data can be output to relevant downstream scenario platforms to implement the task execution of specific scenarios.

[0147] The model interaction method provided in the embodiments of this application can, based on the object request data, call the target object generative pre-trained model generated by any one of the training methods in this application to achieve automatic recognition and response generation of multiple object response tasks, effectively improving the recognition and response efficiency.

[0148] It should be understood that although the steps in the flowcharts involved in the above-mentioned embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps in other steps.

[0149] Based on the inventive concept of the above model training method, as Figure 11 shown, an embodiment of the present application further provides a first model training apparatus 1100 for implementing the above-mentioned model training method. The first model training apparatus 1100 includes: A first acquisition module 1101, configured to acquire an initial baseline model, an initial reward model, and a preset training sample set, where the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; A fine-tuning module 1102, configured to perform supervised fine-tuning on the initial baseline model based on a first training sample set in the preset training sample set to obtain a supervised fine-tuned model and preference data samples; A first training module 1103, configured to perform optimization training on the initial reward model based on the preference data samples to obtain a target reward model, and perform optimization training on the supervised fine-tuned model based on the preference data samples to obtain a preference-optimized model; A generation module 1104, configured to generate a first reward signal based on the preference-optimized model and the first training sample set, and generate a second reward signal based on the preference-optimized model, the target reward model, and a second training sample set in the preset training sample set; A second training module 1105, configured to perform optimization training on the preference-optimized model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

[0150] In one embodiment, the fine-tuning module 1102 is specifically configured to perform multiple rounds of parameter updates on the initial baseline model in a supervised learning manner based on the first training sample set in the preset training sample set to obtain a supervised fine-tuned model; during the process of performing multiple rounds of parameter updates on the initial baseline model, sample the first training sample set to obtain input samples and multiple candidate responses corresponding to each of the input samples; screen the multiple candidate responses corresponding to each of the input samples based on a preset index to obtain a first response and a second response corresponding to each of the input samples, so as to construct paired preference data samples, where the first response is better than the second response.

[0151] In one embodiment, the first training module 1103 is specifically configured to input each of the above input samples included in the above preference data sample, and the above first response and the above second response corresponding to each of the above input samples, into the above initial reward model; generate score values for the above first response and the above second response corresponding to each of the above input samples through the above initial reward model, to obtain a first response score value corresponding to the above first response and a second response score value corresponding to the above second response; construct a score loss function according to the above first response score value and the above second response score value; and iteratively optimize the parameters of the above initial reward model based on the above score loss function to obtain a target reward model.

[0152] In one embodiment, the first training module 1103 is specifically configured to input each of the above preference data samples into the above supervised fine-tuning model; calculate the generation probability information of the above first response and the above second response in each of the above preference data samples by the above supervised fine-tuning model; construct a contrast loss function based on the above generation probability information; and iteratively update the parameters of the above supervised fine-tuning model based on the above contrast loss function to obtain a preference optimization model.

[0153] In one embodiment, the generation module 1104 is specifically configured to, for each first training sample data in the above first training sample set, determine a first predicted response corresponding to the above first training sample data through the above preference optimization model; compare the above first predicted response with the reference output of the above first training sample data, and calculate a difference metric between the above first predicted response and the above reference output; and generate a first reward signal according to the above difference metric.

[0154] In one embodiment, the generation module 1104 is specifically configured to, for each second training sample data in the second training sample set of the above preset training sample set, determine a second predicted response corresponding to the above second training sample data through the above preference optimization model; input the above each second training sample data and the corresponding second predicted response into the above target reward model, so as to perform reward signal generation processing through the above target reward model to generate a second reward signal.

[0155] In one embodiment, the second training module 1105 is specifically configured to perform reinforcement learning on the above preference optimization model based on the above first reward signal to obtain a reinforcement object generative pre-training model; and perform unsupervised training on the above reinforcement object generative pre-training model based on the above second reward signal and the above second training sample set to obtain a target object generative pre-training model.

[0156] In one embodiment, the second training module 1105 is specifically configured to, for each second training sample data in the above-mentioned second training sample set, use the above-mentioned reinforcement object generative pre-training model to generate object response information corresponding to the second training sample data; input the second training sample data and the corresponding object response information into the target reward model to obtain a reward score for reinforcement training; construct a reinforcement learning objective function with the reward score as the return signal; and optimize the parameters of the reinforcement object generative pre-training model based on the reinforcement learning objective function to obtain a trained target object generative pre-training model.

[0157] Each module in the above-mentioned first model training device 1100 can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0158] Based on the inventive concept of the above-mentioned model training method, as Figure 12 shown, an embodiment of the present application further provides a second model training device 1200 for implementing the above-mentioned model training method. The second model training device 1200 includes: A second acquisition module 1201, configured to acquire a target object generative pre-training model and a task training data set corresponding to a preset sub-task; A third training module 1202, configured to use the target object generative pre-training model as an initial basic model and perform fine-tuning training using the task training data set to obtain a target task model for the preset sub-task; wherein, the target object generative pre-training model is a target object generative pre-training model generated by any model training method provided by the present application.

[0159] Each module in the above-mentioned second model training device 1200 can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0160] Based on the inventive concept of the above-mentioned model interaction method, as Figure 14 shown, an embodiment of the present application further provides a model interaction device 1300 for implementing the above-mentioned model interaction method. The model interaction device 1300 includes: A third acquisition module 1301, configured to acquire object request data; An input module 1302 is configured to input the above object request data into a target object generative pre-training model, and analyze and process the above object request data by using the above target object generative pre-training model to generate corresponding object response data; An output module 1303 is configured to output the above object response data; wherein, the above target object generative pre-training model is a target object generative pre-training model generated by any model training method provided by the present application.

[0161] Each module in the above model interaction device 1300 can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0162] An embodiment of the present application further provides an electronic device, which may be a server, and its internal structure diagram may be as Figure 14 shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store activity configuration information. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with an external terminal through a network connection. The processor of the electronic device executes the computer program to implement a model training method or a model interaction method.

[0163] Those skilled in the art can understand that Figure 14 the structure shown in

[0164] The embodiment of the present application also provides a computer storage medium, in which instructions are stored. When the instructions run on a computer or a processor, the computer or the processor is caused to execute one or more steps in the above embodiments. If each component module of the above electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the above computer-readable storage medium.

[0165] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the processes or functions according to the above embodiments of the present application are generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer storage medium or transmitted through the above computer storage medium. The above computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The above computer storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more integrated available media. The above available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a Digital Versatile Disc (DVD)), or a semiconductor medium (for example, a Solid State Disk (SSD)), etc.

[0166] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The foregoing storage media include: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes. Without conflict, the technical features in this embodiment and the implementation solutions can be combined arbitrarily.

[0167] The above-described embodiments are only described in the preferred implementation manners of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present application should fall within the protection scope determined by the claims.

[0168] The above description has been made of specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A model training method, characterized in that, The method includes: Obtaining an initial baseline model, an initial reward model, and a preset training sample set, where the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; Supervisedly fine-tuning the initial baseline model based on a first training sample set in the preset training sample set to obtain a supervised fine-tuning model and preference data samples; Optimally training the initial reward model based on the preference data samples to obtain a target reward model, and optimally training the supervised fine-tuning model based on the preference data samples to obtain a preference optimization model; Generating a first reward signal based on the preference optimization model and the first training sample set, and generating a second reward signal based on the preference optimization model, the target reward model, and a second training sample set in the preset training sample set; Optimally training the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

2. The method according to claim 1, characterized in that, The step of supervisedly fine-tuning the initial baseline model based on a first training sample set in the preset training sample set to obtain a supervised fine-tuning model and preference data samples includes: Updating the parameters of the initial baseline model in multiple rounds in a supervised learning manner based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model; During the process of updating the parameters of the initial baseline model in multiple rounds, sampling the first training sample set to obtain input samples and multiple candidate responses corresponding to each of the input samples; Screening the multiple candidate responses corresponding to each input sample based on a preset metric to obtain a first response and a second response corresponding to each input sample, so as to construct paired preference data samples, where the first response is better than the second response.

3. The method according to claim 2, wherein The step of optimally training the initial reward model based on the preference data samples to obtain a target reward model includes: Inputting each of the input samples included in the preference data samples, and the first response and the second response corresponding to each input sample into the initial reward model; Performing a scoring value generation process on the first response and the second response corresponding to each input sample through the initial reward model to obtain a first response scoring value corresponding to the first response and a second response scoring value corresponding to the second response; Constructing a scoring loss function according to the first response scoring value and the second response scoring value; Iteratively optimizing the parameters of the initial reward model based on the scoring loss function to obtain a target reward model.

4. The method according to claim 2, wherein The step of optimally training the supervised fine-tuning model based on the preference data samples to obtain a preference optimization model includes: Inputting each of the preference data samples into the supervised fine-tuning model; Calculating the generation probability information of the first response and the second response in each of the preference data samples by the supervised fine-tuning model; Constructing a contrast loss function based on the generation probability information; Iteratively updating the parameters of the supervised fine-tuning model based on the contrast loss function to obtain a preference optimization model.

5. The method according to claim 1, characterized in that, Generating a first reward signal based on the preference optimization model and the first training sample set includes: For each first training sample data in the first training sample set, determining a first predicted response corresponding to the first training sample data through the preference optimization model; Comparing the first predicted response with the reference output of the first training sample data, and calculating a difference metric between the first predicted response and the reference output; Generating a first reward signal according to the difference metric.

6. The method according to claim 1, characterized in that, Generating a second reward signal based on the preference optimization model, the target reward model, and the second training sample set in the preset training sample set includes: For each second training sample data in the second training sample set in the preset training sample set, determining a second predicted response corresponding to the second training sample data through the preference optimization model; Inputting the second training sample data and the corresponding second predicted response into the target reward model to perform reward signal generation processing through the target reward model, and generating a second reward signal.

7. The method according to claim 1, characterized in that, Optimally training the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model includes: Performing reinforcement learning on the preference optimization model based on the first reward signal to obtain a reinforced object generative pre-training model; Performing unsupervised training on the reinforced object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model.

8. The method according to claim 7, characterized in that Performing unsupervised training on the reinforced object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model includes: For each second training sample data in the second training sample set, generating object response information corresponding to the second training sample data by using the reinforced object generative pre-training model; Inputting the second training sample data and the corresponding object response information into the target reward model to obtain a reward score for reinforcement training; Constructing a reinforcement learning objective function with the reward score as the return signal; Optimizing the parameters of the reinforced object generative pre-training model based on the reinforcement learning objective function to obtain a trained target object generative pre-training model.

9. A model training method, characterized in that, The method includes: Obtaining a target object generative pre-training model and a task training data set corresponding to a preset subtask; Using the target object generative pre-training model as an initial basic model, and performing fine-tuning training by using the task training data set to obtain a target task model for the preset subtask; Wherein, the target object generative pre-training model is a target object generative pre-training model generated by the model training method according to any one of claims 1 to 8.

10. A model interaction method, characterized in that, The method includes: Obtaining object request data; Input the object request data into the target object generative pre-trained model, and analyze and process the object request data through the target object generative pre-trained model to generate corresponding object response data; Output the object response data; Wherein, the target object generative pre-trained model is a target object generative pre-trained model generated by the model training method according to any one of claims 1 to 8.

11. A model training device, characterized in that, The device includes: A first acquisition module, configured to acquire an initial baseline model, an initial reward model, and a preset training sample set, where the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; A fine-tuning module, configured to perform supervised fine-tuning on the initial baseline model based on a first training sample set in the preset training sample set to obtain a supervised fine-tuned model and preference data samples; A first training module, configured to perform optimization training on the initial reward model based on the preference data samples to obtain a target reward model, and perform optimization training on the supervised fine-tuned model based on the preference data samples to obtain a preference-optimized model; A generation module, configured to generate a first reward signal based on the preference-optimized model and the first training sample set, and generate a second reward signal based on the preference-optimized model, the target reward model, and a second training sample set in the preset training sample set; A second training module, configured to perform optimization training on the preference-optimized model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-trained model.

12. A model training device, characterized in that, The device includes: A second acquisition module, configured to acquire a target object generative pre-trained model and a task training data set corresponding to a preset subtask; A third training module, configured to use the target object generative pre-trained model as an initialization base model and perform fine-tuning training using the task training data set to obtain a target task model for the preset subtask; Wherein, the target object generative pre-trained model is a model generated by the model training method according to any one of claims 1 to 8.

13. An electronic device, characterized in that, Including: A processor and a memory; the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1-10 is implemented.

14. A computer storage medium, characterized in that, The computer storage medium stores multiple instructions, and the instructions are suitable for being loaded and executed by a processor to perform the method according to any one of claims 1-10.

Images