Model training method, device, electronic device and storage medium

Abstract

The present application discloses a model training method, device, electronic device and computer storage medium. The model training method includes: performing supervised fine-tuning on an initial baseline model based on a first training sample set in a preset training sample set to obtain a supervised fine-tuning model and a preference data sample; optimizing and training an initial reward model based on the preference data sample to obtain a target reward model, and optimizing and training a supervised fine-tuning model based on the preference data sample to obtain a preference optimization model; generating a first reward signal based on the preference optimization model and the first training sample set, and generating a second reward signal based on the preference optimization model, the target reward model and a second training sample set in the preset training sample set; optimizing and training the preference optimization model based on the first reward signal, the second reward signal and the second training sample set to obtain a target object generative pre-training model. This method can improve the generalization ability of the model.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a model training method, device, electronic device and storage medium. Background Art

[0002] Background: With the development of on-demand food delivery (OFD) platforms, related platforms need to simultaneously handle multiple tasks such as user intent understanding, product attribute prediction, title rewriting, and relevance matching, which places higher demands on the semantic understanding and generation capabilities of the model.

[0003] However, due to the complex data structures of objects such as goods and stores in OFD scenarios, the diverse tasks involved, and the scarcity of labeled data, traditional models are difficult to adapt directly and struggle to meet the platform's demand for low-resource, highly adaptable semantic models. Therefore, there is an urgent need for an object-generative pre-training model with efficient training capabilities to improve model training efficiency and generalization capabilities. Summary of the Invention

[0004] The embodiments of the present application provide a model training method, device, electronic device, and computer storage medium that can improve the efficiency of model training and the generalization ability of the model. The above technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0006] Obtaining an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks;

[0007] Performing supervised fine-tuning on the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample;

[0008] Optimizing and training the initial reward model based on the preference data sample to obtain a target reward model, and optimizing and training the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model;

[0009] Generating a first reward signal based on the preference optimization model and the first training sample set, and generating a second reward signal based on the preference optimization model, the target reward model, and a second training sample set in the preset training sample set;

[0010] The preference optimization model is optimized and trained based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

[0011] In a possible implementation, the supervised fine-tuning of the initial baseline model based on the first training sample set in the preset training sample set to obtain the supervised fine-tuning model and the preference data sample includes:

[0012] Based on the first training sample set in the preset training sample set, the initial baseline model is updated with multiple rounds of parameters using a supervised learning method to obtain a supervised fine-tuning model;

[0013] During multiple rounds of parameter updates on the initial baseline model, sampling the first training sample set to obtain input samples and multiple candidate responses corresponding to each of the input samples;

[0014] Based on preset indicators, multiple candidate responses corresponding to each of the above input samples are screened to obtain a first response and a second response corresponding to each of the above input samples to construct paired preference data samples, wherein the above first response is better than the above second response.

[0015] In one possible implementation, optimizing and training the initial reward model based on the preference data sample to obtain a target reward model includes:

[0016] Inputting each of the input samples included in the preference data sample, and the first response and the second response corresponding to each of the input samples, into the initial reward model;

[0017] Performing score generation processing on the first response and the second response corresponding to each of the input samples using the initial reward model to obtain a first response score corresponding to the first response and a second response score corresponding to the second response;

[0018] Constructing a scoring loss function based on the first response scoring value and the second response scoring value;

[0019] Based on the above scoring loss function, the parameters of the above initial reward model are iteratively optimized to obtain the target reward model.

[0020] In one possible implementation, optimizing and training the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model includes:

[0021] Input each of the above preference data samples into the above supervised fine-tuning model;

[0022] Calculating the generation probability information of the supervised fine-tuning model for the first response and the second response in each of the preference data samples;

[0023] Construct a contrast loss function based on the above generated probability information;

[0024] Based on the above contrast loss function, the parameters of the supervised fine-tuning model are iteratively updated to obtain a preference optimization model.

[0025] In one possible implementation, generating the first reward signal based on the preference optimization model and the first training sample set includes:

[0026] For each first training sample data in the first training sample set, determining a first predicted response corresponding to the first training sample data by using the preference optimization model;

[0027] Comparing the first predicted response with the reference output of the first training sample data, and calculating a difference metric between the first predicted response and the reference output;

[0028] A first reward signal is generated according to the difference metric.

[0029] In one possible implementation, generating the second reward signal based on the preference optimization model, the target reward model, and the second training sample set in the preset training sample set includes:

[0030] For each second training sample data in the second training sample set in the preset training sample set, determining a second predicted response corresponding to the second training sample data by using the preference optimization model;

[0031] The above-mentioned second training sample data and the corresponding second predicted response are input into the above-mentioned target reward model, so as to perform reward signal generation processing through the above-mentioned target reward model to generate a second reward signal.

[0032] In one possible implementation, the preference optimization model is optimized and trained based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model, including:

[0033] Performing reinforcement learning on the preference optimization model based on the first reward signal to obtain a reinforcement object generative pre-training model;

[0034] Based on the second reward signal and the second training sample set, the reinforcement object generative pre-training model is unsupervisedly trained to obtain a target object generative pre-training model.

[0035] In one possible implementation, performing unsupervised training on the reinforcement object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model includes:

[0036] For each second training sample data in the second training sample set, using the enhanced object generative pre-training model to generate object response information corresponding to the second training sample data;

[0037] Inputting the second training sample data and the corresponding subject response information into a target reward model to obtain a reward score for reinforcement training;

[0038] Construct a reinforcement learning objective function with reward score as the return signal;

[0039] Based on the reinforcement learning objective function, the parameters of the reinforcement object generative pre-training model are optimized to obtain a trained target object generative pre-training model.

[0040] In a second aspect, an embodiment of the present application provides a model training method, comprising:

[0041] Obtain the target object generative pre-trained model and the task training dataset corresponding to the preset subtask;

[0042] The target object generative pre-trained model is used as the initialization basic model, and the task training dataset is used for fine-tuning training to obtain the target task model for the preset subtask;

[0043] Among them, the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by the model training method provided by the first aspect.

[0044] In a third aspect, an embodiment of the present application provides a model interaction method, the method comprising:

[0045] Get object request data;

[0046] Inputting the object request data into a target object generative pre-trained model, analyzing and processing the object request data through the target object generative pre-trained model to generate corresponding object response data;

[0047] Output the above object response data;

[0048] Among them, the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by the model training method provided by the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a model training device, comprising:

[0050] A first acquisition module is configured to acquire an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes a plurality of training sample groups corresponding to a plurality of object response training tasks;

[0051] A fine-tuning module, configured to perform supervised fine-tuning on the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample;

[0052] A first training module is configured to optimize and train the initial reward model based on the preference data sample to obtain a target reward model, and to optimize and train the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model;

[0053] a generation module, configured to generate a first reward signal based on the preference optimization model and the first training sample set, and to generate a second reward signal based on the preference optimization model, the target reward model, and a second training sample set in the preset training sample set;

[0054] The second training module is used to optimize and train the preference optimization model based on the first reward signal, the second reward signal and the second training sample set to obtain a generative pre-training model of the target object.

[0055] In a fifth aspect, an embodiment of the present application provides a model training device, comprising:

[0056] The second acquisition module is used to obtain the target object generative pre-training model and the task training data set corresponding to the preset subtask;

[0057] A third training module is configured to use the target object generative pre-trained model as an initialized basic model and perform fine-tuning training using the task training dataset to obtain a target task model for the preset subtask;

[0058] Among them, the above-mentioned target object generative pre-training model is a model generated by the model training method provided by the first aspect.

[0059] In a sixth aspect, an embodiment of the present application provides a model interaction device, comprising:

[0060] A third acquisition module is used to obtain object request data;

[0061] An input module, configured to input the object request data into a target object generative pre-trained model, analyze and process the object request data using the target object generative pre-trained model, and generate corresponding object response data;

[0062] Output module, used for outputting the above object response data;

[0063] Among them, the above-mentioned target object generative pre-training model is a model generated by the model training method provided by the first aspect.

[0064] In the seventh aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; wherein the above-mentioned memory stores a computer program, and when the processor executes the computer program, it implements the method steps provided in any one of the first aspect, second aspect and third aspect of the embodiment of the present application.

[0065] In an eighth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps provided in any one of the first aspect, the second aspect and the third aspect of the embodiment of the present application.

[0066] In the embodiment of the present application, an initial baseline model, an initial reward model and a preset training sample set are obtained, wherein the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; based on the first training sample set in the preset training sample set, the initial baseline model is supervised and fine-tuned to obtain a supervised fine-tuning model and a preference data sample; based on the preference data sample, the initial reward model is optimized and trained to obtain a target reward model, and based on the preference data sample, the supervised fine-tuning model is optimized and trained to obtain a preference optimization model; based on the preference optimization model and the first training sample set, a first reward signal is generated, and based on the preference optimization model, the target reward model and the second training sample set in the preset training sample set, a second reward signal is generated; based on the first reward signal, the second reward signal and the second training sample set, the preference optimization model is optimized and trained to obtain a target object generative pre-training model. On the one hand, fine-tuning the initial baseline model based on supervised samples can improve the basic capabilities of each task of the model; on the other hand, by constructing preference data samples to jointly train the reward model and the strategy model, the model behavior and human preferences are effectively aligned. Furthermore, the introduction of two types of reward signals—one derived from a reward model and the other from a preference optimization model—integrated into the unsupervised data training phase effectively improves the model's generalization capabilities and significantly reduces its reliance on high-quality manually labeled data. The resulting object-generative pre-training model possesses enhanced object understanding, strategy generation, and interactive response capabilities, making it better suited to the multi-scenario intelligent needs of on-demand food delivery platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1A schematic diagram of the system architecture of a model training method provided by an exemplary embodiment of the present application;

[0069] Figure 2 A flowchart of a model training method provided by an exemplary embodiment of the present application;

[0070] Figure 3 A flowchart of another model training method provided as an exemplary embodiment of the present application;

[0071] Figure 4 A flowchart of another model training method provided as an exemplary embodiment of the present application;

[0072] Figure 5 A flowchart of another model training method provided as an exemplary embodiment of the present application;

[0073] Figure 6 A flowchart of another model training method provided as an exemplary embodiment of the present application;

[0074] Figure 7 A schematic diagram of the overall process of a training and optimization method for a target object generative pre-training model provided by an exemplary embodiment of the present application;

[0075] Figure 8 A schematic diagram of a performance comparison result provided by an exemplary embodiment of the present application;

[0076] Figure 9 A schematic diagram of an ablation experiment result provided by an exemplary embodiment of the present application;

[0077] Figure 10 A schematic diagram of an experimental result provided for an exemplary embodiment of the present application;

[0078] Figure 11 A schematic structural diagram of a model training device provided by an exemplary embodiment of the present application;

[0079] Figure 12 A schematic structural diagram of another model training device provided by an exemplary embodiment of the present application;

[0080] Figure 13 A schematic structural diagram of a model interaction device provided by an exemplary embodiment of the present application;

[0081] Figure 14 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0083] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0084] See also Figure 1 , Figure 1 An exemplary system architecture diagram of a model training method provided in an embodiment of the present application.

[0085] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links. For example, the wired communication link may include an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link may include a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.

[0086] The terminal 101 can interact with the server 103 via the network 102 to receive messages from the server 103 or send messages to the server 103. Alternatively, the terminal 101 can interact with the server 103 via the network 102 to receive messages or data sent to the server 103 by other users. The terminal 101 can be hardware or software. When the terminal 101 is hardware, it can be various electronic devices, including but not limited to tablet computers, laptop computers, and desktop computers. When the terminal 101 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services) or as a single software or software module, which is not specifically limited here.

[0087] In an embodiment of the present application, the terminal 101 first obtains an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks. The preset training sample set may include a first training sample set and a second training sample set. After the sample set and the initial model architecture are prepared, the terminal 101 can supervise and fine-tune the initial baseline model based on the first training sample set to obtain a supervised fine-tuning model and a preference data sample; optimize and train the initial reward model based on the preference data sample to obtain a target reward model, and optimize and train the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model; then, generate a first reward signal based on the preference optimization model and the first training sample set, and generate a second reward signal based on the preference optimization model, the target reward model, and the second training sample set; and then optimize and train the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

[0088] Server 103 can be a server that provides various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module, without further limitation.

[0089] Alternatively, the system architecture may also not include the server 103. In other words, the server 103 may be an optional device in the embodiment of the present application, that is, the method provided in the embodiment of the present application can be applied to a system structure that only includes the terminal 101, and the embodiment of the present application does not limit this.

[0090] It should be understood that Figure 1 The number of terminals, networks, and servers in the figure is only for illustration and any number of terminals, networks, and servers may be used according to implementation requirements.

[0091] See also Figure 2 , Figure 2 A flowchart of a model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application can be a terminal that executes the model training method, a processor in the terminal that executes the model training method, or a training service for a feature extraction model in the terminal that executes the model training method. For ease of description, the specific execution process of the model training method is described below using the execution subject being a processor in a terminal as an example.

[0092] S201: Acquire an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes a plurality of training sample groups corresponding to a plurality of object response training tasks.

[0093] The initial baseline model can be a large language model that has completed general language pre-training and can be used as the starting model for the training process. The initial baseline model can have basic language understanding and generation capabilities, but has not been fine-tuned for specific tasks. For example, the initial baseline model can be an open source large language model, such as the Tongyi Qianwen large model Qwen2.5-7B in the related art. In addition, the initial reward model is an initial reward scorer that can be used to estimate the quality or preference of the model generation results. The initial reward model can be inherited from the initial baseline model and has not been trained with preference data.

[0094] Optionally, the object response training task can be a task that takes information about a certain type of object as input and generates response information specific to that object. The objects can be any of the following types: e-commerce objects (e.g., products, stores, reviews, categories, events), social objects (e.g., social dynamics, topics, tags, follow-up relationships), medical objects (e.g., medical records, checklists, clinical notes), educational objects (e.g., test questions, knowledge points, courses, answer sheets), and transportation objects (e.g., travel routes, locations, stops), etc., depending on the platform or scenario type.

[0095] For example, if the object is a test question, the object response training task can be to generate the corresponding knowledge point label or reference answer based on the question content. If the object is a travel route, the object response training task can be to generate travel suggestions or congestion prediction results based on the route information.

[0096] For example, the following describes this application using a commodity as an example. In the case where the object is a commodity, the multiple object response training tasks can include at least one or more of the following types of tasks: category prediction, title generation, information extraction, product matching, search relevance, and query annotation. Specifically, the category prediction type of object response training task can be used to predict the Standard Product Unit (SPU) category label to which the commodity belongs based on the commodity's basic attribute information. For example, for the commodity "Local Enoki Mushrooms, approximately 150g; Color: White," the model can predict that its SPU is "White Enoki Mushrooms." The title generation type of object response training task can be used to generate a more suitable new title for display based on the commodity's original title, brand, flavor, specifications, and other attributes. For example, given the original title "XX Brand Nighttime Sanitary Napkins, 35cm, 8 pieces / pack," the generated new title can be "XX Brand Nighttime Cotton Soft and Delicate Sanitary Napkins, 350mm, 8 pieces." This task helps improve the standardization and user readability of product titles. Information extraction-type object response training tasks can be used to extract structured attribute information from product titles or product attributes (such as brand, series, and origin). For example, for the title "XXX Brand Vita Moisturizing Face Cream," the model might extract fields such as "Net Content: 150; Packaging: Bottled; Packaging Specifications: 150ml*1 Bottle; Brand: XXX; Quantity: 1 Bottle; Category: Face Cream." Product matching-type object response training tasks can be used to select appropriate Child Standard Product Unit Prediction (CSPU), a more fine-grained product identifier, for a product. For example, for the product "Frozen Bamboo Snail Meat, Approximately 500-550g; Net Content: 500g," the model identifies the CSPU as "Fresh Chilled Bamboo Snail." Search relevance-type object response training tasks can be used to assess the relevance between the user query and the product title in the product context. For example, for the query "Stomach-Nourishing Porridge" and the product title "Stomach-Nourishing Purple Sweet Potato Porridge [Contains Diced Purple Sweet Potato]," the model might determine a strong correlation. Query-labeled object response training tasks can be used to assign part-of-speech tags to user search queries. For example, for the query "tops for men," the tagging result is "tops: product category, attribute: men's clothing." Data from each of these tasks can be organized into corresponding training sample groups and then collectively constituted into the aforementioned pre-set training sample set for subsequent model training, improving the model's comprehensive modeling capabilities in product response generation scenarios.

[0097] Optionally, the preset training sample set may include a first training sample set and a second training sample set. The first training sample set may be a pre-labeled dataset that can be used for subsequent supervised training. The first training sample set may specifically include multiple training sample groups for product response-related tasks, each of which may correspond to a specific object response training task. In other words, the first training sample set may include multiple training sample groups corresponding to multiple object response training tasks. Each training sample group in the first training sample set includes multiple first training sample data and a reference output corresponding to each first training sample data, i.e., a true label. Furthermore, the second training sample set may be a product-related input data set that does not include a reference output (label). Specifically, the second training sample set may include multiple training sample groups for product response-related tasks, each of which may correspond to a specific object response training task. In other words, the second training sample set may include multiple training sample groups corresponding to multiple object response training tasks. Each training sample group in the second training sample set includes multiple second training sample data.

[0098] S202: Perform supervised fine-tuning on the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample.

[0099] In one embodiment, the initial baseline model may be fine-tuned using a low-rank adaptation method (LoRA) fine-tuning or full fine-tuning method to obtain a supervised fine-tuning (SFT) model.

[0100] It's worth noting that during this supervised fine-tuning process, model parameters are continuously optimized through iterative training, allowing for multiple training cycles (epochs) on the complete first training set. Each epoch represents a complete learning of the model's entire first training set. Through multiple rounds of training, the model continuously adjusts its internal representations and parameters to more fully learn the common patterns and fine-grained features across various tasks, thereby improving its ability to generalize to unseen examples. After training is complete, the resulting supervised fine-tuned model possesses multi-task responsiveness.

[0101] In one embodiment, preference data samples may refer to a set of one or more output responses constructed based on the same input data, with the relative preference relationship between "selection" and "rejection" clearly marked between these responses, for training the model to learn human preferences. Optionally, preference data samples can be obtained by extending the error cases of the supervised fine-tuning model in the object response training task. The ultimately generated paired preference data can be used as training data input for the subsequent DPO (Direct Preference Optimization) stage, i.e., preference data samples, which can be used to guide the model to be more inclined to generate high-quality responses under the same input conditions, thereby improving response quality and preference consistency.

[0102] S203: Optimizing and training the initial reward model based on the preference data sample to obtain a target reward model, and optimizing and training the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model.

[0103] The target reward model takes as input a sample and its corresponding response, and outputs a score that measures the quality of the response. This model can reflect the relative quality of responses and can be used to evaluate model generation results in scenarios without a reference output, as well as provide reward signals for the policy model in subsequent reinforcement learning stages.

[0104] Alternatively, the goal of a preference optimization model can be to learn the ability to select high-quality responses and reject low-quality responses. Given the same input, a preference optimization model tends to generate responses that align with human preferences. Compared to supervised fine-tuning models, preference optimization models focus more on the relative differences in output quality and can be used to initialize the reinforcement strategy model in subsequent stages.

[0105] In one embodiment, the preference data sample includes multiple pairs of preference data. Each pair can consist of the same input, a preferred response, and a non-preferred response, representing the "selection" and "rejection" relationships in human preferences. Alternatively, the input sample and the corresponding response (including both preferred and non-preferred responses) can be used as input to train a reward model to predict the score difference between the two responses and minimize the expected ranking error, thereby ensuring that the preferred response is scored higher than the non-preferred response. After training, a target reward model is obtained that quantifies the quality of the response.

[0106] In one embodiment, a supervised fine-tuning model can be trained using direct preference optimization based on the same preference data samples. This process uses the supervised fine-tuning model as the initial policy model and, by introducing a Determined Proportional Opportunity (DPO) loss function, adjusts the model's output distribution, making it more likely to generate preferred responses and less likely to generate non-preferred responses under the same input conditions. After optimization, a preference optimization model is obtained that better aligns with human preference guidance.

[0107] It should be noted that the two optimization processes of optimizing and training the above-mentioned initial reward model based on the above-mentioned preference data samples to obtain the target reward model, and optimizing and training the above-mentioned supervised fine-tuning model based on the above-mentioned preference data samples to obtain the preference optimization model can be executed in parallel or sequentially to respectively output the target reward model and the preference optimization model, thereby providing scoring capabilities and strategy basis for the subsequent reinforcement learning training stage.

[0108] S204: Generate a first reward signal based on the preference optimization model and the first training sample set, and generate a second reward signal based on the preference optimization model, the target reward model, and a second training sample set in the preset training sample set.

[0109] Optionally, the first reward signal can be a fractional reward signal calculated using a difference metric by generating a predicted response based on the preference optimization model on the first training sample set and comparing the predicted response with a known reference output. The difference metric can be, for example, the edit distance or the Recall-Oriented Understudy for Gisting Evaluation (ROUGE). The first reward signal is a rule-based reward with strong task interpretability.

[0110] Optionally, the second reward signal may refer to generating a predicted response on a second training sample set (without reference output) based on the preference optimization model, and inputting the input and response together into the target reward model for scoring to obtain a numerical reward representing the generation quality.

[0111] In one embodiment, on the one hand, a preference optimization model is used to generate corresponding predicted responses for each input data in the first training sample set. Since the first training sample set contains reference outputs, the generated responses can be compared with the reference outputs one by one, and the degree of difference between the responses and the reference outputs can be calculated using difference metrics such as edit distance, and a first reward signal for reinforcement learning can be further generated. The first reward signal can be used to quantify the degree of closeness between the model generation results and the standard answers. On the other hand, a preference optimization model is used to generate responses for the input data in the second training sample set. Since the second training sample set is an unsupervised sample and lacks a reference output, the input data and the generated responses are input into the target reward model together, and the target reward model outputs the scoring result as the second reward signal. The second reward signal can be used to measure the degree of preference matching of the model-generated output in terms of actual semantic quality, fluency, completeness and other dimensions.

[0112] The first and second reward signals are used in different stages of the reinforcement learning phase to guide the policy model to optimize its generation strategy, ensuring that the generated results are in the expected form and improving the response quality to open task inputs and consistency with human preferences.

[0113] S205: Optimize and train the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

[0114] In some embodiments, the preference optimization model can be first constructed as a proximal policy optimization (PPO) model. The PPO model is trained in the first phase based on the first reward signal to achieve preliminary optimization of the reinforcement learning objective. Furthermore, the PPO model can be trained in an additional round based on the second reward signal and the second training sample set to further improve model performance based on the initial optimization, thereby obtaining a target object generative pre-trained model.

[0115] It is understandable that the above-mentioned target object generative pre-training model can be used as a base model for supervised fine-tuning, preference modeling or reinforcement learning optimization of downstream subtasks to improve the accuracy and diversity of object response generation.

[0116] In the embodiment of the present application, an initial baseline model, an initial reward model and a preset training sample set are obtained, wherein the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks; based on the first training sample set in the preset training sample set, the initial baseline model is supervised and fine-tuned to obtain a supervised fine-tuning model and a preference data sample; based on the preference data sample, the initial reward model is optimized and trained to obtain a target reward model, and based on the preference data sample, the supervised fine-tuning model is optimized and trained to obtain a preference optimization model; based on the preference optimization model and the first training sample set, a first reward signal is generated, and based on the preference optimization model, the target reward model and the second training sample set in the preset training sample set, a second reward signal is generated; based on the first reward signal, the second reward signal and the second training sample set, the preference optimization model is optimized and trained to obtain a target object generative pre-training model. On the one hand, fine-tuning the initial baseline model based on supervised samples can improve the basic capabilities of each task of the model; on the other hand, by constructing preference data samples to jointly train the reward model and the strategy model, the model behavior and human preferences are effectively aligned. At the same time, the introduction of two types of reward signals, derived from the reward model and the preference optimization model, and their integration into the unsupervised data training phase effectively improves the model's generalization capabilities and significantly reduces its reliance on high-quality manually labeled data. The resulting target object generative pre-training model has enhanced object understanding, strategy generation, and interactive response capabilities, and is better adapted to the multi-scenario intelligent needs of relevant platforms.

[0117] In one embodiment, Figure 3 As shown, another model training method is provided, including the following steps:

[0118] S301: Acquire an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks.

[0119] Specifically, S301 is consistent with S201 and will not be repeated here.

[0120] S302: Based on the first training sample set in the preset training sample set, the initial baseline model is updated with multiple rounds of parameters using a supervised learning method to obtain a supervised fine-tuning model.

[0121] In one embodiment, the first training sample set in the preset training sample set is first selected. The first training sample set can be a supervised data set with manually annotated output labels, and covers multiple object response training tasks (for example, tasks such as product title generation, attribute extraction, category prediction, query labeling, product matching and search relevance judgment). Each first training sample includes an input sample, a prompt word, and a corresponding expected output result (i.e., a true label). Then, the initial baseline model is used as the initial state of the pre-trained language model, and supervised learning is performed based on the above-mentioned first training sample set. The supervised learning process may include training for multiple epochs. In each epoch, the model outputs a prediction result through forward reasoning, and calculates the loss between the prediction result and the true label (such as cross entropy loss), and then updates the model parameters through the back-propagation algorithm. This process is iterated continuously until the model converges on the validation set.

[0122] Through these multiple rounds of parameter updates, the model developed preliminary representation learning and task response capabilities across multiple product response tasks, ultimately resulting in a supervised fine-tuning model. This supervised fine-tuning model not only possesses a general understanding of various product information processing tasks but also provides a robust initialization foundation for subsequent preference training and reinforcement learning stages.

[0123] S303: During multiple rounds of parameter updates on the initial baseline model, the first training sample set is sampled to obtain input samples and multiple candidate responses corresponding to each of the input samples.

[0124] Optionally, during supervised fine-tuning training, samples can be sampled from the first training sample set based on task category or data distribution, and a higher-temperature sampling strategy can be used to generate multiple candidate responses based on the currently trained supervised fine-tuning model. This sampling strategy helps generate response samples that contain diverse expressions or potential errors, facilitating the subsequent construction of preference data required for training.

[0125] S304: Filter multiple candidate responses corresponding to each of the above input samples based on preset indicators to obtain a first response and a second response corresponding to each of the above input samples to construct a paired preference data sample, wherein the above first response is better than the above second response.

[0126] Furthermore, based on preset indicators (such as edit distance, ROUGE, etc.) or combined with manual scoring rules, multiple candidate responses corresponding to each input sample can be screened, and responses with relatively good semantics, structure or content performance can be selected as the "first response", and responses with language defects or missing information can be selected as the "second response", thereby constructing paired samples with a clear "selection-rejection" preference relationship.

[0127] The preference data samples constructed in the embodiments of the present application can be used to train reward models and optimization strategy models, enabling the models to directly learn human preferences for high-quality responses, thereby achieving more stable and intuitive behavioral adjustments in the subsequent DPO stage, and effectively improving the accuracy, naturalness and adaptability of the generated content.

[0128] S305: Inputting the input samples contained in the preference data sample, and the first response and the second response corresponding to each input sample into the initial reward model.

[0129] Optionally, the constructed preference data sample (input + first response + second response) is input into the initial reward model, which will be used to learn the ability of "preference judgment".

[0130] S306: Perform score generation processing on the first response and the second response corresponding to each of the input samples through the initial reward model to obtain a first response score corresponding to the first response and a second response score corresponding to the second response.

[0131] In one embodiment, forward reasoning can be performed on the first and second responses in the constructed paired preference data based on the initial reward model to generate a score that characterizes the degree of response preference. Specifically, the input sample and the first response are concatenated to form a complete input sequence, which is then fed into the initial reward model to generate a first response score. Similarly, a corresponding second response score is generated based on the second response. The first and second response scores can then be used to construct a scoring loss function to guide the reward model in learning the preference ranking relationship of response quality.

[0132] S307: Constructing a scoring loss function according to the first response scoring value and the second response scoring value.

[0133] Alternatively, a ranking loss can be used to construct a loss function for pairwise learning, where the goal is to encourage the model to give a higher score to the first response. Specifically, the scoring loss function can be expressed as follows:

[0134] (1);

[0135] in, represents the parameters of the reward model, represents the scoring loss function, K represents the number of preference data samples in the preference data set (the number of paired samples is K / 2 pairs). D represents the preference data sample set, which can contain multiple groups (x, y w ,y l ) triples; Represents a triple sampled from the preference data sample set D: input sample x, preference response (first response) , non-preferred response (second response) , represents the first response score value, represents the second response score. σ(·) represents the Sigmoid function, and E(·) represents the mathematical expectation.

[0136] S308: Based on the above-mentioned scoring loss function, iteratively optimize the parameters of the above-mentioned initial reward model to obtain the target reward model.

[0137] Specifically, based on the above-mentioned scoring loss function, the parameters of the initial reward model are updated using optimization algorithms such as gradient descent, and finally a target reward model that can stably distinguish between preferences is trained.

[0138] In this embodiment, by inputting constructed preference data samples into an initial reward model and generating scoring values ​​based on paired responses, a ranking loss function is further constructed and parameter optimization is performed. This allows the reward model to learn the preference ranking relationship between responses, thereby providing a more accurate and stable reward signal for policy optimization in the subsequent reinforcement learning phase. This method not only improves the accuracy of the model's response preference judgment, but also has good scalability and versatility.

[0139] S309: Input each of the above preference data samples into the above supervised fine-tuning model.

[0140] Among them, inputting the above preference data samples into the previously obtained supervised fine-tuning model can be used to further guide the model to understand preference differences and thus make behavioral adjustments.

[0141] S310: Calculating the generation probability information of the supervised fine-tuning model for the first response and the second response in each of the preference data samples.

[0142] Specifically, the supervised fine-tuning model predicts the first response and the second response respectively based on the current parameter state, and calculates the conditional probability information of the supervised fine-tuning model generating the first response and the second response to reflect the model's generation preference for the two candidate outputs.

[0143] S311: Construct a contrast loss function based on the above generation probability information.

[0144] Furthermore, a contrastive loss function can be constructed based on this generation probability information. For example, using the contrastive log-probability difference, this function encourages the model to increase the generation probability of the first response and suppress the generation probability of the second response. This contrastive loss function measures the model's ability to distinguish between two responses under the same input conditions. The optimization goal is to maximize the difference between the generation probability of the preferred response and the generation probability of the second response. By minimizing this loss function, the parameters of the supervised fine-tuning model can be updated, further improving the model's ability to fit human preferences.

[0145] Among them, the DPO loss function can be constructed by the following formula (2):

[0146] (2);

[0147] in, represents the DPO loss function, represents the initial supervised fine-tuning model, which can be used as a reference model. Represents the target to be trained, x represents the input sample in the preferred data sample, Indicates the preferred response chosen by humans, i.e. the first response, represents the second choice response, i.e. the second response. D represents the preference data sample set, which can contain multiple groups (x, y w ,y l ) triples, representing the preferred data samples. β can represent the scaling factor, which is used to control the contrast intensity; σ(·) represents the Sigmoid function, and E(·) represents the mathematical expectation. The goal of the DPO loss function is: if the current model to be trained Compared to the reference model More inclined to generate better response y w , rather than the poor response y l , then the loss is smaller. Conversely, if the model is biased towards poor responses, the loss increases, thereby guiding the model to optimize more in line with the preference.

[0148] S312: Iteratively update the parameters of the supervised fine-tuning model based on the contrast loss function to obtain a preference optimization model.

[0149] Optionally, based on the constructed contrastive loss function, the parameters of the supervised fine-tuning model can be optimized through back-propagation, and ultimately a preference optimization model that can reflect human preference judgments can be obtained.

[0150] S313: Generate a first reward signal based on the preference optimization model and the first training sample set, and generate a second reward signal based on the preference optimization model, the target reward model, and the second training sample set in the preset training sample set.

[0151] Specifically, S313 is consistent with S204 and will not be repeated here.

[0152] S314: Optimize and train the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

[0153] Specifically, S314 is consistent with S205 and will not be repeated here.

[0154] In the embodiment of the present application, by comparative modeling of the generation probabilities of the first response and the second response, the model is guided to tend to generate outputs that the user prefers under the same input conditions, thereby achieving fine-tuning at the behavioral level. As a result, not only is the model's tendency to generate high-quality responses strengthened, but its ability to suppress low-quality responses is also significantly enhanced, which helps to improve the naturalness, consistency, and controllability of the overall response. The resulting preference optimization model is more in line with the preference selection criteria of real users, providing better strategic guidance for downstream generation tasks.

[0155] In one embodiment, Figure 4 As shown, another model training method is provided, including the following steps:

[0156] S401: Acquire an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes a plurality of training sample groups corresponding to a plurality of object response training tasks.

[0157] Specifically, S401 is consistent with S201 and will not be repeated here.

[0158] S402: Perform supervised fine-tuning on the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample.

[0159] Specifically, S403 is consistent with S202 and will not be repeated here.

[0160] S403: Optimizing and training the initial reward model based on the preference data sample to obtain a target reward model, and optimizing and training the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model.

[0161] Specifically, S403 is consistent with S203 and will not be repeated here.

[0162] S404: For each first training sample data in the first training sample set, determine a first predicted response corresponding to the first training sample data by using the preference optimization model.

[0163] In one embodiment, each prompt word in the first training sample data can be input into the preference optimization model to generate a corresponding natural language prediction response, namely a first prediction response, which serves as an input basis for subsequent evaluation of model performance and generation of rewards.

[0164] S405: Compare the first predicted response with the reference output of the first training sample data, and calculate a difference measurement index between the first predicted response and the reference output.

[0165] In one embodiment, the first predicted response generated by the preference optimization model can be compared character by character with the corresponding reference output, and the minimum number of operations can be calculated using the edit distance algorithm in the related art, and a standardized difference ratio can be further obtained as a difference measurement indicator between the above-mentioned first predicted response and the above-mentioned reference output.

[0166] Specifically, the ratio of the edit distance to the character length of the reference output can be used as the normalized edit difference rate to measure the similarity between the predicted response and the reference output (i.e., the difference metric).

[0167] S406: Generate a first reward signal according to the above difference measurement indicator.

[0168] Specifically, the above-mentioned difference metric can be used as input and converted through a preset reward mapping function to generate a corresponding first reward signal. The preset reward mapping function can be determined according to actual needs and is not specifically limited in this application.

[0169] S407: For each second training sample data in the second training sample set included in the preset training sample set, determine a second predicted response corresponding to the second training sample data by using the preference optimization model.

[0170] The second training sample data may include input samples and prompt words.

[0171] Optionally, each second training sample data in the second training sample set may be sequentially input into the preference optimization model, and its corresponding natural language response, ie, the second predicted response, may be output.

[0172] S408: Inputting the above-mentioned second training sample data and the corresponding second predicted response into the above-mentioned target reward model to perform reward signal generation processing through the above-mentioned target reward model to generate a second reward signal.

[0173] In one embodiment, each second training sample data set and the corresponding second predicted response can be input into an optimized target reward model. The target reward model performs semantic matching or relevance scoring on each set of input samples and the second predicted response, and outputs a corresponding reward value. This reward value serves as the second reward signal in the second stage, which is used to further tune the preference optimization model in the reinforcement learning strategy optimization, improving its generation capability and response quality under unsupervised input conditions.

[0174] S409: Optimize and train the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model.

[0175] Specifically, S409 is consistent with S205 and will not be repeated here.

[0176] The embodiment of the present application, through the above-mentioned model training method, can fully integrate the advantages of supervised learning and reinforcement learning, achieve high-quality preliminary modeling based on the first labeled training sample set, and then use paired preference data and reward models to build a refined preference learning path. Then, combined with the unsupervised second training sample set, more actual scenario data is introduced. Through the joint optimization of the two-stage reward signal, the generalization ability and response quality of the object-generating pre-training model in multiple tasks and multiple scenarios are effectively improved. This method not only improves the model's ability to understand and generate complex commodity data, but also significantly reduces its dependence on large-scale manually labeled data, and has stronger scalability and deployment adaptability.

[0177] In one embodiment, Figure 5 As shown, another model training method is provided, including the following steps:

[0178] S501: Acquire an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes a plurality of training sample groups corresponding to a plurality of object response training tasks.

[0179] Specifically, S501 is consistent with S201 and will not be repeated here.

[0180] S502: Acquire an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes a plurality of training sample groups corresponding to a plurality of object response training tasks.

[0181] Specifically, S501 is consistent with S201 and will not be repeated here.

[0182] S503: Optimizing and training the initial reward model based on the preference data sample to obtain a target reward model, and optimizing and training the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model.

[0183] Specifically, S503 is consistent with S203 and will not be repeated here.

[0184] S504: Perform reinforcement learning on the preference optimization model based on the first reward signal to obtain a reinforcement object generative pre-training model.

[0185] In one embodiment, the preference optimization model can be further trained and optimized using a proximal policy optimization algorithm in reinforcement learning based on the first reward signal to obtain a pre-trained model for generative reinforcement learning. During the reinforcement learning process, the policy model parameters are updated using policy gradients, using the first reward signal as feedback, to iteratively optimize the model behavior.

[0186] Optionally, during the reinforcement learning training process, the following formula (3) can be used as the loss function to optimize the policy model parameters:

[0187] (3);

[0188] in, represents the loss function, represents the parameters of the current preference optimization model, represents the expectation under the data distribution generated by the current model, Represents language modeling constraints, which are used to maintain the rationality and fluency of the generated language and prevent semantic collapse. Specifically, Represents an input sample pair, where x is the input of the object response training task (such as product information and related prompt words, etc.), and y is the response text generated by the current model (such as product title, etc.). represents the data distribution sampled from the current model (enhanced object generation pre-trained model), It represents the score of the reward model for the generated response y, which is used to measure the quality, relevance or preference satisfaction of the response. represents the probability that the current model generates response y under input x, It represents the probability of the preference optimization model generating y under the same input x, which serves as a reference baseline. Represents the original language modeling corpus data distribution, and strengthens the probability of the strategy model generating a complete sentence. represents the probability of the enhanced object generative pre-training model generating a complete sentence, and γ represents the regularization strength parameter, which is used to control whether the strength of the language modeling task objective is considered.

[0189] S505: Perform unsupervised training on the reinforcement object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model.

[0190] In one embodiment, the above-mentioned reinforcement object generative pre-training model is unsupervisedly trained based on the above-mentioned second reward signal and the above-mentioned second training sample set to obtain a target object generative pre-training model, including: for each second training sample data in the above-mentioned second training sample set, using the above-mentioned reinforcement object generative pre-training model to generate object response information corresponding to the above-mentioned second training sample data; inputting the above-mentioned second training sample data and the corresponding above-mentioned object response information into the target reward model to obtain a reward score for reinforcement training; constructing a reinforcement learning objective function with the reward score as the return signal; and optimizing the parameters of the above-mentioned reinforcement object generative pre-training model based on the above-mentioned reinforcement learning objective function to obtain the trained target object generative pre-training model.

[0191] Specifically, first, each second training sample data in the second training sample set is sequentially input into the trained reinforcement object generative pre-training model to generate corresponding natural language object response information. Subsequently, the above-mentioned second training sample data and the corresponding object response information are input into the target reward model together. The target reward model can comprehensively evaluate the accuracy, relevance, readability and other dimensions of the response content based on its internal strategy, and output the corresponding reward score as the reward signal of the generated behavior. Furthermore, a reinforcement learning objective function is constructed with the reward score as the core, for example, the expected return maximization objective based on the PPO policy gradient algorithm is adopted. The system encourages strategy exploration by minimizing the policy loss function of negative rewards and combining entropy regularization and other means to continuously optimize the parameters of the reinforcement object generative pre-training model in each round of training. Thus, the trained target object generative pre-training model is finally obtained. The model can generate object response texts that are more in line with human preferences and real-related logic in the absence of clear supervision labels, further improving its practicality and stability in complex scenarios.

[0192] In this application, by introducing a two-stage reinforcement learning mechanism (i.e., using a first reward signal for supervised reinforcement and a second reward signal for unsupervised optimization), the present invention fully exploits the human preferences and semantic information contained in the training data, enabling the generative model to generate object responses that are more closely aligned with user intent, application scenarios, and natural language expression habits. In particular, in unsupervised data scenarios, the target reward model provides scoring feedback on the generated results, guiding the policy model to continuously adjust its generation strategy, significantly improving the model's responsiveness and generalization capabilities for complex queries and ambiguous requirements.

[0193] Furthermore, the generated target object generative pre-trained model can be used in a variety of subsequent object-related tasks to support a variety of application scenarios such as object information structuring, search recommendation optimization, intelligent classification, and question-answer alignment. This can further improve the intelligence and processing efficiency of object content processing in scenarios such as e-commerce platforms or distribution systems. Optionally, the target object generative pre-trained model can be used in the OFD field.

[0194] See also Figure 6 , Figure 6 A flowchart of another model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application can be a terminal that executes the model training method, a processor in the terminal that executes the model training method, or a training service for a feature extraction model in the terminal that executes the model training method. For ease of description, the specific execution process of the model training method is described below using the execution subject being a processor in a terminal as an example.

[0195] S601: Obtain a target object generative pre-training model and a task training dataset corresponding to a preset subtask.

[0196] S601: Use the above-mentioned target object generative pre-training model as the initialization basic model, and use the above-mentioned task training data set for fine-tuning training to obtain a target task model for the preset subtask; wherein, the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by any one of the model training methods provided in the embodiments of the present application.

[0197] Optionally, when the object is a commodity, the preset subtask may be a preset on-demand food delivery subtask.

[0198] In a specific embodiment, a pre-trained target object generative pre-training model is first obtained. The model is trained based on large-scale text data such as product titles, attributes, user queries, comments, etc., and has good product semantic understanding and text generation capabilities. Subsequently, a task training dataset is constructed for the "product title generation" subtask in the OFD scenario. The dataset includes multiple pairs of input product attributes and target titles. The target object generative pre-training model is used as the initialization model and fine-tuned with the task training dataset. During the fine-tuning process, by minimizing the generation loss function of the target title, the model is guided to learn to generate semantically consistent and e-commerce style title expressions from product attributes, and finally a target task model suitable for the title generation subtask is obtained. The model can be directly deployed in real scenarios such as product launches and title optimization to achieve the automatic generation of high-quality product titles.

[0199] Understandably, because the target object generative pre-trained model has been trained on a large amount of task data within the OFD domain and possesses excellent general semantic modeling and task transfer capabilities, the amount of supervised data required for task-specific fine-tuning is significantly lower than that required for direct training from the initial baseline model. Furthermore, thanks to its task-aligned pre-training capabilities, the target object generative pre-trained model requires only a few rounds to achieve convergence during fine-tuning, significantly reducing the computational resource overhead and time cost of model training.

[0200] In the embodiment of the present application, with the help of the above-mentioned target object generative pre-training model, it is possible to achieve rapid adaptation and high-quality response to multiple object response tasks through a small amount of fine-tuning or direct calling without the need to re-train each downstream task on a large scale, thereby significantly reducing training costs, improving system generalization capabilities and actual deployment efficiency.

[0201] It is worth noting that the target object generative pre-training model can be applied to multiple scenarios on the business (Business, B) and consumer (Consumer, C) sides of relevant platforms, and demonstrates strong generalization and adaptability in different tasks. Specifically, in the B-side scenario, the target object generative pre-training model can be used for the following tasks: retail product category prediction, retail product unit click-through rate (CPV) prediction, retail product title rewriting, retail product unit or standard product sub-unit prediction, etc., to improve product clustering and standardization processing effects. In addition, in the C-side scenario, the above-mentioned target object generative pre-training model can also be deployed in the following tasks: query-product (food) relevance evaluation, query-product (retail) relevance evaluation, query-store relevance evaluation, etc. Through the specific deployment of the above tasks, the target object generative pre-training model provided in this application can not only cover B-side data-driven tasks, but also adapt to the C-side intelligent processing needs for user intent understanding, further demonstrating the wide applicability of this model in the OFD field.

[0202] The following describes this application in conjunction with 7. Figure 7 This paper provides a schematic diagram of the overall training and optimization process of the target object generative pre-training model, covering two major steps: multi-task post-training and specific task fine-tuning. Among them, the multi-task post-training step can specifically include three stages: supervised fine-tuning, direct preference optimization, and proximal strategy optimization, reflecting the layer-by-layer optimization path from general capability building to human preference alignment and then to scenario adaptation. The overall data preparation required includes multi-task training data, which contains sample data corresponding to object response training tasks of category prediction type, title generation type, information extraction type, product matching type, search relevance type, and query annotation type.

[0203] In one embodiment, Figure 7 In the supervised fine-tuning phase, the initial baseline model undergoes multiple rounds of parameter optimization based on the multi-task training data to obtain a supervised fine-tuning model with general generation capabilities. In the direct preference optimization phase, preference data is introduced. By constructing paired preference data samples and combining them with a reward model to score different response qualities, the model is trained and optimized using a comparative loss function. This improves the model output to be closer to human expectations in terms of style, format, and content rationality. The reward model can generate scoring signals based on techniques such as edit distance and semantic matching to help the model strengthen its learning of high-quality outputs. In the proximal policy optimization phase, reinforcement learning techniques, such as the PPO algorithm, are further introduced under unsupervised data conditions. The output of the trained reward model is used as a reward signal to guide the model to optimize its response strategy. In each round of optimization, the amplitude of the policy update is controlled to avoid policy instability, thereby improving the generalization and robustness of the model in open input scenarios. After completing the multi-task post-training step above, we can proceed to the specific task fine-tuning step, and further use the supervised dataset of the task to perform lightweight fine-tuning on the optimized basic model for a specific OFD object response subtask (such as title generation, category matching, etc.). At this stage, since the model already has strong general semantic understanding and preference alignment capabilities, it only requires a small amount of specific task data and low computing resources to quickly complete fine-tuning, significantly improving the performance and usability of the task. Figure 7 The process shown effectively combines multi-source heterogeneous data, supervised learning, and reinforcement learning techniques. While building general generation capabilities, it gradually introduces human preferences, ultimately achieving a high-quality, strongly adaptable generative pre-training model for target objects.

[0204] In one embodiment, in order to verify the generalization ability of the target object generative pre-training model on new specific tasks in the OFD field, supervised fine-tuning can be performed on the preset hypernym-hyponym classification (HHC) dataset and title-query alignment (TQA) dataset based on the trained target object generative pre-training model to achieve adaptation to specific tasks. Among them, the HHC dataset can be used to determine whether there is a hyponym-hyponym relationship between two product-related words, and the TQA dataset can be used to determine whether a given product title is semantically matched with a user query. The above two datasets both belong to actual tasks in the OFD field, but are not included in the original multi-task training set of the target object generative pre-training model (i.e., the above preset training sample set).

[0205] Specifically, the performance of the target object generative pre-trained model and the initial baseline model was compared on 12 pre-set online-to-offline (O2O) object response tasks, with the comparison metric being the accuracy (in %) on each task's test set. The tasks involved cover six broad categories: category prediction, title generation, information extraction, product matching, search relevance, and query tagging. These tasks include 12 specific tasks: standardized product unit prediction, standardized product subunit prediction, product category prediction, multi-turn conversational product category prediction, store-based product category prediction, product attribute extraction, query category prediction, query-product (food) relevance assessment, query-product relevance assessment, query term part-of-speech tagging, query-store relevance assessment, and product title generation. The comparison results show that the initial baseline model achieves a reasonable level of accuracy across several typical O2O product response tasks. In contrast, the target object generative pre-training model provided in the embodiment of the present application has better overall performance in various tasks under the same data and training settings, and the average accuracy rate has been improved. Among them, the improvement in individual tasks is particularly significant. For example, the accuracy rate of the query word part-of-speech tagging task can be improved by more than 4%. The comprehensive comparison results show that the target object generative pre-training model, which is pre-trained by integrating multi-task corpus and optimized by reinforcement learning strategy, can achieve higher accuracy in most tasks as the initialization basis for downstream tasks, thereby effectively improving the overall performance of the product response system in actual O2O scenarios.

[0206] In addition, to further verify the performance advantages of the target object generative pre-training model provided by this application under multiple mainstream base models, three large language models from related technologies were selected as initialization base models and compared and evaluated on four typical representative tasks under the same training data and SFT strategy, including the product title generation task, the product sub-standard unit prediction task, the product category prediction task, and the product category prediction task under multi-round dialogue. The comparative evaluation shows that although some large language models in related technologies have advantages in individual tasks, the overall average accuracy still needs to be improved. In contrast, the target object generative pre-training model provided by the embodiment of this application shows higher consistency and accuracy under the same training strategy and evaluation conditions, achieving a high average accuracy level in the above four tasks. It can be seen that the target object generative pre-training model provided by this application not only has good generalization ability in 12 multi-task training tasks, but also shows high task adaptability and overall performance advantages when compared with multiple mainstream base models. This model has strong transferability and is suitable for complex and diverse real-world product response generation tasks in the OFD field, effectively improving the system's comprehensive performance in terms of accuracy, robustness, and response quality.

[0207] Furthermore, the relevant algorithmic modules or functional components used in the entire model optimization process can be removed or replaced one by one to evaluate the actual contribution of each component to the final model performance. Specifically, ablation experiments were conducted on the multi-task post-training step. The experimental results show that, compared to the supervised fine-tuning model, the average accuracy of the supervised fine-tuning model improved by over 1% after adding the direct preference optimization stage, using the performance improvement of the supervised fine-tuning model as a baseline (i.e., the relative performance improvement of the supervised fine-tuning model is 0). This indicates that the direct preference optimization method can effectively align model behavior with human preferences. Furthermore, when the edit distance-based proximal policy optimization is introduced on top of the direct preference optimization, performance is further improved. When the reward model-based proximal policy optimization is introduced, the improvement is even greater, indicating that both reward signals can effectively improve the accuracy and practicality of the model generation results. More significantly, when the edit distance reward and the reward model are simultaneously introduced as a joint reward signal for proximal policy optimization, the average performance of the final model is significantly improved, with the largest improvement compared to the baseline model, indicating the importance of diversity and complementarity of reward signals in the post-training process. It can be seen that this ablation experiment fully verifies the independent and collaborative value of each component in the entire model optimization process. In particular, in the reinforcement learning stage, by introducing multi-dimensional reward signals, the model's generalization ability and adaptability to complex response tasks can be further improved.

[0208] Further, see Figure 8 , Figure 8The performance comparison results of supervised fine-tuning using different basic models on two newly introduced OFD specific tasks are shown, further verifying the generalization ability and training efficiency of the target object generative pre-training model proposed in this application when migrating to new specific tasks. Figure 8 The accuracy performance of the initial baseline model and the target object generative pre-trained model as the base model in the HHC task and TQA task was compared within five consecutive supervised fine-tuning training rounds. The results show that the target task model based on the target object generative pre-trained model is significantly better than the target task model based on the initial baseline model in both tasks. Specifically, whether in the initial training rounds or in subsequent iterations, the fine-tuned model based on the target object generative pre-trained model is significantly better than the similar model based on the traditional initial baseline model. In the early stage of training, the fine-tuned model based on the target object generative pre-trained model shows a higher accuracy, reflecting its good initialization ability and task transfer performance. As the number of training rounds increases, the advantages of the fine-tuned model based on the target object generative pre-trained model continue to be maintained, and ultimately after multiple training cycles, it can still maintain stable and high performance. This result shows that the target object generative pre-trained model provided in the embodiment of the present application not only has good versatility, but also can show better performance in the process of specific task transfer, providing a basis for the efficient deployment of complex commodity response tasks in the OFD field.

[0209] In order to further verify the generalization ability and training efficiency of the target object generative pre-trained model as the domain basic model in the fine-tuning stage, ablation experiments can be carried out on two tasks: commodity category prediction and commodity standardized product sub-unit prediction. The performance under different training rounds and training data ratios is evaluated. The experimental results are as follows: Figure 9 As shown. Figure 9, the horizontal axis is the training round, and the vertical axis is the accuracy. Line a represents the change in the accuracy of the target task model based on the initial baseline model in the commodity category prediction task from the 1st to the 5th training round; line b represents the change in the accuracy of the target task model based on the target object generative pre-training model in the commodity category prediction task from the 1st to the 5th training round. It can be seen that in the commodity category prediction task, compared with the target task model based on the initial baseline model, which always maintains a low level of accuracy, the target task model based on the target object generative pre-training model provided by the embodiment of the present application shows better performance in the initial training stage, and gradually improves in subsequent rounds, showing a faster convergence trend. This shows that the target task model based on the target object generative pre-training model has a stronger initial generalization ability and a faster convergence speed in the commodity category prediction task, and can achieve better performance in the early stages. Line c shows the change in accuracy of the target task model based on the initial baseline model from the first to the fifth training round in the commodity standardized product sub-unit prediction task; line d shows the change in accuracy of the target task model based on the target object generative pre-trained model from the first to the fifth training round in the commodity standardized product sub-unit prediction task. As can be seen, in the commodity standardized product sub-unit prediction task, the target task model based on the initial baseline model has stable accuracy but limited room for improvement, while the target task model based on the target object generative pre-trained model consistently achieves small improvements over multiple training rounds, ultimately reaching a higher level of accuracy. This demonstrates that the target task model based on the target object generative pre-trained model not only has strong expressive power at the beginning of training, but also further improves performance after multiple rounds of training, validating its superiority and higher convergence efficiency in the commodity standardized product sub-unit prediction task.

[0210] Furthermore, in order to evaluate the performance stability and generalization ability of the target task model based on the target object generative pre-training model under different training data volume conditions, we can select two tasks, commodity category prediction and commodity standardized product sub-unit prediction, to conduct accuracy comparison experiments based on changes in the training set ratio. The training set ratios are set to 20%, 40%, 60%, 80% and 100% respectively, and the accuracy performance of the model on the validation set is recorded at each ratio. At the same time, the target task model based on the initial baseline model is set for comparison. The experimental results are as follows. Figure 10 As shown, Figure 10The horizontal axis represents the training set proportion, and the vertical axis represents the accuracy. Line e represents the accuracy change of the target task model based on the initial baseline model in the product category prediction task, and line f represents the accuracy change of the target task model based on the initial baseline model in the product standardized product subunit prediction task. Line g represents the accuracy change of the target task model based on the target object generative pre-training model in the product category prediction task, and line h represents the accuracy change of the target task model based on the target object generative pre-training model in the product standardized product subunit prediction task. Comparing lines e and g shows that, in the product category prediction task, the target task model based on the target object generative pre-training model significantly outperforms the target task model based on the initial baseline model, even with only a small amount of training data. The overall trend shows good stability and adaptability as the amount of data increases. In other words, the target task model based on the target object generative pre-training model has strong low-resource learning capabilities and can still provide superior prediction performance when training data is limited. Comparing lines f and h shows that the target task model of the target object generative pre-training model also performs well in the task of predicting standardized product subunits. The accuracy remains high across different training set ratios and steadily increases with increasing data volume, further demonstrating the excellent expressive power and convergence efficiency of the target task model of the target object generative pre-training model.

[0211] It can be seen that the target task model based on the target object generative pre-training model shows strong adaptability to training data in both types of tasks. In particular, it can still stably surpass the target task model based on the initial baseline model under low resources (such as 40% training set), verifying that it has high generalization ability and data utilization efficiency in downstream tasks as a domain basic model.

[0212] Furthermore, an embodiment of the present application also provides a model interaction method, which at least includes: obtaining object request data; inputting the above-mentioned object request data into the target object generative pre-training model, analyzing and processing the above-mentioned object request data through the above-mentioned target object generative pre-training model to generate corresponding object response data; outputting the above-mentioned object response data; wherein the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by any model training method provided by this application.

[0213] The object request data may be input data received from the relevant platform system or user terminal for generating a response. Specifically, the object request data may be a task instruction in natural language, for example, "Please generate an attractive title for the product 'Honey Grapefruit Tea'." The object request data may also include structured product information, such as the product name, category, and attribute description.

[0214] Optionally, the object request data can be used to perform a specific task corresponding to any one of multiple object response task types. When the object is a commodity, the multiple object response task types include but are not limited to: category prediction type, title generation type, information extraction type, product matching type, search relevance type, and query annotation type.

[0215] In one embodiment, the target object generative pre-training model is generated by any one of the model training methods provided in this application, and can have the ability to understand and generate a variety of object task responses. The above-mentioned object request data is input into the target object generative pre-training model, so that the target object generative pre-training model automatically identifies the corresponding task type based on the learned semantic representation ability and task characteristics, and generates corresponding object response data based on the internally encoded task representation and the domain knowledge obtained through training. Among them, the object response data can be presented in the form of natural language text, or in a structured data format, such as category labels, category codes, matching scores, etc., to adapt to different scenario requirements. Furthermore, the above-mentioned generated object response data can be output to the relevant downstream scenario platform to realize the task execution of specific scenarios.

[0216] The model interaction method provided in the embodiment of the present application can call the target object generation pre-training model generated by any training method of the present application based on object request data, realize automatic recognition and response generation of various object response tasks, and effectively improve recognition and response efficiency.

[0217] It should be understood that, although the steps in the flowcharts of the above-mentioned embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0218] Based on the inventive concept of the above model training method, Figure 11 As shown, the embodiment of the present application further provides a first model training device 1100 for implementing the above-mentioned model training method. The first model training device 1100 includes:

[0219] A first acquisition module 1101 is configured to acquire an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes a plurality of training sample groups corresponding to a plurality of object response training tasks;

[0220] A fine-tuning module 1102 is configured to perform supervised fine-tuning on the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample;

[0221] A first training module 1103 is configured to optimize and train the initial reward model based on the preference data sample to obtain a target reward model, and optimize and train the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model;

[0222] a generating module 1104 for generating a first reward signal based on the preference optimization model and the first training sample set, and generating a second reward signal based on the preference optimization model, the target reward model, and a second training sample set in the preset training sample set;

[0223] The second training module 1105 is used to optimize and train the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a generative pre-training model of the target object.

[0224] In one embodiment, the fine-tuning module 1102 is specifically used to perform multiple rounds of parameter updates on the above-mentioned initial baseline model using a supervised learning method based on the first training sample set in the above-mentioned preset training sample set to obtain a supervised fine-tuning model; in the process of performing multiple rounds of parameter updates on the above-mentioned initial baseline model, the above-mentioned first training sample set is sampled to obtain input samples and multiple candidate responses corresponding to each of the above-mentioned input samples; based on preset indicators, the multiple candidate responses corresponding to each of the above-mentioned input samples are screened to obtain a first response and a second response corresponding to each of the above-mentioned input samples, so as to construct a pair of preference data samples, wherein the above-mentioned first response is better than the above-mentioned second response.

[0225] In one embodiment, the first training module 1103 is specifically used to input each of the above-mentioned input samples contained in the above-mentioned preference data sample, as well as the above-mentioned first response and the above-mentioned second response corresponding to each of the above-mentioned input samples into the above-mentioned initial reward model; perform scoring value generation processing on the above-mentioned first response and the above-mentioned second response corresponding to each of the above-mentioned input samples through the above-mentioned initial reward model to obtain a first response scoring value corresponding to the above-mentioned first response, and a second response scoring value corresponding to the above-mentioned second response; construct a scoring loss function according to the above-mentioned first response scoring value and the above-mentioned second response scoring value; based on the above-mentioned scoring loss function, iteratively optimize the parameters of the above-mentioned initial reward model to obtain a target reward model.

[0226] In one embodiment, the first training module 1103 is specifically used to input each of the above-mentioned preference data samples into the above-mentioned supervised fine-tuning model; calculate the generation probability information of the above-mentioned supervised fine-tuning model for the above-mentioned first response and the above-mentioned second response in each of the above-mentioned preference data samples; construct a contrast loss function based on the above-mentioned generation probability information; and iteratively update the parameters of the above-mentioned supervised fine-tuning model based on the above-mentioned contrast loss function to obtain a preference optimization model.

[0227] In one embodiment, the generation module 1104 is specifically used to determine, for each first training sample data in the first training sample set, a first predicted response corresponding to the first training sample data through the preference optimization model; compare the first predicted response with the reference output of the first training sample data, and calculate a difference measurement index between the first predicted response and the reference output; and generate a first reward signal according to the difference measurement index.

[0228] In one embodiment, the generation module 1104 is specifically used to determine the second predicted response corresponding to each second training sample data in the second training sample set in the above-mentioned preset training sample set through the above-mentioned preference optimization model; input the above-mentioned each second training sample data and the corresponding second predicted response into the above-mentioned target reward model, so as to perform reward signal generation processing through the above-mentioned target reward model to generate a second reward signal.

[0229] In one embodiment, the second training module 1105 is specifically used to perform reinforcement learning on the preference optimization model based on the first reward signal to obtain a reinforcement object generative pre-training model; and perform unsupervised training on the reinforcement object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model.

[0230] In one embodiment, the second training module 1105 is specifically used to generate object response information corresponding to the second training sample data in the second training sample set using the reinforcement object generative pre-training model; input the second training sample data and the corresponding object response information into the target reward model to obtain a reward score for reinforcement training; construct a reinforcement learning objective function with the reward score as the return signal; and optimize the parameters of the reinforcement object generative pre-training model based on the reinforcement learning objective function to obtain a trained target object generative pre-training model.

[0231] Each module in the first model training device 1100 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0232] Based on the inventive concept of the above model training method, Figure 12 As shown, the embodiment of the present application further provides a second model training device 1200 for implementing the above-mentioned model training method. The second model training device 1200 includes:

[0233] The second acquisition module 1201 is used to obtain a target object generative pre-training model and a task training dataset corresponding to a preset subtask;

[0234] The third training module 1202 is used to use the above-mentioned target object generative pre-training model as the initialization basic model, and adopt the above-mentioned task training data set for fine-tuning training to obtain the target task model for the preset subtask; wherein, the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by any model training method provided in this application.

[0235] Each module in the second model training device 1200 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0236] Based on the inventive concept of the above-mentioned model interaction method, Figure 14 As shown, the embodiment of the present application further provides a model interaction device 1300 for implementing the above-mentioned model interaction method. The model interaction device 1300 includes:

[0237] The third acquisition module 1301 is used to obtain object request data;

[0238] An input module 1302 is configured to input the object request data into a target object generative pre-trained model, analyze and process the object request data using the target object generative pre-trained model, and generate corresponding object response data;

[0239] The output module 1303 is used to output the above-mentioned object response data; wherein the above-mentioned target object generative pre-training model is a target object generative pre-training model generated by any model training method provided in this application.

[0240] Each module in the model interaction device 1300 may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0241] The embodiment of the present application also provides an electronic device, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 14 As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the electronic device is used to store active configuration information. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal through a network connection. The processor of the electronic device executes a computer program to implement a model training method or a model interaction method.

[0242] Those skilled in the art will understand that Figure 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0243] The present application also provides a computer storage medium having instructions stored therein that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of the above-described embodiments. If the components of the electronic device described above are implemented as software functional units and sold or used as independent products, they may be stored in the computer-readable storage medium described above.

[0244] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described above in accordance with the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer storage medium or transmitted via the computer storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer storage medium may be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The above-mentioned available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, digital versatile discs (DVDs)), or semiconductor media (for example, solid state drives (SSDs)).

[0245] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. The technical features of this embodiment and the implementation scheme can be combined in any manner unless they conflict.

[0246] The embodiments described above are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the scope of protection determined by the claims.

[0247] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A model training method, characterized in that: The method comprises: Obtain an initial baseline model, an initial reward model, and a preset training sample set, wherein the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks, the object is an e-commerce object, and the object response training task is a training task that uses the relevant information of the object as an input target to generate response information for the object. The multiple object response training tasks include at least one or more of the following types of tasks: category prediction type, title generation type, information extraction type, product matching type, search relevance type, and query annotation type. The category prediction type of object response training task is used to predict the standardized product unit category label to which the product belongs based on the basic attribute information of the product. The information extraction type of object response training task is used to extract structured attribute information from the product title or product attributes. The search relevance type of object response training task is used to evaluate the correlation between the user query and the product title in the context information corresponding to the product. The query annotation type of object response training task is used to assign part-of-speech tags to user search queries. Performing supervised fine-tuning on the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample; Optimizing and training the initial reward model based on the preference data sample to obtain a target reward model, and optimizing and training the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model; A first reward signal is generated based on the preference optimization model and the first training sample set, and a second reward signal is generated based on the preference optimization model, the target reward model, and a second training sample set in the preset training sample set. The first reward signal is a score-type reward signal calculated using a difference metric by generating a first predicted response based on the preference optimization model on the first training sample set and comparing the first predicted response with a reference output. The second reward signal is a numerical reward representing the quality of the generation by generating a second predicted response based on the preference optimization model on the second training sample set and inputting the input sample and the second predicted response into the target reward model for scoring. Optimizing and training the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model; The optimizing and training the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a target object generative pre-training model includes: Performing reinforcement learning on the preference optimization model based on the first reward signal to obtain a reinforcement object generative pre-training model; Unsupervised training is performed on the reinforcement object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model.

2. The method according to claim 1, wherein The supervised fine-tuning of the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample includes: Based on the first training sample set in the preset training sample set, performing multiple rounds of parameter updates on the initial baseline model using a supervised learning method to obtain a supervised fine-tuning model; In a process of performing multiple rounds of parameter updates on the initial baseline model, sampling the first training sample set to obtain input samples and multiple candidate responses corresponding to each of the input samples; Based on preset indicators, multiple candidate responses corresponding to each input sample are screened to obtain a first response and a second response corresponding to each input sample to construct a paired preference data sample, wherein the first response is better than the second response.

3. The method according to claim 2, wherein The optimizing and training the initial reward model based on the preference data sample to obtain a target reward model includes: inputting each input sample included in the preference data sample, and the first response and the second response corresponding to each input sample, into the initial reward model; Performing score generation processing on the first response and the second response corresponding to each of the input samples using the initial reward model to obtain a first response score corresponding to the first response and a second response score corresponding to the second response; Constructing a scoring loss function according to the first response scoring value and the second response scoring value; Based on the scoring loss function, the parameters of the initial reward model are iteratively optimized to obtain a target reward model.

4. The method according to claim 2, wherein The optimizing and training the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model includes: Inputting each of the preference data samples into the supervised fine-tuning model; Calculating generation probability information of the first response and the second response in each of the preference data samples by the supervised fine-tuning model; Constructing a contrast loss function based on the generation probability information; The parameters of the supervised fine-tuning model are iteratively updated based on the contrast loss function to obtain a preference optimization model.

5. The method according to claim 1, wherein Generating a first reward signal based on the preference optimization model and the first training sample set includes: For each first training sample data in the first training sample set, determining a first predicted response corresponding to the first training sample data by using the preference optimization model; Comparing the first predicted response with a reference output of the first training sample data, and calculating a difference metric between the first predicted response and the reference output; A first reward signal is generated according to the difference metric.

6. The method according to claim 1, wherein Generating a second reward signal based on the preference optimization model, the target reward model, and the second training sample set in the preset training sample set includes: For each second training sample data in the second training sample set in the preset training sample set, determining a second predicted response corresponding to the second training sample data by using the preference optimization model; The second training sample data and the corresponding second predicted response are input into the target reward model so as to perform reward signal generation processing through the target reward model to generate a second reward signal.

7. The method according to claim 1, wherein The performing unsupervised training on the reinforcement object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model includes: For each second training sample data in the second training sample set, generating object response information corresponding to the second training sample data using the enhanced object generative pre-training model; Inputting the second training sample data and the corresponding subject response information into a target reward model to obtain a reward score for reinforcement training; Construct a reinforcement learning objective function with reward score as the return signal; Optimizing the parameters of the reinforcement object generative pre-training model based on the reinforcement learning objective function to obtain a trained target object generative pre-training model.

8. A model training method, characterized in that: The method comprises: Obtain the target object generative pre-trained model and the task training dataset corresponding to the preset subtask; The target object generative pre-trained model is used as an initialized basic model, and fine-tuned using the task training dataset to obtain a target task model for the preset subtask; The target object generative pre-training model is a target object generative pre-training model generated by the model training method according to any one of claims 1 to 7.

9. A model interaction method, characterized in that: The method comprises: Get object request data; Inputting the object request data into a target object generative pre-trained model, analyzing and processing the object request data through the target object generative pre-trained model to generate corresponding object response data; Outputting the object response data; The target object generative pre-training model is a target object generative pre-training model generated by the model training method according to any one of claims 1 to 7.

10. A model training device, characterized in that: The device comprises: A first acquisition module is used to acquire an initial baseline model, an initial reward model and a preset training sample set, wherein the preset training sample set includes multiple training sample groups corresponding to multiple object response training tasks, the object is an e-commerce object, and the object response training task is a training task that uses the relevant information of the object as an input target to generate response information for the object. The multiple object response training tasks include at least one or more types of tasks: category prediction type, title generation type, information extraction type, product matching type, search relevance type and query annotation type. The category prediction type object response training task is used to predict the standardized product unit category label to which the product belongs based on the basic attribute information of the product. The information extraction type object response training task is used to extract structured attribute information from the product title or product attributes. The search relevance type object response training task is used to evaluate the correlation between the user query and the product title in the context information corresponding to the product. The query annotation type object response training task is used to assign part-of-speech tags to user search queries. A fine-tuning module, configured to perform supervised fine-tuning on the initial baseline model based on the first training sample set in the preset training sample set to obtain a supervised fine-tuning model and a preference data sample; A first training module is configured to optimize and train the initial reward model based on the preference data sample to obtain a target reward model, and optimize and train the supervised fine-tuning model based on the preference data sample to obtain a preference optimization model; A generation module, configured to generate a first reward signal based on the preference optimization model and the first training sample set, and to generate a second reward signal based on the preference optimization model, the target reward model, and a second training sample set in the preset training sample set, wherein the first reward signal is a score-type reward signal calculated by using a difference metric by generating a first predicted response based on the preference optimization model on the first training sample set and comparing the first predicted response with a reference output; the second reward signal is a score-type reward signal calculated by generating a second predicted response based on the preference optimization model on the second training sample set and inputting the input sample and the second predicted response into the target reward model for scoring to obtain a numerical reward representing the generation quality; A second training module is configured to optimize and train the preference optimization model based on the first reward signal, the second reward signal, and the second training sample set to obtain a generative pre-trained model of the target object; The second training module is specifically used to: perform reinforcement learning on the preference optimization model based on the first reward signal to obtain a reinforcement object generative pre-training model; and perform unsupervised training on the reinforcement object generative pre-training model based on the second reward signal and the second training sample set to obtain a target object generative pre-training model.

11. A model training device, characterized in that: The device comprises: The second acquisition module is used to obtain the target object generative pre-training model and the task training data set corresponding to the preset subtask; A third training module is configured to use the target object generative pre-training model as an initialized basic model and perform fine-tuning training using the task training dataset to obtain a target task model for the preset subtask; The target object generative pre-training model is a model generated by the model training method according to any one of claims 1 to 7.

12. An electronic device, characterized in that: include: A processor and a memory; the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

13. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 9.

Images