A model training method, a training system and related equipment

Abstract

This application provides a model training method, training system, and related equipment. The method includes the following steps: acquiring positive sample data, wherein the positive sample data includes input samples and positive labels, the input samples include questions, and the positive labels include the answers corresponding to the questions; inputting the input samples into a first large model to be fine-tuned, obtaining the first negative label corresponding to the input sample; determining the first score of the first negative label based on the relevance of the first negative label to the question and the quality of the response to the first negative label based on a scoring model; inputting the input samples into the first large model; and fine-tuning the first large model based on a loss function to obtain a second large model. The loss function is used to guide the first large model to improve the prediction probability of positive labels and the prediction probability of negative labels with scores greater than a threshold, so that the model can not only learn standard answers that meet user expectations, but also generate more diverse answers, thereby improving the fine-tuning effect of the large model.

Description

[0001] This application claims priority to Chinese Patent Application No. 202411998539.0, filed with the China National Intellectual Property Administration on December 31, 2024, entitled "A Method and Related Apparatus for Fine-tuning a Large Model", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and in particular to a model training method, training system and related equipment. Background Technology

[0003] Model fine-tuning refers to further training a pre-trained large-scale language model (LLM) using domain-specific data or task requirements. This allows the large model to perform more accurately on specific tasks. Simply put, model fine-tuning involves adjusting the parameters of an already trained large model for specific tasks in a professional domain, enabling it to achieve better performance in that domain as well.

[0004] Typically, when fine-tuning a model, negative samples are used. These negative samples are usually randomly generated answers or randomly selected answers that may be relevant or irrelevant. For example, if the question is "How old is celebrity A this year?", negative samples could include answers like "apple," "rainy weather," or "1977." During fine-tuning, the predicted probability of these negative samples is set to 0. The model adjusts its parameters according to the loss function, making the probability of predicting such negative samples approach 0. This helps the larger model better understand what incorrect answers are and learn how to avoid generating irrelevant or wrong answers.

[0005] However, large models are often used to handle open-domain generation tasks, such as writing and dialogue, which require preserving the diversity of output answers, such as different expression styles and reasonable but not optimal candidate answers. Although some negative sample answers are randomly selected, they may contain some relevant content. If the probability of negative samples is directly set to 0, the model will tend to generate a single safe answer, lose creativity, make the fine-tuning effect of large models poor, make it difficult to improve the output quality of large models, and may even damage the original large model. Summary of the Invention

[0006] This application provides a model training method, training system, and related equipment to address the problem of poor fine-tuning performance of large models.

[0007] Firstly, a model training method is provided, which is applied to a training system. The method includes the following steps: acquiring positive sample data, wherein the positive sample data includes input samples and positive labels, the input samples include questions, and the positive labels include the answers corresponding to the questions; inputting the input samples into a first large model to be fine-tuned, obtaining the first negative label corresponding to the input sample; determining the first score of the first negative label based on the relevance of the first negative label to the question and the quality of the response to the first negative label based on a scoring model; inputting the input samples into the first large model; and fine-tuning the first large model based on a loss function to obtain a second large model, wherein the loss function is used to guide the first large model to improve the prediction probability of positive labels and the prediction probability of negative labels whose scores are greater than a threshold.

[0008] By implementing the method described in the first aspect, the training system can first obtain the positive labels of the input data, then input the input data into the large model to be fine-tuned to obtain the negative labels of the input data, and then use a scoring model to score the negative labels to obtain negative sample scores. Then, the large model is fine-tuned using positive labels, negative labels, and negative sample scores, guiding the model to generate standard content that is closer to the positive labels, reducing the generation of low-quality content, so that the model can not only learn standard answers that meet user expectations, but also understand and quantify a variety of different answers that do not meet user expectations. This allows the large model to generate more diverse answers, improves the fine-tuning effect of the large model, and achieves better performance in professional domains.

[0009] In one possible implementation, the value of the loss function is negatively correlated with the probability of a positive sample predicted by the first largest model with a positive label, and positively correlated with the probability of a negative sample predicted by the first largest model with a negative label and the first gap between the negative sample score.

[0010] In the above implementation, the loss function value is negatively correlated with the positive sample prediction probability. The lower the positive sample prediction probability, the larger the loss function value, and the greater the adjustment range of the large model parameters. This makes the model more inclined to generate positive labels, i.e., the standard answer to the question. The loss function value is positively correlated with the first difference between the negative sample prediction probability and the negative sample score. The larger the first difference, the higher the probability of the model predicting low-scoring negative samples or the lower the probability of predicting high-scoring negative samples. In this case, the loss function value will increase, and the adjustment range of the large model parameters will be greater. This makes the model more inclined to generate high-scoring negative labels and avoid generating low-scoring negative labels. Thus, the model can not only learn the standard answer that meets user expectations but also understand and quantify various different answers that do not meet user expectations. This allows the large model to generate more diverse responses based on the standard answer, improving the fine-tuning effect of the large model.

[0011] In one possible implementation, the loss function includes a first loss function and a second loss function. The first loss function is determined based on the negative sample scores and negative sample labels. The first loss function is used to instruct the first large model to learn the features of the negative labels for different negative sample scores. The second loss function is determined based on the positive labels. The second loss function is used to guide the first large model to learn the features of the positive labels.

[0012] The above implementation guides the model to learn the features of the standard answer through a first loss function and to learn the features of negative samples of varying quality through a second loss function. This enables the model to generate both standard and diverse answers, resulting in more accurate and varied responses in specialized domains and significantly improving the model's fine-tuning performance. Furthermore, the two loss functions guide the model in different directions, and their weights can be dynamically adjusted to guide the training process. For example, initially, the second loss function is increased to guide the model towards generating the standard answer; later, once the model has a foundation for generating standard answers, the weight of the first loss function is increased to guide the model to generate diverse answers. This gradual approach makes model fine-tuning more stable and further improves the performance of large-scale models.

[0013] In one possible implementation, the first loss function includes a negative sample advantage value, which is determined based on a first gap. The negative sample advantage value is used to amplify the impact of the first gap on the first loss function. The first gap is positively correlated with the negative sample advantage value, and the negative sample advantage value is positively correlated with the value of the first loss function.

[0014] The above implementation amplifies the impact of the first gap in the loss function by using the negative sample advantage value, making the loss function more sensitive to changes in the first gap. The larger the first gap, the higher the prediction probability of the large model for low-scoring negative samples and the lower the prediction probability for high-scoring negative samples. After the advantage value amplifies this first gap, the model can strengthen the identification of low-quality negative samples, enhance the learning of features that distinguish these cases, avoid the above-mentioned errors, and improve the fine-tuning effect of the large model.

[0015] In one possible implementation, the first loss function includes a positive sample advantage value, which is determined based on the positive sample prediction probability. The positive sample advantage value is used to amplify the impact of the positive sample prediction probability on the first loss function. The positive sample prediction probability is positively correlated with the positive sample advantage value, and the positive sample advantage value is negatively correlated with the value of the first loss function.

[0016] The above implementation guides the model to generate standard answers through positive sample dominance values. This avoids the model from learning the merits and demerits of diverse answers through negative sample dominance values, which could lead to overly divergent answers. Based on positive sample dominance values, the model can be guided to stay on track with the standard answer, ensuring that the final generated answers are not only diverse but also correct.

[0017] In one possible implementation, the second loss function includes the cross-entropy loss between the predictions obtained by the first large model based on the input samples and the positive labels.

[0018] In the above implementation, the second loss function is based on the cross-entropy loss between the predicted result and the positive label. This loss function is determined entirely based on the positive label. The first loss function is determined based on the positive label, negative label, and negative label scores. The second loss function guides the model to generate a standard answer and, based on that, to generate more diverse answers. During model training, the weight of the two loss functions can be dynamically adjusted based on the learning progress of the large model. For example, in the early stages, the weight of the second loss function can be increased to encourage the model to generate standard answers. In the later stages, after the model has learned how to generate standard answers, the weight of the first loss function can be increased to encourage the model to generate more diverse answers. This gradual learning approach can improve the model's learning efficiency and stability.

[0019] In one possible implementation, the method further includes the following steps: inputting the input sample into the second large model to obtain the second negative label of the input sample; scoring the second negative label based on the scoring model to obtain the second negative sample score; and fine-tuning the second large model using the input sample, positive label, second negative label, and second negative sample score to obtain the third large model.

[0020] The above implementation allows the large model to update its parameters in stages. The negative samples used in each stage are the results predicted using the latest model parameters, which can effectively reflect the current advantages and disadvantages of the large model. This enables the large model to be fine-tuned in each stage, further improving the fine-tuning effect and efficiency of the large model.

[0021] In one possible implementation, the formula for the loss function includes:

[0022] L=E(x,y w ,y l ,s l )[L SFT +λL OSR ]

[0023]

[0024]

[0025]

[0026] Where x is the input sample, y w It's a positive tag, y l It is the first negative label, s l It's the first rating, L SFT It is the second loss function, L OSR For the first loss function, p w The first major model predicts the probability of a positive label based on the input sample, p. l It is the first major model that predicts the probability of a negative label based on the input sample, where α and δ are parameter values.

[0027] It should be understood that the above formula is an example provided in this application. In specific implementations, the above formula can be modified according to the actual training objectives. For example, the absolute value operation on the difference in the formula for negative sample advantage can be replaced by a smoothing function, such as the difference of squares formula. Multi-dimensional scoring can also be introduced, such as expanding the multi-dimensional scoring into a vector form to replace s in the formula. l Furthermore, the formula uses the SIGMOD function for normalization, and other normalization functions can also be used, such as the Tanh function. The above examples provide some possible variations. In some implementation scenarios, other variations may also be used, but this application does not impose any specific limitations.

[0028] It should be understood that traditional negative sample training processes directly set the probability of negative samples to 0. On the one hand, this suppresses the contribution of negative samples to model learning to zero, causing some potentially contributing negative samples to be discarded, resulting in wasted resources and reduced model training efficiency. On the other hand, it makes the model tend to repeat safe phrases, leading to a decline in the quality of model output, an inability to generate diverse answers, or even the output of a single meaningless answer. In contrast, the technical solution of this application introduces negative sample scoring into the loss function, enabling the model to not only learn answers that meet user expectations but also quantify various different answers that do not meet user expectations. This allows large models to handle problems more accurately, provide more diverse answers, and make more reasonable judgments in complex scenarios.

[0029] In this embodiment, the training system can use sample data (including positive sample data, negative sample data, and negative sample scores) and the aforementioned loss function L to fine-tune the first large model to obtain a second large model. Specifically, input sample x into the model to obtain the positive sample probability p. w and negative sample probability p l Then, based on the above formula, calculate the first loss function L respectively. OSR Second loss function L SFTThe loss function value L is obtained by weighted summation. The gradient of the loss function value L with respect to the model parameters θ is calculated. The optimizer is then used to update the parameters θ along the reverse direction of the gradient to reduce the loss function value L, resulting in the updated model parameters θ'. The first loss function L... OSR The guided model distinguishes features of different quality answers, and the second loss function L SFT Ensure the model retains its basic generative capabilities. For each sample, the model parameters θ can be updated following the process described above to improve model performance. Fine-tuning continues until performance no longer improves, at which point the fine-tuning is terminated, resulting in the second-largest model.

[0030] Secondly, this application provides a training system, which includes: a positive sample acquisition unit for acquiring positive sample data, wherein the positive sample data includes input samples and positive labels, the input samples include questions, and the positive labels include answers corresponding to the questions; a negative sample acquisition unit for inputting input samples into a first large model to be fine-tuned, and obtaining a first negative label corresponding to the input sample; a scoring unit for determining a first score for the first negative label based on the relevance of the first negative label to the question and the quality of the response to the first negative label based on a scoring model; and a fine-tuning unit for inputting input samples into the first large model and fine-tuning the first large model based on a loss function to obtain a second large model, wherein the loss function is used to guide the first large model to improve the prediction probability of positive labels and the prediction probability of negative labels whose scores are greater than a threshold.

[0031] In one possible implementation, the value of the loss function is negatively correlated with the probability of a positive sample predicted by the first largest model with a positive label, and positively correlated with the probability of a negative sample predicted by the first largest model with a negative label and the first gap between the negative sample score.

[0032] In one possible implementation, the first loss function includes a negative sample advantage value, which is determined based on a first gap. The negative sample advantage value is used to amplify the impact of the first gap on the first loss function. The first gap is positively correlated with the negative sample advantage value, and the negative sample advantage value is positively correlated with the value of the first loss function.

[0033] In one possible implementation, the first loss function includes a positive sample advantage value, which is determined based on the positive sample prediction probability. The positive sample advantage value is used to amplify the impact of the positive sample prediction probability on the first loss function. The positive sample prediction probability is positively correlated with the positive sample advantage value, and the positive sample advantage value is negatively correlated with the value of the first loss function.

[0034] In one possible implementation, the second loss function includes the cross-entropy loss between the predictions obtained by the first large model based on the input samples and the positive labels.

[0035] In one possible implementation, a negative sample acquisition unit is used to input the input sample into the second large model to obtain the second negative label of the input sample; a scoring unit is used to score the second negative label based on the scoring model to obtain the second negative sample score; and a fine-tuning unit is used to fine-tune the second large model using the input sample, positive label, second negative label, and second negative sample score to obtain the third large model.

[0036] In one possible implementation, the formula for the loss function includes:

[0037] L=E(x,y w ,y l ,s l )[L SFT +λL OSR ]

[0038]

[0039]

[0040]

[0041] Where x is the input sample, y w It's a positive tag, y l It is the first negative label, s l It's the first rating, L SFT It is the second loss function, L OSR For the first loss function, p w The first major model predicts the probability of a positive label based on the input sample, p. l It is the first major model that predicts the probability of a negative label based on the input sample, where α and δ are parameter values.

[0042] Thirdly, this application provides a computing device, which includes a processor, a memory, a communication interface, and a bus. The processor, the memory, and the communication interface are connected via the bus and communicate with each other. The memory stores computer execution instructions. When the processor is running, the processor executes the computer execution instructions in the memory to utilize the hardware resources in the processor to perform the operation steps of the method in the first aspect or any possible implementation of the first aspect.

[0043] Fourthly, this application provides a computing device cluster, which includes multiple computing devices. Each computing device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected to each other via the bus and communicate with each other. The memory is used to store computer execution instructions. When the processor is running, the processor executes the computer execution instructions in the memory to use the hardware resources in the processor to perform the operation steps of the method in the first aspect or any possible implementation of the first aspect.

[0044] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0045] Sixthly, this application provides a computer program product containing instructions that, when run on a computer, causes the computer to perform the methods described above.

[0046] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0047] Figure 1 This is an architecture diagram of a training system provided in this application;

[0048] Figure 2 This is an example diagram of a training system deployed in a cloud environment according to this application;

[0049] Figure 3 This is a flowchart illustrating the steps of a model training method provided in this application;

[0050] Figure 4 This is a schematic diagram of the structure of a loss function provided in this application;

[0051] Figure 5 This is an example diagram of the training process of a model training system provided in this application, showing multiple stages.

[0052] Figure 6 This is a schematic diagram of the software structure of a training system provided in this application;

[0053] Figure 7 This is a schematic diagram of the structure of a computing device provided in this application;

[0054] Figure 8 This is an example diagram of a computing device cluster provided in this application;

[0055] Figure 9 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0056] For ease of description, the terminology used in this application will be introduced first.

[0057] Model fine-tuning refers to further training a pre-trained model using data from a specific domain or task requirements, making the model more accurate in a particular task. Simply put, model fine-tuning is to fine-tune a pre-trained large model for a specific task in a specific domain, adjusting its parameters so that it can have better performance in that domain. Specific domains may include non-general sub-fields such as medicine, law, finance, and academia, and this application does not make any specific limitations.

[0058] Positive samples refer to samples in a dataset that contain the target features or meet the user's expectations. For example, in a binary classification model, correctly classified samples are positive samples. In a recommendation system, products liked by the user are positive samples. In a question-answering system, the correct answer to a user's question is a positive sample.

[0059] Negative samples are those in a dataset that do not contain the target features or do not meet the user's expectations. For example, in a binary classification model, a misclassified sample is a positive sample. In a recommendation system, other products that are not among the user's preferred products are negative samples. In a question-answering system, incorrect answers to a user's question are negative samples.

[0060] The technical problem solved by this application is described below.

[0061] Typically, when fine-tuning a model, negative samples are used. These negative samples are usually randomly generated answers or randomly selected answers to other questions. For example, if the question is "How old is celebrity A this year?", the negative sample answers could be "apple", "rainy weather", "1977", etc. Since most of these randomly selected answers are unrelated to the question, the prediction probability corresponding to these negative samples is set to 0 during fine-tuning. The model adjusts its parameters according to the loss function so that the probability of predicting such negative sample answers approaches 0. This helps the large model better understand what incorrect answers are and learn how to avoid generating irrelevant and wrong answers.

[0062] In fields other than natural language processing, such as image classification, setting the probability of negative samples to 0 is feasible and necessary because the goal in these fields is to provide users with a unique optimal solution, and the demand for diversity is low. However, in the field of natural language processing, large models are often used for open-domain generation (such as writing and dialogue), which requires preserving the diversity of output answers, such as different expression styles and reasonable but not optimal candidate answers. Although the answers of negative samples are randomly drawn or generated, they may contain some related content. If the probability of negative samples is directly set to 0, the model will tend to generate a single safe answer and lose creativity.

[0063] For example, in the previous example, the question was: "How old is celebrity A this year?" Answers like "apple" and "rainy weather" are completely irrelevant, but "1977" might be the celebrity's birth year, which is partially relevant to the question. Directly suppressing these negative samples to 0 might cause the model to miss some important fine-tuning opportunities, leading to a performance degradation.

[0064] At the same time, setting the prediction probability of negative samples to 0 may also have a negative impact on the original prediction ability of the model. This is because the large model has already learned certain knowledge based on the pre-trained model. This knowledge is stored in the form of complex relationships between parameters. If negative samples are forcibly suppressed directly during the fine-tuning stage, the parameters related to negative samples may be over-adjusted, destroying the language patterns and generation rules that the model has already learned, and causing the core capabilities of the model to degrade.

[0065] For example, if the incorrect answer "Mount Everest is located in India" is suppressed, the model may also forget the correct fact that "Mount Everest is located on the border between China and Nepal." This is because the parameters related to geographical entities are globally perturbed, causing the model's performance to decrease after fine-tuning.

[0066] Furthermore, forcibly suppressing the probability of negative samples to 0 may cause the model to only output "safe answers" that frequently appear in the training set, becoming overly "mechanical" and "limited," and losing the ability to generate novel expressions. For example, a dialogue system's response to "recommend a movie" will always be movie A, unable to generate other classic movies such as movie B. Similarly, a medical question-answering system's response to "Should drug B be used for disease A?" is correct but lacks information, such as "Please consult a professional doctor," leading to a degradation in model performance after fine-tuning.

[0067] In summary, although some negative sample answers are randomly selected, they may contain some relevant content. Setting the probability of negative samples directly to 0 will cause the model to tend to generate a single safe answer, losing creativity, resulting in poor fine-tuning of the large model, making it difficult to improve the output quality of the large model, and may even damage the original large model.

[0068] To address the issue of poor fine-tuning performance of large models, this application provides a model training method. This method guides the model to learn the correct standard answer through positive samples and to learn incorrect answers of different qualities through negative samples and negative sample scores. This enables the model to not only understand how to generate standard answers but also to understand incorrect answers of different qualities, resulting in more accurate and diverse model responses and thus improving the fine-tuning performance of large models.

[0069] Specifically, the training system can first obtain the positive labels of the input data, then input the input data into the large model to be fine-tuned to obtain the negative labels of the input data. A scoring model is then used to score the negative labels to obtain negative sample scores. The large model is then fine-tuned using the positive labels, negative labels, and negative sample scores. The value of the loss function used during fine-tuning is negatively correlated with the probability of the large model predicting positive samples with positive labels, thus guiding the model to generate standard content closer to the positive labels. Conversely, the value of the loss function is positively correlated with the difference between the probability of the large model predicting negative samples with negative labels and the negative sample scores, thus guiding the model to reduce the generation of low-quality content. This allows the model to not only learn standard answers that meet user expectations but also understand and quantify various different answers that do not meet user expectations, enabling the large model to generate more diverse responses and improving the fine-tuning effect of the large model, resulting in better performance in specialized fields.

[0070] The technical solution provided in this application will be explained below with reference to the accompanying drawings.

[0071] Figure 1 This is an architecture diagram of a training system provided in this application, such as... Figure 1 As shown, the architecture may include a client 100 and a training system 200, wherein the number of clients 100 that establish connections with the training system 200 may be one or more. Figure 1This application uses one example for illustration, and does not impose specific limitations. A communication connection is established between the client 100 and the training system 200. The communication connection can be a wired connection or a wireless connection. Wired connections include Ethernet, wired lines, cables, etc., while wireless connections include wireless local area networks (Wi-Fi), cellular networks, etc. It can also include the Internet, local area networks (LANs), etc., which simultaneously support wired and wireless connections. Of course, other connection types can also be included, and this application does not impose specific limitations.

[0072] Client 100 is deployed on terminal devices, computing devices, or edge computing devices. Terminal devices include personal computers, smartphones, wearable devices, handheld processors, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, smart conferencing devices, etc., without specific limitations. Computing devices can be bare metal servers (BMS), virtual machines, or containers. BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has its own independent basic input / output system (BIOS), hard drive, and operating system, allowing it to be operated like a physical machine. A container is a portable software unit that combines an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, eliminating the need to build complex environments and simplifying the application development and deployment process. Edge computing devices are hardware devices located close to the data source or data user.

[0073] The training system 200 is deployed on computing devices, computing device clusters, or chip systems. The description of computing devices is provided above and will not be repeated here. A computing device cluster refers to a cluster of devices composed of multiple computing devices, such as data centers or cloud computing platforms; this application does not specifically limit this. A chip system refers to a system composed of multiple chips. Specifically, a chip can be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The aforementioned PLD can be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), a system-on-chip (SoC), or any combination thereof; this application does not specifically limit this. It may also include processors or accelerator cards that have good performance when performing machine learning tasks and can improve the speed and efficiency of neural network training, such as graphics processing units (GPUs), neural network processing units (NPUs), data processing units (DPUs), tensor processing units (TPUs), etc., which are not specifically limited in this application.

[0074] Optionally, client 100 can be software or an application running on a user-controlled terminal device or computing device, such as a personal computer (PC) client, a web client accessed via a browser, an application (APP) client running on a mobile terminal, or a cloud platform console; this application does not impose specific limitations. The user holding client 100 can be the developer of the training model, such as a programmer or AI engineer; this application does not impose specific limitations.

[0075] Optionally, client 100 can be a client for a model service platform or a machine learning framework used to handle machine learning-related tasks. Such clients provide a series of tools and services for the entire process of defining, training, deploying, inferring, and managing machine learning models. The model service platform can be ModelArts, OpenAI, etc., and the machine learning framework can be TensorFlow, PyTorch, MindSpore, etc., without specific limitations in this application. The model training method provided in this application can be a function of the model service platform or machine learning framework used to optimize the fine-tuning effect of large models.

[0076] Optionally, client 100 can also be a client of an acceleration library. An acceleration library is a software library containing functions, algorithms, or interfaces optimized for specific AI tasks. Through the acceleration library, developers can improve the performance of algorithms on specific hardware platforms. The model training method provided in this application can be a function, algorithm, or interface from this acceleration library to improve the fine-tuning effect of large models, resulting in better performance of the fine-tuned large models, enabling them to play a role in specialized vertical fields.

[0077] Optionally, client 100 can also be a comprehensive client that includes natural language processing capabilities. Such clients typically use large models to perform various generation tasks, such as dialogue systems and text generation platforms. This application does not impose specific limitations on these applications. The model training method provided in this application can be used to optimize the fine-tuning effect of large language models on these platforms, resulting in better performance of the fine-tuned large language models.

[0078] Optionally, client 100 can also be a client of a cloud platform, allowing users to purchase and rent various cloud services. The model training method provided in this application can be one of these cloud services, which users can purchase separately to optimize the effect of fine-tuning large models and improve the quality and diversity of the output results after fine-tuning large models. Alternatively, the cloud platform provides users with a comprehensive service, and the model training method provided in this application can be a sub-service of this comprehensive cloud service. For example, if the comprehensive cloud service is an AI cloud service, the model training method provided in this application can be a sub-service of this AI cloud service, used to optimize the effect of fine-tuning large models and improve the quality and diversity of the output results after fine-tuning large models.

[0079] The preceding text has described in detail the possible deployment methods of client 100 and training system 200. In actual deployment, flexible deployment can be carried out according to specific application scenarios and business requirements. The following section provides examples of actual deployment methods for client 100 and training system 200 in specific application scenarios.

[0080] As an example of an application scenario where the training system 200 is deployed on a local device, assuming that the client 100 is an AI training platform client deployed by an enterprise, the enterprise employees can use the client to perform operations such as training and fine-tuning of large models. In this scenario, the client 100 can be deployed on the enterprise's office computer, and the training system 200 is deployed on a server cluster purchased by the enterprise. The nodes in the training system 200 are the servers in the server cluster. The enterprise employees are the users and can use their office computers to run the client 100 and use the technical solution provided in this application to fine-tune the large model, thereby improving the output quality and diversity of the large model.

[0081] As an example of an application scenario where a training system 200 is deployed on a cloud platform, Figure 2 This is an example diagram of a training system deployed in a cloud environment according to this application, such as... Figure 2 As shown, users can initiate a purchase request for AI cloud services through client 100. After client 100 sends the purchase request to the cloud platform, the cloud platform can provide cloud service access to training system 200 to client 100, allowing users to fine-tune large models through client 100 and improve the output quality and diversity of large models.

[0082] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, and security resources, to meet the computing needs of the training system 200 under different scales and loads. Furthermore, these computing resources can be dynamically scaled according to the usage needs of the training system 200 to ensure the stable operation of the training system 200 and provide users with reliable large model fine-tuning services.

[0083] It should be understood that the above application scenarios are for illustrative purposes only. The client 100 and training system 200 can be flexibly deployed according to actual business needs. They will not be listed here.

[0084] The possible system architecture of this application has been described above. The following is a brief explanation of how this system architecture implements the model training method provided in this application.

[0085] In this application, after receiving the model fine-tuning request sent by the client 100, the training system 200 can fine-tune the large model based on the large model information carried in the model fine-tuning request, using the model training method provided in this application. The large model information includes relevant information about the large model that needs to be fine-tuned, such as the storage address of the large model data, the hardware requirements for running the large model, etc., and may also include other relevant information required for running the large model, which is not specifically limited in this application.

[0086] Specifically, the training system 200 can first obtain the training data required for fine-tuning the model designed in this application, then input the training data into the large model, use the loss function designed in this application to fine-tune the parameters of the large model until the performance of the large model meets the expected target, obtain the fine-tuned large model, and then feed back the relevant data of the fine-tuned large model to the client 100.

[0087] Optionally, the training data includes positive sample data, negative sample data, and negative sample scores. Positive sample data indicates the standard answer to a question and includes the question and a first answer, which is the correct and standard answer to the question. Negative sample data indicates non-standard answers to a question and includes the question and a second answer, which is an incorrect or non-standard answer to the question that does not meet user expectations. Specifically, the first answer is a high-quality answer to the question, highly relevant to the question and meeting user expectations. The second answer is a low-quality answer to the question, partially relevant, poorly relevant, or irrelevant to the question and not meeting user expectations. Negative sample scores indicate the quality of the second answer in the negative sample data and can be determined based on the relevance of the second answer to the question and the quality of the answer.

[0088] For example, if the question is "How to repair a bicycle," the first answer from positive sample data would be: "First check if the wheels are damaged, then check if the brakes are damaged, and finally check if the chain is loose." This answer is highly relevant to the question and meets user expectations. The second answer from negative sample data could be: "Don't repair it, just buy a new bicycle." This is a relevant but low-quality answer because it doesn't provide useful information about the question and doesn't meet user expectations. Alternatively, the second answer could be: "The tools needed to repair a bicycle include pliers." This answer is relevant but low-quality because it doesn't fully answer the question; it only mentions the tools but not how to use them, thus failing to meet user expectations. These examples are for illustrative purposes only and are not intended to impose specific limitations in this application.

[0089] In practice, the first answer in the positive sample data can be obtained through manual annotation. Specifically, the question can be answered by experts or business personnel, and their answers can be used as the first answer in the positive sample data. Alternatively, the output of a powerful business model can be used as the first answer in the positive sample data. This application does not impose any specific limitations.

[0090] In practice, the second answer in the negative sample data is obtained after inputting the question into a large model, which is a large language model to be fine-tuned. It should be understood that using the answer output by the large model to be fine-tuned as a negative sample allows the negative sample to reflect the model's understanding and errors in the question to a certain extent. This enables targeted fine-tuning of the large model based on the quality of its output, thereby improving the performance and robustness of the large model.

[0091] In practice, negative sample scores can be obtained manually, specifically by having experts or business personnel score the negative sample data. Alternatively, existing scoring models can be used to score the negative samples. The scoring criteria can be determined according to the task type, and the scores can be based on multiple indicators, such as accuracy, completeness, correctness of reasoning, and rationality. This application does not impose specific limitations on these criteria.

[0092] It should be noted that the training data may also include positive sample scores, where negative sample scores are lower than positive sample scores. For example, positive sample scores can be set to a full score, such as 100 points or 1 point. The specific setting can be determined according to the actual business scenario, and this application does not impose any specific limitations.

[0093] Optionally, the loss function guides the first-level model to improve the prediction probability of positive labels and the prediction probability of high-scoring negative labels. Positive labels refer to the labels of positive sample data, which are the first answer to the question; negative labels refer to the labels of negative sample data, which are the second answer to the question; and high-scoring negative labels can refer to negative labels with scores greater than a threshold. It should be understood that some answers, while not the standard answer, may be close to it. For example, if the question is "Where is the capital of China?" and the answer is "Washington is not the capital of China," although this answer is correct and irrelevant to the question, it is not wrong. If the prediction probability of this answer is suppressed to 0, the model's knowledge of geographical location may be compromised, affecting the overall performance of the model. Therefore, scoring different negative labels allows the model to learn the difference between high-scoring and low-scoring answers, thus enabling it to generate more diverse answers to other questions based on this learned knowledge.

[0094] Optionally, the loss function may include a first loss function, which is used to guide the model to fine-tune the model's parameters based on positive sample data, negative sample data, and negative sample scores. The first loss function enables the model to not only distinguish between positive and negative samples, but also to distinguish between the quality of negative samples, allowing the model to explore high-quality but non-standardized answers, and the output answers are more in line with the complex preferences of humans.

[0095] Furthermore, the first loss function can be obtained based on the positive sample odds and negative sample odds. The positive sample odds indicates the probability that the model generates or predicts a positive label (first answer), while the negative sample odds indicates the probability that the model generates or predicts a negative label (second answer) with different negative sample scores. Positive labels guide the model to learn the features of the standard answer, while negative labels guide the model to learn the features of non-standard answers of varying quality. The positive sample odds are negatively correlated with the first loss function; a larger positive sample odds indicates a higher probability of the model predicting the first answer, and a smaller first loss function value requires less fine-tuning, thus guiding the model to generate high-quality content closer to the positive label. Conversely, the negative sample odds are positively correlated with the first loss function; a larger negative sample odds indicates a higher probability of the model predicting a low-scoring second answer, or a lower probability of predicting a high-scoring second answer, and a larger first loss function value requires more fine-tuning, thus guiding the model to reduce the generation of low-quality content.

[0096] Optionally, the loss function may include a second loss function, which guides the model to fine-tune based on positive sample data. In specific implementations, the second loss function may be a supervised fine-tuning (SFT) loss function, which optimizes model parameters by comparing the predicted results with positive labels.

[0097] In its implementation, the SFT loss function uses cross-entropy loss. The large model generates words one by one, producing a probability distribution for each word. The word with the highest probability is the next to be generated. Therefore, the positive label corresponds to a true probability distribution, and the large model's output corresponds to a predicted probability distribution. The goal of the SFT loss is to make the predicted probability distribution as close as possible to the true probability distribution, and the cross-entropy loss measures the difference between the two. Therefore, the second loss function not only guides the difference between the model's output and the first answer but also guides the difference in the generation process, assigning a high probability to the generation of the correct answer, rather than simply generating the correct answer.

[0098] Understandably, the second loss function ensures that the model generates standard content, while the first loss function ensures that the model generates more diverse content. The synergy between the two allows the model to balance accuracy and diversity, which can significantly improve its usability in some specialized fields.

[0099] It should be understood that traditional negative sample training processes directly set the probability of negative samples to 0, which makes the model tend to repeat safe phrases, resulting in a decline in the quality of model output, an inability to generate diverse answers, or even only outputting a single meaningless answer. However, the technical solution of this application introduces negative sample scoring into the loss function, which enables the model to not only learn answers that meet user expectations, but also to quantify a variety of different answers that do not meet user expectations. This allows large models to handle problems more accurately, provide more diverse answers, and make more reasonable judgments in some complex scenarios.

[0100] Optionally, large model fine-tuning may include multiple stages. In each stage, the problem can be input into the large model to obtain new negative sample data, and the new negative samples can be scored to obtain new negative sample scores. Then, the large model can be fine-tuned using the new negative sample data and the new negative sample scores until the large model reaches the expected performance.

[0101] For example, in the first stage, the sample data V1 used includes the input samples and their corresponding positive labels, negative labels V1, and negative sample scores V1. The input samples are then fed into the first-generation model (V1 version) for fine-tuning, resulting in the first-generation model (V2 version), completing the first-stage fine-tuning. The first-generation model V2 is then used to re-predict the negative labels and update the scores. Specifically, the input samples are fed into the first-generation model V2, the output is used as the new negative label V2, and the scoring model is used to score the negative label V2, obtaining the corresponding negative sample score V2. In the second stage, the sample data V2 used includes the input samples and their corresponding positive labels, negative labels V2, and negative sample scores V2. The input samples are then fed into the first-generation model (V2 version) for fine-tuning, resulting in the first-generation model (V3 version), completing the second-stage fine-tuning. This process can be repeated iteratively to continuously update the first-generation model. After each stage ends and before the next stage begins, the latest first-generation model can be used to update the negative labels and negative sample scores, allowing the model to specifically improve its weak points at each stage, thus steadily improving its performance. After m stages, a large model Vm that meets the training expectations can be obtained. Examples will not be given here.

[0102] It should be understood that by fine-tuning the large model through multiple stages, and by re-sampling negative samples and updating negative sample scores in each stage to form new training data, the large model's capabilities can be gradually improved. At each stage, targeted fine-tuning can be performed based on the quality of the large model's output, avoiding overfitting problems caused by large-scale adjustments at once, and providing a gradual learning process for the large model.

[0103] It should be noted that the positive sample data used in each stage can be the same or different, and this application does not impose specific limitations. For example, multiple early stages can use positive sample data with simple problems, while multiple later stages can use positive sample data with complex problems. This can better achieve gradual improvement, reduce the difficulty in the early stages of training, improve training efficiency, and enable the large model to adapt to complex situations more stably, thereby improving the model's stability and generalization ability.

[0104] In practice, the fine-tuned model can be quantized and deployed. The deployment provides services to the outside world in the form of an application programming interface (API). Clients can call the API service to use the fine-tuned model for inference and obtain inference results.

[0105] In summary, this application provides a training system that first acquires positive labels for the input data, then feeds the input data into a large model to be fine-tuned, obtaining negative labels for the input data. A scoring model is then used to score the negative labels, obtaining negative sample scores. The large model is then fine-tuned using the positive labels, negative labels, and negative sample scores. The loss function used during fine-tuning is negatively correlated with the probability of the large model predicting positive samples with positive labels, thus guiding the model to generate content closer to the standard positive labels. Conversely, the loss function is positively correlated with the difference between the probability of the large model predicting negative samples with negative labels and the negative sample scores, thus guiding the model to reduce the generation of low-quality content. This allows the model to not only learn standard answers that meet user expectations but also understand and quantify various different answers that do not meet user expectations, enabling the large model to generate more diverse responses, improving the fine-tuning effect of the large model, and achieving better performance in specialized fields.

[0106] The training system provided in this application has been described in detail above. The model training method provided in this application will be explained below with reference to the accompanying drawings.

[0107] Figure 3 This is a flowchart illustrating the steps of a model training method provided in this application, as shown below. Figure 3 As shown, the method may include the following steps:

[0108] S310: Client 100 sends a model training request to training system 200.

[0109] Optionally, the model training request is used to obtain a pre-trained first large model, which is then fine-tuned to obtain a second large model. Here, the first large model refers to a large language model (LLM). An LLM is a natural language processing model trained with massive amounts of data and computational resources, capable of understanding, generating, and processing language text. It belongs to a type of deep learning and has a large parameter scale. In specific implementations, the first large model refers to an LLM trained on general text data. While the first large model performs well on various language tasks, it may have limitations for specific domain tasks. For example, the first large model may lack sufficient expertise in professional fields such as finance, medicine, and law, or lack specific processing methods in particular applications (such as sentiment analysis and machine translation). Therefore, after fine-tuning the first large model, the second large model can achieve better performance in the corresponding professional domain and specific application.

[0110] Furthermore, the aforementioned model training request may include the method of obtaining the primary model, such as the name, version, and read address of the primary model. It may also include the necessary information required for fine-tuning the primary model, including the task type, target domain, and hyperparameter information. The task type refers to the task type of the primary model, such as classification or text generation. The target domain refers to the target domain for model fine-tuning, such as legal clauses, medical question answering, or financial analysis. The hyperparameter information refers to the learning rate and batch size when fine-tuning the model. Of course, the aforementioned model training request may also include more or less data, which will not be listed here.

[0111] Optionally, the model training request can also be used to train the first large model from scratch, and then further fine-tune it to generate the second large model. In this case, the model training request not only includes the necessary information required for fine-tuning the large model, but also the necessary information required for training the first large model. For example, the model training request may also include information on the general training sample set required for training the first large model, model architecture definition information, pre-training hyperparameters, etc. The information on the general training sample set may be data path, data format, etc., the model architecture definition information may be the defined structure of the first large model, such as the number of transformer layers, the dimension of hidden layers, etc., and the pre-training hyperparameters include learning rate, number of training steps, etc., which are not specifically limited in this application.

[0112] S320: Training system 200 acquires positive sample data.

[0113] In specific implementation, positive sample data includes input samples and positive labels for the input samples. Input samples may include questions, and positive labels may include the first answer to the question that meets the user's expectations. The input samples can be unimodal or multimodal data; for example, the question may be described by text, text and images, or a combination of text, voice, and images, depending on the actual application scenario. This application does not impose specific limitations. Similarly, the first answer in the positive labels can also be unimodal or multimodal data; for example, the answer may be described by text, text and images, or a combination of text, voice, and images. This application does not impose specific limitations.

[0114] For example, an input sample might be a question consisting of an image (e.g., a dog playing in the grass) and the text: "What breed is this dog?" The label for this input sample could be the corresponding answer: "It's a Golden Retriever." The above example is for illustrative purposes only and is not intended to impose specific limitations.

[0115] Optionally, the user can provide positive sample data that has been labeled with positive tags. In this case, the model training request can include the method of obtaining the positive sample data, such as the storage path, so that the system can read the positive sample data according to the storage path.

[0116] Optionally, users can provide unlabeled input samples, which the system can then label to obtain positive sample data. In this case, the model training request can include the method of obtaining the unlabeled input samples, such as the storage path. The system can obtain the unlabeled input samples according to the model training request, and then obtain the first answer for each question through manual or machine labeling, thereby obtaining positive sample data.

[0117] In practice, if manual annotation is used, the system can first read a large number of input samples according to the storage path carried in the model training request, and then send the input samples to the annotators (experts or business-related personnel). The annotators can make the first answer that meets the user's expectations based on the problem described by the input samples, thereby obtaining a large number of positive sample data.

[0118] If machine annotation is used, the system can input the input sample into a third-largest model to obtain the first answer, which is then used as the positive label for the input sample, thus obtaining positive sample data. The third-largest model can be a large model within the target domain for this model fine-tuning. For example, if the target domain is finance, a commercially available, multiplier-based large model in finance can be used as the third-largest model for labeling positive samples. Of course, other models can also be used as the third-largest model; this application does not impose specific limitations.

[0119] Optionally, the user can provide only the target domain required for model fine-tuning, and the system will obtain the corresponding positive sample data based on the target domain. For example, the system may pre-store positive sample data in multiple domains such as finance, law, and healthcare. After the user selects the finance domain, the system can obtain positive sample data in the finance domain. This application does not impose specific limitations. The positive sample data in each domain pre-stored by the system can be obtained in advance through machine annotation or manual annotation, and this application does not impose specific limitations.

[0120] The above examples illustrate several ways to obtain positive sample data. In specific implementations, positive sample data can also be obtained through other methods, and this application does not impose any specific limitations.

[0121] S330: Training system 200 obtains negative sample data based on the first large model.

[0122] In its implementation, the training system 200 inputs the positive sample data into the first large model to obtain the second answer. This second answer is then used as the negative label for the input sample, resulting in negative sample data. It should be understood that the first large model is the large model to be fine-tuned. Using the first large model to label the input samples and obtain negative sample data allows the self-labeled negative sample data to reveal the weaknesses or errors in the first large model's data processing. This enables the first large model to focus on optimizing these weak points during fine-tuning, thus achieving targeted fine-tuning and improving the effectiveness of model fine-tuning.

[0123] It should be understood that traditional model fine-tuning techniques also use negative sample data, but usually they use randomly selected answers as negative labels for input samples to obtain negative sample data. In this way, the randomly selected answers may be completely irrelevant to the question, and the quality of such samples is poor. The first model is a pre-trained model, which has already learned the knowledge of outputting relevant answers, and the possibility of outputting completely irrelevant answers is low. Using irrelevant answers as negative samples will lead to poor model fine-tuning results.

[0124] The negative labels used in this application are generated by the primary model. These negative labels reflect the strengths and weaknesses of the primary model, allowing for targeted model fine-tuning while avoiding the introduction of external noise. Externally generated negative samples may cause data feature distribution shifts due to differences in labeling rules, while negative samples generated by the primary model naturally fit the current data features, making it easier for the model to learn the true mapping relationship between input and output. Furthermore, this method of obtaining negative samples is highly efficient and low-cost, providing almost zero-cost access to negative samples that perfectly match the current model state, and these negative samples directly address the core areas where the model needs optimization.

[0125] Optionally, when acquiring negative sample data, the input sample can be fed into the large model. The large model then generates a prompt corresponding to the input sample based on the input sample and the task description. Based on the prompt and the input sample, it performs inference to obtain a second answer, which is then used as the negative label for that input sample. This prompt can be used not only to acquire negative sample data but also for all sample data input into the large model during subsequent training. It should be understood that the prompt helps the large model understand the inference objective and clearly specifies the output format, making the output results more in line with user expectations.

[0126] Optionally, the large model can generate a prompt based on the input samples and task description. Specifically, the system can pre-set a prompt template, and the corresponding prompt can be obtained by filling in the template based on the input samples and task description. The template can include multiple fields to be filled, such as role definition, input question, output requirements, basis, and exception handling. Filling in each field based on the task description and input sample yields the prompt corresponding to that input sample. These fields can be designed based on the task objectives to ensure that the prompt allows the large model to better understand the current task objectives.

[0127] Optionally, after the large model generates a prompt for the input data, the prompt containing the input question can be input into the large model, allowing it to complete reasoning based on richer content and obtain more accurate results. For example, without a prompt, if the input patient information is: "Question: 39 degrees Celsius, headache, no cough, what is the possible diagnosis?", the large model outputs: "It may be the flu; rest is recommended." If there is a prompt, which restricts the answer to be based on the "Clinical Practice Guidelines," the large model outputs: "According to the 'Fever Diagnosis and Treatment Guidelines 2023,' fever accompanied by headache may suggest the flu; further blood tests are recommended." Therefore, the prompt can further improve the professionalism of the model output, making it more in line with current professional standards in the field.

[0128] The task description required to generate the prompt may include task definition, input data definition, output data definition, and exception handling, which are not specifically limited in this application. The task definition clearly defines the specific problem type the model needs to solve, such as classification, generation, or question answering, to avoid output deviating from the target. For example, in the legal field, the task description could specify: "Identify illegal clauses in a contract, based on the XXX Contract Law." Input and output data definitions can standardize formats and specify the data structures for model input and return, such as JSON, plain text, or specific fields, ensuring compatibility between upstream and downstream processing. Exception handling defines how the model handles uncertain or ambiguous situations, preventing the model from acting arbitrarily or generating "illusions." For example, it could specify "unknown" if risk preference is not mentioned, prohibiting unauthorized speculation. Of course, the task description may include some custom guidance assessments, such as defining questions containing keywords like "immediately" or "right away" as having an urgency level of 1 for expedited processing. The above examples are for illustration only and are not specifically limited in this application.

[0129] It should be understood that task descriptions can eliminate ambiguities that may arise during model fine-tuning. The general generation capability of the first major model may output too broadly, making it difficult to meet the business needs of a specific domain. By injecting domain rules through task descriptions, the fine-tuned second major model can output answers that better meet business needs.

[0130] In practice, the task description can be manually entered, such as by a product manager or business personnel through the client, and the model training request can carry this task description. Alternatively, the task description can be generated by the system based on information such as the task type, target domain, and hyperparameter information in the model training request; this application does not impose any specific limitations.

[0131] Optionally, a prompt is a text instruction input to the model, used to guide the model to understand the task and generate output that meets the requirements. By explicitly stating instructions (such as "Please answer as a doctor"), the model's role and output style are constrained; by specifying the output structure (such as point-based answers, JSON key-value pairs, etc.), the results are ensured to be programmatically parsable; and other prompt content further constrains the model, improves output quality, avoids the illusion of a large model, and reduces the risk of the model fabricating content.

[0132] In practice, the prompt can be pre-set by the system. For example, the prompt can be a pre-set prompt template containing multiple prompts to be filled in the computer system by the developers based on the task description, or it can be automatically generated by the system based on the task description. The specific requirements can be determined according to the actual business needs, and this application does not impose any specific limitations.

[0133] It should be understood that the task description defines the overall goals, scope, and constraints for fine-tuning, providing a general direction for task execution. The prompt, on the other hand, provides specific operational instructions and contextual guidance to the model, serving as a guide for more detailed task processing. The task description does not require input from the model; it primarily guides the design of the prompt, while the prompt requires input from the model to guide its generation behavior.

[0134] For example, in a medical diagnosis scenario, a task description could be as follows:

[0135] Task objective: To make a preliminary diagnosis of possible diseases based on the patient's symptoms and to provide a confidence level.

[0136] Output requirements: The diagnosis must be a standard disease name in the 2023 Disease Catalogue; if the symptoms involve multiple organ systems, a "multidisciplinary consultation is recommended" message should be displayed.

[0137] Output format: {"diagnosis":str,"confidence":float}.

[0138] Abnormal handling: Providing unverified treatment options is prohibited.

[0139] The corresponding prompt template example is as follows:

[0140] [Role] Doctor

[0141] [Based on] 2023 National Health and Family Planning Commission

[0142] Output: {"diagnosis":"disease name","confidence":0.8}

[0143] [Exception Handling] Returns {"diagnosis":"Further investigation required","confidence":0}

[0144] The aforementioned prompt template defines the role of the large model, namely the physician, and also defines knowledge constraints, namely referring to a specific disease catalog. It further defines the output format: outputting the disease name if the confidence level is greater than 0.8, otherwise outputting text indicating further investigation is required. The above example is for illustrative purposes only and is not intended to impose specific limitations in this application.

[0145] Thus, when the user inputs the question "Fever for 3 days, sore throat", the prompt template can be filled based on the question to obtain the following prompt:

[0146] You are a doctor. Please make a diagnosis based on the following symptoms:

[0147] 1. Extract keywords: fever for 3 days, sore throat.

[0148] 2. Matched according to the 2023 Disease Catalogue: Acute tonsillitis (code J03.901).

[0149] 3. Output format: {"diagnosis":"disease name","confidence":0.8}.

[0150] Inputting such a prompt into the model, compared to directly inputting the question, allows the model to obtain more information and generate answers that better meet the user's needs.

[0151] Understandably, after generating the corresponding prompt based on the fine-tuned task description, the first major model can generate the output answer required by the target domain. Such negative samples can accurately simulate complex error situations in real business scenarios, significantly improving the generalization ability and robustness of the fine-tuned second major model, with remarkable effects in some professional fields.

[0152] It should be noted that the same input sample can be input into the first large model multiple times to obtain multiple different negative labels, because inputting the same problem into the large model usually does not yield completely identical results. Specifically, it can be input into the first large model multiple times at the same time, or it can be input into the first large model multiple times at different times. This application does not make specific limitations.

[0153] S340: Training system 200 obtains negative sample scores of negative sample data.

[0154] It should be understood that a positive label (first answer) represents a high-quality answer to the question, one that is highly relevant to the question and meets the user's expectations. A negative label (second answer) represents a low-quality answer to the question, one that is partially relevant, poorly relevant, or irrelevant, and does not meet the user's expectations. Negative sample scores are determined based on the relevance of the second answer to the question and the quality of the answer. Lower relevance and lower answer quality result in lower scores. Only second answers with both high relevance and high quality receive high negative sample scores.

[0155] For example, the question is "How to repair a bicycle." A positive label would be: "First check if the wheels are damaged, second check if the brakes are damaged, and finally check if the chain is loose." This answer is highly relevant to the question and meets user expectations. A negative label 1 would be: "Don't repair it, buy a new bicycle." This is a relevant but low-quality answer because it doesn't provide useful information about the question and doesn't meet user expectations, resulting in a low sample score. A negative label 2 would be: "Tools needed to repair a bicycle include pliers." This answer is relevant but low-quality because it doesn't fully address the question, only mentioning the tools without explaining how to use them, thus failing to meet user expectations and also resulting in a low sample score. A negative label 3 would be: "The weather is nice today, temperature 18-26 degrees Celsius, humidity 50%, cloudy, windy." This answer describes the weather in great detail and is a high-quality answer, but it is completely irrelevant to the question and doesn't meet user expectations, resulting in a low sample score. The above negative labels are examples provided in this application and are not intended to limit the scope of the problem.

[0156] In practical implementation, the relevance in negative sample scoring refers to the content relevance between the second answer and the question. Content relevance can include semantic matching degree, keyword matching degree, etc. Semantic matching degree can be measured using NLP techniques, such as converting the question and second answer into vectors and calculating their cosine similarity to determine their content relevance. Alternatively, BERT or other pre-trained language models can be used to extract semantic embedding vectors of the second answer and the question, and the cosine similarity of the embedding vectors can be calculated to determine their content relevance. Keyword matching degree can be determined by extracting core entities in the question using the TD-IDF algorithm, checking whether the second answer contains these entities, and determining the content relevance based on entity coverage. Alternatively, the model can automatically extract keywords from the question and answer, and calculate the keyword overlap to determine the content relevance. Of course, semantic matching degree and keyword matching degree can be combined to determine the content relevance, or other methods can be used to calculate the content relevance; this application does not impose specific limitations on these methods.

[0157] In practice, the quality of answers in negative sample scoring includes the completeness of the second answer, the degree of information value, and the degree of grammatical correctness. Specifically, the final negative sample score can be determined by summing, weighted summing, or weighted averaging the completeness, information value, and grammatical correctness. Other types of indicators can also be used to determine the negative sample score, but this application does not make any specific limitations.

[0158] Completeness is used to indicate whether the second answer provides enough information to answer the question. For example, in the above example, the answer "The tools needed to repair a bicycle include pliers" is a low-completeness answer, and the corresponding negative sample score is also low. Completeness can be determined by comparing the second answer with the first answer and calculating the coverage ratio, or by extracting key points from the first answer and determining whether the second answer covers these key points.

[0159] The information value level is used to indicate whether the second answer has practical significance, whether it can substantially answer the question, and whether the answer meets the user's expectations. For example, in the above example, the answer "Don't repair, buy a new bicycle" has a low information value level, and the corresponding negative sample score is also low. When determining the information value level, an intent classification model can be used to judge whether the second answer matches the user's intent. For example, the intent to repair and the intent to buy are clearly mismatched, thus determining the information value level. Alternatively, a domain knowledge base can be used to check whether the answer contains valid keywords. For example, in the above example, a pre-defined domain keyword base can include keywords such as "tools," "steps," "adjust," and "replace." Then, based on the answer, "buy a new bicycle" does not contain any of these keywords, thus determining its low information value. The above examples are for illustrative purposes only and are not intended to limit the scope of this application.

[0160] In this context, grammatical correctness is used to indicate the grammatical correctness of the second answer. For example, in the above example, if the answer is: "Finally check if the chain is loose, first check if the wheel is damaged, then check if the brake is damaged," the word order of this answer is incorrect, meaning it has low grammatical correctness and a correspondingly low negative sample score. It should be understood that the above example is for illustrative purposes only and is not intended to be specific. When determining grammatical correctness, grammar checking tools can be used to detect the number of grammatical errors, and AI models can also be used to determine the perplexity of the second answer; higher perplexity indicates lower grammatical correctness.

[0161] Taking the above example, the question is "What tools are needed to repair a bicycle?" Answer 1 is "pliers (high relevance, low completeness)", Answer 2 is "buy a new bicycle (low relevance)", and Answer 3 is "first check the wheels, then check the chain (grammatical error)". Following the method described above, the scoring results of the three answers can be determined as shown in Table 1 below.

[0162] Table 1. Examples of scoring results for different negative labels

[0163]

[0164]

[0165] In practical implementation, all negative labels can be scored, enabling the model to learn the features of different negative samples with high and low scores. It can also filter out negative samples with excessively low scores to improve the quality of negative samples. Thresholds can also be set for individual indicators, such as a relevance threshold. If the relevance is below the threshold, even if the score is high, it needs to be filtered. Specific filtering conditions can be flexibly set based on scores according to actual business needs, or no filtering may be performed; this application does not impose specific limitations.

[0166] Optionally, the training system 200 can score the negative sample data based on relevance, answer quality, and other factors through manual evaluation. In this case, the training system 200 can send the generated negative sample data to relevant experts for scoring, and determine the score for each negative sample based on the expert feedback.

[0167] Optionally, the training system 200 can automatically score negative samples using a scoring model. This scoring model can comprehensively consider factors such as relevance and answer quality when scoring negative sample data. Alternatively, multiple scoring models can perform preliminary scoring on different aspects, and the final negative sample score can be obtained based on these preliminary scores. The scoring model can be implemented using rules (different conditions correspond to different scores), decision trees (different decision branches correspond to different scores), AI technology (AI model inference scores), etc., and this application does not impose specific limitations. The scoring model can be a pre-trained model or an existing scoring model in a relevant field, and this application does not impose specific limitations.

[0168] S350: The training system 200 generates sample data based on positive sample data, negative sample data, and negative sample scores.

[0169] Optionally, the sample data may also include positive sample scores, where each positive sample score is greater than the score of every negative sample. For example, if a positive sample score is set to 1 and a negative sample score is less than 1, normalization techniques can be used to uniformly map the scores of negative samples to the 0-1 range. This application does not impose specific limitations on this. This allows the model to explicitly learn the goal of obtaining higher scores for positive samples, making the optimization objective clearer.

[0170] In practice, positive and negative sample data can be merged into sample pairs. Each sample label in a pair corresponds to a score, thus obtaining sample data. Specifically, this sample data includes input samples and their corresponding positive and negative labels. Positive labels correspond to positive sample scores, and negative labels correspond to negative sample scores. Each input sample can have one or more positive and negative labels. However, the score corresponding to each label is unique.

[0171] Taking the example shown in Table 1, in the merged sample data, the input sample is the question: "What tools are needed to repair a bicycle?" This input sample corresponds to 1 positive label and 3 negative labels. The positive label is: "First check if the wheels are damaged, then check if the brakes are damaged, and finally check if the chain is loose," with a positive sample score of 1. The first negative label is: "Pliers," with a corresponding score of 0.59. The second negative label is: "Buy a new bicycle," with a corresponding score of 0.43. The third negative label is: "First check the wheels, and finally check the chain," with a score of 0.62. The above example is for illustration only and is not intended to be specific.

[0172] Optionally, the training system 200 can also format the sample data to adapt it to the input format required for model fine-tuning. For example, the input format could be (systemprompt, userprompt, positive sample, positive sample score, negative sample, negative sample score). Here, systemprompt sets the model's role or task context, guiding the model to generate answers suitable for a specific scenario. For example, in a customer service scenario, systemprompt could be "You are a customer service assistant, and you need to answer user questions professionally and friendly in Chinese." In a medical scenario, systemprompt could be "You are a doctor's assistant, and you need to provide preliminary advice based on symptoms." Generally, if the model needs to switch between multiple roles, different systemprompts can be set for different samples; if multiple role switching is not required, the systemprompt for the samples can be the same. Userprompt is the question input by the user, and may also include contextual information from the user's historical input, such as historical records in multi-turn dialogues. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0173] S360: The training system 200 uses sample data and a loss function to fine-tune the first large model to obtain the second large model.

[0174] Optionally, the loss function may include a first loss function, which is used to guide the model to fine-tune the model's parameters based on positive sample data, negative sample data, and negative sample scores. The first loss function enables the model to not only distinguish between positive and negative samples, but also to distinguish between the quality of negative samples, allowing the model to explore high-quality but non-standardized answers, and the output answers are more in line with the complex preferences of humans.

[0175] Optionally, the loss function may include a second loss function, which guides the model to fine-tune based on positive sample data. In specific implementations, the second loss function may be a supervised fine-tuning (SFT) loss function, which optimizes model parameters by comparing the predicted results with positive labels.

[0176] In its implementation, the SFT loss function uses cross-entropy loss. The large model generates words one by one, producing a probability distribution for each word. The word with the highest probability becomes the next word to be generated. Therefore, the positive label corresponds to a true probability distribution, while the large model's output corresponds to a predicted probability distribution. The goal of the SFT loss is to make the predicted probability distribution as close as possible to the true probability distribution, and cross-entropy loss measures the difference between the two. Therefore, the second loss function not only guides the difference between the model's output and the first answer but also guides the difference in the generation process, assigning a high probability to the generation of the correct answer, rather than simply generating the correct answer.

[0177] Understandably, the second loss function ensures the model generates standard content, while the first loss function ensures the model generates more diverse content. Their synergistic effect allows the model to balance accuracy and diversity, significantly enhancing its usability in specialized fields. In practice, the first and second loss functions can be controlled by parameters, allowing the model to flexibly adjust their weights according to different scenarios. For example, in medical scenarios, where more accurate output is required and the model's performance is less important, the second loss function can be weighted at 70%, and the first loss function at 30%. In creative writing scenarios, where more diverse output is needed, the second loss function can be weighted at 30%, and the first loss function at 70%. These examples are for illustrative purposes only and do not constitute a specific limitation in this application.

[0178] It should be noted that the loss function may include only the first loss function, or only the second loss function, or more loss functions, such as the third loss function. Parameters are also used to control the proportion of different loss functions. This application does not impose specific limitations.

[0179] Optionally, the first loss function can be obtained based on the positive sample odds and negative sample odds. The positive sample odds indicate the probability that the model predicts a positive label (first answer), while the negative sample odds indicate the probability that the model predicts a negative label (second answer) with different negative sample scores. The positive sample odds are negatively correlated with the first loss function; a larger positive sample odds indicates a higher probability of the model predicting the first answer, and a smaller first loss function value requires less fine-tuning of the model, thus guiding the model to generate content closer to the standard positive label. Conversely, the negative sample odds are positively correlated with the first loss function; a larger negative sample odds indicates a higher probability of the model predicting a low-scoring second answer, or a lower probability of predicting a high-scoring second answer, and a larger first loss function value requires more fine-tuning of the model, thus guiding the model to reduce the generation of low-quality content.

[0180] In one possible implementation, the positive sample dominance value can be determined based on the probability of a positive sample occurring. A higher probability of a positive sample occurs results in a larger positive sample dominance value and a smaller first loss function value. Specifically, the positive sample dominance value can be determined based on the ratio between the probability of a positive sample occurring and the probability of a positive sample not occurring. It should be understood that the probability ratio is more sensitive to changes in probability, effectively amplifying the positive sample dominance value and thus amplifying the first loss function value, which can incentivize the model to learn high-confidence samples. Furthermore, when judging preferences, humans tend to express them through relative advantage (e.g., A is 3 times better than B) rather than absolute difference (A is 0.2 points higher than B). Using the dominance value to calculate the first loss function allows the model's learning objective to more closely align with human preferences, generating answers that better meet human needs.

[0181] In one possible implementation, the negative sample dominance value can be determined based on the difference between the probability of a negative sample occurring (the probability of generating the second answer) and the negative sample score. The larger the difference, the larger the negative sample dominance value, and consequently, the larger the first loss function value. This difference describes the degree of inconsistency between the model's predicted negative sample probability and the score. A smaller difference indicates a higher probability of the model generating high-quality answers or a lower probability of generating low-quality answers, meaning a high degree of consistency between the score and the probability. A larger difference indicates either a higher probability of the model generating low-quality answers or a lower probability of generating high-quality answers, meaning a lower degree of inconsistency between the score and the probability. It should be understood that the negative sample dominance value determined based on the difference comprehensively considers both the negative sample probability and the negative sample score, guiding the model to learn the characteristics of answers of different qualities, thereby suppressing the generation of low-quality answers and guiding the model to generate diverse high-quality answers.

[0182] Furthermore, since there is a certain gap between the numerical ranges of scores and probabilities, a scaling factor α can be used to scale either the score or the probability, bringing them to the same dimension. This avoids optimization bias caused by the numerical ranges of scores and probabilities, allowing the difference to become very small when the negative sample probability and score are high, thus minimizing the negative sample advantage and reducing the loss function value. For example, assuming the probability is p and the score is s, the difference can be |p-αs|. If the model predicts a high probability p and a high score s (good negative sample quality), the difference will be small, such as 0.1, resulting in a smaller negative sample advantage, a smaller loss function value, and a smaller parameter adjustment range during model fine-tuning. The above example is for illustration only and is not intended to limit the scope of the application.

[0183] Furthermore, the negative sample advantage value can amplify the difference, thereby amplifying the inconsistency between the model's prediction and the score. This allows the model to more sensitively capture error signals and guide more targeted fine-tuning. Specifically, the difference between the probability and the score can be converted into a relative error ratio. For example, if the difference is x, the negative sample advantage value could be x / (1-x). It should be understood that the difference describes the inconsistency between the model's prediction and the score, while the negative sample advantage value x / (1-x) describes the proportion of this inconsistency in the remaining inconsistency space. The remaining inconsistency space can be understood as the gap between the difference and the theoretical maximum consistency (when x=0). Thus, when the difference x increases, the numerator of the negative sample advantage value x / (1-x) increases linearly while the denominator decreases non-linearly, causing the negative sample advantage value x / (1-x) to rise exponentially. This allows the loss function to amplify the penalty signal, enabling the model to more sensitively identify prediction biases and achieve efficient and targeted model fine-tuning. Meanwhile, to avoid computational crashes when the difference is 0, a parameter δ can be set at the numerator, in which case the negative sample advantage value can be (x+δ) / (1-x). The above formula is for illustrative purposes only and is not intended to be specific.

[0184] Optionally, after determining the positive and negative sample dominance values, a first loss function can be obtained based on the ratio between the positive and negative sample dominance values. This ensures that the positive sample dominance value is directly proportional to the first loss function, guiding the model to strive towards generating the standard correct answer (the first answer with a positive label), while the negative sample dominance value is inversely proportional to the first loss function, guiding the model to strive towards generating a high-quality answer (the second answer with a high negative sample score).

[0185] Furthermore, the first loss function can be obtained based on the aforementioned ratio and optimization algorithm. Optionally, the optimization algorithm may include a logarithmic transformation algorithm; that is, the value of the first loss function can be obtained by performing a logarithmic transformation on the ratio. It should be understood that when the advantage of positive samples is much greater than the advantage of negative samples, the ratio may be extremely large, such as 10. 6Using the ratio directly as the loss function can lead to gradient explosion, while a very small ratio can lead to gradient vanishing. Therefore, taking the logarithm of the ratio can compress the numerical range and avoid gradient problems caused by extreme values.

[0186] Optionally, the optimization algorithm may also include a normalization function, such as the SIGMOD function. That is, the first loss function value can be obtained by performing a logarithmic transformation on the comparison value and then inputting the logarithmic transformation result into the SIGMOD function. This compresses the logarithmic value of the real number range to the probability interval of [0,1], while making the first loss function more stable and smooth, maintaining stability during training, avoiding numerical explosion or oversensitivity, and improving training efficiency and optimization effect.

[0187] It should be understood that the above examples provide some optimization algorithms that the first loss function may use. In specific implementations, more or fewer optimization algorithms may be selected based on actual business needs. These will not be listed here.

[0188] The following is combined Figure 4 To summarize the structure of the loss function described above, Figure 4 This is a schematic diagram of the structure of a loss function provided in this application, such as... Figure 4 As shown, the loss function includes a first loss function and a second loss function. The first loss function is determined based on the positive sample advantage and the negative sample advantage, for example, by taking the logarithm of the ratio between the two and inputting it into the sigmoid function. The positive sample advantage is determined based on the prediction probability of the first answer by the first large model, for example, the ratio between the probability of a positive sample occurring and the probability of a positive sample not occurring. Specifically, the negative sample advantage is determined based on the prediction probability of the second answer by the first large model and the score of the second answer, for example, the difference between the probability and the score. The second loss function is determined based on the gap between the first answer and the predicted result for a positive sample, and can specifically be the cross-entropy loss function, which is not specifically limited in this application.

[0189] The determination methods for the first loss function and the second loss function have been described in detail above. The specific formulas are given below to illustrate the loss function used in this application.

[0190] For example, the loss function provided in this application can be a weighted sum of the first loss function and the second loss function. For example, the formula for the loss function L can be as follows: Formula (1):

[0191] L=E(x,y w ,y l ,s l )[L SFT +λL OSR ] Formula (1)

[0192]

[0193]

[0194]

[0195] Formula (1) explains that the loss function L is obtained based on two parts, L SFT This refers to the second loss function, L. OSR It is the first loss function, and the ratio between the two can be controlled by the hyperparameter λ to adapt to various application scenarios.

[0196] Equation (2) explains the first loss function L OSR The odds of positive and negative samples θ (y w |x)) and negative sample odds θ (y l ,s l The ratio between |x) is logarithmically transformed and then input into the SIGMOD function. As described above, the logarithmic transformation and the SIGMOD function are optimization algorithms for the loss function, aiming to make the first loss function smoother and more stable. The first loss function is determined based on the ratio of positive sample advantage to negative sample advantage so that the positive sample advantage value is directly proportional to the first loss function, guiding the model towards generating the standard correct answer (the first answer with a positive label). Conversely, the negative sample advantage value is inversely proportional to the first loss function, guiding the model towards generating a high-quality answer (the second answer with a high negative sample score). For a detailed description, please refer to the above; it will not be repeated here.

[0197] Equation (3) explains the first loss function L OSR Positive sample advantage odds θ (y w |x) is the probability p that the first model predicts as the positive label (first answer). w Specifically, this can be the ratio between the probability of predicting the first answer and the probability of not predicting the first answer. This probability ratio is more sensitive to changes in probability, effectively amplifying the advantage of positive samples and thus amplifying the value of the first loss function, which can incentivize the model to learn from high-confidence samples.

[0198] Equation (4) explains the first loss function L OSR Negative sample advantage odds θ (y l ,s l |x) is the probability p that the first model predicts the negative label (second answer). w and negative sample scores s l The difference between |pl -αs l |Obtained, where in order to ensure probability p w And ratings l For scales with similar dimensions, use the hyperparameter α to reduce the score s. l Typically, α < 1. The difference is then converted to a relative error ratio to amplify its impact on the loss function. As the difference increases, the numerator grows linearly while the denominator shrinks non-linearly, leading to an exponential increase in the negative sample advantage value. This allows the loss function to amplify the penalty signal, making the model more sensitive to prediction bias. Simultaneously, to avoid computational collapse during subsequent logarithmic transformation when the difference is zero, a hyperparameter δ can be set at the numerator. δ can be a very small positive constant.

[0199] Formulas (2) to (4) above are applied to the first loss function L. OSR An explanation was provided regarding the second loss function L. SFT The SFT loss function is obtained based on the model prediction results and positive labels, such as the cross-entropy loss function, or other commonly used SFT functions in the industry. This application does not make specific limitations, and will not elaborate further here.

[0200] It should be noted that the above formulas (1) to (4) are examples of a first loss function provided in this application. In specific implementations, the formulas can be modified according to the actual application scenario. For example, the absolute value operation on the difference in formula (4) for negative sample advantage can be replaced by a smoothing function, such as the squared difference formula. Multi-dimensional scoring can also be introduced, such as extending the multi-dimensional scoring into a vector form to replace s in formula (4). l Furthermore, formula (2) uses the SIGMOD function to achieve normalization. Other normalization functions can also be used, such as the Tanh function. The above examples provide some possible variations. In some implementation scenarios, other variations may also be used. This application does not make any specific limitations.

[0201] For example, formula (1) can be transformed to obtain formula (5):

[0202] L=E(x,y w ,y l ,s l )[L SFT +λ(t)L OSR In simple terms, formula (5) means that λ can change over time, and the initial training phase focuses on the basic task L. SFT Later enhancement mission L OSR This enhances the model's ability to output diverse answers. Specifically, λ(t) can be a function that decreases with time, such as λ(t) = λ0e^(-λt / t). -ktWhere λ0 and k are preset parameters. Or other decreasing functions, this application does not specifically limit them. The above formula (5) is a possible variation, this application does not specifically limit it.

[0203] For example, formula (4) can be transformed to obtain formula (6):

[0204]

[0205] In this equation, δ1 and δ2 are both smoothing factors, with the denominator or numerator being 0. Equation (6) replaces the absolute value of Equation (4) with the difference of squares. The squaring operation can make the gradient change smoother, which is suitable for scenarios that require continuous optimization.

[0206] For example, the score s can be used to introduce multidimensional features to evaluate the quality of negative samples, such as s l =[s1,s2,…,s n ] T , which is a multi-dimensional scoring vector, where each s represents a score in one dimension. For example, the relevance of the second answer to the question is described by s1, the text quality of the second answer is described by s2, and so on, to obtain the final multi-dimensional scoring vector. At this time, formula (4) can be transformed to obtain the following formula (7):

[0207]

[0208] Where, β i It is a learnable weight; when the negative sample score is determined by multiple indicators, the learnable weight β... i This allows the model to automatically focus on important metrics among multiple indicators. For example, if the model consistently generates incorrect answers with low relevance, the weight of the relevance score (s1) will increase. Conversely, if the model consistently generates answers with high relevance but poor text fluency, the weight of the text quality score (s2) will increase. This enables more targeted fine-tuning based on the model's weaknesses.

[0209] For example, formula (2) can be transformed to obtain formula (8):

[0210]

[0211] Here, γ is the boundary offset, a trainable parameter. When it is greater than zero, it increases the positive sample dominance threshold, which is equivalent to requiring the positive sample dominance value to be much higher than the negative sample dominance value. In this case, the model is required to learn positive samples more rigorously, which is suitable for tasks with high accuracy requirements, such as medical diagnosis. When it is less than zero, it decreases the positive sample dominance threshold, which encourages the model to explore diverse answers, and is suitable for tasks with high diversity requirements, such as title generation and advertising slogan design. Of course, γ can also be used to achieve progressive learning from lenient to strict, improving the fine-tuning effect of large models.

[0212] For another example, formula (2) can be transformed to obtain formula (9):

[0213]

[0214] Here, γ is a temperature coefficient that controls the model's sensitivity to the ratio between positive and negative sample dominance, essentially scaling the ratio. It can also be used to achieve progressive learning from lenient to strict. In some possible implementations, equations (8) and (9) can be combined, that is, dual γ jointly control progressive learning, which will not be illustrated here.

[0215] It should be understood that the above formulas (5) to (9) provide possible variations of the loss function. There may be other variations based on formulas (1) to (4), which will not be illustrated here.

[0216] It should be understood that in the loss function provided in this application, the second loss function is obtained based on positive sample data, which guides the model to learn the relevant features of the first answer and ensures that the model generates standard content. The first loss function is obtained based on positive sample data, negative sample data and negative sample scores, which guides the model to learn the relevant features of the second answer of different quality (different scores) and guides the model to generate more diverse content. The synergistic effect of the two can enable the model to take into account both accuracy and diversity, and can significantly improve its use value in some professional sub-fields.

[0217] It should be understood that traditional negative sample training processes directly set the probability of negative samples to 0. On the one hand, this suppresses the contribution of negative samples to model learning to zero, causing some potentially contributing negative samples to be discarded, resulting in wasted resources and reduced model training efficiency. On the other hand, it makes the model tend to repeat safe phrases, leading to a decline in the quality of model output, an inability to generate diverse answers, or even the output of a single meaningless answer. In contrast, the technical solution of this application introduces negative sample scoring into the loss function, enabling the model to not only learn answers that meet user expectations but also quantify various different answers that do not meet user expectations. This allows large models to handle problems more accurately, provide more diverse answers, and make more reasonable judgments in complex scenarios.

[0218] In this embodiment, the training system 200 can use sample data (including positive sample data, negative sample data, and negative sample scores) and the aforementioned loss function L to fine-tune the first large model to obtain a second large model. Specifically, input sample x into the model to obtain positive sample probability p w and negative sample probability p l Then, based on the above formulas (1) to (4), the first loss function L is calculated respectively.OSR Second loss function L SFT The loss function value L is obtained by weighted summation. The gradient of the loss function value L with respect to the model parameters θ is calculated. The optimizer is then used to update the parameters θ along the reverse direction of the gradient to reduce the loss function value L, resulting in the updated model parameters θ'. The first loss function L... OSR The guided model distinguishes features of different quality answers, and the second loss function L SFT Ensure the model retains its basic generative capabilities. For each sample, the model parameters θ can be updated following the process described above to improve model performance. Fine-tuning continues until performance no longer improves, at which point the fine-tuning is terminated, resulting in the second-largest model.

[0219] Optionally, large model fine-tuning may include multiple stages. In each stage, the problem can be input into the large model to obtain new negative sample data, and the new negative samples can be scored to obtain new negative sample scores. Then, the large model can be fine-tuned using the new negative sample data and the new negative sample scores until the large model reaches the expected performance.

[0220] For example, Figure 5 This is an example diagram illustrating the multi-stage training process of a model training system provided in this application, such as... Figure 5 As shown:

[0221] First, in the first stage, the sample data V1 used includes the input samples and their corresponding positive labels, negative labels V1, and negative sample scores V1. Then, the input samples are input into the first large model of version V1 for fine-tuning to obtain the first large model of version V2, thus completing the fine-tuning of the first stage.

[0222] Next, before proceeding to the second stage of fine-tuning, the first large model V2 is used to re-predict the negative labels and update the scores. Specifically, the input samples are input into the first large model V2 to obtain the output results, which are used as the new negative labels V2. Then, the scoring model is used to score the negative labels V2 to obtain the corresponding negative sample scores V2.

[0223] Then, in the second stage, the sample data V2 used includes the input samples and the corresponding positive labels, negative labels V2, and negative sample scores V2. The input samples are then fed into the first large model of version V2 for fine-tuning to obtain the first large model of version V3, thus completing the fine-tuning of the second stage.

[0224] The first large model can then be continuously updated using the above cycle. After each stage ends and before the next stage begins, the latest first large model can be used to update the negative labels and negative sample scores. This allows the model to specifically improve its weak points at each stage, and the model's performance can be steadily improved. After m stages, a large model Vm that meets the training expectations can be obtained. Examples are not provided here.

[0225] It should be understood that by fine-tuning the large model through multiple stages, and by re-sampling negative samples and updating negative sample scores in each stage to form new training data, the large model's capabilities can be gradually improved. At each stage, targeted fine-tuning can be performed based on the quality of the large model's output, avoiding overfitting problems caused by large-scale adjustments at once, and providing a gradual learning process for the large model.

[0226] It should be noted that the positive sample data used in each stage can be the same or different, and this application does not impose specific limitations. For example, multiple stages in the early stage of training can use positive sample data with simple problems, while multiple stages in the later stage can use positive sample data with complex problems. This can better achieve gradual improvement, reduce the difficulty in the early stage of training, improve training efficiency, and enable the large model to adapt to complex situations more stably, thereby improving the model's stability and generalization ability.

[0227] S370: The training system 200 sends information about the second largest model to the client 100.

[0228] Optionally, the information of the second major model may include metadata information of the second major model, including but not limited to training time, final loss value, validation set accuracy, model storage path, version number, log link, etc. The client 100 can download the model file and deploy related applications based on the information of the second major model.

[0229] Optionally, the training system 200 can quantize the second-largest model to obtain a quantized model, then validate the quantized model. After confirming that the accuracy loss is within a threshold range, it sends the information of the second-largest model to the client 100. This information includes, but is not limited to, quantization method, accuracy loss, quantized model storage path, comparison of inference speed before and after quantization, supported hardware, and supported inference frameworks. The client 100 can download the quantized model file based on this information and deploy relevant applications. Model quantization can reduce the memory usage and inference latency of the model.

[0230] Optionally, after obtaining the quantized model, the training system 200 can automatically deploy it in a user-pre-configured production environment. This includes encapsulating the model into an independently runnable inference service module, selecting a deployment framework such as TensorFlow or Torchserve, choosing a deployment method such as containerized deployment or cloud platform deployment, designing the API interface (e.g., defining inputs and outputs), and implementing production-grade safeguards such as security encryption, traffic control, and monitoring alerts. Finally, based on the selected framework, deployment method, API design, and production-grade safeguards, the quantized model can be deployed. After deployment, information about the second major model is sent to the client 100. This information includes the API documentation of the deployed model, including service endpoints, request methods, and content types, as well as detailed technical parameters such as input data specifications, output field descriptions, and performance metrics. Further information may also be included, but this application does not specify a particular scope. The client 100 can use this information to call the API service to run the quantized model, complete inference, and obtain the inference results.

[0231] In summary, this application provides a model training method that first obtains positive labels for the input data, then feeds the input data into a large model to be fine-tuned, obtaining negative labels for the input data. A scoring model is then used to score the negative labels to obtain negative sample scores. The large model is then fine-tuned using positive labels, negative labels, and negative sample scores. The value of the loss function used during fine-tuning is negatively correlated with the probability of the large model predicting positive samples with positive labels, thus guiding the model to generate content closer to the standard positive labels. Conversely, the value of the loss function is positively correlated with the difference between the probability of the large model predicting negative samples with negative labels and the negative sample scores, thus guiding the model to reduce the generation of low-quality content. This allows the model to not only learn standard answers that meet user expectations but also understand and quantify various different answers that do not meet user expectations, enabling the large model to generate more diverse responses, improving the fine-tuning effect of the large model, and achieving better performance in specialized fields.

[0232] The model training method and training system provided in this application have been described in detail above. The following section will combine... Figure 6 The software structure of the training system provided in this application will be explained.

[0233] Figure 6 This is a schematic diagram of the software structure of a training system provided in this application, such as... Figure 6 As shown, the training system 200 includes a positive sample acquisition unit 610, a negative sample acquisition unit 620, a scoring unit 630, and a fine-tuning unit 640. It should be understood that... Figure 6 This is an exemplary partitioning method. The training system 200 can be divided into more or fewer unit modules, and this application does not make any specific limitations.

[0234] The positive sample acquisition unit 610, negative sample acquisition unit 620, scoring unit 630, and fine-tuning unit 640 can all be implemented in software or hardware. For example, the implementation of fine-tuning unit 640 will be described below. Similarly, the implementation of positive sample acquisition unit 610, negative sample acquisition unit 620, and scoring unit 630 can refer to the implementation of fine-tuning unit 640.

[0235] As an example of a software functional unit, the fine-tuning unit 640 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the fine-tuning unit 640 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0236] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0237] As an example of a hardware functional unit, the fine-tuning unit 640 may include at least one computing device, such as a server. Alternatively, the fine-tuning unit 640 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-a-chip (SoC), an offload card, an inference card, or any combination thereof.

[0238] The multiple computing devices included in the fine-tuning unit 640 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the fine-tuning unit 640 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the fine-tuning unit 640 can be distributed in the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, and inference cards.

[0239] It should be noted that, in other embodiments, the steps implemented by the positive sample acquisition unit 610, the negative sample acquisition unit 620, the scoring unit 630, and the fine-tuning unit 640 can be specified as needed. The positive sample acquisition unit 610, the negative sample acquisition unit 620, the scoring unit 630, and the fine-tuning unit 640 respectively implement different steps in the model training method to realize all the functions of the training system 200.

[0240] The functions of the positive sample acquisition unit 610, negative sample acquisition unit 620, scoring unit 630, and fine-tuning unit 640 are explained below.

[0241] Positive sample acquisition unit 610 is used to acquire positive sample data, wherein the positive sample data includes input samples and positive labels. The input samples include questions, and the positive labels include the answers corresponding to the questions. Specifically, it is used to implement... Figure 3 S310 to S320 and their optional steps in the embodiments.

[0242] The negative sample acquisition unit 620 is used to input the input sample into the first large model to be fine-tuned and obtain the first negative label corresponding to the input sample. Specifically, it is used to implement... Figure 3 S330 of the embodiment and its optional steps.

[0243] Scoring unit 630 is used to determine the first score of the first negative label based on the relevance of the first negative label to the question and the quality of the response to the first negative label, specifically for implementing... Figure 3 S340 of the embodiment and its optional steps.

[0244] The fine-tuning unit 640 is used to input the input samples into the first large model and fine-tune the first large model based on the loss function to obtain the second large model. The loss function guides the first large model to improve the prediction probability of positive labels and the prediction probability of negative labels with scores greater than a threshold. Specifically, it is used to implement... Figure 3 S350 to S370 of the embodiment and its optional steps Figure 5 The steps described in the embodiments.

[0245] In one possible implementation, the value of the loss function is negatively correlated with the probability of a positive sample predicted by the first largest model with a positive label, and positively correlated with the probability of a negative sample predicted by the first largest model with a negative label and the first gap between the negative sample score.

[0246] In one possible implementation, the loss function includes a first loss function and a second loss function. The first loss function is determined based on the negative sample scores and negative sample labels. The first loss function is used to instruct the first large model to learn the features of the negative labels for different negative sample scores. The second loss function is determined based on the positive labels. The second loss function is used to guide the first large model to learn the features of the positive labels.

[0247] In one possible implementation, the first loss function includes a negative sample advantage value, which is determined based on a first gap. The negative sample advantage value is used to amplify the impact of the first gap on the first loss function. The first gap is positively correlated with the negative sample advantage value, and the negative sample advantage value is positively correlated with the value of the first loss function.

[0248] In one possible implementation, the first loss function includes a positive sample advantage value, which is determined based on the positive sample prediction probability. The positive sample advantage value is used to amplify the impact of the positive sample prediction probability on the first loss function. The positive sample prediction probability is positively correlated with the positive sample advantage value, and the positive sample advantage value is negatively correlated with the value of the first loss function.

[0249] In one possible implementation, the second loss function includes the cross-entropy loss between the predictions obtained by the first large model based on the input samples and the positive labels.

[0250] In one possible implementation, the negative sample acquisition unit 620 is used to input the input sample into the second large model to obtain the second negative label of the input sample; the scoring unit 630 is used to score the second negative label based on the scoring model to obtain the second negative sample score; and the fine-tuning unit 640 is used to fine-tune the second large model using the input sample, positive label, second negative label, and second negative sample score to obtain the third large model. See details for further information. Figure 5 The relevant descriptions of the embodiments will not be repeated here.

[0251] In one possible implementation, the input sample includes a question, the first negative label includes the answer to the question, and the scoring model is used to determine a first score based on the relevance of the first negative label to the question and the quality of the response to the first negative label. The response quality includes one or more of completeness, information value, and grammatical correctness.

[0252] In one possible implementation, the formula for the loss function includes the above formulas (1) to (4), which can be referred to in the previous content and will not be repeated here.

[0253] In summary, this application provides a model training method that first obtains positive labels for the input data, then feeds the input data into a large model to be fine-tuned, obtaining negative labels for the input data. A scoring model is then used to score the negative labels to obtain negative sample scores. The large model is then fine-tuned using positive labels, negative labels, and negative sample scores. The value of the loss function used during fine-tuning is negatively correlated with the probability of the large model predicting positive samples with positive labels, thus guiding the model to generate content closer to the standard positive labels. Conversely, the value of the loss function is positively correlated with the difference between the probability of the large model predicting negative samples with negative labels and the negative sample scores, thus guiding the model to reduce the generation of low-quality content. This allows the model to not only learn standard answers that meet user expectations but also understand and quantify various different answers that do not meet user expectations, enabling the large model to generate more diverse responses, improving the fine-tuning effect of the large model, and achieving better performance in specialized fields.

[0254] The above text describes in detail the model training method and training system provided in this application. The following section will combine... Figures 7-9 The computing device provided in this application will be explained.

[0255] Figure 7 This is a schematic diagram of the structure of a computing device provided in this application, such as... Figure 7 As shown, the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, memory 706, and communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server. It should be understood that this application does not limit the number of processors, memory, and accelerator cards in the computing device 700. The computing device 700 can be used to deploy the aforementioned... Figures 1-8 In the training system 200 of this embodiment, memory 706 is used to store instructions, and processor 704 is used to execute the instructions stored in memory 706. Memory 706 stores program code, and processor 704 can call the program code stored in memory 706 to perform the following operations:

[0256] Obtain positive sample data, which includes input samples and positive labels. The input samples include questions, and the positive labels include the answers to the questions.

[0257] Input the input sample into the first large model to be fine-tuned to obtain the first negative label corresponding to the input sample;

[0258] The scoring model determines the first score for the first negative label based on its relevance to the question and the quality of the response to the first negative label.

[0259] The input samples are fed into the first large model, and the first large model is fine-tuned based on the loss function to obtain the second large model. The loss function is used to guide the first large model to improve the prediction probability of positive labels and the prediction probability of negative labels whose scores are greater than the threshold.

[0260] It should be understood that the memory 706 stores executable program code, and the processor 704 executes the executable program code to achieve the above. Figures 3-5 The embodiments describe the operational steps of the model training method. The program code may include the code of one or more software units, which may be... Figure 6 The diagram shows a positive sample acquisition unit, a negative sample acquisition unit, a scoring unit, and a fine-tuning unit. The positive sample acquisition unit is used to acquire positive sample data, specifically for implementing… Figure 3 S310, S320, and their optional steps in the embodiment. The negative sample acquisition unit is used to input the input sample into the first large model to be fine-tuned, and obtain the first parent label corresponding to the input sample, specifically for implementing... Figure 3S330 of the embodiment and its optional steps. The scoring unit is used to score the first negative label based on the scoring model to obtain a first score, specifically used to implement... Figure 3 S340 of the embodiment and its optional steps. The fine-tuning unit is used to input the input samples into the first large model, and to fine-tune the first large model based on the loss function to obtain the second large model. Specifically, it is used to implement... Figure 3 S350 to S370 of the embodiment and its optional steps, and Figure 5 The steps described in the embodiments will not be repeated here.

[0261] The 702 bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 702 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 702 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708). The unified bus may also be called the Lingqu bus.

[0262] The processor 704 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an accelerator card.

[0263] Memory 706 may include volatile memory, such as random access memory (RAM). Memory 706 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 706 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0264] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 706, or two or more types of storage media can be configured to realize the function of memory 706. This application does not limit this.

[0265] The accelerator card 707 can be implemented through computing units such as a data processing unit (DPU), a neural processing unit (NPU), a graphics processing unit (GPU), an extended processing unit (XPU), and a system-on-chip (SoC). The accelerator card 707 is used to implement the training and inference of AI models, such as the training and inference of the diffusion model of this application. The accelerator card 707 includes a processing unit and a power supply circuit. The power supply circuit is used to supply power to the processing unit, enabling the processing unit to perform AI tasks, such as model fine-tuning tasks.

[0266] It should be understood that the accelerator card 707 can be understood as the processor used to train AI models as mentioned above, while the processor 704 refers to the processing core in the computing device 700, such as the CPU. The two are different concepts.

[0267] The communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0268] It should be understood that the computing device 700 described above in this application may correspond to this application. Figures 1-8 The training system 200 in the present application can be used to perform training according to the present application. Figures 3 to 5The corresponding entities in the method, and the above and other operations and / or functions of each module in the computing device 700, are respectively for the purpose of implementing Figures 3 to 5 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0269] As one possible implementation method, Figures 3 to 5 The method described herein can be implemented either by the processor in the computing device 700 executing program code in the memory, or by the accelerator card 707 in the computing device 700. The accelerator card 707 can be implemented by computing devices such as DPU, NPU, GPU, XPU, SoC, or AI chips; this application does not specifically limit its implementation. The accelerator card 707 includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to execute the above-described... Figures 3 to 5 The operational steps of the model training method described in each embodiment are as follows.

[0270] As one possible implementation method, Figures 3 to 5 The method described above can be implemented either by the processor in the computing device 700 executing program code in memory, or by the accelerator card 707, or by a combination of the processor 704 and the accelerator card 707 in the computing device 700. Figures 3 to 5 The operation steps performed by the corresponding entity in the model training method described herein. Specifically, the program code stored in memory 706 includes... Figure 6 The shown unit modules enable the processor 704 to execute the program code to implement some operation steps of the model training method, and the power supply circuit in the accelerator card 707 supplies power to the processing unit, enabling the processing unit to execute another part of the operation.

[0271] This application also provides a cluster of computing devices. For example... Figure 8 As shown, Figure 8 This is an example diagram of a computing device cluster provided in this application, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0272] As one possible implementation, the same implementation can be stored in the memory 706 of one or more computing devices 700 in a computing device cluster. Figures 3 to 5 The instructions describe the operational steps of the model training method. Alternatively, different implementations may be stored separately in the memory 706 of one or more computing devices 700. Figures 3 to 5The instructions for the operational steps of the described model training method; in other words, a combination of one or more computing devices 700 can jointly execute the instructions for the operational steps required to implement the model training method.

[0273] As one possible implementation, the same implementation can be stored in the accelerator cards 707 of one or more computing devices 700 in the computing device cluster. Figures 3 to 5 The instructions describe the operational steps of the model training method. Alternatively, different implementations may be stored in one or more accelerator cards 707 of the computing device 700. Figures 3 to 5 The instructions for the operational steps of the described model training method; in other words, a combination of one or more computing devices 700 can jointly execute the instructions for the operational steps required to implement the model training method.

[0274] As one possible implementation, the same implementation is stored in the memory 706 of one or more computing devices 700 in the computing device cluster. Figures 3 to 5 The instructions for some operation steps in the described model training method are stored in the accelerator card 707, which contains the same instructions for implementing other operation steps. Alternatively, different implementations may be stored in the memory 706 of one or more computing devices 700. Figures 3 to 5 The instruction for one part of the operation steps of the described data method is stored in the accelerator card 707, which contains different instructions for implementing another part of the operation steps. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for the operation steps required to implement the model training method.

[0275] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, Figure 9 This is a schematic diagram of another computing device cluster structure provided in this application. Two computing devices, 700A and 700B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the accelerator card 707 in computing device 700A implements the functions described by the negative sample acquisition unit and the fine-tuning unit, while the memory 706 in computing device 700B stores instructions for the positive sample acquisition unit and the scoring unit for execution by the processor 704. Figure 9 The connection method between the computing device clusters shown can be based on the consideration that the training system 200 can be a distributed system. In order to reduce the processing pressure of each device, the unit modules are distributed on different computing devices.

[0276] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0277] This application also provides a chip or a chip system including multiple chips. The chip may include a processing unit and a power supply circuit. The power supply circuit can supply power to the processing unit, enabling the power supply unit to perform the AI-related operation steps that need to be processed in the above-mentioned training system.

[0278] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the model training method described above.

[0279] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned model training method.

[0280] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method is applied to a training system, and the method includes: Obtain positive sample data, wherein the positive sample data includes input samples and positive labels, the input samples include questions, and the positive labels include the answers corresponding to the questions; The input sample is input into the first large model to be fine-tuned to obtain the first negative label corresponding to the input sample; The first score of the first negative label is determined based on the relevance of the first negative label to the question and the quality of the response to the first negative label, according to the scoring model. The input sample is input into the first large model, and the first large model is fine-tuned based on the loss function to obtain the second large model. The loss function is used to guide the first large model to improve the prediction probability of the positive label and the prediction probability of the negative label whose score is greater than the threshold.

2. The method according to claim 1, characterized in that, The value of the loss function is negatively correlated with the positive sample prediction probability of the positive label predicted by the first large model, and positively correlated with the negative sample prediction probability of the first negative label predicted by the first large model and the first gap between the negative sample score.

3. The method according to claim 1 or 2, characterized in that, The loss function includes a first loss function and a second loss function; The first loss function is determined based on the negative sample score and the negative sample label. The first loss function is used to instruct the first large model to learn the features of the negative labels for different negative sample scores. The second loss function is determined based on the positive label, and the second loss function is used to guide the first large model to learn the features of the positive label.

4. The method according to claim 3, characterized in that, The first loss function includes a negative sample advantage value, which is determined based on the first gap. The negative sample advantage value is used to amplify the impact of the first gap on the first loss function. The first gap is positively correlated with the negative sample advantage value, and the negative sample advantage value is positively correlated with the value of the first loss function.

5. The method according to claim 3 or 4, characterized in that, The first loss function includes a positive sample advantage value, which is determined based on the positive sample prediction probability. The positive sample advantage value is used to amplify the impact of the positive sample prediction probability on the first loss function. The positive sample prediction probability is positively correlated with the positive sample advantage value, and the positive sample advantage value is negatively correlated with the value of the first loss function.

6. The method according to any one of claims 3 to 5, characterized in that, The second loss function includes the cross-entropy loss between the prediction result obtained by the first large model based on the input sample and the positive label.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The input sample is input into the second large model to obtain the second negative label of the input sample; The second negative sample score is obtained by scoring the second negative label based on the scoring model; The second major model is fine-tuned using the input samples, the positive labels, the second negative labels, and the scores of the second negative samples to obtain the third major model.

8. The method according to any one of claims 1 to 7, characterized in that, The formula for the loss function includes: L=E(x,y w ,y l ,s l )[THE SFT +λL OSR ] Where x is the input sample, and y w It is the positive label, the y l It is the first negative label, the s l It is the first score, the L SFT It is the second loss function, the L OSR For the first loss function, p w The first large model predicts the probability of a positive label based on the input sample, where p is... l It is the probability that the first large model predicts a negative label based on the input sample, where α and δ are parameter values.

9. A training system, characterized in that, The system includes: A positive sample acquisition unit is used to acquire positive sample data, wherein the positive sample data includes input samples and positive labels, the input samples include questions, and the positive labels include the answers corresponding to the questions; The negative sample acquisition unit is used to input the input sample into the first large model to be fine-tuned and obtain the first negative label corresponding to the input sample. A scoring unit is used to determine a first score for the first negative label based on a scoring model, according to the relevance of the first negative label to the question and the quality of the response to the first negative label. The fine-tuning unit is used to input the input sample into the first large model and fine-tune the first large model based on the loss function to obtain a second large model. The loss function is used to guide the first large model to improve the prediction probability of the positive label and the prediction probability of the negative label whose score is greater than a threshold.

10. The system according to claim 9, characterized in that, The value of the loss function is negatively correlated with the positive sample prediction probability of the positive label predicted by the first large model, and positively correlated with the negative sample prediction probability of the first negative label predicted by the first large model and the first gap between the negative sample score.

11. The system according to claim 9 or 10, characterized in that, The first loss function includes a negative sample advantage value, which is determined based on the first gap. The negative sample advantage value is used to amplify the impact of the first gap on the first loss function. The first gap is positively correlated with the negative sample advantage value, and the negative sample advantage value is positively correlated with the value of the first loss function.

12. The system according to any one of claims 9 to 11, characterized in that, The first loss function includes a positive sample advantage value, which is determined based on the positive sample prediction probability. The positive sample advantage value is used to amplify the impact of the positive sample prediction probability on the first loss function. The positive sample prediction probability is positively correlated with the positive sample advantage value, and the positive sample advantage value is negatively correlated with the value of the first loss function.

13. A computing device, characterized in that, The computing device includes a processor and a memory, the memory being used to store code, the code being executed by the processor, such that the computing device performs the operational steps of the method as described in any one of claims 1 to 8.

14. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 8.

15. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the operational steps of the method as described in any one of claims 1 to 8.

Images