Large model training method and device, electronic equipment and storage medium

Abstract

The invention provides a large model training method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, in particular to the fields of large models, artificial intelligence and deep learning. According to the specific implementation scheme, a sample data set is obtained, sample data comprises a task instruction and a sample pair, and the sample pair comprises a positive sample and a negative sample; based on the sample data set, training the large model to obtain direct preference optimization (DPO) loss information and supervised fine tuning (SFT) loss information of the large model; and according to the DPO loss information and the SFT loss information, carrying out gradient adjustment on the model parameters of the large model to obtain a target large model, and improving the model training precision.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to the fields of large models, artificial intelligence and deep learning, and particularly to a training method, apparatus, electronic device and storage medium for large models. Background Technology

[0002] Model training is the process of using specific algorithms and data to repeatedly adjust the internal parameters of a model so that it can accurately predict or classify new data. Current training methods for large models are difficult to generate text that is both natural and fluent and conforms to human expectations and value standards in complex scenarios, and cannot accurately achieve high-quality intelligent generation and decision assistance. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for training large models.

[0004] According to one aspect of this disclosure, a method for training a large model is provided, the method comprising: Obtain a set of sample data, wherein the sample data includes task instructions and sample pairs, and the sample pairs include positive samples and negative samples; Based on the sample data set, a large model is trained to obtain the Direct Preference Optimization (DPO) loss information and supervised fine-tuning SFT loss information of the large model. Based on the DPO loss information and SFT loss information, the model parameters of the large model are adjusted by gradient to obtain the target large model.

[0005] According to another aspect of this disclosure, a training apparatus for a large model is provided, comprising: The acquisition module is used to acquire a sample data set, wherein the sample data includes task instructions and sample pairs, and the sample pairs include positive samples and negative samples. The training module is used to train the large model based on the sample data set to obtain the direct preference optimization (DPO) loss information and supervised fine-tuning (SFT) loss information of the large model. The optimization module is used to perform gradient adjustment on the model parameters of the large model based on the DPO loss information and SFT loss information to obtain the target large model.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect embodiment.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect embodiment.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect embodiment.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of a large model training method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a large model training method provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of a large model training method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a large model training device provided in an embodiment of this disclosure; Figure 5 A schematic block diagram of an electronic device used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] Data processing is the collection, storage, retrieval, processing, transformation, and transmission of data; its purpose is to extract and deduce data that is valuable and meaningful to certain specific people from large amounts of data that may be disorganized and difficult to understand.

[0013] Artificial intelligence (AI) is a new technological science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. It aims to understand the nature of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0014] Large models refer to a class of artificial intelligence models with a large number of parameters built by artificial neural networks. They are characterized by a large number of parameters, large training data, and large computing resources, and have the ability to solve general tasks, follow human instructions, and perform complex reasoning.

[0015] Deep learning specifically refers to machine learning based on deep neural network models and methods. Its most important technical feature is the ability to automatically extract features. The extracted features are also called deep features or deep feature representations. Compared with manually designed features, deep features have stronger and more robust representation capabilities.

[0016] The two main model training methods in existing technologies are Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). SFT focuses on language expression optimization, while DPO focuses on preference alignment, and the two conflict in their optimization objectives. Using only one method can lead to a performance degradation in another dimension. When the model relies entirely on DPO for training, it is prone to overfitting preference labels in the early stages, resulting in degraded output language, logical inconsistencies, or semantic patterning. Existing technologies typically use a fixed loss structure throughout training, failing to dynamically adjust the weights of language quality and preference learning according to the training stage, thus limiting the model's generalization and stability. DPO relies on manually or semi-automatically labeled data. If the corpus distribution is homogeneous or the weights are fixed, the model may only conform to human judgment in certain scenarios, while deviating from expectations in complex contexts.

[0017] Figure 1 This is a schematic diagram illustrating a large model training method provided in an embodiment of this disclosure. For example... Figure 1 As shown, the method includes the following steps: S101, Obtain the sample data set.

[0018] The sample data includes task instructions and sample pairs, with each sample pair consisting of positive and negative samples. Task instructions can be user-inputted instructions or questions. In each sample pair, positive samples are preferred samples, and negative samples are non-preferred samples.

[0019] In some embodiments, the sample data in the sample dataset can be processed by word segmentation, padding, and masking before being used as training sample data.

[0020] S102, based on the sample data set, trains a large model to obtain the direct preference optimization (DPO) loss information and supervised fine-tuning (SFT) loss information of the large model.

[0021] DPO learns preference patterns by comparing data with humans, making the output more in line with user expectations. Understandably, the optimization goal of DPO is not language correctness, but "preference difference". The model only needs to make the relevance probability of positive samples higher than that of negative samples to obtain a lower loss. Training based solely on DPO loss information lacks language distribution constraints during the training process. The model may deviate from natural language during the optimization process, resulting in semantic degradation, language degradation and preference overfitting.

[0022] SFT is a model trained under supervision with high-quality human-written corpus samples, which makes the generated texts perform well in terms of language structure, grammatical logic and coherence. It mainly optimizes language expression ability, but lacks understanding and adaptation to human subjective preferences.

[0023] Optionally, sample data from the sample dataset can be input into a large model. Based on the comparison of the preference probabilities between the predicted sample data and the sample data output by the large model, DPO loss information can be obtained. Based on the loss between the predicted positive samples and the positive samples in the sample data, SFT loss information can be determined. Based on the DPO loss information and SFT loss information, the large model can be adjusted and optimized in a coordinated manner to improve the model training accuracy.

[0024] S103. Based on the DPO loss information and SFT loss information, the model parameters of the large model are adjusted by gradient to obtain the target large model.

[0025] Optionally, the DPO loss information and SFT loss information can be fused to obtain the total loss information. The model parameters of the large model can be adjusted based on the total loss information until the total loss information converges or the preset number of iterations is reached, at which point the model training is stopped and the target large model is obtained.

[0026] Optionally, the total loss information can be a weighted fusion of DPO loss information and SFT loss information, or the average loss of DPO loss information and SFT loss information, and the specific fusion method is not limited.

[0027] In this embodiment, a sample dataset is acquired, and a large model is trained based on the sample dataset to obtain the DPO loss information and SFT loss information of the large model. According to the DPO loss information and SFT loss information, the model parameters of the large model are adjusted by gradient to obtain the target large model. The SFT loss information optimizes language expression ability, and the DPO loss information makes the output of the large model more in line with user expectations. Through the combination and optimization of DPO loss information and SFT loss information, each focuses on different objectives, improving language expression ability while learning value preferences. The final optimization training effect of the large model is better.

[0028] Figure 2 This is a schematic diagram illustrating a large model training method provided in an embodiment of this disclosure. For example... Figure 2 As shown, the method includes the following steps: S201, Obtain the sample data set.

[0029] In this embodiment of the disclosure, the method for implementing step S201 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0030] S202, based on the sample data set, trains a large model to obtain the predicted sample pairs output by the large model.

[0031] The predicted sample pairs include predicted positive samples and predicted negative samples.

[0032] S203, based on the sample pairs and predicted sample pairs, obtain the DPO loss information of the large model.

[0033] In some embodiments, for any pair of samples in the sample data, the deviation information of the large model after the current training relative to the initial large model is determined, and the DPO loss information of the large model is determined based on the deviation information, thereby improving the accuracy of DPO loss information acquisition.

[0034] Optionally, a first prediction information for the sample data output by the large model during current training can be determined; the first prediction information includes at least the first positive sample prediction probability for positive samples and the first negative sample prediction probability for negative samples.

[0035] The second prediction information of the sample data output by the initial large model is determined. The initial large model is the reference large model in this embodiment. The second prediction information includes at least the second positive sample prediction probability and the second negative sample prediction probability.

[0036] Further, deviation information is determined based on the first prediction information and the second prediction information; optionally, local deviation information corresponding to the sample data can be determined based on the first prediction information and the second prediction information; the local deviation information corresponding to the first sample data is weighted to obtain the deviation information.

[0037] Understandably, based on the first prediction information, the first positive sample prediction probability and the first negative sample prediction probability can be determined; based on the second prediction information, the second positive sample prediction probability and the second negative sample prediction probability can be determined; in this embodiment, based on the first positive sample prediction probability and the second positive sample prediction probability, the positive sample deviation information corresponding to the sample data is determined; based on the first negative sample prediction probability and the second negative sample prediction probability, the negative sample deviation information corresponding to the sample data is determined; based on the positive sample deviation information and the negative sample deviation information, the local deviation information corresponding to the sample data is determined.

[0038] Furthermore, the local deviation information is weighted to obtain the deviation information, and the DPO loss information of the large model is obtained based on the deviation information. In this embodiment, the DPO loss information can be expressed as:

[0039] in, This is the DPO loss function, which is also the DPO loss information of the large model in this embodiment; Enter information; Positive samples; Negative samples; Predict the probability for the first positive sample; Predict the probability for the first negative sample; Predict the probability for the second positive sample; Predict the probability for the second negative sample; This indicates deviation information from positive samples; This provides information on negative sample deviation. This is information about local deviations; Temperature coefficient; The activation function is the sigmoid function; To Expected calculation.

[0040] DPO optimizes the consistency between the model output and human evaluation by using human preference samples, making the model output more in line with human preferences and values.

[0041] S204: Based on the positive samples and the predicted positive samples, obtain the SFT loss information.

[0042] Alternatively, the SFT loss information can be represented as:

[0043] in, This is the SFT loss function, which is also the SFT loss information of the large model in this embodiment; Enter information; For high-quality positive samples, the SFT loss information is determined by the deviation between the predicted positive samples and the preset high-quality positive samples.

[0044] SFT mimics high-quality positive samples, enabling the model to maintain consistency with the distribution of natural language in terms of grammatical structure, syntactic dependencies, and semantic logic. The prediction of each token depends on contextual information, forcing the model to learn lexical rules, semantic collocations, and sentence logical structures, thereby generating grammatically correct, semantically coherent, and logically consistent text. It can generate natural text that conforms to human language norms, ensuring quality at the language level.

[0045] S205. Based on the DPO loss information and SFT loss information, the model parameters of the large model are adjusted by gradient to obtain the target large model.

[0046] In this embodiment of the disclosure, the method for implementing step S205 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0047] In this embodiment, after obtaining the sample data set, the large model is trained based on the sample data set to obtain the predicted sample pairs output by the large model. Based on the prediction probabilities of positive and negative samples in the predicted sample pairs and the prediction information of the initial large model, deviation information is obtained. DPO loss information is obtained based on the deviation information to improve the accuracy of DPO loss calculation. SFT loss information is obtained based on high-quality positive samples and predicted positive samples. By imitating high-quality samples, the model can maintain consistency with the distribution of natural language in terms of grammatical structure, syntactic dependencies, and semantic logic. The model parameters of the large model are adjusted by gradient based on accurate DPO loss information and SFT loss information to obtain an accurate target large model. SFT keeps the language manifold from collapsing, and DPO adjusts the output direction to be more human-like, thus achieving a balanced optimization of language ability and value consistency.

[0048] Figure 3 This is a schematic diagram illustrating a large model training method provided in an embodiment of this disclosure. For example... Figure 3 As shown, the method includes the following steps: S301, Obtain the sample data set.

[0049] In this embodiment of the disclosure, the method for implementing step S301 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0050] S302, based on the sample data set, train the large model to obtain the predicted sample pairs output by the large model.

[0051] In this embodiment of the disclosure, the method for implementing step S302 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0052] S303, based on the sample pairs and predicted sample pairs, obtains the DPO loss information of the large model.

[0053] In this embodiment of the disclosure, the method for implementing step S303 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0054] S304: Based on the positive samples and the predicted positive samples, obtain the SFT loss information.

[0055] In this embodiment of the disclosure, the method for implementing step S304 can be implemented in any of the various embodiments of the disclosure, and no limitation is made here, nor will it be described in detail.

[0056] S305, Recognizing the training phase of a large model.

[0057] Optionally, the training phase of a large model can include two phases: the initial training phase and the middle and later training phases. Different weights can be assigned to the DPO loss information and the SFT loss information in different training phases to improve the training effect of the large model.

[0058] S306: Based on the training phase, the DPO loss information and SFT loss information are weighted to obtain the global loss information of the large model.

[0059] In this embodiment, the weights corresponding to DPO loss information and SFT loss information are determined according to the training phase.

[0060] Optionally, during the initial training phase of the large model, the weights corresponding to the DPO loss information are determined within a first set range, for example, the first set range is 0.3-0.5; based on the weights corresponding to the DPO loss information, the weights corresponding to the SFT loss information are determined, and the weights corresponding to the SFT loss information are expressed as follows: ,in The weights are the values ​​corresponding to the DPO loss information.

[0061] Understandably, in the initial training phase, the weights corresponding to the DPO loss information and the SFT loss information are kept unchanged, with SFT dominating the training of the large model to ensure semantic and syntactic stability.

[0062] Optionally, during the non-initial training phase of the large model, the weights corresponding to the DPO loss information are determined within a second set range, and the weights corresponding to the SFT loss information are determined based on the weights corresponding to the DPO loss information; wherein, the upper limit of the first set range is less than the lower limit of the second set range.

[0063] For example, if the second setting range is 0.8-0.9, then according to Determine the weights corresponding to the SFT loss information; use DPO to lead the training of the large model, so that the model can further align with human preferences on the basis of language stability.

[0064] Optionally, during the non-initial training phase of a large model, the weights corresponding to the DPO loss information increase as the training phase progresses; while the weights corresponding to the SFT loss information decrease as the training phase progresses. That is, the weights corresponding to the DPO loss information gradually increase during training, and correspondingly, the weights corresponding to the SFT loss information gradually decrease. This embodiment can automatically adjust the weights according to the perplexity and the alignment score to achieve adaptive balance, effectively preventing the semantic degradation problem caused by DPO training alone, and enhancing the grammatical robustness and semantic consistency of the model in long text generation through the language constraints of SFT.

[0065] Furthermore, based on the DPO loss information and SFT loss information and their respective weights, the global loss information of the large model is determined. The global loss information is expressed as:

[0066] in, This provides global loss information for the large model. Information on DPO loss; For SFT loss information; The weights corresponding to the DPO loss information; The weights are the values ​​corresponding to the SFT loss information.

[0067] SFT loss provides language stability constraints to ensure the correctness and fluency of the model in terms of syntax, semantics, and logical structure. DPO loss provides value guidance constraints to make the model's output more in line with human preferences and value orientations. Based on the synergy of DPO loss information and SFT loss information, a balanced optimization of "language ability and value consistency" is achieved.

[0068] S307: Based on the global loss information, the model parameters of the large model are adjusted using gradients to obtain the target large model.

[0069] Understandably, the model parameters of the large model are adjusted using gradients based on global loss information until the global loss information converges or the preset number of iterations is reached, at which point model training is stopped to obtain the target large model, thus preventing overfitting or waste of resources during model training.

[0070] In this embodiment, a sample dataset is acquired, and a large model is trained based on the sample dataset to obtain predicted sample pairs output by the large model. DPO loss information and SFT loss information are acquired based on the sample pairs and predicted sample pairs to improve the accuracy of DPO and SFT loss information acquisition. A weighted sum is performed based on the accurate DPO and SFT loss information to obtain global loss information. The weights of the DPO and SFT loss information are dynamically adjusted during the training phase to effectively prevent semantic degradation caused by DPO training alone. The linguistic constraints of SFT enhance the model's syntactic robustness and semantic consistency in long text generation. The model parameters of the large model are gradient-adjusted based on the more accurate global loss information to obtain the target large model. SFT loss information optimizes language expression ability, while DPO loss information makes the large model output more consistent with user expectations, jointly achieving a balanced optimization of language ability and value consistency. While learning value preferences, language expression ability is improved, resulting in better final optimization training effects for the large model.

[0071] Figure 4 This is a schematic diagram of the structure of a large model training device provided in an embodiment of this disclosure. Figure 4 As shown, the training device 400 for this large model includes: The acquisition module 401 is used to acquire a sample data set, wherein the sample data includes task instructions and sample pairs, and the sample pairs include positive samples and negative samples. Training module 402 is used to train a large model based on a sample dataset to obtain the direct preference optimization (DPO) loss information and supervised fine-tuning (SFT) loss information of the large model. The optimization module 403 is used to perform gradient adjustment on the model parameters of the large model based on the DPO loss information and SFT loss information to obtain the target large model.

[0072] In some embodiments, the training module 402 is configured to: Based on the sample data set, the large model is trained to obtain the predicted sample pairs output by the large model. The predicted sample pairs include predicted positive samples and predicted negative samples. Based on the sample pairs and predicted sample pairs, the DPO loss information of the large model is obtained; Based on the positive samples and the predicted positive samples, the SFT loss information is obtained.

[0073] In some embodiments, the optimization module 403 is configured to: Identify the training phase of a large model; Based on the training phase, the DPO loss information and SFT loss information are weighted to obtain the global loss information of the large model; Based on the global loss information, the model parameters of the large model are adjusted using gradients to obtain the target large model.

[0074] In some embodiments, the optimization module 403 is configured to: Based on the training phase, determine the weights corresponding to the DPO loss information and the SFT loss information respectively. The global loss information of the large model is determined based on the DPO loss information and SFT loss information and their respective weights.

[0075] In some embodiments, the optimization module 403 is configured to: In the initial training phase of a large model, the weights corresponding to the DPO loss information are determined within a first set range. The weights corresponding to the SFT loss information are determined based on the weights corresponding to the DPO loss information.

[0076] In some embodiments, the optimization module 403 is configured to: During the initial training phase, the weights corresponding to the DPO loss information and the SFT loss information are kept unchanged.

[0077] In some embodiments, the optimization module 403 is configured to: During the non-initial training phase of the large model, the weights corresponding to the DPO loss information are determined within a second set range. Based on the weights corresponding to the DPO loss information, determine the weights corresponding to the SFT loss information. The upper limit of the first set range is less than the lower limit of the second set range.

[0078] In some embodiments, the optimization module 403 is configured to: In the early training phase of a large model, the weights corresponding to the DPO loss information increase as the training phase progresses; while the weights corresponding to the SFT loss information decrease as the training phase progresses.

[0079] In some embodiments, the training module 402 is configured to: For any pair of samples in the sample data, determine the deviation information of the large model relative to the initial large model after the current training; Based on the deviation information, determine the DPO loss information of the large model.

[0080] In some embodiments, the training module 402 is configured to: Determine the first predictive information of the sample data output by the large model during the current training phase; Determine the second predictive information of the sample data output by the initial large model; Based on the first and second prediction information, the deviation information is determined.

[0081] In some embodiments, the training module 402 is configured to: Based on the first and second prediction information, determine the local deviation information corresponding to the sample data; The local deviation information corresponding to the first sample data is weighted to obtain the deviation information.

[0082] In some embodiments, the training module 402 is configured to: Based on the first prediction information, determine the first positive sample prediction probability of positive samples and the first negative sample prediction probability of negative samples; Based on the second prediction information, determine the second positive sample prediction probability and the second negative sample prediction probability of the positive sample; Based on the predicted probabilities of the first and second positive samples, determine the positive sample deviation information corresponding to the sample data; Based on the predicted probability of the first negative sample and the predicted probability of the second negative sample, determine the negative sample deviation information corresponding to the sample data; Based on the deviation information of positive samples and the deviation information of negative samples, the local deviation information corresponding to the sample data is determined.

[0083] In this embodiment, a sample dataset is acquired, and a large model is trained based on the sample dataset to obtain predicted sample pairs output by the large model. DPO loss information and SFT loss information are acquired based on the sample pairs and predicted sample pairs to improve the accuracy of DPO and SFT loss information acquisition. A weighted sum is performed based on the accurate DPO and SFT loss information to obtain global loss information. The weights of the DPO and SFT loss information are dynamically adjusted during the training phase to effectively prevent semantic degradation caused by DPO training alone. The linguistic constraints of SFT enhance the model's syntactic robustness and semantic consistency in long text generation. The model parameters of the large model are gradient-adjusted based on the more accurate global loss information to obtain the target large model. SFT loss information optimizes language expression ability, while DPO loss information makes the large model output more consistent with user expectations, jointly achieving a balanced optimization of language ability and value consistency. While learning value preferences, language expression ability is improved, resulting in better final optimization training effects for the large model.

[0084] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0085] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0086] Figure 5 A schematic block diagram of an electronic device for implementing embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0087] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0088] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0089] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as large model training methods. For example, in some embodiments, the large model training method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the large model training method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform large model training methods by any other suitable means (e.g., by means of firmware).

[0090] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0094] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0095] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0096] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a large model, wherein, The method includes: Obtain a set of sample data, wherein the sample data includes task instructions and sample pairs, and the sample pairs include positive samples and negative samples; Based on the sample data set, a large model is trained to obtain the Direct Preference Optimization (DPO) loss information and supervised fine-tuning SFT loss information of the large model. Based on the DPO loss information and SFT loss information, the model parameters of the large model are adjusted by gradient to obtain the target large model.

2. The method according to claim 1, wherein, The process of training a large model based on the sample dataset to obtain the direct preference optimization (DPO) loss information and supervised fine-tuning SFT loss information of the large model includes: Based on the sample data set, the large model is trained to obtain the predicted sample pairs output by the large model, wherein the predicted sample pairs include predicted positive samples and predicted negative samples. Based on the sample pairs and the predicted sample pairs, the DPO loss information of the large model is obtained; The SFT loss information is obtained based on the positive samples and the predicted positive samples.

3. The method according to claim 1, wherein, The step of adjusting the model parameters of the large model based on the DPO loss information and SFT loss information to obtain the target large model includes: Identify the training phase of the large model; Based on the training phase, the DPO loss information and SFT loss information are weighted to obtain the global loss information of the large model; Based on the global loss information, the model parameters of the large model are adjusted using gradients to obtain the target large model.

4. The method according to claim 3, wherein, The step of weighting the DPO loss information and SFT loss information according to the training phase to obtain the global loss information of the large model includes: Based on the training phase, determine the weights corresponding to the DPO loss information and the SFT loss information respectively; Based on the DPO loss information and SFT loss information and their respective weights, the global loss information of the large model is determined.

5. The method according to claim 4, wherein, The step of determining the weights corresponding to the DPO loss information and SFT loss information based on the training phase includes: In the initial training phase of the large model, the weights corresponding to the DPO loss information are determined within a first set range. The weights corresponding to the SFT loss information are determined based on the weights corresponding to the DPO loss information.

6. The method according to claim 5, wherein, The method further includes: During the initial training phase, the weights corresponding to the DPO loss information and the SFT loss information are kept unchanged.

7. The method according to claim 5, wherein, The step of determining the weights corresponding to the DPO loss information and SFT loss information based on the training phase includes: During the non-initial training phase of the large model, the weights corresponding to the DPO loss information are determined within a second set range. Based on the weights corresponding to the DPO loss information, determine the weights corresponding to the SFT loss information; Wherein, the upper limit of the first set range is less than the lower limit of the second set range.

8. The method according to claim 7, wherein, The method further includes: During the non-initial training phase of the large model, the weights corresponding to the DPO loss information increase as the training phase progresses; while the weights corresponding to the SFT loss information decrease as the training phase progresses.

9. The method according to claim 2, wherein, The step of obtaining the DPO loss information of the large model based on the sample pair and the predicted sample pair includes: For any pair of samples in the sample data, determine the deviation information of the large model relative to the initial large model after the current training; Based on the deviation information, the DPO loss information of the large model is determined.

10. The method according to claim 9, wherein, Determining the deviation information of the large model relative to the initial large model after the current training includes: Determine the first prediction information of the sample data output by the large model during the current training; Determine the second prediction information of the sample data output by the initial large model; The deviation information is determined based on the first prediction information and the second prediction information.

11. The method according to claim 10, wherein, Determining the deviation information based on the first prediction information and the second prediction information includes: Based on the first prediction information and the second prediction information, determine the local deviation information corresponding to the sample data; The deviation information is obtained by weighting the local deviation information corresponding to the first sample data.

12. The method according to claim 11, wherein, The step of determining the local deviation information corresponding to the sample data based on the first prediction information and the second prediction information includes: Based on the first prediction information, determine the first positive sample prediction probability of the positive sample and the first negative sample prediction probability of the negative sample; Based on the second prediction information, determine the second positive sample prediction probability and the second negative sample prediction probability of the predicted positive sample; Based on the first positive sample prediction probability and the second positive sample prediction probability, determine the positive sample deviation information corresponding to the sample data; Based on the first negative sample prediction probability and the second negative sample prediction probability, the negative sample deviation information corresponding to the sample data is determined; Based on the positive sample deviation information and the negative sample deviation information, the local deviation information corresponding to the sample data is determined.

13. A training device for a large model, comprising: An acquisition module is used to acquire a set of sample data, wherein the sample data includes task instructions and sample pairs, and the sample pairs include positive samples and negative samples. The training module is used to train the large model based on the sample data set to obtain the direct preference optimization (DPO) loss information and supervised fine-tuning (SFT) loss information of the large model. The optimization module is used to perform gradient adjustment on the model parameters of the large model based on the DPO loss information and SFT loss information to obtain the target large model.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-12.

Images