A method and device for fine-tuning a large model

Abstract

The embodiments of this specification provide a method and device for fine-tuning a large model, the method comprising: obtaining a first question and a first answer corresponding to the first question, and obtaining a first score for the first answer based on the first large model; determining whether the first answer is a high-quality answer based on the first score, and if the first answer is a high-quality answer, obtaining a second answer to the first question and a second score for the second answer based on the first large model and the first question, the second score being lower than the first score; determining a first difference based on the probability of obtaining the first answer after inputting the first question into the second large model and the probability of obtaining the second answer after inputting the first question into the second large model, and determining a second difference based on the first score and the second score; and updating the parameters of the second large model with the goal of reducing the difference between the first difference and the second difference.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of large model technology, and more particularly, to a method and apparatus for fine-tuning a large model. Background Art

[0002] A large language model (LLM) is a deep learning model based on natural language processing, trained on large text corpora and containing hundreds of millions or more parameters. Users can engage in conversations with the LLM and receive answers to their questions. Fine-tuning a large model involves retraining a pre-trained LLM using a specific dataset. Its core purpose is to retain the general capabilities of the LLM while improving its specialized output capabilities in specific domains or scenarios. Summary of the Invention

[0003] The embodiments in this specification aim to provide a method and apparatus for fine-tuning a large model, which can fine-tune the large language model based on a quantitative assessment of the user's preference for the answer, so that the output of the fine-tuned large language model is more in line with the expectations of human users, improve the fine-tuning effect of the large language model, and address the shortcomings of the existing technology.

[0004] According to a first aspect, a method for fine-tuning a large model is provided, comprising:

[0005] Obtaining a first question and a first answer corresponding to the first question, and obtaining a first score for the first answer based on a first large model; determining whether the first answer is a high-quality answer based on the first score; and if the first answer is a high-quality answer, obtaining a second answer to the first question and a second score for the second answer based on the first large model and the first question, wherein the second score is lower than the first score;

[0006] A first difference is determined based on the probability of obtaining a first answer after inputting the first question into the second largest model, and a probability of obtaining a second answer after inputting the first question into the second largest model, and a second difference is determined based on the first score and the second score; and the parameters of the second largest model are updated with the goal of reducing the difference between the first difference and the second difference.

[0007] In one possible implementation, determining the first difference based on a probability of obtaining a first answer after inputting the first question into the second largest model and a probability of obtaining a second answer after inputting the first question into the second largest model includes:

[0008] The first value is determined based on the logarithm of the ratio of the probability of obtaining the first answer after inputting the first question into the second largest model to the probability of obtaining the first answer after inputting the first question into the third largest model; the second value is determined based on the logarithm of the ratio of the probability of obtaining the second answer after inputting the first question into the second largest model to the probability of obtaining the second answer after inputting the first question into the third largest model; and the first difference is determined based on the first value and the second value.

[0009] In a possible implementation, updating the parameters of the second largest model with the goal of reducing the difference between the first difference and the second difference includes:

[0010] The parameters of the second largest model are updated with the goal of reducing the expected value of the square of the deviation between the first difference and the second difference.

[0011] In one possible implementation, obtaining a second answer to the first question and a second score for the second answer based on the first model and the first question includes:

[0012] The first question, the first answer, and the prompt for generating a low-quality answer based on the first question and the first answer are input into the first large model to obtain a second answer; the first question, the second answer, and the prompt for scoring the second answer are input into the first large model to obtain a second score for the second question and answer.

[0013] In one possible implementation, obtaining a second answer to the first question and a second score for the second answer based on the first model and the first question includes:

[0014] The first question, the first answer, and an indicator word for generating a low-quality answer and a score for the low-quality answer based on the first question and the first answer are input into a first large model to obtain a second answer and the second score.

[0015] In one possible implementation, determining whether the first answer is a high-quality answer based on the first score includes:

[0016] If the first score is higher than a preset threshold, the first answer is determined to be a high-quality answer.

[0017] In a possible implementation, the first question is a question that indicates an answer based on a multi-step process or logical reasoning, or a question associated with a specified context.

[0018] According to a second aspect, there is provided an apparatus for fine-tuning a large model, the apparatus comprising:

[0019] an acquisition unit configured to acquire a first question and a first answer corresponding to the first question, obtain a first score for the first answer based on a first large model; determine whether the first answer is a high-quality answer based on the first score, and if the first answer is a high-quality answer, obtain a second answer to the first question and a second score for the second answer based on the first large model and the first question, wherein the second score is lower than the first score;

[0020] The fine-tuning unit is configured to determine a first difference based on the probability of obtaining a first answer after inputting the first question into the second largest model and the probability of obtaining a second answer after inputting the first question into the second largest model, and determine a second difference based on the first score and the second score; and update the parameters of the second largest model with the goal of reducing the difference between the first difference and the second difference.

[0021] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0022] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0023] By utilizing one or more of the methods, apparatuses, computing devices, and storage media in the above aspects, the large language model can be fine-tuned based on a quantitative assessment of the user's preference for the answer, so that the output of the fine-tuned large language model better meets the expectations of human users. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic diagram illustrating a scheme for fine-tuning a large model;

[0026] Figure 2 A flowchart showing a method for fine-tuning a large model according to an embodiment of this specification;

[0027] Figure 3 A schematic diagram illustrating obtaining training samples for fine-tuning a large model according to an embodiment of this specification;

[0028] Figure 4A schematic diagram illustrating a fine-tuned large model according to an embodiment of the present specification;

[0029] Figure 5 A structural diagram of a device for fine-tuning a large model according to an embodiment of this specification is shown. DETAILED DESCRIPTION

[0030] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0031] Throughout this document, a Large Language Model (LLM) may also be referred to as a large model. A large language model is a natural language processing model based on deep learning technology. Its parameters typically range from billions to hundreds of billions, or even higher, and it possesses powerful language understanding and generation capabilities. Large language models can adopt the Transformer architecture or its variants (such as GPT and BERT). This architecture utilizes the attention mechanism to achieve global modeling of sequential data, efficiently handling long-range dependencies and thus excelling in natural language tasks. Large language models are pre-trained on large corpora to learn the statistical characteristics and semantic relevance of language, enabling them to generalize well. The core capabilities of large language models include, but are not limited to, understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. They are typically used in two modes: direct inference and fine-tuning. In direct inference mode, users design prompts to guide the large language model to generate specific outputs. The prompts can be textual descriptions of tasks or instructions, stimulating the semantic understanding and generation capabilities of large language models. The powerful generalization and flexibility of large language models make them a crucial tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and comprehension.

[0032] In some embodiments, large language models may also have the ability to understand and generate data from other modalities (such as vision, audio, etc.). In this case, large language models may also be referred to as multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of inputs and outputs such as text, images, and sound. The core advantage of MLLMs is that they can process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze a picture and generate descriptive text, or generate a corresponding image based on the text description. This cross-modal understanding and generation capability gives MLLMs broad application prospects in multiple fields. It should be noted that the key technologies of large language models can be found in the detailed description in the paper "ASurvey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), which will not be repeated in this manual.

[0033] Fine-tuning refers to retraining a pre-trained large language model using a specific dataset. Its core purpose is to retain the general capabilities of the large language model while improving its specialized output capabilities in specific domains or scenarios. Preference alignment fine-tuning is a fine-tuning solution for large language models. Existing preference alignment fine-tuning solutions obtain user feedback on the preferences of model-generated content. The parameters of the large language model are then fine-tuned based on this feedback. For example, preference pairs are constructed based on this feedback and used to fine-tune the large language model. This ensures that the text generated by the fine-tuned large language model better meets user expectations. Figure 1 A schematic diagram showing a solution for fine-tuning a large model. Figure 1 As shown, for example, a question can be input into a large language model to obtain an output answer. Based on the user's feedback on the answer (such as like or dislike), a preference sample pair including the question, the preferred answer, or the disliked answer is generated, and the large language model is fine-tuned using the preference sample pairs.

[0034] However, this preference alignment fine-tuning scheme typically judges user preferences for answers based on absolute likes or dislikes, ignoring the fact that user preferences in real-world scenarios are often non-absolute. That is, users' judgments on answers to questions are usually not absolute likes or dislikes, but only to a certain degree. In particular, the degree of preference for answers varies significantly between preference sample pairs corresponding to different questions. This difference is even more pronounced in scenarios where answers are generated for complex questions, such as those requiring multi-step processing or processed through logical reasoning. Therefore, the effectiveness of fine-tuning large language models using existing preference alignment fine-tuning schemes, especially for complex questions, remains suboptimal.

[0035] In order to solve the above technical problems, the embodiments of this specification provide a method for fine-tuning a large model. First, a question pair including a question and an answer corresponding to the question can be obtained, and the quality score of the answer can be obtained through the auxiliary large model, and whether the answer is a high-quality answer can be determined based on the quality score. If it is a high-quality answer, a low-quality answer to the question and a quality score for the low-quality answer can be obtained based on the high-quality answer and its quality score through the auxiliary large model. Thereafter, a first difference can be determined based on the probability of obtaining a high-quality answer after inputting the question into the target large model and the probability of obtaining a low-quality answer after inputting the question into the target large model, and a second difference can be determined based on the quality score of the high-quality answer and the quality score of the low-quality answer; with the goal of reducing the difference between the first difference and the second difference, the parameters of the target large model are updated.

[0036] The advantage of this method is that it can quantitatively assess user preferences for answers by scoring high-quality answers and low-quality answers. Furthermore, based on this quantitative assessment of user preferences, the large language model can be fine-tuned to ensure that the output of the fine-tuned large language model better meets the expectations of human users.

[0037] The detailed process of this method is further explained below. Figure 2 A flow chart of a method for fine-tuning a large model according to an embodiment of this specification is given. Figure 2 Said method comprises at least the following steps:

[0038] Step S201: Obtain a first question and a first answer corresponding to the first question, obtain a first score for the first answer based on a first large model; determine whether the first answer is a high-quality answer based on the first score; if the first answer is a high-quality answer, obtain a second answer to the first question and a second score for the second answer based on the first large model and the first question, where the second score is lower than the first score;

[0039] Step S203: Determine a first difference based on the probability of obtaining a first answer after inputting the first question into the second largest model, and the probability of obtaining a second answer after inputting the first question into the second largest model; determine a second difference based on the first score and the second score; and update the parameters of the second largest model with the goal of reducing the difference between the first difference and the second difference.

[0040] First, in step S201, a first question and a first answer corresponding to the first question may be obtained, and a first score for the first answer may be obtained based on a first large model. Based on the first score, it is determined whether the first answer is a high-quality answer. If the first answer is a high-quality answer, a second answer to the first question and a second score for the second answer may be obtained based on the first large model and the first question, with the second score being lower than the first score. The specific method for obtaining the first question and the first answer may vary in different embodiments, and this specification does not limit this. In one embodiment, for example, a question input by a user and the answer to the question input by the large model may be obtained.

[0041] After obtaining the first question and the first answer, a first score for the first answer can be obtained based on the first large model. In one embodiment, for example, a prompt containing the first question, the first answer, and an instruction to score the first answer can be input into the first large model to obtain the first score. The first large model is a large language model used to score answers to questions and generate low-quality answers based on high-quality answers. In different embodiments, the first large model can be a large language model of different specific types or with different neural network structures, and this description does not limit this.

[0042] After obtaining the first score, whether the first answer is a high-quality answer can be determined based on the first score. In one embodiment, if the first score is higher than a preset threshold, the first answer can be determined to be a high-quality answer. In different specific embodiments, the value of the preset threshold can vary.

[0043] If the first answer is determined to be a high-quality answer, a second answer to the first question and a second score for the second answer can be obtained based on the first large model and the first question, and the second score is lower than the first score.

[0044] In different embodiments, the specific method for obtaining the second answer and the second score may vary. In one embodiment, the first question, the first answer, and a prompt for generating a low-quality answer based on the first question and the first answer may be input into the first large model to obtain the second answer; the first question, the second answer, and a prompt for scoring the second answer may be input into the first large model to obtain the second score for the second question and answer. The specific form or text contained in the prompt for scoring the second answer may vary in different embodiments and is not limited in this specification.

[0045] In another embodiment, the first question, the first answer, and an indicator word for generating a low-quality answer and a score for the low-quality answer based on the first question and the first answer may be input into the first large model to obtain a second answer and the second score.

[0046] Figure 3 FIG. 1 is a schematic diagram showing a method for obtaining training samples for fine-tuning a large model according to an embodiment of the present specification. Figure 3 As shown, for example, the first question x and the first answer y can be input into the first large model to obtain a first score s for the first answer. If s is greater than a preset threshold td, the first answer can be determined as a high-quality answer. , for the rating of high-quality answers Then, the first question x and the high-quality answer to question x can be , ratings for high-quality answers , and according to x, 、 Generate a low-quality answer to x and a prompt word indicating the score of the low-quality answer, input it into the first large model, and obtain a low-quality answer to x and for Rating In the subsequent steps, the training sample pairs (x, , , , ) Fine-tune large language models.

[0047] Then, in step S203, a first difference can be determined based on the probability of obtaining the first answer after inputting the first question into the second large model, and the probability of obtaining the second answer after inputting the first question into the second large model. A second difference can also be determined based on the first score and the second score. The parameters of the second large model are updated with the goal of reducing the difference between the first and second differences. The second large model is the large language model that is desired to be fine-tuned. In different embodiments, the second large model can also be a large language model of a different specific type or having a different neural network structure.

[0048] The specific method for determining the first difference may vary in different embodiments. In one embodiment, the first value may be determined based on the logarithm of the ratio of the probability of receiving the first answer after inputting the first question into the second largest model to the probability of receiving the first answer after inputting the first question into the third largest model. The second value may be determined based on the logarithm of the ratio of the probability of receiving the second answer after inputting the first question into the second largest model to the probability of receiving the second answer after inputting the first question into the third largest model. The first difference is determined based on the first and second values. The third largest model is a large language model used to assist in retraining (fine-tuning) the second largest model. During the fine-tuning process of the second largest model, the parameters of the third largest model itself do not change.

[0049] In this embodiment, the first difference is determined based on the first value determined by the logarithmic value of the ratio of the probability of obtaining the first answer after inputting the first question into the second largest model to the probability of obtaining the first answer after inputting the first question into the third largest model, and the second value determined by the logarithmic value of the ratio of the probability of obtaining the second answer after inputting the first question into the second largest model to the probability of obtaining the second answer after inputting the first question into the third largest model. While improving the output quality of the second largest model for complex problems, the overfitting problem of the fine-tuned second largest model for the fine-tuned samples can be avoided, thereby avoiding the problem of the second largest model's output quality for basic problems being reduced. In addition, the second and third largest models output the logarithmic value of the probability rather than the probability value itself, which can avoid directly calculating the probability value, thereby reducing computing consumption and improving reasoning speed.

[0050] After the first difference, a second difference can be determined based on the first score and the second score. Furthermore, the parameters of the second large model can be updated with the goal of reducing the difference between the first difference and the second difference. In a specific embodiment, the parameters of the second large model can be updated with the goal of reducing the expected value of the square of the deviation between the first difference and the second difference.

[0051] Figure 4 FIG. 1 shows a schematic diagram of a fine-tuned large model according to an embodiment of the present specification. Figure 4 As shown, the first question x can be input into the second and third largest models, and the second largest model outputs a high-quality answer The logarithm of the probability and the second largest model outputs low-quality answers The logarithm of the probability , and the third largest model outputs high-quality answers The logarithm of the probability and the third largest model outputs low-quality answers The logarithm of the probability .in, Represents the second largest model output The probability of Represents the third largest model output The probability of Represents the second largest model output The probability of Represents the third largest model output The probability of represents the logarithmic function. Then, the first difference can be calculated ,in Represents a hyperparameter. You can also Rating and for low-quality answers Rating , and get the second difference ( ). For example, according to , calculated Afterwards, the training loss can be determined based on the first difference and the second difference, and the parameters of the second largest model can be updated based on the training loss. In a specific example, the training loss can be expressed as:

[0052]

[0053] in, is the training loss, Indicates expected value.

[0054] Furthermore, the specific method for updating the parameters of the large model based on the training loss may vary in different embodiments. In one embodiment, all parameters of the second large model may be updated using a backpropagation algorithm based on the training loss. In another embodiment, some parameters of the second large model may be updated based on the training loss, such as one or more parameters of newly added layers, newly added low-rank parameters, or parameters of some existing layers.

[0055] According to yet another embodiment, a device for fine-tuning a large model is provided. Figure 5 A structural diagram of a device for fine-tuning a large model according to an embodiment of this specification is shown, Figure 5 As shown, the apparatus 500 includes:

[0056] The acquisition unit 502 is configured to acquire a first question and a first answer corresponding to the first question, obtain a first score for the first answer based on a first large model, determine whether the first answer is a high-quality answer based on the first score, and if the first answer is a high-quality answer, obtain a second answer to the first question and a second score for the second answer based on the first large model and the first question, where the second score is lower than the first score;

[0057] The first processing unit 504 is configured to determine a first difference based on a probability of obtaining a first answer after inputting the first question into the second largest model and a probability of obtaining a second answer after inputting the first question into the second largest model, and to determine a second difference based on the first score and the second score; and to update the parameters of the second largest model with the goal of reducing the difference between the first difference and the second difference.

[0058] Another aspect of the embodiments of this specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute any one of the above methods.

[0059] On the other hand, the embodiments of this specification provide a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the above methods is implemented.

[0060] It should be understood that the descriptions such as “first” and “second” in this article are only used to distinguish similar concepts for the sake of simplicity of description and do not have any other limiting effect.

[0061] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using physical hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using software called a "logic compiler." This is similar to the software compilers used during program development. Before compilation, the original code must be written in a specific programming language, called a Hardware Description Language (HDL). There are many types of HDL, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that simply by programming a method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0062] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller purely in computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0063] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0064] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is only one way of executing the steps among many, and does not represent the only execution order. When a device or terminal product is actually executed, the method sequence shown in the embodiments or figures can be executed sequentially or in parallel (for example, in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, product, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, product, or device. Without further limitation, it does not exclude the presence of other identical or equivalent elements in the process, method, product, or device that includes the elements. For example, if words such as first and second are used to indicate names, they do not indicate any particular order.

[0065] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0066] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0069] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0070] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0071] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0072] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as methods, systems, or computer program products. Thus, one or more embodiments of this specification may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0073] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0074] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0075] The foregoing description is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. Those skilled in the art will appreciate that various modifications and variations of one or more embodiments of this specification are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification are intended to be included within the scope of the claims.

Claims

1. A method for fine-tuning a large model, comprising: Obtaining a first question and a first answer corresponding to the first question, and obtaining a first score for the first answer based on a first model; determining whether the first answer is a high-quality answer based on the first score, and if the first answer is a high-quality answer, inputting the first question, the first answer, and an indicator word for generating a low-quality answer and a score for the low-quality answer based on the first question and the first answer into a first large model to obtain a second answer and a second score, wherein the second score is lower than the first score; A first difference is determined based on the probability of obtaining a first answer after inputting the first question into the second largest model, and a probability of obtaining a second answer after inputting the first question into the second largest model, and a second difference is determined based on the first score and the second score; and the parameters of the second largest model are updated with the goal of reducing the difference between the first difference and the second difference.

2. The method according to claim 1, wherein Determining a first difference based on a probability of obtaining a first answer after inputting the first question into the second largest model and a probability of obtaining a second answer after inputting the first question into the second largest model includes: The first value is determined based on the logarithm of the ratio of the probability of obtaining the first answer after inputting the first question into the second largest model to the probability of obtaining the first answer after inputting the first question into the third largest model; the second value is determined based on the logarithm of the ratio of the probability of obtaining the second answer after inputting the first question into the second largest model to the probability of obtaining the second answer after inputting the first question into the third largest model; and the first difference is determined based on the first value and the second value.

3. The method according to claim 1, wherein With the goal of reducing the difference between the first difference and the second difference, updating the parameters of the second largest model includes: The parameters of the second largest model are updated with the goal of reducing the expected value of the square of the deviation between the first difference and the second difference.

4. The method according to claim 1, wherein Determining whether the first answer is a high-quality answer according to the first score includes: If the first score is higher than a preset threshold, the first answer is determined to be a high-quality answer.

5. The method according to claim 1, wherein The first question is a question indicating that an answer is obtained based on a multi-step process or logical reasoning, or a question associated with a specified context.

6. A device for fine-tuning a large model, the device comprising: an acquiring unit configured to acquire a first question and a first answer corresponding to the first question, and obtain a first score for the first answer based on a first large model; determining whether the first answer is a high-quality answer based on the first score, and if the first answer is a high-quality answer, inputting the first question, the first answer, and an indicator word for generating a low-quality answer and a score for the low-quality answer based on the first question and the first answer into a first large model to obtain a second answer and a second score, wherein the second score is lower than the first score; The fine-tuning unit is configured to determine a first value based on the probability of obtaining a first answer after inputting the first question into the second largest model, and determine a second value based on the probability of obtaining a second answer after inputting the first question into the second largest model; determine a first difference based on the first value and the second value, and determine a second difference based on the first score and the second score; and update the parameters of the second largest model with the goal of reducing the difference between the first difference and the second difference.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.

8. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 5 is implemented.

Images