Model alignment method and device

Abstract

The embodiment of the invention provides a model alignment method and equipment. The method comprises the following steps: determining a reward value corresponding to each candidate output in a group of candidate outputs and a relative advantage value in the group based on a reward model; based on the reward value corresponding to each candidate output, the intra-group relative advantage value and the strategy model, determining the alignment loss of the large model; through the standard alignment loss item in the alignment loss, when the reward value of the candidate output is smaller than the preset reward threshold value, the strategy model is biased to the learning standard output, the unification of a supervised fine adjustment target and a reinforcement learning target can be realized in the model alignment process, and the reasoning ability of a large model is improved.

Description

Technical Field

[0001] This specification relates to the field of large model technology, and in particular to a model alignment method and electronic equipment. Background Art

[0002] With the development of artificial intelligence technology, the application of LLMs (Large Language Models) is becoming more and more widespread. Since model alignment can significantly improve the reasoning ability of large models, how to align large models has become a focus of attention.

[0003] In related technical solutions, large models are aligned based on reinforcement learning, for example, through RLHF (Reinforcement Learning with Human Feedback), so that the output of the large model conforms to human preferences. However, in this technical solution, reinforcement learning adjusts model behavior by optimizing the policy model. Because reinforcement learning relies on a reference model, it is difficult to reason beyond the existing capabilities of the reference model, which limits the reasoning ability of the large model. For example, if the reference model itself lacks the knowledge to solve a certain type of problem, it will be difficult to generate a correct response even through reinforcement learning optimization.

[0004] Therefore, how to improve the reasoning ability of large models has become a technical problem that needs to be solved urgently.

[0005] The content of the background technology section is merely information known to the inventor personally, and does not mean that the above information has entered the public domain before the filing date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the Invention

[0006] This specification provides a model alignment method and electronic device, which can unify the goals of supervised fine-tuning and reinforcement learning during the model alignment process, thereby improving the reasoning ability of large models.

[0007] In a first aspect, this specification provides a model alignment method applied to a large model, wherein the large model includes a policy model and a reward model, and the method includes:

[0008] For each input query, generating a set of candidate outputs using the policy model;

[0009] Determining a reward value corresponding to each of the candidate outputs based on the reward model;

[0010] Determine the relative advantage value within the group corresponding to each candidate output based on the reward value corresponding to each candidate output;

[0011] determine, based on the reward value corresponding to each of the candidate outputs, the relative advantage value within the group, and the policy model, an alignment loss of the large model, the alignment loss comprising a standard alignment loss term for causing the policy model to be biased to learn a standard output when the reward value of the candidate output is less than a preset reward threshold; and

[0012] adjust model parameters of the policy model based on the alignment loss.

[0013] In some example embodiments, based on the above scheme, the alignment loss further comprises a policy change term for measuring a change of a new policy function corresponding to the policy model compared to an old policy function corresponding to the policy model, and the standard alignment loss term comprises a standard policy function corresponding to the policy model for causing the policy model to be biased to learn a standard output when the reward value of the candidate output is less than or equal to a preset reward threshold.

[0014] In some example embodiments, based on the above scheme, the determining, based on the reward value corresponding to each of the candidate outputs, the relative advantage value within the group, and the policy model, an alignment loss of the large model comprises:

[0015] determining the policy change term based on the relative advantage value within the group corresponding to the candidate output, and a policy ratio of the new policy function to the old policy function; and

[0016] determining the standard alignment loss term based on the reward value, the size of the preset reward threshold, and the standard policy function.

[0017] In some example embodiments, based on the above scheme, the policy change term further comprises a clipping term for limiting a change of the policy ratio when the policy ratio is greater than a predetermined range, and the determining the policy change term based on the relative advantage value within the group corresponding to the candidate output, and a policy ratio of the new policy function to the old policy function comprises:

[0018] if the policy ratio is greater than the predetermined range, taking an upper limit of the predetermined range as a target policy ratio, and determining the policy change term based on the relative advantage value within the group and the target policy ratio;

[0019] if the policy ratio is less than the predetermined range, taking a lower limit of the predetermined range as the target policy ratio, and determining the policy change term based on the relative advantage value within the group and the target policy ratio.

[0020] In some example embodiments, based on the above scheme, the determining the standard alignment loss term based on the reward value of the candidate output, the size of the preset reward threshold, and the standard policy function comprises:

[0021] If the reward value of the candidate output is less than or equal to the preset reward threshold, the standard alignment loss term is determined based on the reward value corresponding to the candidate output and the standard policy function; and

[0022] If the reward value of the candidate output is greater than the preset reward threshold, the standard alignment loss term is 0.

[0023] In some example embodiments, based on the above scheme, the standard alignment loss term comprises a policy ratio term of a new policy function corresponding to the policy model and a standard policy function corresponding to the policy model, and the standard alignment loss term is used to make the policy model learn the standard output when the reward value of the candidate output is less than or equal to the preset reward threshold, and make the policy model learn the candidate output when the reward value of the candidate output is greater than the preset reward threshold.

[0024] In some example embodiments, based on the above scheme, the determining the alignment loss of the policy model based on the reward value corresponding to each candidate output, the intra-group relative advantage value, and the policy model comprises:

[0025] Determining a weight of the policy ratio term corresponding to each candidate output based on the size of the reward value corresponding to the candidate output and the preset reward threshold and the intra-group relative advantage value;

[0026] Performing a weighted operation on the policy ratio term based on the weight of the policy ratio term to determine the alignment loss of the policy model.

[0027] In some example embodiments, based on the above scheme, the determining the weight of the policy ratio term corresponding to each candidate output based on the size of the reward value corresponding to the candidate output and the preset reward threshold and the intra-group relative advantage value comprises:

[0028] If the reward value corresponding to the candidate output is greater than the preset reward threshold, the weight of the policy ratio term is determined based on the intra-group relative advantage value corresponding to the candidate output;

[0029] If the reward value corresponding to the candidate output is less than or equal to the preset reward threshold, the weight of the policy ratio term is determined based on a negative value of the intra-group relative advantage value corresponding to the candidate output.

[0030] In some example embodiments, based on the above scheme, the determining, for each of the candidate outputs, the in-group relative advantage value corresponding to the candidate output based on the reward value corresponding to the candidate output comprises:

[0031] determining, for the group of candidate outputs, a reward mean value and a reward standard deviation based on the reward value corresponding to each of the candidate outputs;

[0032] determining, for each of the candidate outputs, the in-group relative advantage value corresponding to the candidate output based on the reward value corresponding to the candidate output, the reward mean value, and the reward standard deviation.

[0033] In some example embodiments, based on the above scheme, the alignment loss further comprises a distribution difference loss, the distribution difference loss being configured to constrain a distribution difference between the new policy function of the policy model and the reference policy function of the reference model.

[0034] In some example embodiments, based on the above scheme, the method further comprises:

[0035] averaging the alignment loss corresponding to each of the candidate outputs in the group of candidate outputs to determine an expected value of the alignment loss corresponding to the input query.

[0036] In a second aspect, the present specification also provides an electronic device, comprising: at least one storage medium storing at least one instruction set for performing model alignment processing; and at least one processor in communication connection with the at least one storage medium, wherein when the electronic device is running, the at least one processor reads the at least one instruction set, and executes the model alignment method according to the indication of the at least one instruction set.

[0037] According to the technical solution, the model alignment method and device provided by the embodiments of the present specification can, on the one hand, determine the reward value corresponding to each candidate output in a group of candidate outputs and the relative superiority value within the group based on the reward model, and can perform reinforcement learning on the strategy model based on the relative superiority value within the group; on the other hand, the strategy model is biased towards learning the standard output when the reward value of the candidate output of the strategy model is less than the preset reward threshold through the standard alignment loss term, which can enable the strategy model to perform supervised learning according to the standard output when the inference result is problematic, and adjust the learning direction of the sampling space explored by the strategy model according to the size of the reward value; on the other hand, the alignment loss of the large model is determined based on the reward value corresponding to each candidate output, the relative superiority value within the group and the strategy model, and since the respective advantages of the efficient knowledge injection of supervised fine-tuning SFT and the active exploration of reinforcement learning RL are combined, the inference ability beyond the capability range of the reference model is learned while the exploration of the sampling space of the model is maintained, the unification of the goals of supervised fine-tuning SFT and reinforcement learning RL is achieved in the model alignment process, and the inference ability of the large model is improved.

[0038] Other functions of the model alignment method and device provided by the present specification will be partially listed in the following description. According to the description, the following numbers and examples will be apparent to those of ordinary skill in the art. The creative aspects of the model alignment method and device provided by the present specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present specification, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present specification, and those of ordinary skill in the art can also obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 A schematic diagram of an implementation environment of a model alignment method provided by an embodiment of the present specification is shown;

[0041] Figure 2 A hardware structure schematic diagram of an electronic device 200 provided according to an embodiment of the present specification is shown;

[0042] Figure 3 A flowchart schematic diagram of a model alignment method provided according to some embodiments of the present specification is shown;

[0043] Figure 4 A flowchart schematic diagram of a model alignment method provided according to some embodiments of the present specification is shown; and

[0044] Figure 5 A flow diagram illustrating a model alignment method according to some embodiments of the present specification is shown. DETAILED DESCRIPTION

[0045] The following description provides specific applications and requirements of the present specification, which is intended to enable a person skilled in the art to manufacture and use the contents of the present specification. Various modifications to the disclosed embodiments are apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the present specification. Therefore, the present specification is not limited to the embodiments shown, but is consistent with the widest scope of the claims.

[0046] The terms used herein are used only for the purpose of describing specific example embodiments, and are not limiting. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an", and "the" can also include the plural forms. When used in the present specification, the terms "comprise", "include" and / or "contain" mean that the associated integer, step, operation, element and / or component exists, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components and / or groups.

[0047] These features of the present specification and other features, as well as the operation and function of related elements of the structure, and the economy of combination and manufacture of components can be significantly improved in view of the following description. Referring to the drawings, all of which form part of the present specification. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of the present specification. It should also be understood that the drawings are not drawn to scale.

[0048] The flowchart used in the present specification shows the operation of the system implementation according to some embodiments of the present specification. It should be clearly understood that the operations of the flowchart can not be implemented in sequence. On the contrary, the operations can be implemented in reverse order or simultaneously. In addition, one or more other operations can be added to the flowchart. One or more operations can be removed from the flowchart.

[0049] First, the nomenclature related to one or more embodiments of the present specification is explained.

[0050] Large Language Model (LLM): that is, a large model, a large model is a neural network model capable of understanding and generating natural language text obtained by training on a large amount of text data, such as a neural network model based on Transformer, a ChatGPT model or a DeepSeek model, etc.

[0051] Learning: In large model training, refers to the way of knowledge injection, such as SFT (Supervised fine-tuning) training method.

[0052] SFT (Supervised fine-tuning): refers to using labeled data to adjust a pre-trained large model to make it more suitable for a specific task, so as to improve its performance on the task.

[0053] RL (Reinforcement Learning) is a machine learning paradigm whose core idea is to let machine learning models such as large models learn optimal decision-making strategies through interaction with the environment to maximize cumulative rewards.

[0054] Exploration: In large model training, it refers to the training method that stimulates the inference ability of the model, such as RL (Reinforcement Learning) training method.

[0055] Alignment: generally refers to aligning the output of the large model to the direction expected by humans, and the process usually includes several supervised fine-tuning SFT and reinforcement learning RL processes.

[0056] In order to enable the LLM model to complete the specific task of the downstream, it is necessary to align the pre-trained LLM model. In the related technical solution, the LLM model is aligned in stages, for example, the stages include the supervised learning-based fine-tuning stage SFT and the reinforcement learning RL-based fine-tuning stage. The alternation between the supervised fine-tuning SFT and the reinforcement learning RL process can enhance the inference ability of the model through knowledge injection and reinforcement exploration. However, in this technical solution, the supervised fine-tuning SFT has poor generalization effect due to supervised training, and the reinforcement learning RL relies on the reference model, making it difficult to break through the existing ability range of the reference model for inference. Therefore, how to realize the unification of the goals of supervised fine-tuning SFT and reinforcement learning RL in the model alignment process has become a technical problem to be solved.

[0057] Based on the above, the model alignment method and the electronic device provided by the embodiments of the present specification can determine the reward value corresponding to each candidate output in a group of candidate outputs and the relative superiority value within the group based on the reward model, and can perform reinforcement learning on the policy model based on the relative superiority value within the group; on the other hand, the policy model is biased towards learning the standard output when the reward value of the candidate output of the policy model is less than the preset reward threshold through the standard alignment loss term, which can enable the policy model to perform supervised learning according to the standard output when the inference result is problematic, and adjust the learning direction of the sampling space explored by the policy model according to the size of the reward value; on the other hand, based on the reward value corresponding to each candidate output, the relative superiority value within the group and the policy model, the alignment loss of the large model is determined, which combines the respective advantages of the efficient knowledge injection of supervised fine-tuning SFT and the active exploration of reinforcement learning RL, while learning the inference ability beyond the capability range of the reference model, it maintains the exploration of its own sampling space, and realizes the unification of the supervised fine-tuning SFT goal and the reinforcement learning RL goal in the model alignment process, and improves the inference ability of the large model.

[0058] In the following, the technical solutions of the embodiments of the present specification will be described in detail with reference to the accompanying drawings.

[0059] Figure 1 A schematic diagram of an implementation environment of a model alignment method provided by an embodiment of the present specification is shown.

[0060] Referring to Figure 1 As shown in the figure, the implementation environment 100 can include a terminal 110, a server 130 and a database 140.

[0061] The terminal 110 is connected to the server 130 through a wireless network or a wired network 120. The terminal 110 can be a tablet computer, a notebook computer or a desktop computer, but is not limited thereto.

[0062] The terminal 110 can store data or instructions for executing the model alignment method described in the present specification. The terminal 110 can include a hardware device with data information processing capability and the necessary programs required to drive the hardware device to work.

[0063] The server 130 is a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The server 130 provides background services for the application programs running on the terminal 110.

[0064] The server 130 is equipped with an integrated development platform. The integrated development platform, also known as an integrated development environment (IDE), is an application program for providing a program development environment, generally including a code editor, a compiler, a debugger, and a graphical user interface, etc. A developer can write program code (i.e., program development) on the integrated development platform. The integrated development platform server can be a computing device specially used for implementing the model alignment method by the integrated development platform. The server 130 can respectively communicate data with the terminal 110 and the database 140.

[0065] In addition, the server 130 can store data or instructions for executing the model alignment method described in the present specification. The server 130 can include a hardware device with data information processing capability and necessary programs required to drive the hardware device to work. Of course, the server 130 can also be only a hardware device with data processing capability, or only a program running in the hardware device. In some embodiments, the server 130 can also be a plug-in and deployed in the terminal 110, at this time, the server 130 stores data or instructions for executing the model alignment method corresponding to the terminal 110 described in the present specification.

[0066] The database 140 can store data and / or instructions. In some embodiments, the database 140 can store tool features generated by the Agent proxy corresponding to tool descriptions of a plurality of tools, etc. In some embodiments, the database 140 can store data and / or instructions for the server 130 to execute or use to execute the model alignment method described in this specification. The terminal 110 and the server 130 have access to the database 140, and the terminal 110 and the server 130 can access data or instructions stored in the database 140 through a network. In some embodiments, the database 140 can be directly connected to the terminal 110 and the server 130. In some embodiments, the database 140 can be part of the server 130. In some embodiments, the database 140 can include a mass storage, a removable storage, a volatile read-write memory, a read-only memory (ROM), or the like, or any combination thereof. An exemplary mass storage can include a non-transitory storage medium such as a magnetic disk, an optical disk, a solid-state drive, etc. An exemplary removable storage can include a flash drive, a floppy disk, an optical disk, a memory card, a zip disk, a magnetic tape, etc. A typical volatile read-write memory can include a random access memory (RAM). An exemplary RAM can include a dynamic RAM (DRAM), a double date rate synchronous dynamic RAM (DDR SDRAM), a static RAM (SRAM), a thyristor RAM (T-RAM), and a zero-capacitor RAM (Z-RAM), etc. An exemplary ROM can include a mask ROM (MROM), a programmable ROM (PROM), a virtually programmable ROM (PEROM), an electronically programmable ROM (EEPROM), a compact disk (CD ROM), and a digital versatile disk ROM, etc.

[0067] A person skilled in the art can know that the number of terminals described above can be more or less. For example, the terminal described above is only one, or the terminal described above is dozens or hundreds, or more, and other terminals are included in the implementation environment described above. The number of terminals and the type of equipment are not limited in the embodiments of the present specification.

[0068] After introducing the implementation environment of the embodiments of the present specification, the application scenarios of the embodiments of the present specification will be introduced in combination with the implementation environment described above. In the following description, the terminal is the terminal 110 in the implementation environment described above, and the server is the server 130 in the implementation environment described above. The technical solutions provided by the embodiments of the present specification can be applied in the scenario of model alignment of a large model, such as a chat large model, a shopping assistant large model, and a travel assistant large model, etc.

[0069] Taking the application of the technical solution provided in the embodiments of this specification in the model alignment scenario of the chat big model as an example, the chat big model includes a strategy model and a reward model, the reward model is a pre-trained model, and the strategy model is a model to be aligned and optimized, and the terminal 110 generates a set of candidate outputs for each input query through the strategy model; determines the reward value corresponding to each candidate output based on the reward model; determines the relative advantage value within the group corresponding to each candidate output based on the reward value corresponding to each candidate output; determines the alignment loss of the big model based on the reward value corresponding to each candidate output, the relative advantage value within the group and the strategy function of the strategy model, the alignment loss includes a standard alignment loss term, and the standard alignment loss term is used to make the strategy model learn the standard output when the reward value of the candidate output is less than or equal to the preset reward threshold, and to make the strategy model learn the candidate output when the reward value of the candidate output is greater than the preset reward threshold; and adjusts the model parameters of the strategy model based on the alignment loss.

[0070] It should be noted that the above is explained by taking the application of the technical solution provided in the embodiment of this specification in the model alignment scenario of the chat big model as an example. The technical solution provided in the embodiment of this specification can also be applied in other appropriate scenarios, such as the model alignment scenario of the financial assistant big model or the shopping assistant big model, etc. The implementation process belongs to the same inventive concept as the above description and will not be repeated here.

[0071] It should be noted that the steps in the model alignment method in the example embodiment of this specification can be partially executed by the client, partially executed by the server, or all executed by the server or all executed by the client, and this specification does not specifically limit this.

[0072] based on Figure 1 The implementation environment shown below will be combined with Figures 2-5 , the model alignment method and electronic device provided in the embodiments of this specification are described in detail. It should be noted that the above implementation environment is only provided to facilitate understanding of the spirit and principles of this specification, and the embodiments of this specification are not limited in this respect. On the contrary, the embodiments of this specification can be applied to any applicable scenario.

[0073] Figure 2 This is a schematic diagram of the structure of an electronic device 200 provided according to some embodiments of this specification. The electronic device 200 can execute the model alignment method described in this specification. The model alignment method is introduced in other parts of this specification. The electronic device 200 can be a general-purpose computer or a special-purpose computer. For example, the electronic device 200 can be a server, a personal computer, a portable computer (such as a notebook computer, a tablet computer, etc.), or other electronic devices with computing capabilities. Of course, the electronic device can be Figure 1The terminal 110 and / or the server 130 can also be a terminal device used by a plurality of developers to develop programs on an integrated development platform.

[0074] The electronic device in the present specification can include one or more of the following components: a processor 210, a memory 220, an input device 230, an output device 240, and a bus 250. The processor 210, the memory 220, the input device 230, and the output device 240 can be connected through the bus 250.

[0075] The processor 210 can include one or more processing cores. The processor 210 connects various parts within the entire electronic device with various interfaces and lines, executes the model alignment method described in the present specification by running or executing instructions, programs, code sets, or instruction sets stored in the memory 220, and calls data stored in the memory 220. Alternatively, the processor 210 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 210 can be integrated with a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, the user interface, and the application program; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 210, but be realized by a separate communication chip.

[0076] The memory 220 can include random access memory (RAM) and can also include read-only memory (ROM). Optionally, the memory 220 includes a non-transitory computer-readable storage medium. The memory 220 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 220 can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the following method embodiments, etc., and the operating system can be an Android system, an IOS system, or other systems.

[0077] In order to enable the operating system to distinguish the specific application scenarios of the third-party application, it is necessary to open up the data communication between the third-party application and the operating system, so that the operating system can obtain the current scenario information of the third-party application at any time, and then perform targeted system resource adaptation based on the current scenario.

[0078] The input device 230 is configured to receive input instructions or data, and the input device 230 includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 240 is configured to output instructions or data, and the output device 240 includes but is not limited to a display device and a speaker. In one example, the input device 230 and the output device 240 can be combined, and the input device 230 and the output device 240 are a touch display screen.

[0079] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above-described drawings does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than those shown, or combine certain components, or different component arrangements. For example, the electronic device also includes radio frequency circuitry, an input unit, a sensor, audio circuitry, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, and the like, which are not described here.

[0080] Figure 3A flowchart of a model alignment method according to an embodiment of the present specification is shown. As before, the electronic device 200 can perform the model alignment method of the embodiments of the present specification. Specifically, the processor 210 can read the instruction set stored in its local storage medium, and then perform the model alignment method of the embodiments of the present specification according to the provisions of the instruction set. Below, steps S310 to S350 in the model alignment method will be described in detail in conjunction with the drawings.

[0081] Referring to Figure 3 As shown in step S310, for each input query, a set of candidate outputs is generated by the policy model of the large model.

[0082] In an example embodiment, the large model includes a policy model. The policy model is responsible for generating candidate outputs based on input queries, and the policy model is usually a pre-trained large language model, such as the DeepSeek model or the ChatGPT model. The electronic device 200 generates G candidate outputs {O1, O2, …, OG} through the policy model for each output query q, where G = G(q), and each candidate output can be an action sequence such as text, code, or a diagnosis report. Taking a medical large model as an example, the input query can be patient detection information, and the policy model of the medical large model generates multiple candidate diagnosis suggestions based on the patient detection information. G

[0083] For example, for each input query, the electronic device 200 generates multiple candidate outputs such as 4-16 candidate outputs by random sampling from the old policy (OldPolicy) model, and the multiple candidate outputs form a small group G. The candidate output generation process can be represented by the following formula (1):

[0084]

[0085] where Oi represents the i-th candidate output, G represents the number of grouping elements, i where fθ(q) represents the old policy function corresponding to the policy model, θ is the model parameter of the policy model, and q represents the input query.

[0086] Further, the diversity and quality of the candidate outputs can be controlled by adjusting the sampling parameters. For example, the electronic device 200 often uses techniques such as temperature sampling, Top-k sampling, or Nucleus sampling (Top-p) to avoid generating completely identical answers. Among them, the temperature parameter in temperature sampling is used to control the randomness of the generated output, the higher the temperature value, the more random the output; the lower the temperature value, the more concentrated the output. Top-k sampling selects the next token from the top k tokens with the highest probability, reducing long-tail noise. ​​

[0087] In step S320, the reward value corresponding to each candidate output is determined based on the reward model.

[0088] In an example embodiment, the large model includes a reward model. The reward model (RM) is a pre-trained model that evaluates the quality of generated outputs. The reward model is trained based on preference data (e.g., pairs of human-labeled good and bad answers) and outputs a scalar reward value Ri representing the quality of the candidate output O i The electronic device 200 inputs each candidate output O i into the reward model to obtain the reward value R i corresponding to the candidate output O i . The reward value guides the optimization of the policy model to generate higher-quality outputs.

[0089] The reward value can be determined based on the following factors: (1) correctness: whether the answer is correct (e.g., whether the result of a math problem is accurate); format: whether it conforms to a specific format (e.g., contains reasoning steps, labels, etc.); content quality: logicality, completeness, language fluency, etc. The reward function of the reward model can be flexibly designed, and common types include: correctness reward: based on task objectives (e.g., whether the result of a math problem is correct); format reward: check if the output conforms to a specific format (e.g., contains reasoning steps, labels, etc.); content quality reward: evaluate language fluency, logicality, etc.

[0090] The reward model is usually trained using preference data (e.g., pairs of human-labeled good and bad answers). The model structure of the reward model can be a classification model (outputting a binary classification label) or a regression model (outputting a continuous reward value). The reward model generates a reward value by integrating multiple reward signals, for example, the electronic device 200 inputs each candidate output O i into the reward model, and the reward model generates the reward value R i corresponding to the candidate output O i by weighted combination of multiple reward functions (e.g., correctness, format, length).

[0091] In step S330, the in-group relative advantage value corresponding to each candidate output is determined based on the reward value corresponding to each candidate output.

[0092] In an example embodiment, the group-wise relative advantage value represents a normalized reward value of the candidate output in the group, reflecting its advantage over the average performance in the group. Through group-wise comparison and normalization processing, the reward value corresponding to each candidate output is converted into a group-wise relative advantage value (Advantage), thereby providing a stable learning signal for policy optimization. The electronic device 200 performs normalization processing on the reward value of each candidate output based on the reward value corresponding to each candidate output, and determines the group-wise relative advantage value corresponding to each candidate output.

[0093] For example, the electronic device 200 determines a reward mean value and a reward standard deviation corresponding to a group of candidate outputs based on the reward value corresponding to each candidate output; and determines the group-wise relative advantage value corresponding to each candidate output based on the reward value corresponding to each candidate output, the reward mean value, and the reward standard deviation. As shown in the following formula (2):

[0094]

[0095] wherein Ai represents the group-wise relative advantage value corresponding to the i-th candidate output O i , R i is the reward value of the i-th candidate output, mean({R1, R2, …, R G}) and std({R1, R2, …, R G}) are respectively the reward mean value and the reward standard deviation of a group of reward values. Ai greater than 0 indicates that the reward of the candidate output is higher than the average in the group, and Ai less than 0 indicates that the reward value of the candidate output is lower than the average in the group. The normalized Ai reflects the relative advantages and disadvantages of the candidate output in the group.

[0096] Through group-wise comparison, the reward value is normalized to the group-wise relative advantage value Ai, and the policy update is guided by the group-wise relative advantage value, so that the model is more inclined to generate high-advantage outputs, rather than relying on the single reward value of the candidate output. The advantage value calculated through group-wise reward normalization reflects the advantages and disadvantages of a single candidate output relative to the average level in the group. The group-wise relative advantage value can replace the traditional value network, dynamically estimate the baseline through group-wise comparison, reduce the variance, and improve the training stability.

[0097] In step S340, based on the reward value corresponding to each candidate output, the group-wise relative advantage value, and the policy model, the alignment loss of the policy model is determined.

[0098] In an example embodiment, the alignment loss represents an objective function for optimizing the model parameters of the policy model. The alignment loss includes a standard alignment loss term for biasing the policy model to learn the standard output when the reward value of the candidate output is less than a preset reward threshold. The standard output represents a high-quality output labeled by an expert or a high-reward output filtered by the reward model score. The standard output serves as an "anchor point" for model learning, ensuring that low-reward outputs converge to the standard output. A reward threshold τ can be set according to historical data or dynamically adjusted strategy. Through the contrastive learning mechanism of the standard alignment loss term, low-reward outputs are aligned to the standard output, thereby improving the overall performance of the policy model.

[0099] The electronic device 200 determines the learning direction of the policy model based on the reward value corresponding to each candidate output, for example, biasing the policy model to learn the standard output when the reward value of the candidate output is less than or equal to the preset reward threshold, and biasing the policy model to learn the candidate output when the reward value of the candidate output is greater than the preset reward threshold. Based on the above learning direction, the relative advantage value within the group, and the policy function corresponding to the policy model, the alignment loss of the policy model is determined. In the following, the steps of determining the alignment loss will be described in detail in conjunction with the embodiments.

[0100] In some example embodiments, the alignment loss further includes a policy change term for measuring the change of the new policy function corresponding to the policy model compared to the old policy function corresponding to the policy model. The new policy function represents the policy function corresponding to the policy model after updating the model parameters of the policy model; the old policy function represents the policy function corresponding to the policy model before updating the model parameters of the policy model. Let the policy change term be K i,t , then refer to the following formula (3) shown:

[0101]

[0102] Wherein, K i,t represents the policy change term corresponding to the time step t of the candidate output O i , π θ represents the new policy function of the policy model after updating the parameters, represents the old policy function of the policy model before the reference update, o i,t represents the candidate output token corresponding to the time step t of the candidate output O i , t represents the position of the output token corresponding to the time step of the generated sequence.

[0103] Further, the standard alignment loss term includes a standard policy function corresponding to the policy model, and the standard policy function is used to make the policy model learn to favor the learning of the standard output when the reward value of the candidate output is less than the preset reward threshold. The electronic device 200 determines the policy change term based on the in-group relative advantage value corresponding to the candidate output, the policy ratio of the new policy function and the old policy function; and determines the standard alignment loss term based on the size of the reward value and the preset reward threshold and the standard policy function.

[0104] For example, if the reward value of the candidate output is less than or equal to the preset reward threshold, the electronic device 200 determines the standard alignment loss term based on the reward value corresponding to the candidate output and the standard policy function; and if the reward value of the candidate output is greater than the preset reward threshold, the standard alignment loss term is 0.

[0105] Referring to the following formula 4:

[0106]

[0107] wherein J GRAO (θ) represents the alignment loss or objective function of the large model; is an indicator function, which is 1 when , and 0 otherwise; π θ (y i,t | q, o i,<t ) represents the standard policy function; represents the in-group relative advantage value corresponding to the candidate output O i ; E represents the expected value, which is the average calculation of various possible cases; q represents the input query y represents the label, and P(Q, Y) represents the joint distribution of the input query and the label; O i represents the candidate output output from the old policy function, and G represents the number of grouping elements, represents the average of a group of candidate outputs, represents the average of multiple time steps of the candidate output Oi; |oi| represents the length (measured in token number) of the output sequence oi.

[0108] The above formula (4) represents the expected value of the alignment loss or objective function under the condition that the joint distribution of the input query and the label and a group of candidate outputs corresponding to the old policy function are given.

[0109] In some example embodiments, the policy change item further comprises a clipping item for limiting the change of the policy ratio if the policy ratio is greater than a predetermined range, wherein the electronic device 200 takes the upper limit of the predetermined range as a target policy ratio, and determines the policy change item based on the relative advantage value within the group and the target policy ratio if the policy ratio is greater than the predetermined range; and takes the lower limit of the predetermined range as the target policy ratio if the policy ratio is less than the predetermined range, and determines the policy change item based on the relative advantage value within the group and the target policy ratio.

[0110] For example, the clipping item is used to limit the clipping range of the change parameter ∈ of the policy ratio if the policy ratio is greater than a predetermined range [1-ε, 1+ε], and the clipping range can be taken as 0.1-0.2. If the policy ratio is greater than the predetermined range, the upper limit 1+ε of the predetermined range is taken as the target policy ratio; and if the policy ratio is less than the predetermined range, the lower limit 1-ε of the predetermined range is taken as the target policy ratio.

[0111] The clipping item limits the change range of the new and old policy probability ratio, and prevents instability caused by excessive policy update.

[0112] In some example embodiments, the standard alignment loss item comprises a policy ratio item of a new policy function corresponding to the policy model and a standard policy function corresponding to the policy model, and the standard alignment loss item is used to make the policy model biased to learn the standard output if the reward value of the candidate output is less than or equal to a preset reward threshold, and make the policy model biased to learn the candidate output if the reward value of the candidate output is greater than the preset reward threshold. Let K i,t For the policy ratio item, refer to the following formula (5):

[0113]

[0114] wherein, π θ (o i,t |q,o i,<t ) represents the new policy function corresponding to the policy model, π θ (y i,t |q,y i,<t ) represents the standard policy function corresponding to the policy model, and y i,t represents the label corresponding to the input query q.

[0115] Further, in the example embodiment, the electronic device 200 determines the weight of the policy ratio term corresponding to the candidate output based on the size of the reward value corresponding to the candidate output and the preset reward threshold value and the relative advantage value within the group; and performs weighted operation on the policy ratio term based on the weight of the policy ratio term to determine the alignment loss of the policy model. For example, if the reward value corresponding to the candidate output is greater than the preset reward threshold value, the electronic device 200 determines the weight of the policy ratio term based on the relative advantage value within the group corresponding to the candidate output; and if the reward value corresponding to the candidate output is less than or equal to the preset reward threshold value, the weight of the policy ratio term is determined based on the negative value of the relative advantage value within the group corresponding to the candidate output. Referring to the following formula (6) shown:

[0116]

[0117] wherein K i,t is the above-mentioned policy ratio term, I() is an indicator function, which is 1 when the condition in the bracket is met, and 0 otherwise; represents the relative advantage value within the group corresponding to the candidate output O i When , the weight of the policy ratio term is When , the weight of the policy ratio term is

[0118] Further, the electronic device 200 sums and averages the alignment loss corresponding to each candidate output in a group of candidate outputs to determine the expected value of the alignment loss corresponding to the input query. Referring to the above formula (4) and (6), the alignment loss corresponding to each candidate output Oi is obtained by summing and averaging each time step t of each candidate output in a group of candidate outputs; and the expected value of the alignment loss corresponding to the input query q is obtained by summing and averaging the alignment loss corresponding to each candidate output.

[0119] According to the technical solutions in the above example embodiments, when the number of samples is large enough, according to the Law of Large Numbers, the sample average value converges to the expected value, so that the expected reward can be maximized.

[0120] In step S350, the model parameters of the policy model are adjusted based on the alignment loss.

[0121] In the example embodiment, the electronic device 200 updates the model parameters θ of the policy model by updating the model parameters of the policy model by gradient descent based on the alignment loss, which can dynamically combine reinforcement learning and supervised learning according to the reward value. Under the condition that the reward value is higher than the predetermined reward threshold value, reinforcement learning is biased to maintain the diversity of generation; and under the condition that the reward value is lower than the predetermined reward threshold value, supervised learning is biased to quickly repair the model defects.

[0122] For example, the electronic device 200 derives the gradient of the model parameters θ by deriving the alignment loss, i.e., the objective function J GRAO (θ) to obtain the gradient of the model parameters θ, and updates the model parameters of the policy model by the gradient descent method. For example, the electronic device 200 updates the model parameters of the policy model based on the alignment loss using an Adam optimizer.

[0123] According to the technical solutions in the example embodiments, Figure 3 On the one hand, the technical solutions in the example embodiments can determine the reward value corresponding to each candidate output in the set of candidate outputs and the relative superiority value in the set based on the reward model, and can perform reinforcement learning on the policy model based on the relative superiority value in the set. On the other hand, the technical solutions in the example embodiments can make the policy model learn the standard output when the reward value of the candidate output of the policy model is less than the preset reward threshold through the standard alignment loss term, so that the policy model can perform supervised learning according to the standard output when the inference result is problematic, and can adjust the learning direction of the sampling space explored by the policy model according to the size of the reward value. On the other hand, the technical solutions in the example embodiments can determine the alignment loss of the large model based on the reward value corresponding to each candidate output, the relative superiority value in the set, and the policy model, which combines the respective advantages of the efficient knowledge injection of supervised fine-tuning SFT and the active exploration of reinforcement learning RL, learns the inference ability beyond the capability range of the reference model while maintaining the exploration of the sampling space of the model, and realizes the unification of the goals of supervised fine-tuning SFT and reinforcement learning RL in the model alignment process, thereby improving the inference ability of the large model.

[0124] Further, in the example embodiments, the alignment loss further includes a distribution difference loss, which is used to constrain the distribution difference between the new policy function of the policy model and the reference policy function of the reference model. The reference model, i.e., the reference policy, is usually the model after initial supervised fine-tuning (SFT) and is used for KL divergence constraint. The distribution difference loss can be KL divergence, which constrains the update amplitude of the policy model to ensure that the updated new policy function does not deviate too much from the reference policy function, thereby avoiding unstable behavior caused by excessive updates.

[0125] Figure 4 A flowchart of a model alignment method according to yet some embodiments of the present specification is shown.

[0126] Referring to FIG. 2, Figure 4 As shown in FIG. 2, the electronic device 200 inputs an input query q into the policy model to generate G candidate outputs {O1, O2, …, OG}. The input query q comes from a training data distribution P(Q, Y), which represents the joint distribution of the input query and the label y. For each input query q, G candidate outputs are sampled from the old policy function of the policy model.

[0127] The electronic device 200 inputs each candidate output O i to the reward model to obtain a reward value R i corresponding to each candidate output.

[0128] The electronic device 200 inputs the candidate output to the reference model to generate a reference output, and determines a reference policy function π ref based on the reference output of the reference model. The electronic device 200 determines a distribution difference, such as a KL divergence, between the reference policy function and a new policy function, and uses the KL divergence to constrain the update range of the policy model, so as to ensure that the updated new policy function does not deviate too much from the reference policy function, and avoid unstable behavior caused by excessive update.

[0129] Further, the alignment loss includes a policy change term and a standard alignment loss term. The policy change term is used to measure the change of the new policy function corresponding to the policy model compared with the old policy function corresponding to the policy model. The standard alignment loss term is used to make the policy model deviate to learn the standard output when the reward value of the candidate output is less than a preset reward threshold. The electronic device 200 determines the alignment loss, i.e., the target function, based on the policy change term, the standard alignment loss term, and the above-mentioned KL divergence. Let the policy change term be K i,t , and refer to the following formula:

[0130]

[0131] wherein J GRAO (θ) represents the alignment loss or the target function of the large model; K i,t is the above-mentioned policy ratio; is an indicator function, which is 1 when , and 0 otherwise; π θ (y i,t | q, o i,<t ) represents the standard policy function; clip(K i,t , 1-ε, 1+ε) represents a clipping term, which is used to limit the change of the policy ratio when the policy ratio is greater than a predetermined range [1-ε, 1+ε]; DKL is a distribution difference term, which is used to constrain the update range of the policy model, so as to ensure that the updated new policy function π θ does not deviate too much from the reference policy function π ref ; and β represents a KL regularization coefficient, which controls the degree of deviation of the policy from the reference model.

[0132] Referring to the following formula: Figure 4 when the reward value of the candidate output is less than the preset reward threshold, the policy model deviates to learn the standard output, and the policy ratio K When , it indicates that the model is not capable of answering the current input query q, then the supervised fine-tuning SFT target is started for knowledge injection, that is, the alignment loss term includes the standard alignment loss term -log(π θ (y i,t ∣q,o i,<t )). When the reward value of the candidate output When , it indicates that the model’s answer can meet the user’s preference requirements, then the standard alignment loss term is 0, that is, no knowledge injection is performed.

[0133] according to Figure 4 The technical solution in the example embodiment directly aligns the standard output, i.e., the standard preference, through the standard alignment loss term when the reward value of the candidate output is less than the preset reward threshold. This enables the policy model to perform supervised learning based on the standard output when problems occur in the inference result, and adjusts the learning direction of the sampling space explored by the policy model according to the size of the reward value. Due to the combination of the respective advantages of efficient knowledge injection of supervised fine-tuning SFT and active exploration of reinforcement learning RL, the reasoning ability of the large model is improved from learning the standard output output to surpassing the standard output output.

[0134] Figure 5 A flowchart of a model alignment method provided according to some other embodiments of this specification is shown.

[0135] Figure 5 The model alignment process is similar to Figure 4 The implementation principle and effect of the same steps in the model alignment process are similar and will not be described here. Figure 5 and Figure 4 The model alignment method is different. Figure 5 In an example embodiment, the standard alignment loss term in the alignment loss includes a policy ratio term of a new policy function corresponding to the policy model and a standard policy function corresponding to the policy model. The standard alignment loss term is used to bias the policy model toward learning the standard output when the reward value of the candidate output is less than or equal to a preset reward threshold, and to bias the policy model toward learning the candidate output when the reward value of the candidate output is greater than the preset reward threshold.

[0136] The electronic device 200 determines the weight of the strategy ratio item corresponding to the candidate output based on the difference between the reward value corresponding to each candidate output and the preset reward threshold and the relative advantage value within the group; performs a weighted operation on the strategy ratio item based on the weight of the strategy ratio item to determine the alignment loss of the strategy model. Let the strategy change item be K i,t , as shown below:

[0137]

[0138] Among them, π θ (oi,t |q,o i,<t ) represents the new policy function corresponding to the policy model, π θ (y i,t |q,y i,<t ) represents the standard policy function corresponding to the policy model, y i,t Indicates the label corresponding to the input query q; I() is an indicator function, which is 1 if the conditions in the brackets are met, otherwise it is 0; Represents the candidate output O i The corresponding relative advantage value within the group; when When , the weight of the strategy ratio term is when When , the weight of the strategy ratio term is

[0139] Referring to the above formula, through the G group and To determine the direction and degree of this optimization, for The variables after standardization or normalization obey the standard normal distribution. When the reward is greater than the predetermined threshold τ, the model's answer is considered to meet the user's preference requirements, and the optimization direction is biased towards the candidate output generated by the current model, with a degree of That is, the greater the relative advantage value within the group, the greater the weight of the strategy ratio item. When the reward is less than or equal to the predetermined reward threshold τ, the model's answer is considered to be unable to meet the user's preference requirements, and the optimization direction is biased towards the standard output, that is, the input corresponding label label, the degree is That is, the smaller the relative advantage value within the group, the greater the weight of the strategy ratio item, and the more important it is to learn the standard output.

[0140] according to Figure 5 Figure 5 The technical solution in the example embodiment can adaptively adjust the weights and directions of knowledge learning and exploration of the large model through alignment loss, thereby achieving a process of "learning-exploration-surpassing", and ultimately leaning towards the results generated by the model itself.

[0141] In another aspect of the present specification, there is provided a non-transitory storage medium storing at least one set of instructions for performing a model alignment process. When the above-mentioned executable instructions are executed by a processor, the above-mentioned executable instructions direct the processor to perform the steps of the model alignment method described above. In some possible embodiments, the aspects of the present specification can also be implemented in the form of a program product including a program code. When the above-mentioned program product is run on the electronic device 200, the above-mentioned program code is used to make the electronic device 200 perform the steps of the model alignment method described in the present specification. The program product for implementing the above-mentioned method can include a program code in a portable compact disc read-only memory (CD-ROM) and can be run on the electronic device 200. However, the program product of the present specification is not limited to this, and in the present specification, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system. The above-mentioned program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The above-mentioned computer readable storage medium can include a data signal propagating in a baseband or as a carrier wave in a propagated signal, in which the readable program code is borne. Such a propagated data signal can take on multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium can also be any readable medium that is not a storage medium that can send, propagate, or transmit the program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, and the like, or any suitable combination of the above. The program code for performing the operations of the present specification can be written in any combination of one or more programming languages, including an object-oriented programming language, such as Java, C++, and the like, and a conventional procedural programming language, such as the "C" programming language or similar programming languages. The program code can be executed entirely on the electronic device 200, partially on the electronic device 200, as a standalone software package, partially on the electronic device 200 and partially on a remote computing device, or entirely on a remote computing device.

[0142] The above described embodiments of the disclosure have been described. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still accomplish desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0143] In light of the above, those skilled in the art will appreciate that the foregoing detailed description of the present disclosure is susceptible to various modifications and / or revisions without departing from the spirit and scope of the present disclosure. Although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement which is calculated to achieve the same purpose can be substituted for the specific embodiments shown. This disclosure is intended to cover any adaptations or variations of the present disclosure. Therefore, it is intended that the application be protected by: the broadest interpretation of the appended claims to take into account unforeseen equivalents and alternatives based on current knowledge, or future knowledge.

[0144] In addition, certain terminology has been used to describe embodiments of the disclosure. For example, "one embodiment," "an embodiment," and / or "some embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. Therefore, it is understood that the use of "embodiment" or "one embodiment" or "an embodiment" or "some embodiments” in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner on one or more embodiments without limitation.

[0145] It should be understood that in the foregoing description of embodiments of the disclosure, various features are sometimes grouped together in a single embodiment, figure, or description of a figure for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various aspects, embodiments, and / or features. However, this should not be interpreted as a requirement that these features must be provided together in order to form an embodiment of the disclosure. In fact, some embodiments of the disclosure can provide only a subset of these features. In addition, streams of features from different embodiments can be combined to provide an embodiment of the disclosure. In some cases, features from one embodiment can be combined with features from another embodiment to provide an embodiment of the disclosure. In some cases, features from one embodiment can be combined with features from another embodiment to provide an embodiment of the disclosure.

[0146] Each patent, patent application, publication of a patent application, and other material, for example articles, books, specifications, publications, documents, things, etc. that are cited in this application can be indicative of the level of skill of those skilled in the art to which this application pertains, and can be considered by those skilled in the art to represent the state of this art prior to the filing date of this application. None is admitted to be prior art to the claimed application, except as may be otherwise specifically admitted by the inventor or may be otherwise specifically admitted by the inventor or may be otherwise specifically admitted by the inventor in the disclosure. All such materials are incorporated herein by reference to the same extent as if each individual publication, application, document, thing or item was specifically and individually indicated to be incorporated by reference.

[0147] Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the present application. Other modifications that fall within the scope of the present description can also be made. Accordingly, the disclosure of embodiments should not be construed as limiting the scope of the description.

Claims

1. A model alignment method, applied to a large model, wherein the large model includes a policy model and a reward model, the method comprising: For each input query, generating a set of candidate outputs using the policy model; Determining a reward value corresponding to each of the candidate outputs based on the reward model; Determine the relative advantage value within the group corresponding to each candidate output based on the reward value corresponding to each candidate output; Determining an alignment loss for the large model based on the reward value corresponding to each candidate output, the relative advantage value within the group, and the policy model, wherein the alignment loss includes a standard alignment loss term, and the standard alignment loss term is used to bias the policy model toward learning a standard output when the reward value of the candidate output is less than a preset reward threshold; as well as Model parameters of the policy model are adjusted based on the alignment loss.

2. The method according to claim 1, wherein The alignment loss also includes a policy change term, which is used to measure the change of the new policy function corresponding to the policy model compared to the old policy function corresponding to the policy model. The standard alignment loss term includes the standard policy function corresponding to the policy model, and the standard policy function is used to bias the policy model towards learning the standard output when the reward value of the candidate output is less than or equal to a preset reward threshold.

3. The method according to claim 2, wherein: The determining the alignment loss of the large model based on the reward value corresponding to each candidate output, the relative advantage value within the group, and the strategy model includes: Determining the policy change term based on the relative advantage value within the group corresponding to the candidate output and a policy ratio between the new policy function and the old policy function; and The standard alignment loss term is determined based on the magnitude of the reward value and the preset reward threshold and the standard policy function.

4. The method according to claim 3, wherein: The policy change term further includes a clipping term, wherein the clipping term is used to limit the change of the policy ratio when the policy ratio is greater than a predetermined range. The policy change term is determined based on the relative advantage value within the group corresponding to the candidate output and the policy ratio of the new policy function to the old policy function, including: If the strategy ratio is greater than the predetermined range, taking the upper limit of the predetermined range as the target strategy ratio, and determining the strategy change item based on the relative advantage value within the group and the target strategy ratio; If the strategy ratio is smaller than the predetermined range, the lower limit of the predetermined range is taken as the target strategy ratio, and the strategy change item is determined based on the relative advantage value within the group and the target strategy ratio.

5. The method according to claim 3, wherein The determining the standard alignment loss term based on the magnitude of the reward value and the preset reward threshold and the standard strategy function includes: If the reward value of the candidate output is less than or equal to the preset reward threshold, determining the standard alignment loss term based on the reward value corresponding to the candidate output and the standard policy function; and If the reward value of the candidate output is greater than the preset reward threshold, the standard alignment loss term is 0.

6. The method according to claim 1, wherein The standard alignment loss term includes a strategy ratio term between the new strategy function corresponding to the strategy model and the standard strategy function corresponding to the strategy model. The standard alignment loss term is used to bias the strategy model towards learning the standard output when the reward value of the candidate output is less than or equal to the preset reward threshold, and to bias the strategy model towards learning the candidate output when the reward value of the candidate output is greater than the preset reward threshold.

7. The method according to claim 6, wherein: The determining the alignment loss of the policy model based on the reward value corresponding to each candidate output, the relative advantage value within the group, and the policy model includes: Determining the weight of the strategy ratio item corresponding to each candidate output based on the magnitude of the reward value corresponding to each candidate output and the preset reward threshold and the relative advantage value within the group; A weighted operation is performed on the strategy ratio item based on the weight of the strategy ratio item to determine the alignment loss of the strategy model.

8. The method according to claim 7, wherein: The determining the weight of the strategy ratio item corresponding to each candidate output based on the difference between the reward value corresponding to each candidate output and the preset reward threshold and the relative advantage value within the group includes: If the reward value corresponding to the candidate output is greater than the preset reward threshold, determining the weight of the strategy ratio item based on the relative advantage value within the group corresponding to the candidate output; If the reward value corresponding to the candidate output is less than or equal to the preset reward threshold, the weight of the strategy ratio item is determined based on the negative value of the relative advantage value within the group corresponding to the candidate output.

9. The method according to any one of claims 1 to 8, wherein The determining, based on the reward value corresponding to each candidate output, the relative advantage value within the group corresponding to each candidate output, includes: Determine a reward mean and a reward standard deviation corresponding to the set of candidate outputs based on the reward value corresponding to each candidate output; The relative advantage value within the group corresponding to each candidate output is determined based on the reward value, the reward mean, and the reward standard deviation corresponding to each candidate output.

10. The method according to claim 9, wherein: The alignment loss also includes a distribution difference loss, which is used to constrain the distribution difference between the new policy function of the policy model and the reference policy function of the reference model.

11. The method according to claim 9, wherein The method further comprises: The alignment losses corresponding to the candidate outputs in the set of candidate outputs are summed and averaged to determine an expected value of the alignment loss corresponding to the input query.

12. An electronic device comprising: at least one storage medium storing at least one instruction set for performing model alignment processing; as well as at least one processor, in communication with the at least one storage medium; Wherein, when the electronic device is running, the at least one processor reads the at least one instruction set and executes the model alignment method according to any one of claims 1 to 11 according to the instructions of the at least one instruction set.

Images