Model training method, instruction processing method, electronic device and storage medium

Abstract

The present disclosure relates to the technical field of large models. Disclosed are a model training method, an instruction processing method, an electronic device and a storage medium. The model training method comprises: on the basis of an original instruction and a principle set associated with the original instruction, constructing an instruction set, wherein the principle set comprises positive principles and negative principles, the positive principles being used for instructing the generation of responses having a positive association with a target requirement included in the original instruction, and the negative principles being used for instructing the generation of responses having a reverse association with the target requirement; using an initial policy model to perform policy sampling on the instruction set, so as to obtain a response set; and on the basis of the instruction set and the response set, training the initial policy model, so as to obtain a target policy model, wherein the target policy model is used for performing policy sampling on the target instruction, so as to obtain a target response. The present disclosure solves the technical problems of a model alignment method in the relevant art of high costs and limitations in terms of generating high-accuracy and on-policy preference signals.

Description

Model training methods, instruction processing methods, electronic devices and storage media Technical Field

[0001] This disclosure relates to the field of large model technology, and more specifically, to a model training method, an instruction processing method, an electronic device, and a storage medium. Background Technology

[0002] In recent years, large language models (LLMs) have made significant progress in the field of natural language processing. Their powerful generation and understanding capabilities are being widely applied in various fields such as dialogue systems, text generation, and intelligent writing assistance. However, as the capabilities of LLMs increasingly approach or even surpass human levels, ensuring that the output responses of LLMs are consistent with human values ​​and preferences—that is, achieving model alignment—has become a critical problem that urgently needs to be solved.

[0003] Current model alignment methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on expensive human-labeled data, are costly, and have limitations in generating high-accuracy, on-policy preference signals.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This disclosure provides a model training method, instruction processing method, electronic device, and storage medium to at least address the technical problems of high cost and limitations in generating highly accurate and consistent preference signals in related technologies.

[0006] According to one aspect of the present disclosure, a model training method is provided, comprising: constructing an instruction set based on an original instruction and a set of principles associated with the original instruction, wherein the set of principles includes positive principles and negative principles, the positive principles being used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and the negative principles being used to indicate the generation of a response that is negatively correlated with the target requirement; performing policy sampling on the instruction set using an initial policy model to obtain a response set; and training the initial policy model based on the instruction set and the response set to obtain a target policy model, wherein the target policy model is used to perform policy sampling on target instructions to obtain target responses.

[0007] According to another aspect of the embodiments of this disclosure, an instruction processing method is also provided, including: acquiring a target instruction; sampling the target instruction using a target policy model to obtain a target response; wherein the target policy model is obtained by training an initial policy model based on a response set, the response set is obtained by sampling a comparison instruction using the initial policy model, and the comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction, the set of principles including positive principles and negative principles, positive principles being used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and negative principles being used to indicate the generation of a response that is negatively correlated with the target requirement.

[0008] According to another aspect of the embodiments of this disclosure, an instruction processing method is also provided, including: acquiring programming assistance instructions; performing programming analysis on the programming assistance instructions using a target programming assistance model to obtain programming assistance responses; wherein, the target programming assistance model is obtained by training an initial programming assistance model based on a response set, the response set is obtained by performing programming analysis on comparison instructions using the initial programming assistance model, and the comparison instructions are constructed based on the original instructions and a set of principles associated with the original instructions, the set of principles including positive principles and negative principles, positive principles being used to indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, and negative principles being used to indicate the generation of responses that are negatively correlated with the target requirements.

[0009] According to another aspect of the embodiments of this disclosure, an instruction processing method is also provided, comprising: obtaining an instruction processing request through a first application programming interface, wherein the request data carried in the instruction processing request includes a target instruction; and returning an instruction processing response through a second application programming interface, wherein the response data carried in the instruction processing response includes a target response, wherein the target response is obtained by sampling the target instruction using a target policy model, the target policy model is obtained by training an initial policy model based on a response set, the response set is obtained by sampling a comparison instruction using the initial policy model, and the comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction, the set of principles including positive principles and negative principles, the positive principles being used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and the negative principles being used to indicate the generation of a response that is negatively correlated with the target requirement.

[0010] According to another aspect of the embodiments of this disclosure, an instruction processing method is also provided, comprising: acquiring a currently input instruction processing dialogue request, wherein the request data carried in the instruction processing dialogue request includes: a target instruction; responding to the instruction processing dialogue request, returning an instruction processing dialogue response, wherein the information carried in the instruction processing dialogue response includes: a target response, wherein the target response is obtained by sampling the target instruction using a target policy model, the target policy model is obtained by training an initial policy model based on a response set, the response set is obtained by sampling a comparison instruction using the initial policy model, the comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction, the set of principles includes positive principles and negative principles, positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement; and displaying the target response in a graphical user interface.

[0011] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any of the above-described model training methods or instruction processing methods during runtime.

[0012] According to another aspect of the embodiments of this disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to execute any of the above-described model training method or instruction processing method.

[0013] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described model training methods or instruction processing methods.

[0014] In this embodiment, an instruction set including positive and negative principles is constructed based on the original instruction and the set of principles associated with it. The positive principles indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while the negative principles indicate that the generated response is negatively correlated with the target requirement. Then, an initial policy model is used to sample the policy from the instruction set to obtain a response set. The initial policy model is then trained based on the instruction set and the response set to obtain a target policy model. This target policy model is used to sample the policy from the target instruction to obtain the target response, thereby significantly improving the model alignment effect. This achieves low-cost training to obtain the target policy model. The target policy model improves the quality of generated data while aligning the model, exhibiting significant advantages in terms of data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0015] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0017] Figure 1 is a schematic diagram of an application scenario of a model training method according to an embodiment of the present disclosure;

[0018] Figure 2 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0019] Figure 3 is a schematic diagram of the G-function design according to an embodiment of the present disclosure;

[0020] Figure 4 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0021] Figure 5 is a flowchart of an instruction processing method according to an embodiment of the present disclosure;

[0022] Figure 6 is a flowchart of an instruction processing method according to an embodiment of the present disclosure;

[0023] Figure 7 is a flowchart of an instruction processing method according to an embodiment of the present disclosure;

[0024] Figure 8 is a flowchart of an instruction processing method according to an embodiment of the present disclosure;

[0025] Figure 9 is a schematic diagram of a model training device according to an embodiment of the present disclosure;

[0026] Figure 10 is a schematic diagram of another instruction processing apparatus according to an embodiment of the present disclosure;

[0027] Figure 11 is a schematic diagram of another instruction processing apparatus according to an embodiment of the present disclosure;

[0028] Figure 12 is a schematic diagram of another instruction processing apparatus according to an embodiment of the present disclosure;

[0029] Figure 13 is a schematic diagram of another instruction processing apparatus according to an embodiment of the present disclosure;

[0030] Figure 14 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] The technical solution disclosed herein is primarily implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models, also known as foundation models, are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models are adaptable to a wide range of downstream tasks and exhibit good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0034] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely applied in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the explanation will focus on instruction processing using a corresponding policy model in an instruction processing scenario.

[0035] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0036] Large Language Models (LLMs): Language models with a large number of parameters that can perform well on a variety of tasks.

[0037] Reinforcement Learning with Human Feedback (RLHF): A method that trains a reward model using human feedback and aligns the model using reinforcement learning.

[0038] Direct Preference Optimization (DPO): An alignment method that directly maximizes the probability gap between the selection and rejection responses.

[0039] Preference signals: Preference data used to guide model alignment, generally including a query, a better response, and a worse response.

[0040] On-policy: The data is highly consistent with the model's current policy, meaning the current model is capable of sampling this data.

[0041] Self-Steering Optimization (SSO): This disclosure presents a novel alignment method that improves model alignment by optimizing the generated preference signal.

[0042] Automatic alignment: Aligning models by generating preference data through automated methods, reducing reliance on human-labeled data.

[0043] The model alignment methods of related technologies have the following drawbacks.

[0044] Defect 1: Both the RLHF and DPO methods require a large amount of high-quality human-annotated data, which is costly. Furthermore, the RLHF method is difficult to scale, and the quality of the preference signal generated by the DPO method gradually declines, making it difficult to continuously improve the alignment effect. In other words, the model alignment methods of these techniques are costly and have limitations in generating high-accuracy and on-policy preference signals.

[0045] To address the aforementioned deficiencies, no effective solution has been proposed prior to this disclosure.

[0046] According to embodiments of this disclosure, a model training method is provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the model training method provided in this disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the large model, thereby implementing the method provided in this disclosure.

[0048] In this embodiment of the disclosure, the system consisting of a client device and a server can perform the following steps: the client device performs steps such as acquiring the original instructions for training the target policy model and the principle set associated with the original instructions input by the user in the graphical user interface, and sending the original instructions and the principle set associated with the original instructions to the server; the server performs steps such as sampling the instruction set based on the acquired original instructions and the principle set associated with the original instructions, using an initial policy model to obtain a response set, training the initial policy model based on the instruction set and the response set to obtain the target policy model, and returning the target policy model to the client device. It should be noted that, provided that the operating resources of the client device can meet the deployment and operation conditions of a large model, this embodiment of the disclosure can be performed on the client device.

[0049] Under the above operating environment, this disclosure provides a model training method as shown in Figure 2. Figure 2 is a flowchart of a model training method according to an embodiment of this disclosure. As shown in Figure 2, the method may include the following steps:

[0050] Step S21: Based on the original instruction and the set of principles associated with the original instruction, construct an instruction set. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of responses that are positively associated with the target requirements contained in the original instruction, and negative principles are used to indicate the generation of responses that are negatively associated with the target requirements.

[0051] Step S22: Use the initial policy model to sample the instruction set to obtain the response set;

[0052] Step S23: Train the initial policy model based on the instruction set and response set to obtain the target policy model, wherein the target policy model is used to sample the target instruction to obtain the target response.

[0053] In this embodiment of the disclosure, the original instruction can be understood as a given instruction input by the user, i.e., a query (Q). The original instruction may contain a target requirement, which can be understood as the user's specific requirement. That is, the original instruction can be understood as an instruction containing a target requirement input by the user to the model, and the user expects the model to return a response that can solve or achieve the target requirement based on the original instruction.

[0054] The set of principles associated with the original instruction can be understood as the principles most relevant to the query. This set includes the Good Principle and the Bad Principle. The Good Principle instructs the generation of responses that are positively correlated with the target requirements contained in the original instruction. In other words, the Good Principle ensures that the generated response is consistent with and has a positive relationship to the target requirements contained in the original instruction. That is, the Good Principle ensures that the generated response is helpful to the target requirements, providing a truly beneficial and specific response.

[0055] The negative principle is used to indicate that the generated response is inversely related to the target requirement. This can be understood as the generated response being the opposite of the target requirement in the original instruction. For example, if the target requirement is to seek a certain solution or expect a certain result, then a response guided by the negative principle may generate irrelevant or contrary-to-expectation content. In other words, the negative principle indicates that the generated response contradicts the target requirement, does not help the target requirement, and provides a vague, irrelevant, or unhelpful response.

[0056] Constructing an instruction set based on the original instruction and the set of principles associated with it can be understood as building the instruction set according to the original instruction input by the user and the positive and negative principles most relevant to that original instruction. For example, the instruction set includes the original instruction, as well as other instructions, such as comparison instructions, i.e., good and bad instructions constructed based on the positive and negative principles; this is not limited here.

[0057] The initial policy model can be understood as the policy model before model training. The policy model can determine the best action or decision to take based on the input state or conditions. For example, the initial policy model can be a large model or other deep learning models, which is not limited here.

[0058] An initial policy model is used to sample the instruction set to obtain a response set. This response set can include the original response corresponding to the original instruction, good principle responses corresponding to positive principles (i.e., good principle responses corresponding to good instructions), and bad principle responses corresponding to negative principles (i.e., bad principle responses corresponding to bad instructions). In other words, this embodiment can directly generate high-quality, diverse positive and negative responses based on the initial policy model.

[0059] The initial policy model is trained based on the instruction set and response set to obtain the target policy model. This can be understood as constructing training data based on the constructed instruction set and the obtained response set, training the initial policy model based on the constructed training data, adjusting the model parameters of the initial policy model, and thus obtaining the target policy model. In other words, the embodiments of this disclosure can generate high-quality, diverse positive and negative responses from the initial policy model to construct training data, significantly reducing training costs.

[0060] The target policy model is used to sample the target instruction to obtain the target response. In other words, the trained target policy model can sample the policy according to the input target instruction, thereby accurately obtaining a high-quality target response corresponding to the target instruction. That is, the target response output by the target policy model can be consistent with human preferences, thus achieving model alignment.

[0061] As can be seen, this disclosure proposes a self-steering optimization (SSO) method. Based on the original instruction and the set of principles associated with it, an instruction set including positive and negative principles is constructed. Positive principles indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that the generated response is negatively correlated with the target requirement. Then, an initial policy model is used to sample the policy from the instruction set to obtain a response set. The initial policy model is then trained based on the instruction set and the response set to obtain a target policy model. This target policy model is used to sample the policy from the target instruction to obtain the target response. Therefore, by using the SSO method proposed in this disclosure, multiple responses (i.e., a response set) are directly generated from the initial policy model using a set of principles highly correlated with the original instruction. These responses are then used to construct training data to train the initial policy model. This reduces reliance on human-labeled data, lowers training costs, and improves the quality of the generated data while aligning the model, thus significantly improving the model's alignment performance. It demonstrates clear advantages in terms of the quality of the generated data and its on-policy nature.

[0062] The model training method provided in this disclosure can be applied, but is not limited to, to application scenarios involving strategy model training in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, strategy model training for e-commerce services, strategy model training for education services, and strategy model training for medical services are not limited here.

[0063] This embodiment of the invention constructs an instruction set including positive and negative principles based on the original instruction and the set of principles associated with it. Positive principles indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that the generated response is negatively correlated with the target requirement. Then, an initial policy model is used to sample the policy from the instruction set to obtain a response set. The initial policy model is then trained based on the instruction set and the response set to obtain a target policy model. This target policy model is used to sample the policy from the target instruction to obtain the target response, thereby significantly improving the model alignment effect. This achieves low-cost training to obtain the target policy model. The target policy model improves the quality of generated data while aligning the model, exhibiting significant advantages in terms of data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0064] In an optional embodiment, in step S21, an instruction set is constructed based on the original instruction and the set of principles associated with the original instruction, including the following method steps:

[0065] Step S211: Construct comparison instructions based on the original instructions and the set of principles associated with the original instructions;

[0066] Step S212: Construct an instruction set using the original instruction and the comparison instruction.

[0067] In this embodiment of the disclosure, when constructing the instruction set based on the original instruction and the set of principles associated with the original instruction, comparison instructions can be constructed based on the original instruction and the set of principles associated with the original instruction. The comparison instructions can include good instructions and bad instructions, that is, good instructions constructed according to positive principles and bad instructions constructed according to negative principles.

[0068] Good instructions, constructed according to the positive principle, can be understood as instructions that enable the model to more accurately understand the task, pattern, or rule. For example, good instructions typically contain clear and specific information, thereby helping the model learn and generalize. Bad instructions, constructed according to the negative principle, can be understood as instructions that may mislead the model or cause the model to learn incorrect patterns. For example, bad instructions may contain noise, irrelevant information, or instructions that contradict the task objective.

[0069] After constructing the comparison instructions, an instruction set can be built based on the original instructions and the comparison instructions. This instruction set includes three types of instructions: original instructions, good instructions, and bad instructions. Therefore, by comparing the differences between different instructions, the initial policy model to be trained can be strengthened in its understanding of the meaning of the instructions, thus improving the model alignment effect.

[0070] In an optional embodiment, in step S211, a comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction, including the following method steps:

[0071] Step S2111: Generate a positive instruction based on the original instruction and the positive principle, and generate a negative instruction based on the original instruction and the negative principle;

[0072] Step S2112: Construct a comparison instruction using positive and negative instructions.

[0073] In this embodiment of the disclosure, when constructing comparison instructions based on the original instructions and the set of principles associated with the original instructions, positive instructions (i.e., good instructions) can be generated based on the original instructions and positive principles, and negative instructions (i.e., bad instructions) can be generated based on the original instructions and negative principles. Then, comparison instructions are constructed based on the positive and negative instructions.

[0074] Positive instructions can be understood as clear, explicit, and specific instructions that the model can easily understand and execute accurately. Negative instructions can be understood as vague, ambiguous, unclear, or containing erroneous information, making them difficult for the model to understand or execute. It is understandable that by comparing the responses to good and bad instructions, the model can learn how to identify and respond to high-quality instructions, while avoiding being misled by low-quality instructions, thereby helping to improve the model's ability to understand and execute instructions.

[0075] In an optional embodiment, in step S22, the comparison instructions are sampled using an initial policy model to obtain a response set, including the following method steps:

[0076] Step S221: Use the initial policy model to sample the original instruction to obtain the original response; use the initial policy model to sample the positive instruction to obtain the positive response; and use the initial policy model to sample the negative instruction to obtain the negative response.

[0077] Step S222: Determine the response set based on the original response, positive response, and negative response.

[0078] In this embodiment of the disclosure, when using an initial policy model to sample the comparison instructions to obtain a response set, the initial policy model can be used to sample the original instructions to obtain the original response, the initial policy model can be used to sample the positive instructions to obtain the positive response, and the initial policy model can be used to sample the negative instructions to obtain the negative response. Then, the response set is determined based on the obtained original response, positive response, and negative response.

[0079] Understandably, policy sampling using a policy model is essentially a process of extracting or generating an action or decision from the policy model. This process is based on the policy learned by the policy model, i.e., the instructions received by the policy model. During policy sampling, the policy model uses the received instructions as the basis for decision-making and then calculates the probability distribution of all possible actions or decisions in the current state. This probability distribution reflects the likelihood that the model believes each action or decision should be taken in the current state. Subsequently, based on the calculated probability distribution of actions or decisions, the policy model selects the action or decision with the higher probability, executes it, and outputs the corresponding response.

[0080] As can be seen, in the model training process, this disclosure helps the initial policy model learn how to distinguish different types of instructions and how to respond to different types of instructions by having the initial policy model sample different responses, thereby improving the quality and diversity of the generated data and ensuring that the generated responses are of high quality and diversity.

[0081] In an optional embodiment, in step S23, the initial policy model is trained based on the instruction set and the response set to obtain the target policy model, including the following method steps:

[0082] Step S231: Generate multiple preference pairs based on the instruction set and response set;

[0083] Step S232: Using multiple preference pairs, perform model alignment training on the initial policy model to obtain the target policy model.

[0084] In this embodiment of the disclosure, when training the initial policy model based on the instruction set and the response set to obtain the target policy model, multiple preference pairs can be generated based on the instruction set and the response set. For example, taking the generation of three preference pairs as an example, the three preference pairs may include a preference pair consisting of (original instruction, positive response, negative response) based on the original instruction, positive response, and negative response; a preference pair consisting of (good instruction, positive response, negative response) based on the good instruction, positive response, and negative response; and a preference pair consisting of (bad instruction, original response, positive response) based on the bad instruction, original response, and positive response.

[0085] Then, the training data is constructed using the multiple preference pairs generated above, and the initial policy model is trained to align the model, thereby obtaining the target policy model. This can improve the quality of the generated data while aligning the model, ensuring that the generated data has high accuracy and diversity, and thus significantly improving the alignment effect of the model.

[0086] In an optional embodiment, step S231 generates multiple preference pairs based on the instruction set and the response set, including the following method steps:

[0087] Step S2311: In response to the instruction set and response set satisfying preset conditions, a first preference pair is generated based on the original instruction, the positive response corresponding to the original response, and the negative response corresponding to the original response. A second preference pair is generated based on the positive instruction, the positive response, and the negative response. A third preference pair is generated based on the negative instruction, the original response, and the positive response, resulting in multiple preference pairs. The original response is the response corresponding to the original instruction. The preset conditions are used to indicate that the instruction set contains the original instruction, the positive instruction, and the negative instruction, and the response set contains the original response, the positive response, and the negative response.

[0088] In this embodiment of the disclosure, when generating multiple preference pairs based on the instruction set and the response set, if the instruction set contains the original instruction, the positive instruction corresponding to the original instruction, and the negative instruction corresponding to the original instruction, and the response set contains the original response corresponding to the original instruction, the positive response corresponding to the original response, and the negative response corresponding to the original response, then a first preference pair, namely (original instruction, positive response, negative response) preference pair, can be generated based on the original instruction, the positive response, and the negative response. This can also be denoted as (x, y). + y - ), where x represents the original instruction, y + Indicates a positive response, y - This indicates a negative response.

[0089] Simultaneously, a second preference pair can be generated based on positive instructions, positive responses, and negative responses, namely the (good instruction, positive response, negative response) preference pair, which can also be denoted as (x + y+ y - ), where x + This indicates a good instruction.

[0090] Furthermore, a third preference pair can be generated based on the negative instruction, the original response, and the positive response, namely the (bad instruction, original response, positive response) preference pair, which can also be denoted as (x - y o y + ), where y o This indicates the original response.

[0091] Thus, based on the first preference pair, the second preference pair, and the third preference pair mentioned above, multiple preference pairs are obtained.

[0092] In an optional embodiment, in step S232, multiple preference pairs are used to perform model alignment training on the initial policy model to obtain the target policy model, including the following method steps:

[0093] Step S2321: The preference optimization loss is calculated using the first preference pair, and the self-directed loss is calculated using the second and third preference pairs. The preference optimization loss is used to optimize the initial policy model to the direction of the positive response, and the self-directed loss is used to control the gap between the positive and negative responses.

[0094] Step S2322: Calculate the target loss based on the preference optimization loss and the self-guided loss;

[0095] Step S2323: Update the model parameters of the initial policy model based on the target loss to obtain the target policy model.

[0096] In this embodiment of the disclosure, when training the initial policy model using multiple preference pairs to align the model and obtain the target policy model, the preference optimization loss can be calculated using the first preference pair. The preference optimization loss is used to optimize the initial policy model towards a positive response direction, meaning the output response of the initial policy model is more biased towards a positive response. The preference optimization loss can be denoted as L. base (x, y) + y - ).

[0097] For example, the L function can be a regular DPO loss or an Identity Preference Optimization (IPO) loss, with IPO loss being more recommended, but there is no restriction here.

[0098] Simultaneously, the self-guided loss is calculated using the second and third preference pairs. This self-guided loss is used to control the difference between the positive and negative responses, i.e., to control y. + and y - The difference between them, the self-guided loss can be denoted as G.

[0099] Then, based on the calculated preference optimization loss and self-guiding loss, the target loss can be calculated, where the target loss can be denoted as L. sso .

[0100] Finally, the model parameters of the initial policy model are updated based on the target loss, thereby obtaining the target policy model.

[0101] In an optional embodiment, in step S2321, the self-guided loss is calculated using the second preference pair and the third preference pair, including the following method steps:

[0102] Step S23211: Obtain the first loss of the second preference pair and the second loss of the third preference pair;

[0103] Step S23212: The self-guiding loss is calculated using the first loss and the second loss.

[0104] In this embodiment of the disclosure, when calculating the self-guiding loss using the second preference pair and the third preference pair, a first loss of the second preference pair and a second loss of the third preference pair can be obtained, wherein the first loss of the second preference pair can be denoted as L. base (x + y + y - The second loss for the third preference pair can be denoted as L. base (x - y o y + ).

[0105] Then, the self-guiding loss is calculated using the first loss and the second loss, which is the self-guiding loss G.

[0106] It should be noted that this disclosure takes into account the use of x when designing the G function. + and x - The loss is constructed as an instruction, and its corresponding response is taken as a positive response, i.e., G = L base (x + y + y - )+L base (x - y - y +This form introduces a backdoor problem: with carefully crafted prompts, the initial policy model can easily be manipulated into undesirable behavior, leading to unpredictable results such as outputting toxic text. Therefore, this disclosure makes an adjustment by designing a self-guided loss G(x, p) + p - y + y - ) = L base (x + y + y - )+L base (x - y o y + ), where p represents the principle, p + This represents the positive principle, p - This represents the negative principle. It can be seen that this disclosure uses y... o Adjust the loss L as the target response base (x - y - y + This prevents the model from using p. - Used as a backdoor.

[0107] Figure 3 is a schematic diagram of the G function design according to an embodiment of the present disclosure. As shown in Figure 3, taking the L function as an example of selecting the DPO loss function, if the G function is designed as G = L... DPO (x + y + y - )+L DPO (x - y - y + If, after training, the policy model learns positive responses and outputs better positive responses, learns the original response and outputs a positive response, and learns negative responses and outputs a worse negative response, then the policy model will output a more dangerous negative response, leading to unpredictable results. Therefore, this disclosure designs the G function as G = L DPO (x + y + y - )+L DPO (x - y o y + After the policy model is trained, it will learn positive responses and output better positive responses, learn the original responses and output positive responses, and learn negative responses and output the original responses. This prevents the policy model from using negative principles as backdoors, which could lead to unpredictable results.

[0108] In an optional embodiment, step S2322, the target loss is calculated based on the preference optimization loss and the self-guided loss, including the following method steps:

[0109] Step S23221: Calculate the target weight value using the first preference pair;

[0110] Step S23222: Calculate the target loss based on the target weight value, preference optimization loss, and self-guided loss.

[0111] In this embodiment of the disclosure, when calculating the target loss based on the preference optimization loss and the self-guided loss, the target weight value can be calculated using the first preference pair. The target weight value is used to control the on-policy behavior of the initial policy model and can be denoted as W(x, y). + y - ).

[0112] Then, based on the target weight value W(x, y) + y - ), preference optimization loss L base (x, y) + y - ) and self-guided loss G(x, p + p - y + y - ) = L base (x + y + y - )+L base (x - y o y + The target loss L is calculated. sso .

[0113] It should be noted that in designing the W function, this disclosure uses a simple function that directly uses y + and y - The average log probability, which can be expressed as: Furthermore, this disclosure defines the target weight value. Here, θ is a parameter that controls the weights of the self-guided loss G, and α is a hyperparameter that can be set to 0.6, which is not restricted here.

[0114] In an optional embodiment, in step S23222, the target loss is calculated based on the target weight value, the preference optimization loss, and the self-guided loss, including the following method steps:

[0115] Step S232221: Define the weight control parameters corresponding to the self-guided loss;

[0116] Step S232222: Calculate the third loss based on the weight control parameters, self-guided loss, and preference optimization loss;

[0117] Step S232223: The target loss is calculated using the third loss and the target weight value.

[0118] In this embodiment of the disclosure, when calculating the target loss based on the target weight value, preference optimization loss, and self-guiding loss, a custom weight control parameter corresponding to the self-guiding loss can be defined, which can be denoted as θ. Then, a third loss is calculated based on the weight control parameter, self-guiding loss, and preference optimization loss, wherein the third loss can be denoted as θ·G(x, p) + p - y + y - )+L base (x, y) + y - ).

[0119] Finally, the third loss θ·G(x, p) is used. + p - y + y - )+L base (x, y) + y - ) and the target weight value W(x, y) + y - The target loss L was calculated. sso =W(x, y) + y - )[θ·G(x,p + p - y + y - )+L base (x, y) + y - It can be seen that the target loss L sso In the middle, the target weight value W(x, y) + y - ) can be understood as a weighting function used for learnable and on-policy signals, θ·G(x, p) + p - y + y - L can be understood as a self-guided loss function used for precise signals. base (x, y) + y - This can be understood as the basic loss function used to optimize the model.

[0120] Furthermore, the target loss calculated in the embodiments of this disclosure can also be L. sso =θ·W(x,y) + y - )·G(x, p + p - y + y - )+L base (x, y) + y - The target loss function is expressed in the form of a custom weight control parameter derived from the product of the W and G functions, which is used to determine the target weight value of the precise signal and the self-guided loss function. Experiments have verified that this form of target loss also exhibits excellent model alignment performance.

[0121] Figure 4 is a flowchart of the model training method according to an embodiment of the present disclosure. As shown in Figure 4, when training the policy model (i.e., training the initial policy model to obtain the target policy model), the first step of the present disclosure is to construct comparison instructions and sample responses. Specifically, given an original instruction, i.e., a query, the initial policy model first determines the features and principles most relevant to the query, i.e., positive principles and negative principles (which can be implemented by an additional instruction, not described here). Then, comparison instructions are constructed based on the obtained original instruction and the most relevant principles, and the constructed comparison instructions and the original instruction are combined to form an instruction set (i.e., including the original instruction, positive instructions, and negative instructions). Next, the initial policy model is used to sample the instruction set to obtain the response corresponding to each instruction, thus obtaining a response set (i.e., the original response y). o Positive response y + and negative response y - ).

[0122] The second step involves optimizing the model based on the response pairs. Specifically, three preference pairs are formed from the three responses obtained to align the model. This involves training the initial policy model based on the instruction set and the response set, ultimately obtaining the target policy model. Specifically, during the training of the initial policy model, the weight function is adjusted so that, under the original response preference pair, the weight of the positive response is greater than the weight of the negative response; under the negative response preference pair, the weight of the original response is greater than the weight of the positive response; and under the positive response preference pair, the weight of the positive response is greater than the weight of the negative response, thus obtaining the target policy model.

[0123] It is easy to understand that the beneficial effects of the instruction processing method provided in this disclosure include the following points.

[0124] Beneficial Effect (1): The SSO method proposed in this disclosure improves the quality and on-policy degree of the generated preference signal by introducing two additional optimization objectives (i.e., a weight function W and an additional self-guided loss G). Specifically, this disclosure defines a weight function W and uses the average generation probability of the two responses as the loss to control the on-policy degree of the generated response, ensuring that the generated preference signal is highly consistent with the current policy of the model. At the same time, a self-guided loss G is also defined to further optimize the quality of the generated signal, ensuring that the generated preference signal has high accuracy and diversity.

[0125] Beneficial Effect (2): The SSO method proposed in this disclosure improves the synthetic data process. Specifically, it uses detailed contrast features and principles highly correlated with instructions to directly generate positive and negative responses from the policy model, and uses these positive and negative responses to construct training data, thereby improving the quality of the generated data while aligning the model. That is, this disclosure uses detailed contrast principles and templates to ensure that the generated responses are of high quality and diversity.

[0126] Beneficial effects (3): Experimental results show that the SSO method proposed in this embodiment exhibits superiority on multiple models and datasets, significantly improving the alignment effect of the models, and outperforming traditional alignment methods in both subjective and objective benchmarks. In particular, the SSO method proposed in this embodiment shows significant advantages in terms of the quality of generated data and the degree of on-policy. That is, through multiple rounds of screening and verification, it has been verified that the SSO method proposed in this embodiment can obtain high-quality generated data, that is, the generated preference signals have high accuracy and on-policy degree, providing reliable data support for model alignment.

[0127] Beneficial effect (4): The SSO method proposed in this embodiment also has wide applicability, can be extended to other DPO losses, is applicable to a variety of models and datasets, and has good generalization ability.

[0128] Beneficial effect (5): The SSO method proposed in this embodiment can train a harmless large language model by fine-tuning the large language model when resources are scarce. That is, it can automatically generate data to train the reward model when there is a lack of labeled data, and then use it for further training of the policy model.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0130] Furthermore, it should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0132] According to embodiments of this disclosure, an instruction processing method is also provided as shown in FIG5. FIG5 is a flowchart of an instruction processing method according to an embodiment of this disclosure. As shown in FIG5, the method includes:

[0133] Step S51: Obtain the target instruction;

[0134] Step S52: The target policy model is used to sample the target instruction to obtain the target response. The target policy model is obtained by training the initial policy model based on the response set. The response set is obtained by sampling the comparison instruction based on the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively associated with the target requirement contained in the original instruction. Negative principles are used to indicate the generation of a response that is negatively associated with the target requirement.

[0135] In this embodiment of the disclosure, the target instruction can be understood as an instruction input by the user that contains the target requirement. The expected target strategy model can perform strategy sampling based on the original instruction and return a response that can solve or achieve the target requirement.

[0136] The target policy model is used to sample the target instruction to obtain the target response. The target response can be understood as the target response corresponding to the target instruction. The target policy model can be understood as the policy model trained according to the SSO method proposed in this embodiment. The target policy model is obtained by training an initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement.

[0137] A raw instruction can be understood as a given command input by the user, i.e., a query (Q). The raw instruction may contain a target requirement, which can be understood as the user's specific needs.

[0138] The set of principles associated with the original instruction can be understood as the principles most relevant to the query. This set includes the Good Principle and the Bad Principle. The Good Principle instructs the generation of responses that are positively correlated with the target requirements contained in the original instruction. In other words, the Good Principle ensures that the generated response is consistent with and has a positive relationship to the target requirements contained in the original instruction. That is, the Good Principle ensures that the generated response is helpful to the target requirements, providing a truly beneficial and specific response.

[0139] The negative principle is used to indicate that the generated response is inversely related to the target requirement. This can be understood as the generated response being the opposite of the target requirement in the original instruction. For example, if the target requirement is to seek a certain solution or expect a certain result, then a response guided by the negative principle may generate irrelevant or contrary-to-expectation content. In other words, the negative principle indicates that the generated response contradicts the target requirement, does not help the target requirement, and provides a vague, irrelevant, or unhelpful response.

[0140] An instruction set can be constructed based on the original instruction and the set of principles associated with it. This can be understood as constructing the instruction set based on the user-inputted original instruction and the positive and negative principles most relevant to that original instruction. For example, the instruction set includes the original instruction, as well as other instructions, such as comparison instructions—that is, good and bad instructions constructed based on positive and negative principles. This is not limited here.

[0141] The initial policy model can be understood as the policy model before model training. The policy model can determine the best action or decision to take based on the input state or conditions. For example, the initial policy model can be a large model or other deep learning models, which is not limited here.

[0142] By sampling the instruction set using an initial policy model, a response set can be obtained. This response set can include the original response corresponding to the original instruction, good principle responses corresponding to positive principles (i.e., good principle responses corresponding to good instructions), and bad principle responses corresponding to negative principles (i.e., bad principle responses corresponding to bad instructions). In other words, this embodiment can directly generate high-quality, diverse positive and negative responses based on the initial policy model.

[0143] Training data is constructed based on the established instruction set and the obtained response set. The initial policy model is then trained using this training data, and its parameters are adjusted to obtain the target policy model. In other words, this embodiment of the present disclosure can generate high-quality, diverse positive and negative responses from the initial policy model to construct training data, significantly reducing training costs.

[0144] For details, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0145] As can be seen, the embodiments of this disclosure obtain target instructions, use a target policy model to sample the target instructions, and obtain target responses. The target policy model is obtained by training an initial policy model based on a response set. The response set is obtained by sampling the comparison instructions using the initial policy model. The comparison instructions are constructed based on the original instructions and a set of principles associated with the original instructions. The set of principles includes positive and negative principles. Positive principles indicate that a response is positively correlated with the target requirement contained in the original instructions, while negative principles indicate that a response is negatively correlated with the target requirement. Therefore, using the SSO method proposed in this disclosure, multiple responses (i.e., a response set) are directly generated through the initial policy model using a set of principles highly correlated with the original instructions. These responses are then used to construct training data to train the initial policy model. This reduces reliance on human-labeled data, lowers training costs, and improves the quality of generated data while aligning the model, thus significantly improving the model's alignment effect. It demonstrates clear advantages in terms of generated data quality and on-policy level.

[0146] The instruction processing method provided in this disclosure can be applied, but is not limited to, application scenarios involving instruction processing in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, instruction processing for e-commerce services, instruction processing for education services, instruction processing for medical services, etc., are not limited here.

[0147] By employing the embodiments of this disclosure, a target instruction is obtained, and a target policy model is used to sample the target instruction to obtain a target response. The target policy model is obtained by training an initial policy model based on a response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction. The set of principles includes positive and negative principles. Positive principles indicate that a response is generated that is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that a response is generated that is negatively correlated with the target requirement. This significantly improves the model alignment effect, thereby achieving low-cost training to obtain the target policy model. This target policy model can improve the quality of generated data while aligning the model, exhibiting significant advantages in terms of the quality of generated data and the degree of on-policy. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0148] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0149] According to an embodiment of this disclosure, an instruction processing method is also provided as shown in FIG6. FIG6 is a flowchart of an instruction processing method according to an embodiment of this disclosure. As shown in FIG6, the method includes:

[0150] Step S61: Obtain programming assistance instructions;

[0151] Step S62: The target programming assistance model is used to perform programming analysis on the programming assistance instructions to obtain programming assistance responses. The target programming assistance model is obtained by training the initial programming assistance model based on the response set. The response set is obtained by performing programming analysis on the comparison instructions using the initial programming assistance model. The comparison instructions are constructed based on the original instructions and the set of principles associated with the original instructions. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, and negative principles are used to indicate the generation of responses that are negatively correlated with the target requirements.

[0152] In this embodiment of the disclosure, the programming assistance instruction can be understood as an instruction input by the user that contains the target requirement, used to request the target programming assistance model to provide programming assistance based on the programming assistance instruction.

[0153] A target programming assistance model is used to analyze programming assistance instructions to obtain programming assistance responses. The programming assistance response can be understood as the target response corresponding to the programming assistance instruction. The target programming assistance model can be understood as a policy model trained according to the SSO method proposed in this embodiment. The target programming assistance model is obtained by training an initial programming assistance model based on a response set. The response set is obtained by sampling comparison instructions using the initial programming assistance model. The comparison instructions are constructed based on the original instruction and a set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles indicate the generation of responses that are positively correlated with the target requirement contained in the original instruction, while negative principles indicate the generation of responses that are negatively correlated with the target requirement.

[0154] A raw instruction can be understood as a given command input by the user, i.e., a query (Q). The raw instruction may contain a target requirement, which can be understood as the user's specific needs.

[0155] The set of principles associated with the original instruction can be understood as the principles most relevant to the query. This set includes the Good Principle and the Bad Principle. The Good Principle instructs the generation of responses that are positively correlated with the target requirements contained in the original instruction. In other words, the Good Principle ensures that the generated response is consistent with and has a positive relationship to the target requirements contained in the original instruction. That is, the Good Principle ensures that the generated response is helpful to the target requirements, providing a truly beneficial and specific response.

[0156] The negative principle is used to indicate that the generated response is inversely related to the target requirement. This can be understood as the generated response being the opposite of the target requirement in the original instruction. For example, if the target requirement is to seek a certain solution or expect a certain result, then a response guided by the negative principle may generate irrelevant or contrary-to-expectation content. In other words, the negative principle indicates that the generated response contradicts the target requirement, does not help the target requirement, and provides a vague, irrelevant, or unhelpful response.

[0157] An instruction set can be constructed based on the original instruction and the set of principles associated with it. This can be understood as constructing the instruction set based on the user-inputted original instruction and the positive and negative principles most relevant to that original instruction. For example, the instruction set includes the original instruction, as well as other instructions, such as comparison instructions—that is, good and bad instructions constructed based on positive and negative principles. This is not limited here.

[0158] The initial programming aid model can be understood as a policy model prior to model training. The policy model determines the optimal action or decision based on the input state or conditions. For example, the initial programming aid model can be a large model or other deep learning models; this is not a limitation.

[0159] By sampling the instruction set using an initial programming aid model, a response set can be obtained. This response set can include the original response corresponding to the original instruction, good principle responses corresponding to positive principles (i.e., good principle responses corresponding to good instructions), and bad principle responses corresponding to negative principles (i.e., bad principle responses corresponding to bad instructions). In other words, this embodiment of the disclosure can directly generate high-quality, diverse positive and negative responses based on the initial programming aid model.

[0160] Training data is constructed based on the established instruction set and the obtained response set. The initial programming aid model is then trained using this training data, and its parameters are adjusted to obtain the target programming aid model. In other words, this embodiment of the present disclosure can generate high-quality, diverse positive and negative responses from the initial programming aid model to construct training data, significantly reducing training costs.

[0161] For details, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0162] As can be seen, this embodiment of the present disclosure obtains programming assistance instructions, performs programming analysis on the instructions using a target programming assistance model, and obtains programming assistance responses. The target programming assistance model is obtained by training an initial programming assistance model based on a response set. The response set is obtained by performing programming analysis on comparison instructions using the initial programming assistance model. The comparison instructions are constructed based on the original instructions and a set of principles associated with them. The set of principles includes positive and negative principles. Positive principles indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, while negative principles indicate the generation of responses that are negatively correlated with the target requirements. Therefore, using the SSO method proposed in this embodiment of the present disclosure, multiple responses (i.e., a response set) are directly generated through the initial policy model by using a set of principles highly correlated with the original instructions. These responses are then used to construct training data to train the initial policy model. This reduces reliance on human-labeled data, lowers training costs, and improves the quality of generated data while aligning the model, thus significantly improving the model's alignment effect. It demonstrates clear advantages in terms of the quality of generated data and the degree of on-policy.

[0163] The instruction processing method provided in this disclosure can be applied, but is not limited to, application scenarios involving instruction processing in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, instruction processing for e-commerce services, instruction processing for education services, instruction processing for medical services, etc., are not limited here.

[0164] By employing the embodiments of this disclosure, programming assistance instructions are obtained, and a target programming assistance model is used to perform programming analysis on the instructions to obtain programming assistance responses. The target programming assistance model is obtained by training an initial programming assistance model based on a response set. The response set is obtained by performing programming analysis on comparison instructions using the initial programming assistance model. The comparison instructions are constructed based on the original instructions and a set of principles associated with them. The set of principles includes positive and negative principles. Positive principles indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that the generated response is negatively correlated with the target requirement. This significantly improves the model alignment effect, enabling low-cost training to obtain the target policy model. This target policy model improves the quality of generated data while aligning the model, demonstrating significant advantages in terms of data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0165] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0166] According to an embodiment of this disclosure, an instruction processing method is also provided as shown in FIG7. FIG7 is a flowchart of an instruction processing method according to an embodiment of this disclosure. As shown in FIG7, the method includes:

[0167] Step S71: Obtain an instruction processing request through the first application programming interface, wherein the request data carried in the instruction processing request includes: the target instruction;

[0168] Step S72: Return the instruction processing response through the second application programming interface. The response data carried in the instruction processing response includes: the target response, which is obtained by sampling the target instruction using the target policy model. The target policy model is obtained by training the initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction. Negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement.

[0169] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0170] In this embodiment of the disclosure, an instruction processing request can be understood as a request to process a target instruction, and the request data carried in the instruction processing request includes the target instruction. An instruction processing response can be understood as an instruction processing response corresponding to the target instruction, and the response data carried in the instruction processing response includes the target response.

[0171] The target response is obtained by sampling the target instruction using the target policy model. The target policy model is obtained by training the initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, while negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement.

[0172] A raw instruction can be understood as a given command input by the user, i.e., a query (Q). The raw instruction may contain a target requirement, which can be understood as the user's specific needs.

[0173] The set of principles associated with the original instruction can be understood as the principles most relevant to the query. This set includes the Good Principle and the Bad Principle. The Good Principle instructs the generation of responses that are positively correlated with the target requirements contained in the original instruction. In other words, the Good Principle ensures that the generated response is consistent with and has a positive relationship to the target requirements contained in the original instruction. That is, the Good Principle ensures that the generated response is helpful to the target requirements, providing a truly beneficial and specific response.

[0174] The negative principle is used to indicate that the generated response is inversely related to the target requirement. This can be understood as the generated response being the opposite of the target requirement in the original instruction. For example, if the target requirement is to seek a certain solution or expect a certain result, then a response guided by the negative principle may generate irrelevant or contrary-to-expectation content. In other words, the negative principle indicates that the generated response contradicts the target requirement, does not help the target requirement, and provides a vague, irrelevant, or unhelpful response.

[0175] An instruction set can be constructed based on the original instruction and the set of principles associated with it. This can be understood as constructing the instruction set based on the user-inputted original instruction and the positive and negative principles most relevant to that original instruction. For example, the instruction set includes the original instruction, as well as other instructions, such as comparison instructions—that is, good and bad instructions constructed based on positive and negative principles. This is not limited here.

[0176] The initial policy model can be understood as the policy model before model training. The policy model can determine the best action or decision to take based on the input state or conditions. For example, the initial policy model can be a large model or other deep learning models, which is not limited here.

[0177] By sampling the instruction set using an initial policy model, a response set can be obtained. This response set can include the original response corresponding to the original instruction, good principle responses corresponding to positive principles (i.e., good principle responses corresponding to good instructions), and bad principle responses corresponding to negative principles (i.e., bad principle responses corresponding to bad instructions). In other words, this embodiment can directly generate high-quality, diverse positive and negative responses based on the initial policy model.

[0178] Training data is constructed based on the established instruction set and the obtained response set. The initial policy model is then trained using this training data, and its parameters are adjusted to obtain the target policy model. In other words, this embodiment of the present disclosure can generate high-quality, diverse positive and negative responses from the initial policy model to construct training data, significantly reducing training costs.

[0179] For details, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0180] As can be seen, this disclosure obtains an instruction processing request through a first application programming interface (API), wherein the request data carried in the instruction processing request includes a target instruction. Then, it returns an instruction processing response through a second API, wherein the response data carried in the instruction processing response includes a target response. The target policy model is obtained by training an initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the principle set associated with the original instruction. The principle set includes positive and negative principles. Positive principles indicate that a response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that a response is negatively correlated with the target requirement. Therefore, using the SSO method proposed in this disclosure, multiple responses (i.e., a response set) are directly generated from the initial policy model by using a principle set highly correlated with the original instruction. These responses are then used to construct training data to train the initial policy model. This reduces reliance on human-labeled data, lowers training costs, and improves the quality of generated data while aligning the model, thus significantly improving the model's alignment effect. It demonstrates clear advantages in terms of generated data quality and on-policy degree.

[0181] The instruction processing method provided in this disclosure can be applied, but is not limited to, application scenarios involving instruction processing in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, instruction processing for e-commerce services, instruction processing for education services, instruction processing for medical services, etc., are not limited here.

[0182] In this embodiment, an instruction processing request is obtained through a first application programming interface (API), wherein the request data carried in the instruction processing request includes a target instruction. Then, an instruction processing response is returned through a second API, wherein the response data carried in the instruction processing response includes a target response. The target response is obtained by sampling the target instruction using a target policy model. The target policy model is obtained by training an initial policy model based on a response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles indicate that a response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that a response is negatively correlated with the target requirement. This significantly improves the model alignment effect, thereby achieving low-cost training to obtain the target policy model. This target policy model improves the quality of generated data while aligning the model, exhibiting significant advantages in terms of generated data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0183] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0184] According to an embodiment of this disclosure, an instruction processing method is also provided as shown in FIG8. FIG8 is a flowchart of an instruction processing method according to an embodiment of this disclosure. As shown in FIG8, the method includes:

[0185] Step S81: Obtain the currently input instruction processing dialog request, wherein the request data carried in the instruction processing dialog request includes: target instruction;

[0186] Step S82: In response to the instruction processing dialogue request, return the instruction processing dialogue reply. The information carried in the instruction processing dialogue reply includes: the target response, which is obtained by sampling the target instruction using the target policy model. The target policy model is obtained by training the initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the principle set associated with the original instruction. The principle set includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively associated with the target requirement contained in the original instruction. Negative principles are used to indicate the generation of a response that is negatively associated with the target requirement.

[0187] Step S83: Display the target response within the graphical user interface.

[0188] In this embodiment of the disclosure, the instruction processing dialogue request can be understood as a dialogue request initiated by the user when conversing with the artificial intelligence, used to request instruction processing of a target instruction. The request data carried in the instruction processing dialogue request includes the target instruction. The instruction processing dialogue response can be understood as the artificial intelligence's response to the instruction processing dialogue request. The information carried in the instruction processing dialogue response includes the target response, which is the target response obtained after processing the target instruction.

[0189] The target response is obtained by sampling the target instruction using the target policy model. The target policy model is obtained by training the initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, while negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement.

[0190] A raw instruction can be understood as a given command input by the user, i.e., a query (Q). The raw instruction may contain a target requirement, which can be understood as the user's specific needs.

[0191] The set of principles associated with the original instruction can be understood as the principles most relevant to the query. This set includes the Good Principle and the Bad Principle. The Good Principle instructs the generation of responses that are positively correlated with the target requirements contained in the original instruction. In other words, the Good Principle ensures that the generated response is consistent with and has a positive relationship to the target requirements contained in the original instruction. That is, the Good Principle ensures that the generated response is helpful to the target requirements, providing a truly beneficial and specific response.

[0192] The negative principle is used to indicate that the generated response is inversely related to the target requirement. This can be understood as the generated response being the opposite of the target requirement in the original instruction. For example, if the target requirement is to seek a certain solution or expect a certain result, then a response guided by the negative principle may generate irrelevant or contrary-to-expectation content. In other words, the negative principle indicates that the generated response contradicts the target requirement, does not help the target requirement, and provides a vague, irrelevant, or unhelpful response.

[0193] An instruction set can be constructed based on the original instruction and the set of principles associated with it. This can be understood as constructing the instruction set based on the user-inputted original instruction and the positive and negative principles most relevant to that original instruction. For example, the instruction set includes the original instruction, as well as other instructions, such as comparison instructions—that is, good and bad instructions constructed based on positive and negative principles. This is not limited here.

[0194] The initial policy model can be understood as the policy model before model training. The policy model can determine the best action or decision to take based on the input state or conditions. For example, the initial policy model can be a large model or other deep learning models, which is not limited here.

[0195] By sampling the instruction set using an initial policy model, a response set can be obtained. This response set can include the original response corresponding to the original instruction, good principle responses corresponding to positive principles (i.e., good principle responses corresponding to good instructions), and bad principle responses corresponding to negative principles (i.e., bad principle responses corresponding to bad instructions). In other words, this embodiment can directly generate high-quality, diverse positive and negative responses based on the initial policy model.

[0196] Training data is constructed based on the established instruction set and the obtained response set. The initial policy model is then trained using this training data, and its parameters are adjusted to obtain the target policy model. In other words, this embodiment of the present disclosure can generate high-quality, diverse positive and negative responses from the initial policy model to construct training data, significantly reducing training costs.

[0197] For details, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0198] Once the target response is obtained, it can be displayed to the user within the graphical user interface.

[0199] As can be seen, this disclosure processes dialogue requests by acquiring the currently input instruction. The request data carried in the instruction processing dialogue request includes the target instruction. Then, in response to the instruction processing dialogue request, an instruction processing dialogue reply is returned. The information carried in the instruction processing dialogue reply includes the target response. The target response is obtained by sampling the target instruction using a target policy model. The target policy model is obtained by training an initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction. Negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement. Finally, the target response is displayed in the graphical user interface. Therefore, by using the SSO method proposed in this disclosure, multiple responses (i.e., a response set) can be directly generated from the initial policy model by using a set of principles highly related to the original instructions, and training data can be constructed using these responses to train the initial policy model. This reduces the reliance on human-labeled data, lowers training costs, and improves the quality of generated data while aligning the model, thereby significantly improving the alignment effect of the model. It shows obvious advantages in terms of the quality of generated data and the degree of on-policy.

[0200] The instruction processing method provided in this disclosure can be applied, but is not limited to, application scenarios involving instruction processing in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, instruction processing for e-commerce services, instruction processing for education services, instruction processing for medical services, etc., are not limited here.

[0201] By employing the embodiments of this disclosure, a dialogue request is processed by acquiring the currently input instruction. The request data carried in the instruction processing dialogue request includes a target instruction. Then, in response to the instruction processing dialogue request, an instruction processing dialogue response is returned. The information carried in the instruction processing dialogue response includes a target response. The target policy model is obtained by training an initial policy model based on a response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the principle set associated with the original instruction. The principle set includes positive principles and negative principles. Positive principles are used to indicate the generation of responses that are positively correlated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of responses that are negatively correlated with the target requirement. Finally, the target response is displayed in the graphical user interface. This achieves the goal of significantly improving the model alignment effect, thereby realizing the low-cost training of the target policy model. This target policy model can improve the quality of generated data while aligning the model. It shows obvious advantages in terms of the quality of generated data and the degree of on-policy. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0202] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0203] According to embodiments of this disclosure, an apparatus embodiment for implementing the above-described model training method is also provided. Figure 9 is a schematic structural diagram of a model training apparatus according to an embodiment of this disclosure. As shown in Figure 9, the apparatus includes:

[0204] The construction module 901 is configured to construct an instruction set based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively associated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of a response that is negatively associated with the target requirement.

[0205] The first sampling module 902 is configured to perform policy sampling on the instruction set using the initial policy model to obtain the response set;

[0206] Training module 903 is configured to train the initial policy model based on the instruction set and response set to obtain the target policy model, wherein the target policy model is used to perform policy sampling on the target instruction to obtain the target response.

[0207] Optionally, the aforementioned construction module 901 is further configured to: construct comparison instructions based on the original instructions and the set of principles associated with the original instructions; and construct an instruction set using the original instructions and the comparison instructions.

[0208] Optionally, the aforementioned construction module 901 is further configured to: generate a positive instruction based on the original instruction and the positive principle, and generate a negative instruction based on the original instruction and the negative principle; and construct a comparison instruction using the positive and negative instructions.

[0209] Optionally, the first sampling module 902 is further configured to: use an initial policy model to perform policy sampling on the original instruction to obtain the original response; use the initial policy model to perform policy sampling on the positive instruction to obtain the positive response; and use the initial policy model to perform policy sampling on the negative instruction to obtain the negative response; and determine the response set based on the original response, the positive response, and the negative response.

[0210] Optionally, the training module 903 is further configured to: generate multiple preference pairs based on the instruction set and response set; and use the multiple preference pairs to perform model alignment training on the initial policy model to obtain the target policy model.

[0211] Optionally, the training module 903 is further configured to: generate a first preference pair based on the original instruction, the positive response corresponding to the original response, and the negative response corresponding to the original response in response to the instruction set and the response set satisfying a preset condition; generate a second preference pair based on the positive instruction, the positive response, and the negative response; and generate a third preference pair based on the negative instruction, the original response, and the positive response, thereby obtaining multiple preference pairs. The original response is the response corresponding to the original instruction, and the preset condition is used to indicate that the instruction set contains the original instruction, the positive instruction, and the negative instruction, and the response set contains the original response, the positive response, and the negative response.

[0212] Optionally, the training module 903 is further configured to: calculate the preference optimization loss using the first preference pair, and calculate the self-guided loss using the second and third preference pairs, wherein the preference optimization loss is used to optimize the initial policy model toward the direction of the positive response, and the self-guided loss is used to control the gap between the positive and negative responses; calculate the target loss based on the preference optimization loss and the self-guided loss; and update the model parameters of the initial policy model based on the target loss to obtain the target policy model.

[0213] Optionally, the training module 903 is further configured to: obtain the first loss of the second preference pair and the second loss of the third preference pair; and calculate the self-guided loss using the first loss and the second loss.

[0214] Optionally, the training module 903 is further configured to: calculate the target weight value using the first preference pair; and calculate the target loss based on the target weight value, preference optimization loss, and self-guided loss.

[0215] Optionally, the training module 903 is further configured to: define the weight control parameters corresponding to the self-guided loss; calculate the third loss based on the weight control parameters, the self-guided loss, and the preference optimization loss; and calculate the target loss using the third loss and the target weight value.

[0216] This embodiment of the invention constructs an instruction set including positive and negative principles based on the original instruction and the set of principles associated with it. Positive principles indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that the generated response is negatively correlated with the target requirement. Then, an initial policy model is used to sample the policy from the instruction set to obtain a response set. The initial policy model is then trained based on the instruction set and the response set to obtain a target policy model. This target policy model is used to sample the policy from the target instruction to obtain the target response, thereby significantly improving the model alignment effect. This achieves low-cost training to obtain the target policy model. The target policy model improves the quality of generated data while aligning the model, exhibiting significant advantages in terms of data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0217] It should be noted that the aforementioned construction module 901, first sampling module 902, and training module 903 correspond to steps S21 to S23 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the aforementioned modules or units may be hardware or software components stored in memory and processed by one or more processors, and the aforementioned modules may also run in the server 10 provided in the embodiments.

[0218] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described instruction processing method is also provided. FIG10 is a schematic structural diagram of another instruction processing apparatus according to an embodiment of this disclosure. As shown in FIG10, the apparatus includes:

[0219] The first acquisition module 1001 is configured to acquire target instructions;

[0220] The second sampling module 1002 is configured to use a target policy model to sample the target instruction to obtain a target response. The target policy model is obtained by training an initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement.

[0221] By employing the embodiments of this disclosure, a target instruction is obtained, and a target policy model is used to sample the target instruction to obtain a target response. The target policy model is obtained by training an initial policy model based on a response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction. The set of principles includes positive and negative principles. Positive principles indicate that a response is generated that is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that a response is generated that is negatively correlated with the target requirement. This significantly improves the model alignment effect, thereby achieving low-cost training to obtain the target policy model. This target policy model can improve the quality of generated data while aligning the model, exhibiting significant advantages in terms of the quality of generated data and the degree of on-policy. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0222] It should be noted that the first acquisition module 1001 and the second sampling module 1002 mentioned above correspond to steps S51 and S52 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0223] According to embodiments of this disclosure, another embodiment of an apparatus for implementing the above-described instruction processing method is also provided. FIG11 is a schematic structural diagram of another instruction processing apparatus according to an embodiment of this disclosure. As shown in FIG11, the apparatus includes:

[0224] The second acquisition module 1101 is configured to acquire programming auxiliary instructions;

[0225] Analysis module 1102 is configured to perform programming analysis on programming assistance instructions using a target programming assistance model to obtain programming assistance responses. The target programming assistance model is obtained by training an initial programming assistance model based on a response set. The response set is obtained by performing programming analysis on comparison instructions using the initial programming assistance model. The comparison instructions are constructed based on the original instructions and a set of principles associated with the original instructions. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, while negative principles are used to indicate the generation of responses that are negatively correlated with the target requirements.

[0226] By employing the embodiments of this disclosure, programming assistance instructions are obtained, and a target programming assistance model is used to perform programming analysis on the instructions to obtain programming assistance responses. The target programming assistance model is obtained by training an initial programming assistance model based on a response set. The response set is obtained by performing programming analysis on comparison instructions using the initial programming assistance model. The comparison instructions are constructed based on the original instructions and a set of principles associated with them. The set of principles includes positive and negative principles. Positive principles indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that the generated response is negatively correlated with the target requirement. This significantly improves the model alignment effect, enabling low-cost training to obtain the target policy model. This target policy model improves the quality of generated data while aligning the model, demonstrating significant advantages in terms of data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0227] It should be noted that the second acquisition module 1101 and analysis module 1102 mentioned above correspond to steps S61 and S62 in the embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0228] According to embodiments of this disclosure, another embodiment of an apparatus for implementing the above-described instruction processing method is also provided. FIG12 is a schematic structural diagram of another instruction processing apparatus according to an embodiment of this disclosure. As shown in FIG12, the apparatus includes:

[0229] The third acquisition module 1201 is configured to acquire an instruction processing request through the first application programming interface, wherein the request data carried in the instruction processing request includes: the target instruction;

[0230] The first return module 1202 is configured to return an instruction processing response through a second application programming interface. The response data carried in the instruction processing response includes: a target response, which is obtained by sampling the target instruction using a target policy model. The target policy model is obtained by training an initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement.

[0231] In this embodiment, an instruction processing request is obtained through a first application programming interface (API), wherein the request data carried in the instruction processing request includes a target instruction. Then, an instruction processing response is returned through a second API, wherein the response data carried in the instruction processing response includes a target response. The target response is obtained by sampling the target instruction using a target policy model. The target policy model is obtained by training an initial policy model based on a response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles indicate that a response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that a response is negatively correlated with the target requirement. This significantly improves the model alignment effect, thereby achieving low-cost training to obtain the target policy model. This target policy model improves the quality of generated data while aligning the model, exhibiting significant advantages in terms of generated data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0232] It should be noted that the third acquisition module 1201 and return module 1202 mentioned above correspond to steps S71 and S72 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0233] According to embodiments of the present disclosure, another embodiment of an apparatus for implementing the above-described instruction processing method is also provided. FIG13 is a schematic structural diagram of another instruction processing apparatus according to an embodiment of the present disclosure. As shown in FIG13, the apparatus includes:

[0234] The fourth acquisition module 1301 is configured to acquire the currently input instruction processing dialog request, wherein the request data carried in the instruction processing dialog request includes: the target instruction;

[0235] The second return module 1302 is configured to respond to an instruction processing dialogue request and return an instruction processing dialogue response. The information carried in the instruction processing dialogue response includes: a target response, which is obtained by sampling the target instruction using a target policy model. The target policy model is obtained by training an initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement.

[0236] Display module 1303 is configured to display the target response within the graphical user interface.

[0237] In this embodiment of the disclosure, a dialogue request is processed by acquiring the currently input instruction. The request data carried in the instruction processing dialogue request includes a target instruction. Then, in response to the instruction processing dialogue request, an instruction processing dialogue response is returned. The information carried in the instruction processing dialogue response includes a target response, which is obtained by sampling the target instruction using a target policy model. The target policy model is obtained by training an initial policy model based on a response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction. The set of principles includes positive principles and negative principles. The positive principle is used to indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while the negative principle is used to indicate that the generated response is negatively correlated with the target requirement. Finally, the target response is displayed in the graphical user interface, thereby significantly improving the model alignment effect and realizing low-cost training to obtain the target policy model. This target policy model can improve the quality of the generated data while aligning the model, and shows obvious advantages in terms of the quality of the generated data and the degree of on-policy. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0238] It should be noted that the fourth acquisition module 1301, the second return module 1302, and the display module 1303 mentioned above correspond to steps S81 to S83 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0239] It should be noted that the preferred implementation schemes involved in the above embodiments of this disclosure are the same as the schemes, application scenarios and implementation processes provided in the embodiments, but are not limited to the schemes provided in the embodiments.

[0240] Embodiments of this disclosure can provide an electronic device, which can be any one of a group of electronic devices. Optionally, in this embodiment, the electronic device can also be replaced by a terminal device such as a mobile terminal. Optionally, in this embodiment, the electronic device can be located in at least one of a plurality of network devices in a computer network.

[0241] In this embodiment, the aforementioned electronic device can execute the program code for the following steps in the model training method: constructing an instruction set based on the original instruction and the principle set associated with the original instruction, wherein the principle set includes positive principles and negative principles, positive principles are used to indicate the generation of responses that are positively correlated with the target requirement contained in the original instruction, and negative principles are used to indicate the generation of responses that are negatively correlated with the target requirement; sampling the instruction set using an initial policy model to obtain a response set; training the initial policy model based on the instruction set and the response set to obtain a target policy model, wherein the target policy model is used to sample the target instruction to obtain the target response.

[0242] Optionally, FIG14 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG14, taking electronic device A as an example, electronic device A may include: one or more (only one is shown in the figure) processors 1402, memory 1404, memory controller, and peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0243] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method and apparatus in this embodiment. The processor executes various functional applications and data processing by running the stored software programs and modules, thereby realizing the aforementioned model training method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic device A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0244] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: Constructing an instruction set based on the original instructions and the set of principles associated with those instructions, wherein the set of principles includes positive principles and negative principles. Positive principles are used to indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, and negative principles are used to indicate the generation of responses that are negatively correlated with the target requirements; Sampling the instruction set using an initial policy model to obtain a response set; Training the initial policy model based on the instruction set and the response set to obtain a target policy model, wherein the target policy model is used to sample the target instructions to obtain the target response.

[0245] This embodiment of the invention constructs an instruction set including positive and negative principles based on the original instruction and the set of principles associated with it. Positive principles indicate that the generated response is positively correlated with the target requirement contained in the original instruction, while negative principles indicate that the generated response is negatively correlated with the target requirement. Then, an initial policy model is used to sample the policy from the instruction set to obtain a response set. The initial policy model is then trained based on the instruction set and the response set to obtain a target policy model. This target policy model is used to sample the policy from the target instruction to obtain the target response, thereby significantly improving the model alignment effect. This achieves low-cost training to obtain the target policy model. The target policy model improves the quality of generated data while aligning the model, exhibiting significant advantages in terms of data quality and on-policy degree. This solves the technical problems of high cost and limitations in generating high-accuracy and policy-related preference signals in related technologies.

[0246] Those skilled in the art will understand that the structure shown in Figure 14 is merely illustrative, and electronic device A can also be a smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. Figure 14 does not limit the structure of the aforementioned electronic device. For example, electronic device A may include more or fewer components (such as network interfaces, display devices, etc.) than shown in Figure 14, or have a different configuration than shown in Figure 14.

[0247] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0248] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the model training method or instruction processing method provided in Embodiment 1.

[0249] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0250] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: constructing an instruction set based on the original instruction and the principle set associated with the original instruction, wherein the principle set includes positive principles and negative principles, the positive principles being used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction, and the negative principles being used to indicate the generation of a response that is negatively correlated with the target requirement; sampling the instruction set using an initial policy model to obtain a response set; training the initial policy model based on the instruction set and the response set to obtain a target policy model, wherein the target policy model is used to sample the target instruction to obtain the target response.

[0251] Embodiments of this disclosure also provide a computer program product comprising a computer program that, when executed by a processor, implements any of the above-described instruction processing methods.

[0252] Optionally, in this embodiment, when the computer program product is executed by the processor, the program code is used to perform the following steps: constructing an instruction set based on the original instructions and the principle set associated with the original instructions, wherein the principle set includes positive principles and negative principles, positive principles are used to indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, and negative principles are used to indicate the generation of responses that are negatively correlated with the target requirements; using an initial policy model to perform policy sampling on the instruction set to obtain a response set; training the initial policy model based on the instruction set and the response set to obtain a target policy model, wherein the target policy model is used to perform policy sampling on target instructions to obtain target responses.

[0253] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0254] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0255] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0256] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0257] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0258] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0259] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

A model training method, comprising: Based on the original instruction and the set of principles associated with the original instruction, an instruction set is constructed, wherein the set of principles includes positive principles and negative principles. The positive principles are used to indicate the generation of a response that is positively associated with the target requirement contained in the original instruction, and the negative principles are used to indicate the generation of a response that is negatively associated with the target requirement. The initial policy model is used to sample the instruction set to obtain the response set; The initial policy model is trained based on the instruction set and the response set to obtain a target policy model, wherein the target policy model is used to sample the target instruction to obtain the target response. According to the model training method of claim 1, wherein, Based on the original instruction and the set of principles associated with the original instruction, constructing the instruction set includes: Based on the original instruction and the set of principles associated with the original instruction, a comparison instruction is constructed; The instruction set is constructed using the original instruction and the comparison instruction. According to claim 2, the model training method, wherein, Based on the original instruction and the set of principles associated with the original instruction, the comparison instruction is constructed as follows: A positive instruction is generated based on the original instruction and the positive principle, and a negative instruction is generated based on the original instruction and the negative principle; The comparison instruction is constructed using the positive instruction and the negative instruction. According to the model training method of claim 3, wherein, The initial policy model is used to sample the comparison instructions to obtain the response set, which includes: The initial policy model is used to sample the original instruction to obtain the original response; the initial policy model is used to sample the positive instruction to obtain the positive response; and the initial policy model is used to sample the negative instruction to obtain the negative response. The response set is determined based on the original response, the positive response, and the negative response. The model training method according to any one of claims 1-4, wherein, The initial policy model is trained based on the instruction set and the response set to obtain the target policy model, which includes: Multiple preference pairs are generated based on the instruction set and the response set; The initial policy model is trained using the multiple preference pairs to obtain the target policy model. According to the model training method of claim 5, wherein, Generating the plurality of preference pairs based on the instruction set and the response set includes: In response to the instruction set and the response set satisfying a preset condition, a first preference pair is generated based on the original instruction, the positive response corresponding to the original response, and the negative response corresponding to the original response; a second preference pair is generated based on the positive instruction, the positive response, and the negative response; and a third preference pair is generated based on the negative instruction, the original response, and the positive response, thus obtaining the plurality of preference pairs. The original response is the response corresponding to the original instruction, and the preset condition indicates that the instruction set includes the original instruction, the positive instruction, and the negative instruction, and the response set includes the original response, the positive response, and the negative response. According to the model training method of claim 6, wherein, Using the multiple preference pairs, the initial policy model is trained to achieve model alignment, resulting in the target policy model, which includes: The preference optimization loss is calculated using the first preference pair, and the self-guided loss is calculated using the second preference pair and the third preference pair, wherein the preference optimization loss is used to optimize the initial policy model to the direction of the positive response, and the self-guided loss is used to control the gap between the positive response and the negative response; The target loss is calculated based on the preference optimization loss and the self-guided loss. The initial policy model is updated based on the target loss to obtain the target policy model. According to the model training method of claim 7, wherein, The self-guiding loss calculated using the second preference pair and the third preference pair includes: Obtain the first loss for the second preference pair and the second loss for the third preference pair; The self-guiding loss is calculated using the first loss and the second loss. According to the model training method of claim 7, wherein, The target loss, calculated based on the preference optimization loss and the self-guided loss, includes: The target weight value is calculated using the first preference pair; The target loss is calculated based on the target weight value, the preference optimization loss, and the self-guided loss. According to claim 9, the model training method, wherein, Based on the target weight value, the preference optimization loss, and the self-guiding loss, the target loss is calculated as follows: Customize the weight control parameters corresponding to the self-guiding loss; The third loss is calculated based on the weight control parameters, the self-guided loss, and the preference optimization loss. The target loss is calculated using the third loss and the target weight value. An instruction processing method, comprising: Obtain the target instruction; The target instruction is sampled using a target policy model to obtain the target response; The target strategy model is obtained by training the initial strategy model based on the response set. The response set is obtained by sampling the comparison instructions using the initial strategy model. The comparison instructions are constructed based on the original instructions and the principle set associated with the original instructions. The principle set includes positive principles and negative principles. The positive principles are used to indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, and the negative principles are used to indicate the generation of responses that are negatively correlated with the target requirements. An instruction processing method, comprising: Get programming assistance commands; The programming assistance instructions are analyzed using a target programming assistance model to obtain the programming assistance response; The target programming assistance model is obtained by training the initial programming assistance model based on the response set. The response set is obtained by performing programming analysis on the comparison instructions using the initial programming assistance model. The comparison instructions are constructed based on the original instructions and the principle set associated with the original instructions. The principle set includes positive principles and negative principles. The positive principles are used to indicate the generation of responses that are positively correlated with the target requirements contained in the original instructions, and the negative principles are used to indicate the generation of responses that are negatively correlated with the target requirements. An instruction processing method, comprising: The instruction processing request is obtained through the first application programming interface, wherein the request data carried in the instruction processing request includes: the target instruction; The instruction processing response is returned through the second application programming interface. The response data carried in the instruction processing response includes: a target response, which is obtained by sampling the target instruction using a target policy model. The target policy model is obtained by training an initial policy model based on the response set. The response set is obtained by sampling the comparison instruction using the initial policy model. The comparison instruction is constructed based on the original instruction and the principle set associated with the original instruction. The principle set includes positive principles and negative principles. The positive principles are used to indicate the generation of a response that is positively associated with the target requirement contained in the original instruction. The negative principles are used to indicate the generation of a response that is negatively associated with the target requirement. An instruction processing method, comprising: The dialog request is processed by acquiring the currently input instruction, wherein the request data carried in the dialog request includes: the target instruction; In response to the instruction processing dialogue request, an instruction processing dialogue reply is returned, wherein the information carried in the instruction processing dialogue reply includes: a target response, which is obtained by sampling the target instruction using a target policy model; the target policy model is obtained by training an initial policy model based on a response set; the response set is obtained by sampling the comparison instruction using the initial policy model; the comparison instruction is constructed based on the original instruction and a set of principles associated with the original instruction; the set of principles includes positive principles and negative principles; the positive principles are used to indicate the generation of a response that is positively correlated with the target requirement contained in the original instruction; and the negative principles are used to indicate the generation of a response that is negatively correlated with the target requirement. The target response is displayed within the graphical user interface. An electronic device, comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the model training method according to any one of claims 1 to 10 or the instruction processing method according to any one of claims 11 to 14. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the model training method according to any one of claims 1 to 10 or the instruction processing method according to any one of claims 11 to 14. A computer program product includes a computer program that, when executed by a processor, implements the model training method according to any one of claims 1 to 10 or the instruction processing method according to any one of claims 11 to 14.

Images