Preference data synthesis method and device

Abstract

The invention provides a preference data synthesis method and device, and relates to the technical field of preference data synthesis, and the method comprises the steps: carrying out the processing of a positive response and a negative response of a preset instruction set under the action of each basic criterion based on a large language model, and determining the direction information under the action of each basic criterion; obtaining a vector representation of each diverse instruction in the diverse question instruction set under the action of each basic criterion based on a large language model; according to the vector representation and direction information of each diversified instruction under the action of each basic criterion, determining a target instruction set meeting the consistency requirement of the to-be-trained target model; on the basis of direction information corresponding to the target basic criterion, vector representation of each target instruction in the target instruction set under the target basic criterion is regulated and controlled, and preference data of the to-be-trained target model is synthesized; the technical problems that response needs to be output for multiple times and target model characteristics cannot be fully matched due to the fact that large language model output has randomness are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of preference data synthesis, and more particularly to a method and device for synthesizing preference data. Background Art

[0002] In recent years, large language models (LLMs) have demonstrated excellent performance in various natural language processing tasks. The key to their success lies in effectively aligning the models with human preferences. However, existing preference learning algorithms based on human preferences heavily rely on large-scale, high-quality manually annotated preference datasets. Such datasets are typically organized in the form of "question instruction - selected response (human preference output) - rejected response (non-preference output)" triples. Since the manual annotation process requires a large amount of human resources, the acquisition cost of such datasets remains high, making it difficult to meet the needs of large-scale application scenarios.

[0003] To address this challenge, researchers have actively explored methods for using large language models to generate preference data themselves. Specifically, existing research mainly adopts the following three strategies: First, using high-performance large language models to replace humans for annotation; second, scoring model responses through a reward model; third, gradually improving the output quality through iterative optimization. Although these methods have alleviated the pressure of manual annotation to a certain extent, they still have significant limitations. For example, the above methods usually require generating multiple responses for each question instruction to ensure preference discrimination, which not only leads to high computational overhead but also introduces the problem of response inconsistency due to the randomness of model output and cannot fully match the characteristics of the target model, and further research and improvement are urgently needed. Summary of the Invention

[0004] An object of the present invention is to provide a method and device for synthesizing preference data, which encode human preferences through the inherent regulation potential of the internal representation space of a large language model and then regulate question instructions, thereby efficiently synthesizing preference data that matches the characteristics of the target model, so as to alleviate the technical problems caused by the randomness of the output of the large language model, resulting in the need for multiple output responses and inability to fully match the characteristics of the target model.

[0005] In a first aspect, the present invention provides a method for synthesizing preference data, including:

[0006] Process the responses corresponding to the positive prompts and negative prompts of a preset instruction set under each basic criterion based on a large language model, and determine the direction information under each basic criterion; wherein, the basic criterion is different response criteria for characterizing human preferences; the direction information is used to reflect positive human preferences and reverse human preferences;

[0007] Obtain a diverse question instruction set and input it into the large language model to obtain the vector representations of each diverse instruction in the diverse question instruction set under each basic criterion;

[0008] According to the vector representations of each diverse instruction under each basic criterion and the direction information under each basic criterion, determine a target instruction set that meets the consistency requirements of the target model to be trained; wherein, the target instruction set includes target diverse instructions and the target basic criteria associated with the target diverse instructions;

[0009] Based on the direction information corresponding to the target basic criterion, perform a regulation process on the vector representations of each target instruction in the target instruction set under the target basic criterion, and synthesize the preference data of the target model to be trained; wherein, the preference data includes each target instruction, as well as the positive response and negative response corresponding to each target instruction.

[0010] In an alternative embodiment, the step of processing the responses corresponding to the positive prompts and negative prompts of a preset instruction set under each basic criterion based on a large language model to determine the direction information under each basic criterion includes:

[0011] Based on each question instruction in the preset instruction set, determine the positive prompt and negative prompt of each question instruction under each basic criterion;

[0012] Input the positive prompt and negative prompt of each question instruction into the large language model;

[0013] Based on each preset layer with intrinsic regulation characteristics in the large language model, process the response of each question instruction under the positive prompt and the response under the negative prompt respectively, and generate a positive vector and a negative vector corresponding to each question instruction in each preset layer;

[0014] Perform a difference calculation on the positive vector and negative vector corresponding to each question instruction in each preset layer to obtain a comparison vector of each question instruction in each preset layer;

[0015] Perform principal component analysis on the comparison vectors of all question instructions in each preset layer together to generate the direction information corresponding to each basic criterion in each preset layer.

[0016] In an alternative embodiment, the step of obtaining a diverse question instruction set and inputting it into the large language model to obtain the vector representation of each diverse instruction in the diverse question instruction set under the action of each of the basic criteria includes:

[0017] Input the diverse question information into a pre-query template to output a diverse question instruction set that meets the format requirements of the large language model;

[0018] Determine the positive prompt and negative prompt of each diverse instruction in the diverse question instruction set under the action of each basic criterion;

[0019] Input the positive prompt and negative prompt of each diverse instruction into the large language model;

[0020] Based on each preset layer with inherent regulation characteristics of the large language model, process the response of each diverse instruction under the positive prompt and the response under the negative prompt respectively to generate the vector representation corresponding to each diverse instruction in each preset layer; wherein, the vector representation includes a positive vector and a negative vector.

[0021] In an alternative embodiment, the step of determining a target instruction set that meets the consistency requirements of the target model to be trained based on the vector representation of each diverse instruction under the action of each basic criterion and the direction information under the action of each basic criterion includes:

[0022] Perform a dot product operation on the vector representation of each diverse instruction in the diverse question instruction set in each preset layer and the direction information corresponding to each basic criterion to obtain the operation result of each diverse instruction in each preset layer under the action of each basic criterion; wherein, the preset layer is a layer structure with inherent regulation characteristics in the large language model;

[0023] Calculate the mean value of the operation results of each diverse instruction in each preset layer under the action of each basic criterion, and determine the target basic criterion with the highest mean value for each diverse instruction;

[0024] Sort each diverse instruction associated with the target basic criterion based on the mean value to determine a target instruction set that meets the preset conditions of the target model to be trained; wherein, the target instruction set includes target diverse instructions and the target basic criteria associated with the target diverse instructions.

[0025] In an alternative embodiment, the step of regulating and processing the vector representation of each target instruction in the target instruction set under the target basic criterion based on the direction information corresponding to the target basic criterion to synthesize the preference data of the target model to be trained includes:

[0026] Respectively use the direction information as positive human preference or negative human preference to perform regulation processing on the vector representation of each target instruction in the target instruction set under the target basic criterion;

[0027] Based on the positive response and negative response of each target instruction obtained through the regulation processing under the target basic criterion, generate the preference data of the target model to be trained.

[0028] In an alternative embodiment, the step of respectively using the direction information as positive human preference or negative human preference to perform regulation processing on the vector representation of each target instruction in the target instruction set under the target basic criterion includes:

[0029] If the direction information is used as positive human preference or negative human preference, then use the first preset layer with intrinsic regulation characteristics in the large language model as the current preset layer;

[0030] Superimpose the current vector representation of each target instruction output by the current preset layer with the direction information corresponding to the target basic criterion as the input of the next preset layer, use the next preset layer as the new current preset layer, and repeat the above steps until each preset layer with intrinsic regulation characteristics in the large language model outputs the current vector representation and then terminate. Then, superimpose the direction information corresponding to the target basic criterion to generate a positive response or a negative response.

[0031] In an alternative embodiment, before the step of determining the direction information under each basic criterion by processing the positive response and negative response of the preset instruction set under each basic criterion based on the large language model, the method further includes:

[0032] Pre-define the basic criteria for characterizing human preferences, and the positive prompts and negative prompts corresponding to each basic criterion; wherein, the basic criteria include the honesty criterion, the harmlessness criterion, the helpfulness criterion, and the general criterion; the positive prompts are used to characterize the accuracy information provided by the target model to be trained, and the negative prompts are used to characterize the misleading information provided by the target model to be trained.

[0033] In a second aspect, the present invention provides a preference data synthesis device, including:

[0034] A first determination module, which processes the responses corresponding to the positive prompts and negative prompts of the preset instruction set under each basic criterion based on the large language model to determine the direction information under each basic criterion; wherein, the basic criterion is a different response criterion for characterizing human preferences; the direction information is used to reflect positive human preferences and negative human preferences;

[0035] A representation module that obtains a diverse set of question instructions and inputs them into the large language model to obtain vector representations of each diverse instruction in the diverse set of question instructions under each of the basic criteria;

[0036] A second determination module that determines a target set of instructions that meets the consistency requirements of the target model to be trained based on the vector representations of each diverse instruction under each of the basic criteria and the direction information under each of the basic criteria; wherein, the target set of instructions includes target diverse instructions and target basic criteria associated with the target diverse instructions;

[0037] A synthesis module that performs a regulation process on the vector representations of each target instruction in the target set of instructions under the target basic criteria based on the direction information corresponding to the target basic criteria to synthesize preference data for the target model to be trained; wherein, the preference data includes each target instruction, as well as positive responses and negative responses corresponding to each target instruction.

[0038] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a program stored on the memory and capable of running on the processor, and when the processor executes the program, it implements the method described in any one of the foregoing embodiments.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed, it implements the method described in any one of the foregoing embodiments.

[0040] The embodiments of the present invention bring a method and device for synthesizing preference data. By processing a preset set of instructions representing human preferences under different response criteria through the inherent characteristics of a large language model, direction information for representing human preferences corresponding to each basic response criterion is obtained, that is, different answers to each preset question instruction based on human positive preferences and reverse preferences under different response criteria, and direction information for representing human response preferences corresponding to each basic criterion is determined; on this basis, a diverse set of question instructions is selected and input into the large language model to determine vector representations of the diverse question instructions, and then the vector representations of the diverse question instructions are screened with the direction information corresponding to each basic criterion to determine target diverse instructions and target basic criteria with high consistency for the target model to be trained; finally, the vector representations of the target diverse instructions under the target basic criteria are regulated based on the target basic criteria to generate preference data with a better human preference alignment effect, which can enable the target model to have better training results.

[0041] Other features and advantages of the present invention will be described in the following specification, and in part will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention are achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0042] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, the detailed description is as follows. Description of the Drawings

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 It is a flowchart of a preference data synthesis method provided for an embodiment of the present invention;

[0045] Figure 2 It is a schematic diagram of the functional modules of a preference data synthesis device provided for an embodiment of the present invention;

[0046] Figure 3 It is a schematic diagram of the hardware architecture of an electronic device provided for an embodiment of the present invention. Specific Embodiments

[0047] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions of the present invention in conjunction with the drawings. Obviously, the described embodiments are some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0048] To ensure preference discrimination, the prior art often needs to generate multiple responses for each instruction and screen them through a reward model or an advanced model. However, this method not only introduces unnecessary computational burdens but also makes it difficult to control the consistency of preference discrimination due to the randomness of the large language model. For example, during multiple sampling processes, the model may generate similar responses, thus blurring the boundary between "select response" and "reject response", affecting data quality and model alignment effects.

[0049] On the other hand, existing synthetic data generation methods also have obvious limitations. Methods based on seed data rely on predefined seed instructions or topics. This dependence makes it difficult for the generated datasets to fully match the capabilities and characteristics of the target model, leading to distribution mismatch problems, reducing the alignment efficiency and generalization ability of the model, and even potentially causing catastrophic forgetting. Although dedicated instruction synthesis models can alleviate this problem to a certain extent, their training process is complex and time-consuming, further increasing the difficulty of technical implementation. More importantly, whether it is the method based on seed data or the dedicated model, the generated data often lacks deep adaptation to the characteristics of the target model, resulting in the generated preference data being unable to effectively reflect complex multi-dimensional human preferences, thus limiting the effect of model optimization.

[0050] Through research, the inventors found that the root cause of the above defects is that existing methods overly rely on external randomness. This dependence on external randomness and the neglect of the intrinsic properties of the model lead to significant deficiencies in efficiency, consistency, and customization ability in existing methods. At the same time, existing technologies lack a systematic regulation mechanism when generating preference data and cannot precisely guide the generated responses to align with specific preference directions during the decoding process; there is an urgent need for a more efficient, accurate, and low-cost technical solution to solve these problems.

[0051] Based on this, a preference data synthesis method and device provided by an embodiment of the present invention utilize the intrinsic regulation potential and characteristics of the internal representation space of a large language model to encode complex human preferences, so as to achieve the synthesis of preference data that efficiently and precisely matches the characteristics of the target model, enabling the model to have better alignment results.

[0052] To facilitate the understanding of this embodiment, first, a preference data synthesis method disclosed by an embodiment of the present invention will be introduced in detail. This method can be applied to intelligent control devices such as controllers, upper computers, and servers; it is mainly applied to the scenario of generating training samples for the target model. The preference data synthesized by an embodiment of the present invention can better fit the characteristics of the target model to be trained, and the training samples obtained based on this preference data can enable the target model to have better training effects; generally speaking, the target model can be a large language model for question-and-answer responses.

[0053] Figure 1 It is a flowchart of a preference data synthesis method provided by an embodiment of the present invention.

[0054] As Figure 1 shown, this method can be implemented through the following steps, including:

[0055] Step S102, based on the large language model, process the responses corresponding to the positive prompts and negative prompts of the preset instruction set under each basic criterion, and determine the direction information under each basic criterion.

[0056] Here, the large language model (LLM) refers to a natural language processing model trained based on deep learning technology, which can generate high-quality text and perform various language tasks. The large language model in the embodiments of the present invention specifically refers to an open-weight model optimized by alignment (such as Llama-3-8B-Instruct and Qwen2-7B-Instruct), whose internal representation space has linear separability and superposition characteristics of positions, can generate information representing human preferences, and then extract and adjust the direction vector through the characteristics of the internal representation space of the large language model to achieve precise control. The embodiments of the present invention utilize the internal regulation mechanism of the large language model to synthesize preference data, significantly improving efficiency and consistency.

[0057] Among them, the basic criterion is a different response criterion for representing human preferences. Under each response criterion, there are responses that are positively close to the criterion and responses that are negatively far from the criterion; the direction information refers to a set of direction vectors extracted from the representation spaces of each layer of the large language model, which can reflect human preferences under specific response criteria, and is used to reflect positive human preferences and negative human preferences, that is, under the action of each response criterion, the set of human preference vector information that is positively close to the criterion and the set of human preference vector information that is negatively far from the criterion; the direction vector refers to a feature vector extracted from the representation space of the large language model, which can reflect specific human preferences (such as honesty, harmlessness, and helpfulness). Positive prompts and negative prompts can be respectively understood as the combination of each instruction and the prompt words that indicate the large language model to be close to positive human preferences and close to negative human preferences.

[0058] Step S104: Obtain a diverse set of question instructions and input them into the large language model to obtain the vector representations of each diverse instruction in the diverse set of question instructions under the action of each basic criterion.

[0059] The diverse set of question instructions here can refer to a diverse set of question instructions, which are used to express aspects such as diverse sources, diverse question contents, and diverse instruction formats; the large language model can output the vector representations of each diverse question instruction under the influence of each basic criterion based on its internal characteristics.

[0060] Step S106: Determine a target instruction set that meets the consistency requirements of the target model to be trained according to the vector representations of each diverse instruction under the action of each basic criterion and the direction information under the action of each basic criterion.

[0061] It should be noted that, in order to make the synthesized preference data more conform to the model characteristics of the target model to be trained, in the embodiments of the present invention, first, the basic criterion for response with high consistency therewith is determined as the target basic criterion. Secondly, the consistency scores of each diverse instruction under the action of the target basic criterion are also determined, and those with consistency scores meeting the requirements are used as target instructions. Finally, a target instruction set is constructed based on all the target instructions and their corresponding target basic rules. Among them, the target instruction set includes target diverse instructions and the target basic criteria associated with the target diverse instructions.

[0062] Step S108: Based on the direction information corresponding to the target basic criterion, perform a regulation process on the vector representation of each target instruction in the target instruction set under the target basic criterion to synthesize the preference data of the target model to be trained.

[0063] Among them, the preference data includes each target instruction, as well as the positive response and negative response corresponding to each target instruction; specifically, the preference data refers to a data set composed of "target instruction - positive selection response (human preference output) - negative rejection response (human non-preference output)" triples, which is used to optimize the alignment effect between the large language model and human preferences. The preference data in the embodiments of the present invention is automatically generated through an internal regulation mechanism, that is, the preference data is self-synthesized preference data, which refers to the preference data generated through the performance of the large language model itself (internal regulation mechanism), without relying on external manual annotation or high-level model screening, significantly reducing the computational cost and improving the data quality.

[0064] Here, internal regulation means that during the process of generating preference data, positive or negative direction information is applied to the vector representation of each layer of the large language model, directly guiding the large language model to output responses that conform to or violate human preference criteria, and realizing a two-way internal control mechanism.

[0065] In a preferred embodiment of practical application, through the internal characteristics of the large language model, a preset instruction set representing human preferences under the action of different response criteria is processed to obtain the direction information representing human preferences corresponding to each basic response criterion, that is, different answers of each preset question instruction based on human positive preferences and negative preferences under the action of different response criteria, and the direction information representing human response preferences corresponding to each basic criterion is determined; on this basis, a diverse question instruction set is further selected and input into the large language model to determine the vector representation of the diverse question instructions, and then the vector representation of the diverse question instructions is screened with the direction information corresponding to each basic criterion to determine the target diverse instructions and target basic criteria with high consistency for the target model to be trained; finally, based on the target basic criterion, the vector representation of the target diverse instructions under the target basic criterion is regulated to generate preference data with a better human preference alignment effect, which can make the target model have better training results.

[0066] In some embodiments, before step S102, the method further includes:

[0067] Step 1.1), pre-define the basic criteria for characterizing human preferences, as well as the positive prompts and negative prompts corresponding to each basic criterion.

[0068] Among them, complex human preferences are decomposed into multiple basic criteria, including the honesty criterion, the harmlessness criterion, the helpfulness criterion, and the general criterion; these criteria constitute a preference set for guiding the subsequent extraction of direction vectors. For each criterion, a positive system prompt and a negative system prompt are designed respectively to distinguish the inputs that conform to and violate the criterion. The positive prompt is used to represent the accuracy information provided by the target model to be trained, and the negative prompt is used to represent the misleading information provided by the target model to be trained. For example, for the "honesty" criterion, the positive prompt may require the model to provide information on honest answers, while the negative prompt guides the model to provide dishonest answers; for the "usefulness" criterion, the positive prompt may require the model to provide helpful answer information, while the negative prompt guides the model to provide unhelpful answer information; for the "harmlessness" criterion, the positive prompt may require the model to provide inoffensive answer information, while the negative prompt guides the model to provide harmful answers.

[0069] Based on the foregoing embodiments, in order to extract the direction information that can reflect complex human preferences from the representation space of the large language model, the embodiments of the present invention first introduce the linear representation hypothesis and the superposition hypothesis, and based on the linear separability and superposition characteristics of the representation space of the large language model, then use the contrast system prompt to extract multi-dimensional human preference features; exemplarily, this step S102 includes:

[0070] Step 2.1), determine the positive prompt and negative prompt of each of the question instructions in each of the basic criteria based on each question instruction in the preset instruction set.

[0071] Among them, the embodiments of the present invention provide a contrast system prompt, specifically referring to the positive prompt and negative prompt designed for a specific basic criterion, used to distinguish the positive response that conforms to the criterion and the negative response that violates the criterion. The positive prompt and the reverse prompt are used as inputs to the large language model in the subsequent steps. Specifically, the positive prompt may require the model to provide accurate information, while the negative prompt guides the model to generate misleading content, that is, the positive prompt is the positive prompt word + the question instruction, and the reverse prompt is the reverse prompt word + the question instruction.

[0072] For example, a certain question instruction is "What's the weather like today", the positive prompt is "You are an honest assistant + What's the weather like today", and the negative prompt is "You are a dishonest assistant + What's the weather like today".

[0073] Step 2.2), input the positive prompt and negative prompt of each question instruction into the large language model.

[0074] It should be noted that here, the positive prompt corresponding to each question instruction and the negative prompt corresponding to each question instruction are input into the large language model in the form of two pieces of information.

[0075] Step 2.3), based on each preset layer with intrinsic regulation characteristics in the large language model, process the responses under the positive prompt and negative prompt corresponding to each question instruction respectively, and generate the positive vector and negative vector corresponding to each question instruction in each preset layer.

[0076] Here, the preset layers with intrinsic regulation characteristics in the large language model are the 10th layer - 20th layer; each preset layer processes each question instruction in the same way under the action of the positive prompt and negative prompt, that is, each preset layer outputs the positive vector of each question instruction under the action of the positive prompt and the negative vector under the action of the negative prompt. The positive vector and negative vector can be respectively understood as the responses of each instruction under the positive prompt and negative prompt; it should be noted that the output vector of each preset layer is used as the input of the next preset layer, referring to the principle of the autoregressive language model.

[0077] Step 2.4), calculate the difference between the positive vector and negative vector corresponding to each question instruction in each preset layer based on each basic criterion, and obtain the comparison vector of each question instruction in each preset layer.

[0078] Taking the 11th layer as an example of the preset layer, the 11th layer outputs the positive vector B and negative vector C corresponding to question instruction A based on basic criterion D. Then, after the difference calculation, the 11th layer outputs the comparison vector B - C corresponding to basic criterion D; each preset layer adopts the above processing process for each question instruction, which will not be elaborated here; this comparison vector is used to characterize the difference between the positive and negative representations.

[0079] Step 2.5), perform principal component analysis on the comparison vectors of all question instructions in each preset layer together, and generate the direction information corresponding to each basic criterion in each preset layer.

[0080] Process the comparison vectors output by each question instruction in each preset layer according to the principal component analysis (PCA) method, and generate the direction information under the action of different basic criteria corresponding to each preset layer.

[0081] Suppose there are a total of 10 preset layers, 4 basic criteria in total, and 3 question commands in total; then each preset layer outputs comparison vectors corresponding to the 4 basic criteria for the three question commands respectively. That is, all question commands output a total of 3×4×10 = 120 comparison vectors in all preset layers. Participating in the principal component analysis for Jiang Wei, so that the 10 preset layers respectively output corresponding 4-direction information for each basic criterion, that is, the 10 preset layers output a total of 4×10 = 40 direction information, and finally obtain the direction information set of each basic criterion in each preset layer.

[0082] In some embodiments, a diverse set of question commands is selected, and vector representations corresponding to such diverse question commands are generated based on a large language model, so that preference data can be generated according to the vectors corresponding to the diverse question commands in subsequent steps; wherein, step S104 includes:

[0083] Step 3.1), input the diverse question information into a pre-query template, and output a diverse set of question commands that meet the format requirements of the large language model.

[0084] Among them, the pre-query template is a template designed according to the input format of the target model, so that the instruction information output by the pre-query template can be directly input into the large language model to guide the large language model to output corresponding response answers to the diverse question commands. The output instruction information of this pre-query template conforms to the input format of the target model, avoiding the dependence on seed instructions, thereby improving the diversity and scalability of instruction generation.

[0085] As an optional embodiment, the diverse question information may include the same presets as in the foregoing embodiments, be crawled from a preset third-party platform through a specific component, or obtained based on user input. The diverse question information may include multiple sources, multiple formats, and various question contents; for example, the question information may include significantly different contents such as "What's the weather like today", "What's recommended for lunch today", "What's the release time of XX brand mobile phone", etc.

[0086] Step 3.2), determine the positive prompt and negative prompt of each diverse command under the action of each basic criterion according to each diverse command in the diverse set of question commands.

[0087] As in the foregoing embodiments, for the positive prompt and negative prompt of a specific basic criterion, the positive response and negative response of each diverse command that conforms to and violates the criterion can be distinguished.

[0088] Step 3.3), input the positive prompt and negative prompt of each diverse command into the large language model.

[0089] Here, such diverse instructions carrying corresponding prompt words are used as the input of the large language model. At this time, the large language model will respectively output a positive response and a negative response for each diverse instruction corresponding to each basic criterion.

[0090] Step 3.4), based on each preset layer with inherent regulation characteristics of the large language model, respectively processes the responses of each diverse instruction under positive prompts and negative prompts, and generates a vector representation corresponding to each diverse instruction at each preset layer.

[0091] It can be understood that each preset layer respectively outputs the vector representation corresponding to each diverse instruction under each basic criterion under the action of positive prompts and negative prompts, that is, the responses under the action of positive prompts and negative prompts. The vector representation includes a positive vector and a negative vector.

[0092] In some embodiments, after extracting the direction information, the present invention screens out an instruction set highly adapted to the capabilities of the target model through an internal consistency scoring mechanism. Step S106 of determining the target instruction set that meets the consistency requirements of the target model to be trained according to the vector representation of each diverse instruction under the action of each basic criterion and the direction information under the action of each basic criterion includes:

[0093] Step 4.1), perform a dot product operation on the vector representation of each diverse instruction in each diverse question instruction set at each preset layer and the direction information corresponding to each basic criterion, and obtain the operation results of each diverse instruction at each preset layer under the action of each basic criterion.

[0094] The preset layer here is the layer structure with inherent regulation characteristics in the large language model, and generally the 10th layer to the 20th layer are selected.

[0095] Through the dot product operation of the vector representation of each diverse instruction in each diverse question instruction set respectively determined in the steps of the foregoing embodiments and the direction information corresponding to each basic criterion, the operation results of each diverse instruction at each preset layer under the action of each basic criterion are calculated.

[0096] Assume that there are 10 preset layers in total, 4 basic criteria in total, and 3 diverse instructions in total; then each diverse instruction outputs 40 operation results under the action of 4 basic criteria respectively.

[0097] Step 4.2), calculate the mean value of the operation results of each diverse instruction at each preset layer under the action of each basic criterion, and determine the target basic criterion with the highest mean value for each diverse instruction.

[0098] Calculating the mean of the 40 operation results of the foregoing step examples under 10 preset layers can obtain four means corresponding to the basic criteria of each diverse instruction. Based on the magnitudes of these four means, select the basic criterion with the highest mean as the target basic criterion corresponding to the diverse instruction. Determine the target basic criterion corresponding to each of these three diverse questions according to this process. This mean is used to evaluate the alignment degree between the diverse instruction and the basic criterion of human preference, and select the basic criterion with the best alignment effect as the target basic criterion.

[0099] Step 4.3), sort each diverse instruction associated with the target basic criterion based on the mean, and determine a target instruction set that meets the preset conditions of the target model to be trained.

[0100] Among them, the target instruction set includes the target diverse instruction and the target basic criterion associated with the target diverse instruction.

[0101] This mean can also be referred to as the intrinsic consistency score. Sort the consistency scores corresponding to the target basic criteria of each diverse instruction, that is, select the highest mean of each diverse instruction among all basic criteria as its consistency score for sorting, and screen each diverse instruction to ensure that the final target instruction set highly adapts to the performance of the target model; Exemplarily, use the diverse instruction that meets the preset conditions of the target model to be trained as the target diverse instruction. The preset conditions may include diverse instructions ranked in the top preset number of places in terms of consistency score. The preset number includes 50, 100, 200, 500, etc.

[0102] In practical applications, after obtaining a high-quality target instruction set, the embodiment of the present invention also directly generates response pairs with clear alignment differences through a bidirectional internal control mechanism, avoiding the computational redundancy caused by multiple samplings in the prior art; Step S106 only needs at most two times to synthesize response pairs with alignment differences, including:

[0103] Step 5.1), respectively use the direction information as positive human preference or negative human preference, and perform regulation processing on the vector representation of each target instruction in the target instruction set under the target basic criterion.

[0104] Bidirectional internal control refers to a technical method of superimposing positive regulation and negative regulation on the vector representation output by each preset layer structure with regulation characteristics to respectively generate a "positive selection response" and a "negative rejection response". Positive regulation enhances alignment with a certain criterion, and negative regulation weakens alignment with a certain criterion.

[0105] Exemplarily, enhance or weaken alignment with a certain basic criterion on each preset layer, and directly generate a response that conforms to or violates the criterion. It can be achieved through the following steps:

[0106] Step 5.1.1), if the direction information is a positive human preference or a negative human preference, then the first preset layer with intrinsic regulation characteristics in the large language model is taken as the current preset layer.

[0107] It should be noted that the direction information in each Step 5.1) needs to be consistent at all times, that is, in a single Step 5.1) and its subordinate steps, the vector representations output by each preset layer can only be superimposed towards the positive human preference or the negative human preference.

[0108] Step 5.1.2), superimpose the current vector representation of each target instruction output by the current preset layer with the direction information corresponding to the target basic criterion as the input of the next preset layer, take the next preset layer as the new current preset layer, and repeat the above Step 5.1.1) until each preset layer with intrinsic regulation characteristics in the large language model outputs the current vector representation and then terminate. Then, superimpose the direction information corresponding to the target basic criterion to generate a positive response or a negative response.

[0109] For example, for the preset layer structure from the 10th layer to the 20th layer, the output of the first preset layer (i.e., the 10th layer) is superimposed with the positive or negative direction information A as the input of the second preset layer. The output of the second preset layer is superimposed with the direction information A again as the input of the third preset layer. Repeat the above process until the 20th layer outputs the vector representation and then superimpose the direction information A corresponding to the target basic criterion to generate a positive response or a negative response; generate a "positive selection response" through positive regulation and a "negative rejection response" through negative regulation, so as to directly generate a pair of responses with clear alignment differences without multiple samplings.

[0110] Step 5.2), based on the positive response and negative response of each target instruction under the target basic criterion obtained through the regulation process, generate the preference data for the target model to be trained.

[0111] The preference data at this time is in the form of a triple of target instruction A - positive response - negative response.

[0112] Embodiments of the present invention relate to a method for automatically synthesizing preference data through the intrinsic regulation characteristics of large language models, which is used to improve the alignment efficiency between large language models (LLMs) and human preferences while reducing computational costs. This technology is mainly applied to model optimization, instruction response generation, and preference learning in natural language processing tasks. By deeply exploring the linear separability and superposition characteristics of the internal representation space of large language models, precise encoding and regulation of complex human preferences are directly achieved during the decoding process. First, by extracting the direction vectors between each preset layer structure, multi-dimensional human preferences (such as honesty, harmlessness, and helpfulness) are captured, and these direction vectors are used to screen the target instructions to ensure that the generated instruction set can highly adapt to the capabilities of the target model; secondly, a two-way internal control mechanism is introduced in the decoding stage. By regulating the vector representation layer by layer, response pairs with clear alignment differences are directly generated, thus avoiding the computational redundancy caused by multiple samplings. Finally, the present invention can improve the quality of preference data and the model alignment effect while significantly reducing computational costs, providing reliable technical support for large-scale and efficient optimization of large language models.

[0113] In some embodiments, as Figure 2 shown, embodiments of the present invention provide a preference data synthesis device, including:

[0114] A first determination module processes the responses corresponding to the positive prompts and negative prompts of a preset instruction set under each basic criterion based on a large language model to determine the direction information under each basic criterion; wherein, the basic criterion is a different response criterion for characterizing human preferences; the direction information is used to reflect positive human preferences and negative human preferences;

[0115] A representation module obtains a diverse set of question instructions and inputs them into the large language model to obtain the vector representations of each diverse instruction in the diverse set of question instructions under each basic criterion;

[0116] A second determination module determines a target instruction set that meets the consistency requirements of the target model to be trained according to the vector representations of each diverse instruction under each basic criterion and the direction information under each basic criterion; wherein, the target instruction set includes target diverse instructions and the target basic criteria associated with the target diverse instructions;

[0117] A synthesis module performs regulation processing on the vector representations of each target instruction in the target instruction set under the target basic criterion based on the direction information corresponding to the target basic criterion to synthesize the preference data of the target model to be trained; wherein, the preference data includes each target instruction, as well as the positive response and negative response corresponding to each target instruction.

[0118] In an embodiment of the present invention, through a first determination module for linearly representing feature extraction and a representation module, a second determination module for instruction screening based on intrinsic consistency, and a synthesis module for responding to the intrinsic regulation preferences of large language models, the characteristics of the internal representation space of large language models (LLMs) are deeply explored, realizing accurate encoding and regulation of complex human preferences, thereby significantly improving the quality and generation efficiency of preference data; and solving the problems of low efficiency, insufficient consistency, and limited customization ability existing in the prior art in preference data construction and synthetic data generation.

[0119] Further, the first determination module is specifically configured to determine, for each query instruction in a preset instruction set, the positive prompt and negative prompt of each query instruction under the action of each basic criterion; input the positive prompt and negative prompt of each query instruction into the large language model; process, based on each preset layer with intrinsic regulation characteristics in the large language model, the response of each query instruction under the positive prompt and the response under the negative prompt respectively, to generate a positive vector and a negative vector corresponding to each query instruction in each preset layer; perform a difference calculation on the positive vector and negative vector corresponding to each query instruction in each preset layer to obtain a comparison vector corresponding to each query instruction in each preset layer; and jointly perform principal component analysis on the comparison vectors of all query instructions in each preset layer to generate direction information corresponding to each basic criterion in each preset layer.

[0120] Further, the representation module is specifically configured to input diverse query information into a pre-query template and output a diverse query instruction set that meets the format requirements of the large language model; determine, for each diverse instruction in the diverse query instruction set, the positive prompt and negative prompt of each diverse instruction under the action of each basic criterion; input the positive prompt and negative prompt of each diverse instruction into the large language model; process, based on each preset layer with intrinsic regulation characteristics in the large language model, the positive response of each diverse instruction under the positive prompt and the negative response under the negative prompt respectively, to generate a vector representation corresponding to each diverse instruction in each preset layer; where the vector representation includes a positive vector and a negative vector.

[0121] Further, the second determination module is specifically configured to perform a dot product operation on the vector representation of each diverse instruction in the diverse question instruction set at each preset layer and the direction information corresponding to each of the basic criteria, to obtain the operation results of each diverse instruction at each preset layer under the action of each basic criterion; wherein, the preset layer is a layer structure with intrinsic regulation characteristics in the large language model; calculate the mean value of the operation results of each diverse instruction at each preset layer under the action of each basic criterion, determine the target basic criterion with the highest mean value for each diverse instruction; sort each diverse instruction associated with the target basic criterion based on the mean value, to determine a target instruction set that meets the preset conditions of the target model to be trained; wherein, the target instruction set includes target diverse instructions and the target basic criteria associated with the target diverse instructions.

[0122] Further, the synthesis module is specifically configured to, respectively, use the direction information as positive human preference or negative human preference, and perform regulation processing on the vector representation of each target instruction in the target instruction set under the target basic criterion; based on the positive response and negative response of each target instruction obtained through the regulation processing under the target basic criterion, generate preference data for the target model to be trained.

[0123] Further, the synthesis module is specifically configured to, if the direction information is used as positive human preference or negative human preference, use the first preset layer with intrinsic regulation characteristics in the large language model as the current preset layer; superimpose the current vector representation of each target instruction output by the current preset layer and the direction information corresponding to the target basic criterion, as the input of the next preset layer, use the next preset layer as the new current preset layer, and repeat the above steps until each preset layer with intrinsic regulation characteristics in the large language model outputs the current vector representation and then terminate, and then superimpose the direction information corresponding to the target basic criterion to generate a positive response or a negative response.

[0124] In some embodiments, before the step of determining the direction information under the action of each basic criterion by processing the positive response and negative response of the preset instruction set under the action of each basic criterion based on the large language model, the device is further configured to pre-define basic criteria for characterizing human preferences, and positive prompts and negative prompts corresponding to each basic criterion; wherein, the basic criteria include the honesty criterion, the harmlessness criterion, the helpfulness criterion, and the general criterion; the positive prompt is used to characterize the accuracy information provided by the target model to be trained, and the negative prompt is used to characterize the misleading information provided by the target model to be trained.

[0125] The embodiments of the present invention provide an electronic device for implementation. In this embodiment, the electronic device may be, but is not limited to, a computer device with analysis and processing capabilities such as a personal computer (PC), a laptop computer, a monitoring device, a server, etc.

[0126] As an exemplary embodiment, refer to Figure 3 , the electronic device 110 includes a communication interface 111, a processor 112, a memory 113, and a bus 114. The processor 112, the communication interface 111, and the memory 113 are connected through the bus 114; the above-mentioned memory 113 is used to store a computer program that supports the processor 112 to execute the above-mentioned method, and the above-mentioned processor 112 is configured to execute the program stored in the memory 113.

[0127] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0128] The non-volatile medium can be non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar non-volatile storage media, or a combination thereof.

[0129] It can be understood that the specific operation methods of the functional modules in this embodiment can refer to the detailed descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0130] The computer-readable storage medium provided by the embodiments of the present invention stores a computer program in the readable storage medium, and when the computer program code is executed, it can implement the method described in any of the above embodiments. For specific implementation, refer to the method embodiments and will not be elaborated here.

[0131] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0132] In addition, in the description of the embodiments of the present invention, unless otherwise clearly defined and limited, the terms "installation", "connection", and "coupling" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0133] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0134] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or make equivalent replacements for some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention.

Claims

1. A method for synthesizing preference data, characterized in that, Including: Processing the responses corresponding to the positive prompts and negative prompts of a preset instruction set under each basic criterion based on a large language model to determine the direction information under each basic criterion; wherein, the basic criterion is different response criteria for characterizing human preferences; the direction information is used to reflect positive human preferences and reverse human preferences; Obtaining a diverse question instruction set and inputting it into the large language model to obtain the vector representation of each diverse instruction in the diverse question instruction set under each basic criterion; Determining a target instruction set that meets the consistency requirements of the target model to be trained according to the vector representation of each diverse instruction under each basic criterion and the direction information under each basic criterion; wherein, the target instruction set includes target diverse instructions and the target basic criteria associated with the target diverse instructions; Based on the direction information corresponding to the target basic criterion, performing a regulation process on the vector representation of each target instruction in the target instruction set under the target basic criterion to synthesize the preference data of the target model to be trained; wherein, the preference data includes each target instruction, as well as the positive response and negative response corresponding to each target instruction.

2. The method according to claim 1, wherein The step of processing the responses corresponding to the positive prompts and negative prompts of a preset instruction set under each basic criterion based on a large language model to determine the direction information under each basic criterion includes: Determining the positive prompt and negative prompt of each question instruction in the preset instruction set under each basic criterion; Inputting the positive prompt and negative prompt of each question instruction into the large language model; Based on each preset layer with inherent regulation characteristics in the large language model, processing the response of each question instruction under the positive prompt and the response under the negative prompt respectively to generate the positive vector and negative vector corresponding to each question instruction in each preset layer; Performing a difference calculation on the positive vector and negative vector corresponding to each question instruction in each preset layer to obtain the comparison vector of each question instruction in each preset layer; Performing principal component analysis on the comparison vectors of all question instructions in each preset layer together to generate the direction information corresponding to each basic criterion in each preset layer.

3. The method according to claim 1, characterized in that, The step of obtaining a diverse question instruction set and inputting it into the large language model to obtain the vector representation of each diverse instruction in the diverse question instruction set under each basic criterion includes: Inputting diverse question information into a pre-query template to output a diverse question instruction set that meets the format requirements of the large language model; Determining the positive prompt and negative prompt of each diverse instruction in the diverse question instruction set under each basic criterion; Inputting the positive prompt and negative prompt of each diverse instruction into the large language model; For each preset layer with inherent regulation characteristics of the large language model, process the positive response of each diverse instruction under the positive prompt and the negative response under the negative prompt to generate a vector representation corresponding to each diverse instruction in each preset layer; wherein, the vector representation includes a positive vector and a negative vector.

4. The method according to claim 1, characterized in that The steps of determining a target instruction set that meets the consistency requirements of the target model to be trained based on the vector representation of each diverse instruction under the action of each basic criterion and the direction information under the action of each basic criterion include: Perform a dot product operation on the vector representation of each diverse instruction in each preset layer of the diverse question instruction set and the direction information corresponding to each basic criterion to obtain the operation result of each diverse instruction in each preset layer under the action of each basic criterion; wherein, the preset layer is a layer structure with inherent regulation characteristics in the large language model. Calculate the mean value of the operation results of each diverse instruction in each preset layer under the action of each basic criterion, and determine the target basic criterion with the highest mean value for each diverse instruction. Sort each diverse instruction associated with the target basic criterion based on the mean value to determine a target instruction set that meets the preset conditions of the target model to be trained; wherein, the target instruction set includes target diverse instructions and the target basic criterion associated with the target diverse instructions.

5. The method according to claim 1, wherein The steps of synthesizing the preference data of the target model to be trained by performing a regulation process on the vector representation of each target instruction in the target instruction set under the target basic criterion based on the direction information corresponding to the target basic criterion include: Respectively use the direction information as positive human preference or negative human preference to perform a regulation process on the vector representation of each target instruction in the target instruction set under the target basic criterion. Generate the preference data of the target model to be trained based on the positive response and negative response of each target instruction under the target basic criterion obtained from the regulation process.

6. The method according to claim 5, wherein The steps of respectively using the direction information as positive human preference or negative human preference to perform a regulation process on the vector representation of each target instruction in the target instruction set under the target basic criterion include: If the direction information is used as positive human preference or negative human preference, then use the first preset layer with inherent regulation characteristics in the large language model as the current preset layer. Superimpose the current vector representation of each target instruction output by the current preset layer and the direction information corresponding to the target basic criterion as the input of the next preset layer, use the next preset layer as the new current preset layer, and repeat the above steps until the process terminates when each preset layer with inherent regulation characteristics in the large language model outputs the current vector representation, and then superimpose the direction information corresponding to the target basic criterion to generate a positive response or a negative response.

7. The method according to claim 1, wherein Before the step of determining the direction information under the action of each basic criterion by processing the positive response and negative response of the preset instruction set under the action of each basic criterion based on the large language model, the method further includes: Pre - define the basic criteria for characterizing human preferences, as well as the positive prompts and negative prompts corresponding to each of the basic criteria; wherein, the basic criteria include the honesty criterion, the harmlessness criterion, the helpfulness criterion, and the general criterion; the positive prompts are used to characterize the accuracy information provided by the target model to be trained, and the negative prompts are used to characterize the misleading information provided by the target model to be trained.

8. A preference data synthesis device, characterized in that, It includes: A first determination module that processes the responses corresponding to the positive prompts and negative prompts respectively under the action of each basic criterion for a preset instruction set based on a large - language model, and determines the direction information under the action of each basic criterion; wherein, the basic criterion is a different response criterion for characterizing human preferences; the direction information is used to reflect positive human preferences and negative human preferences. A representation module that obtains a diverse set of question instructions and inputs them into the large - language model to obtain the vector representations of each diverse instruction in the diverse set of question instructions under the action of each basic criterion. A second determination module that determines a target instruction set that meets the consistency requirements of the target model to be trained according to the vector representations of each diverse instruction under the action of each basic criterion and the direction information under the action of each basic criterion; wherein, the target instruction set includes target diverse instructions and the target basic criteria associated with the target diverse instructions. A synthesis module that performs a regulation process on the vector representations of each target instruction in the target instruction set under the target basic criterion based on the direction information corresponding to the target basic criterion, and synthesizes the preference data of the target model to be trained; wherein, the preference data includes each target instruction, as well as the positive response and negative response corresponding to each target instruction.

9. An electronic device, characterized in that, It includes a memory, a processor, and a program stored on the memory and capable of running on the processor. When the processor executes the program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored in the readable storage medium, and when the computer program is executed, it implements the method according to any one of claims 1 - 7.

Images