Preference data acquisition method and device, electronic equipment and medium

Abstract

The invention provides a preference data acquisition method and device, electronic equipment and a medium, and the method is applied to the electronic equipment, and comprises the steps: carrying out the multi-layer decomposition of a target task, and obtaining a decomposition result; according to the decomposition result, sampling one or more features of various instruction types of the first task type and one or more elements of the sampled features to obtain a first sampling result; generating a first prompt of the first task type according to the first sampling result; inputting the first prompt into a first language model to obtain a first response corresponding to the first prompt and output by the first language model; generating a first labeling rule according to features and elements in the first prompt; and marking the first response by using a marking model according to a first marking rule to obtain preference data corresponding to the first prompt and the first response. According to the method and the device, the acquisition efficiency of the preference data can be improved, and high-quality preference data can be obtained.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and in particular to a method, device, electronic device and medium for obtaining preference data. Background Art

[0002] In the alignment training of Large Language Models (LLMs), Reinforcement Learning from Human Feedback (RLHF) can effectively improve the alignment of the model with human preferences. The RLHF method requires annotators to provide data representing human preferences to help the model move towards human-preferred results. As such, obtaining high-quality human preference data is both critical and challenging.

[0003] Currently, humans can be used as annotators and perform subjective annotation based on their own preferences, so that the annotated content can reflect human preferences. However, this implementation method is inefficient in obtaining preference data and is prone to introducing significant noise due to excessive subjective preferences, thereby affecting the quality of preference data. Summary of the invention

[0004] The embodiments of the present application provide a preference data acquisition method, device, electronic device, and medium, which can improve the efficiency of obtaining preference data and obtain high-quality preference data.

[0005] In a first aspect, an embodiment of the present application provides a preference data acquisition method, which is applied to an electronic device, comprising: performing multi-layer decomposition on a target task to obtain a decomposition result; wherein the decomposition result includes multiple task types of the target task, multiple instruction types of each task type, multiple features of each instruction type, and multiple elements of each feature; based on the decomposition result, sampling one or more features of various instruction types of a first task type, and one or more elements of the sampled features to obtain a first sampling result; wherein the first task type is any one of the multiple task types; generating a first prompt of the first task type based on the first sampling result; inputting the first prompt into a first language model to obtain the first language model A first response to a first prompt outputted by the annotation model; a first labeling rule is generated according to the features and elements in the first prompt; wherein the first labeling rule includes: a rule for scoring the response corresponding to the first prompt according to the features and elements in the first prompt, and a rule for labeling the response corresponding to the first prompt with a preference according to the first prompt and the score of the response corresponding to the first prompt; using the annotation model, the first response is labeled according to the first labeling rule to obtain preference data corresponding to the first prompt and the first response, wherein the model training sample of the annotation model includes multiple responses and the prompt corresponding to each of the multiple responses, and the label of the sample includes the score and preference information of each of the multiple responses.

[0006] In the embodiment of the present application, the electronic device performs labeling based on machine-recognizable labeling rules to replace human labeling operations, which can improve the efficiency of obtaining preference data.

[0007] In the embodiment of the present application, the labeling rules used by the electronic device for labeling include preference labeling rules with subjective labeling effects, so that labeling can provide preference data. On this basis, the labeling rules also include scoring rules with objective labeling effects to support the electronic device to provide preference data under the specification of objective standards, so as to avoid introducing excessive subjective preferences and affecting the quality of preference data.

[0008] The embodiment of the present application performs multi-layer decomposition of the target task and samples the features and elements in the decomposition results, which helps to generate a large number of diverse prompts that can accurately reflect the characteristics of the task, and accurate prompts help the model generate accurate responses that are convenient for labeling preference data. Accurate responses further help electronic devices accurately label preferences.

[0009] The embodiment of the present application generates prompts and labeling rules based on the multi-layer decomposition results of the target task, so that the prompts, the responses generated according to the prompts, and the labeling rules used to label the responses are associated with each other, thereby supporting electronic devices to replace humans in accurate preference labeling.

[0010] Optionally, the target task is decomposed into multiple layers, including: enumerating the features of each instruction type to obtain a feature set of each instruction type; enumerating the elements of each feature in the feature set to obtain an element set of each feature in the feature set; sampling one or more features of various instruction types of the first task type, and one or more elements of the sampled features, including: randomly selecting one or more features from the feature set of each instruction type of the first task type, respectively; and randomly selecting one or more elements from the element set of the selected features.

[0011] By enumerating features and elements and sampling them by random sampling, it helps the electronic device to generate a large number of diverse prompts that can accurately reflect the characteristics of the task.

[0012] Optionally, based on the first sampling result, a first prompt of the first task type is generated, including: generating a prompt of the first instruction type based on sampled features of the first instruction type and sampled elements of the sampled features; wherein the first instruction type is any instruction type of the first task type; and generating a first prompt based on prompts of various instruction types of the first task type.

[0013] In the embodiment of the present application, a prompt can be generated by a two-step method, that is, firstly generating a prompt of a specific instruction type based on the features and elements, and then generating a prompt of a specific type of task by combining the prompts of each instruction type. Compared with the implementation method of directly generating prompts based on scattered key words such as features and elements, the text expression fluency of the prompt generated by the two-step method can be better, which helps the language model to accurately identify the content of the prompt, thereby accurately outputting the response.

[0014] Optionally, the rule for performing preference marking on the responses corresponding to the first prompt includes: a rule for determining a sorting result of the responses corresponding to the first prompt; wherein the sorting result is used to describe an arrangement sequence in the multiple responses corresponding to the first prompt.

[0015] By sorting to mark preferences, the sorting results can clearly reflect the differentiated preference levels of multiple responses to the same prompt, which helps to accurately reflect subjective preferences.

[0016] Optionally, the preference data acquisition method also includes: modifying the first prompt to obtain a second prompt, wherein the features and elements in the first prompt are consistent with the features and elements in the second prompt; inputting the second prompt into the first language model to obtain a second response corresponding to the first prompt output by the first language model; and annotating the second response according to the first annotation rule to obtain preference data corresponding to the first prompt and the second response.

[0017] By varying the prompt while keeping the key information of the prompt the same, multiple responses to the same prompt can be obtained, which helps to increase the diversity of responses.

[0018] Optionally, the preference data acquisition method also includes: inputting the first prompt and the first instruction into the first language model, and obtaining a third response corresponding to the first prompt output by the first language model; wherein the first instruction is an instruction that affects the thinking logic of the first language model; and labeling the third response according to the first labeling rule to obtain preference data corresponding to the first prompt and the third response.

[0019] By adding instructions that influence the language model's thinking, multiple responses to the same prompt can be obtained, which helps improve the diversity of responses.

[0020] Optionally, the preference data acquisition method also includes: inputting the first prompt into the second language model, and obtaining a fourth response corresponding to the first prompt output by the second language model; wherein the first language model and the second language model are different language models obtained by training the same model (or the two models are based on the same or similar base); annotating the fourth response according to the first annotation rule, and obtaining preference data corresponding to the first prompt and the fourth response.

[0021] By using different models based on the same or similar bases, multiple responses to the same prompt can be obtained, which helps to increase the diversity of responses.

[0022] Optionally, a first labeling rule is generated based on the features and elements in the first prompt, including: obtaining a labeling rule template for the target task; wherein the labeling rule template for the target task includes: a rule for scoring responses corresponding to the prompt based on the features and elements in the prompt, and a rule for giving preference labels to responses based on the scores of the responses; and the features and elements in the first prompt are added to the labeling rule template to obtain the first labeling rule.

[0023] By generating instances of labeling rules according to the features and elements in a specific prompt, the electronic device can accurately label the response to the specific prompt based on the generated labeling rules, which helps to improve the efficiency of obtaining preference data.

[0024] Optionally, the target task is a writing task; the multiple task types of the writing task include a generation task, an extended writing task, and a rewriting task; the multiple instruction types of each task type include text information and writing requirements. In this way, high-quality preference data can be provided for the writing task.

[0025] Optionally, the method for obtaining preference data further includes: performing alignment training on the third language model according to the preference data; the first language model and the third language model have the same model manufacturer, which helps to improve model performance.

[0026] In a second aspect, an embodiment of the present application provides a preference data acquisition device, which is applied to an electronic device, including:

[0027] A task decomposition module is used to perform multi-layer decomposition on the target task to obtain a decomposition result; wherein the decomposition result includes multiple task types of the target task, multiple instruction types of each task type, multiple features of each instruction type, and multiple elements of each feature;

[0028] A sampling module, used for sampling one or more features of various instruction types of a first task type and one or more elements of the sampled features according to the decomposition result, to obtain a first sampling result; wherein the first task type is any one of the multiple task types;

[0029] A prompt generating module, configured to generate a first prompt of a first task type according to the first sampling result;

[0030] a response acquisition module, configured to input the first prompt into the first language model, and obtain a first response corresponding to the first prompt output by the first language model;

[0031] A labeling rule generation module is used to generate a first labeling rule according to the features and elements in the first prompt; wherein the first labeling rule includes: a rule for scoring the response corresponding to the first prompt according to the features and elements in the first prompt, and a rule for performing a preference labeling on the response corresponding to the first prompt according to the first prompt and the score of the response corresponding to the first prompt;

[0032] A labeling module is used to use a labeling model to label the first response according to a first labeling rule to obtain preference data corresponding to the first prompt and the first response, wherein the model training sample of the labeling model includes multiple responses and prompts corresponding to each of the multiple responses, and the label of the sample includes a score and preference information of each of the multiple responses.

[0033] In a third aspect, an embodiment of the present application provides a chip, comprising: a processor, which is used to execute computer program instructions stored in a memory, wherein when the computer program instructions are executed by the processor, the chip is triggered to execute any method as in the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising at least one processor, wherein the processor and a memory are coupled, the memory is used to store computer program instructions, and the processor is used to execute the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute a method as described in any one of the first aspects.

[0035] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the computer executes any method in the first aspect.

[0036] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes any method in the first aspect.

[0037] The technical effects of the aforementioned aspects can be referenced to each other and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0039] Figure 1 A flowchart of a method for obtaining preference data provided in an embodiment of the present application;

[0040] Figure 2 A block diagram of a preference data acquisition device provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0043] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0044] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0045] It should be understood that the term "at least one" used in this article refers to one or more, and "plurality" refers to two or more. The term "and / or" used in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. Among them, A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple.

[0046] It should be understood that, although the terms first, second, etc. may be used to describe the set thresholds in the embodiments of the present application, these set thresholds should not be limited to these terms. These terms are only used to distinguish the set thresholds from each other. For example, without departing from the scope of the embodiments of the present application, the first set threshold may also be referred to as the second set threshold, and similarly, the second set threshold may also be referred to as the first set threshold.

[0047] In addition to being powerful, LLMs (i.e., large language models) are also crucial to being reliable and ethical. To ensure that LLMs can achieve these goals, the output of LLMs needs to be consistent with human intentions, that is, LLMs should be aligned with human preferences. Therefore, it is necessary to perform alignment training on LLMs.

[0048] Exemplarily, the alignment training methods may include RLHF (i.e., human feedback reinforcement learning), supervised fine-tuning, direct policy optimization, etc. RLHF may include algorithms such as proximal policy optimization (PPO) and direct preference optimization (DPO).

[0049] In the alignment training of LLMs, RLHF can effectively improve the alignment of the model with human preferences. The RLHF method requires the annotator to provide data that represents human preferences to help the model move towards the results that human preferences prefer. As such, obtaining high-quality human preference data is both critical and challenging.

[0050] Based on the tedious and expensive nature of manual labeling, considering efficiency and cost issues, AI Feedback (AIF) can be used for labeling to replace human feedback with artificial intelligence (AI) feedback, and the data obtained can be used as human preference data. For example, LLM can be used to collect data.

[0051] The AIF method is usually applicable to relatively objective tasks, such as tasks in the fields of mathematics and science, such as coding and logical reasoning. However, for more creative open-ended generative tasks, such as creative writing tasks (referred to as writing tasks), because they involve complex human preferences and have no clear answers, creative generation requirements are difficult to objectively evaluate, making it difficult to use AI-assisted annotation, and AI-based annotation often does not achieve the desired results. In this way, for open-ended generative tasks, human input is still required to provide preference data.

[0052] At present, for open-ended generative tasks, humans can act as annotators and perform subjective annotations based on their own preferences, so that the annotated content can reflect human preferences. However, this implementation method is inefficient in obtaining preference data. In addition, in order to reduce human resource investment, non-professionals are usually used as annotators to provide preference data based on their own personalized preferences. However, different groups of people may have different preferences for open-ended generative tasks, and excessive subjective preferences will introduce significant noise, making task learning difficult and affecting the quality of the provided preference data.

[0053] In order to improve the efficiency of obtaining preference data while ensuring the quality of preference data, see Figure 1 An embodiment of the present application provides a method for obtaining preference data, which is applied to an electronic device and may include the following steps 101 to 106.

[0054] Step 101, perform multi-layer decomposition on the target task to obtain a decomposition result; wherein the decomposition result includes multiple task types of the target task, multiple instruction types of each task type, multiple features of each instruction type, and multiple elements of each feature.

[0055] Feasibly, the target task may be any one of the open generation tasks, such as a writing task, a question-answering task, etc. In other feasible implementations, the target task may also be a relatively objective task.

[0056] Compared with the single-layer decomposition method of tasks, for example, multiple task types can be enumerated, but the key characteristics of various task types are not clear enough. By decomposing tasks in multiple layers, the key characteristics of various task types can be reflected through features and prompts, which will help electronic devices to generate a large number of diverse prompts based on the multi-layer decomposition tasks.

[0057] In one embodiment of the present application, a multi-layer decomposition is performed on the target task, including: enumerating the features of each instruction type to obtain a feature set of each instruction type; enumerating the elements of each feature in the feature set to obtain an element set of each feature in the feature set.

[0058] By enumerating features and elements, it helps to improve the comprehensiveness of task decomposition and fully extract the key features of the task, so that high-quality prompts can be generated, and then high-quality responses can be generated based on the prompts. In addition, high-quality prompts and their responses can support electronic devices to accurately label preferences.

[0059] By performing multi-layer decomposition of the task, the electronic device can generate a prompt for the task based on the decomposition result of the task. The prompt can be used to guide the language model to generate text or instructions that are compatible with the prompt as the response content. Afterwards, the electronic device can mark the response with preference.

[0060] In one embodiment of the present application, the target task is a writing task; the various task types of writing tasks include generating tasks, extending writing tasks, and rewriting tasks (for example, rewriting can be achieved through style conversion); the various instruction types of each task type include text information and writing requirements.

[0061] The text information may be text related to the response content generated by the model, and the writing requirement may be requirement information related to the text or instruction generated by the language model.

[0062] In one example, the multi-layer decomposition result of the writing task may be shown in the following Table 1. The features and elements shown in Table 1 are only examples, and the decomposition result may also include other features and other elements not shown in Table 1.

[0063] Table 1

[0064]

[0065] In other feasible embodiments, the multi-layer decomposition results of the writing task may also include other task types and other instruction types not shown in Table 1, and this application does not limit this. For example, the features of the extended writing text may include not only the theme, background, and personality, but also roles, plots, etc. The features of the extended writing requirements may include not only style and details, but also emotional colors, conflicts, target readers, language characteristics, etc. The elements of the "theme" feature of the extended writing text may include not only love, but also friendship, courage, freedom, etc.

[0066] In one embodiment of the present application, a target task is decomposed into multiple layers, including: using a task decomposition model to decompose the target task into multiple layers; wherein a model training sample of the task decomposition model includes multiple tasks, and a label of the sample includes a multi-layer decomposition result of each of the multiple tasks.

[0067] In one embodiment, the labels of the model training samples of the task decomposition model may be information obtained by a human annotator performing multi-layer decomposition of the task according to professional knowledge. Preferably, the human annotator may be a domain expert to improve the accuracy of task decomposition.

[0068] Since the subsequent preference annotation process involves the annotator's reference to annotation rules, domain experts can perform multi-layer decomposition of the task based on the key points to be considered in the annotation process. This allows the electronic device to generate high-quality instructions based on the task decomposition results during the instruction generation phase, and to generate appropriate annotation rules based on the prompts during the annotation rule generation phase, thereby helping the electronic device to perform high-quality preference annotation on the response generated by the language model according to the instructions based on the annotation rules.

[0069] In a feasible implementation, the electronic device can train the LLM in a supervised fine-tuning manner based on the training data set to obtain a task decomposition model. Exemplarily, the trained LLM can be a transformer-based LLM that has been pre-trained and fine-tuned for general capabilities. Among them, the Transformer is a deep learning model architecture that uses an attention mechanism to increase the speed of model training.

[0070] The training data set used may include multiple training data, each of which is a triple data including a task, a content feature, and a requirement feature to reflect the decomposition result of the task. Among them, the "task" in a triple data may be a specific task type, that is, a task type instance; the "content feature" in a triple data may be one or more features related to the content of the task type instance; and the "requirement feature" in a triple data may be one or more features related to the requirements of the task type instance.

[0071] Feasibly, the “content features” and “requirement features” in the triple data can be extracted by human experts.

[0072] Exemplarily, the model training process for training a task decomposition model may include: taking the task decomposition requirements as questions and the task decomposition results as answers to form supervised fine-tuning training data, and then fine-tuning the LLM based on this to train the LLM, thereby obtaining an LLM with task decomposition capabilities as a task decomposition model.

[0073] After the task decomposition model is trained using the triple data, the decomposition result obtained by multi-layer decomposition of the target task using the task decomposition model corresponds to the triple data to ensure the accuracy of the task decomposition. For example, referring to the decomposition results shown in Table 1, taking the extended writing task of the creative writing task as an example, the extended writing task is further decomposed into extended writing text and extended writing requirements. The features of the extended writing text correspond to the "content features" in the triple data, and the features of the extended writing requirements correspond to the "requirement features" in the triple data.

[0074] In other embodiments, the labels of the model training samples of the task decomposition model may be information obtained by performing task decomposition using AI technology (such as using a language model with task decomposition function) to replace human decomposition with AI decomposition, and then confirmed and optimized as needed by a human annotator. Preferably, the human annotator may be a domain expert.

[0075] It is feasible to perform model training based on the model training samples and their labels to obtain the above-mentioned task decomposition model. In one embodiment, model training can be performed on the LLM to obtain the task decomposition model.

[0076] Since the task decomposition logic reflected by the labels of the model training samples of the task decomposition model depends on human logic, when the task decomposition model is used for task decomposition, the results of the model decomposition can usually accurately reflect human decomposition needs to ensure the accuracy of task decomposition and significantly improve the efficiency of task decomposition.

[0077] Step 102, based on the decomposition result, sample one or more features of various instruction types of the first task type and one or more elements of the sampled features to obtain a first sampling result; wherein the first task type is any one of the multiple task types.

[0078] In one embodiment, the electronic device can sample features and elements by running a preset code program with a sampling function.

[0079] After multi-layer decomposition of the task, features and elements in the decomposition results can be sampled for generating prompts.

[0080] For example, referring to Table 1, the instruction type of the extended writing task may include an extended writing text and an extended writing requirement, and the electronic device may sample one or more features of the extended writing text, such as the three features of the extended writing text, the background, and the personality, and then sample one or more elements of each feature of the extended writing text. In addition, the electronic device may sample one or more features of the extended writing requirement, such as the two features of the style and details of the extended writing requirement, and then sample one or more elements of each feature of the extended writing requirement.

[0081] Preferably, the specific sampling method used can be designed according to different requirements.

[0082] In a feasible implementation, if it is necessary to obtain a sampling result with the highest possible diversity, a random sampling method can usually be used so that as many elements as possible can be sampled, thereby improving the diversity of the sampling results.

[0083] In other feasible implementations, other sampling methods can also be used as needed, for example, sampling can be performed according to the frequency of occurrence of each element in the task. For example, in a certain type of literary creation task, the electronic device can select some existing human works, count the frequency of occurrence of elements in the task decomposition results in these human works, and then sample according to the frequency. This sampling method helps to generate responses with similar element distributions to human works when a large language model is used to generate responses later, making machine-synthesized data and natural language data closer.

[0084] In one embodiment of the present application, the target task is decomposed in multiple layers, including: enumerating the features of each instruction type to obtain a feature set of each instruction type; enumerating the elements of each feature in the feature set to obtain an element set of each feature in the feature set. Based on this, sampling one or more features of various instruction types of the first task type and one or more elements of the sampled features include: randomly selecting one or more features from the feature set of each instruction type of the first task type; and randomly selecting one or more elements from the element set of the selected features.

[0085] The feature set obtained by enumerating features ensures the comprehensiveness of the features, and randomly sampling features from the feature set for generating prompts helps to ensure the diversity and comprehensiveness of prompt generation.

[0086] The combination of elements obtained by enumerating the elements of the features ensures the comprehensiveness of the elements, and randomly sampling elements from the element set for generating prompts helps to ensure the diversity and comprehensiveness of prompt generation.

[0087] In this way, by enumerating features and elements and sampling features and elements by random sampling, it is helpful for the electronic device to generate a large number of diverse prompts that can accurately reflect the characteristics of the task, and the generated prompts can comprehensively cover various features and various elements of the task. By generating a large number of comprehensive and diverse prompts, the language model can be supported to generate a large number of comprehensive and diverse responses based on the prompts, so as to facilitate the annotation model to perform preference annotation on multiple responses corresponding to the same prompt, so as to obtain rich preference annotation data.

[0088] Step 103: Generate a first prompt of a first task type according to the first sampling result.

[0089] Feasibly, after sampling the features and elements involved in the task, LLM can be used to generate smooth and natural prompts based on the sampling results.

[0090] In the field of artificial intelligence, prompts can be used to guide models to generate text or instructions for specific outputs. If a prompt has good clarity, relevance, and accuracy, it can effectively convey the user's intention and guide the model to generate the expected response content. For example, by inputting the generated prompt into LLM, LLM can output a response (or response content, response result) that matches the prompt.

[0091] In one example, the instruction types of the above-mentioned generation task include generating text and generating requirements. The electronic device can sample the feature of "theme" by sampling the features of the generated text, and then sample the elements of the feature of "theme", and can sample the three elements of "dealing with the complex emotions of adolescence and the difficulties of sex education", "elements of the blacksmith club", and "campus life, family education and craft inheritance". In addition, the electronic device can sample the feature of "story type" by sampling the features of the generation requirements, and then sample the elements of the feature of "story type", and can sample the element of "novel". Based on the sampled features and elements, the electronic device can generate a prompt for the above-mentioned generation task, such as recorded as prompt X.

[0092] Tip X: Please write a campus novel about a student who is deeply influenced by the school's family background and how they interact with different classmates and clubs. Describe how students deal with the complex emotions of puberty and the difficulties of sex education. Try to incorporate elements of the blacksmith club, such as scenes where students discuss forging techniques, metal material selection, or the process of ironwork manufacturing. This will be a rich story that combines campus life, family education, and craft inheritance.

[0093] It is feasible that different instruction types may have the same features, for example, both the extended writing text and the extended writing requirements may have the feature of "language characteristics". Exemplarily, for the feature of "language characteristics" of the extended writing text, the element of "local dialect" may be sampled, and for the feature of "language characteristics" of the extended writing requirements, the element of "network language" may be sampled. That is, "local dialect" is a restriction on the text content, and "network language" is a restriction on the writing requirements, so as to restrict the text content of the response generated by the model to contain "local dialect", and the writing requirements of the response to comply with "network language".

[0094] Since different instruction types can have the same features, in order to avoid the situation where the required prompts may not be accurately generated based on the sampled features and elements, thereby failing to obtain the required responses, a two-step method can be used to generate prompts.

[0095] In the two-step process of generating prompts, the features and elements of a specific instruction type are first combined into a prompt for that specific instruction type, so that different instruction types have correspondingly different prompt contents, to clarify the prompt contents of different instruction types, and then the prompts of each instruction type are combined into a final prompt.

[0096] In one embodiment of the present application, step 103 may include: generating a prompt of the first instruction type based on sampled features of the first instruction type and sampled elements of the sampled features; wherein the first instruction type is any instruction type of the first task type; and generating a first prompt based on prompts of various instruction types of the first task type.

[0097] In this embodiment, the electronic device can generate the first prompt through a two-step method, namely, first generating a prompt of a specific instruction type based on the sampled features and elements (i.e., the first step), and then generating a prompt of a specific type of task by combining the prompts of each instruction type (i.e., the second step).

[0098] In one embodiment, the execution subject of the two-step method for generating prompts may be a language model and / or a preset code program with an enhancement generation function. The language model may be, for example, any LLM with relatively strong capabilities.

[0099] Taking the above-mentioned extended writing task as an example, the instruction type of the extended writing task includes extended writing text and extended writing requirements. The electronic device can sample the features and elements of the extended writing text, and generate a prompt for the extended writing text based on the sampled features and elements, and can sample the features and elements of the extended writing requirements, and generate a prompt for the extended writing requirements based on the sampled features and elements, and then generate a prompt for the extended writing text based on the prompt of the extended writing text and the prompt of the extended writing requirements. The electronic device inputs the prompt of the extended writing text into the language model, and the language model can generate and output a response to the extended writing text based on the input prompt.

[0100] Compared with the implementation method of directly generating prompts based on scattered key words such as features and elements, the prompts generated by the two-step method can be more accurate and the text expression of the prompts can be more fluent, which helps the language model to accurately identify the content of the prompts and thus accurately output the response.

[0101] In one embodiment, a two-step method may be used to generate the first prompt in combination with a large language model, and step 103 may include the following steps 1301 to 1302:

[0102] Step 1301: if the first instruction type is a text instruction (such as extended writing text), then the sampled features of the first instruction type and the sampled elements of the sampled features are combined, and the combined result is input into the large language model to obtain a prompt output by the large language model according to the combined result as a prompt of the first instruction type;

[0103] If the first instruction type is a requirement instruction (such as an extended writing requirement), the sampled features of the first instruction type and the sampled elements of the sampled features are combined, and the combined result is used as a prompt of the first instruction type;

[0104] The first instruction type is any instruction type of the first task type.

[0105] For example, the electronic device expands the features of the writing text by sampling, and samples four text features, namely, "theme", "role", "word limit", and "language characteristics", and samples each of these four text features to obtain an element. Among them, the element sampled for "theme" is "loyalty and betrayal", the element sampled for "role" is "innocent children", the element sampled for "word limit" is "short story", and the element sampled for "language characteristics" is "local dialect".

[0106] Afterwards, the electronic device can combine these features and elements of the extended writing text by running the code program to obtain a combination result, the content of which is as follows:

[0107] Please create a text that meets the following requirements. Please do not return redundant content, but directly return the text you created. The creation requirements are: The theme of the content is loyalty and betrayal. The main characters involved are innocent children. The content of the creation is short (within 200 words). The language characteristics of the creation are local dialects.

[0108] Then, the electronic device inputs the above combination result into the LLM and obtains the prompt output by the LLM as a prompt for expanding the writing text. Exemplarily, the above combination result is input into the Baichuan model, and the prompt that the Baichuan module can output is as follows:

[0109] "That afternoon, Xiao Ming and Xiao Hong were playing the game of "I am a detective" in the village. They used their old bicycles as clue trackers, wore big hats, and held fake handcuffs. Suddenly, there was a noise at the head of the village, and the two children immediately ran to check. An adult was pointing at a teenager and yelling, "You actually betrayed me! You stole my things!" The teenager lowered his head, tears in his eyes. A group of children gathered around to watch the fun. Xiao Ming frowned and asked Xiao Hong, "Is betrayal a bad thing?" Xiao Hong nodded, "Yes, we have to be loyal people, not like that bad brother." Xiao Ming nodded fiercely and continued to play their detective game, as if all the disputes of loyalty and disloyalty had nothing to do with them, and only the innocent friendship was with them."

[0110] For example, electronic devices expand the characteristics of writing requirements by sampling, and sample the three required characteristics of "style", "style", and "language characteristics", and sample each of these three required characteristics to obtain an element. Among them, the element sampled for "style" is "classicism", the element sampled for "details" is "psychological activities", and the element sampled for "language characteristics" is "Internet language".

[0111] Afterwards, the electronic device can combine these features and elements of the extended writing requirements by running the code program to obtain a combined result, the content of which is as follows:

[0112] "Please expand the above given text. The expansion needs to meet the following requirements: the style of the expansion should be classicism. The new details that need to be added after the expansion include psychological activities. The language style of the expanded content should be Internet language."

[0113] Step 1302, concatenate the prompts of the first instruction type, and input the concatenation result into the large language model, and obtain the prompt output by the large language model according to the concatenation result as the first prompt of the first task type.

[0114] Optionally, when the electronic device inputs the concatenation result into the large language model, it may also input an instruction into the large voice model to instruct the large voice model to generate a prompt according to the prompt generation requirement specified in the instruction. For example, the instruction input into the large voice model may be as follows:

[0115] “Rewrite a given prompt to make it more fluent and natural, and more in line with the Chinese way of expression:

[0116] 1. This prompt is a prompt for an extended writing task. In the prompt, a text is given first, and then the instructions for the extended writing task are given. The modified prompt is still a prompt for an extended writing task.

[0117] 2. Please make some adjustments to the order of the instructions in the prompt, and you can also modify the wording of the prompt to make it smoother;

[0118] 3. Do not modify the instructions in the prompt, and do not add additional new instructions. Keep the original instructions!

[0119] 4. Please output the rewritten prompt directly, do not output any extra content! Do not include the word prompt! "

[0120] Based on the above example content, the electronic device can run the code program to splice the above extended writing text prompts and extended writing requirements prompts, input the splicing results into the Baichuan model, and obtain the prompts output by the Baichuan module. The content of the prompts is as follows:

[0121] "That afternoon, Xiao Ming and Xiao Hong were happily playing the game of "I am a detective" in the village. They used a shabby bicycle as a clue tracker, wore big hats, waved homemade fake handcuffs in their hands, and looked very excited. Just as they were immersed in the world of detectives, there was a fierce noise at the entrance of the village. The two immediately rushed to the source of the sound out of curiosity, and found that an adult was pointing at a teenager with his head down emotionally, and shouted: "How can you betray me! You actually stole my things!" The teenager's face was full of tears, he lowered his head and said nothing, while the children watching were talking about it with great interest. Xiao Ming asked Xiao Hong in confusion: "Is betrayal a bad thing?" Xiao Hong answered affirmatively: "Yes, we have to be trustworthy and sincere people, and we must not become like that unreliable brother. "After hearing this, Xiao Ming nodded firmly. He and she continued to pursue their dreams of being detectives. All the disputes around them seemed to have nothing to do with them. Only the innocent friendship and carefree childhood were left to accompany them. Please expand this story in the classical style and add new psychological details while maintaining the style of Internet language."

[0122] It can be seen that by combining the natural language capabilities of LLM and generating prompts for specific task types through a two-step method, the elements sampled from specific task types can be changed into text and instruction information that is closer to natural language.

[0123] Step 104: input the first prompt into the first language model, and obtain a first response corresponding to the first prompt output by the first language model.

[0124] Optionally, the first language model may be any large language model.

[0125] In practice, one prompt may correspond to one or more responses. To reflect human preferences, one prompt may correspond to multiple responses, and the electronic device may mark different preference data for different responses to the same prompt to reflect different preferences for different response contents.

[0126] It is feasible to obtain multiple responses to the same prompt by modifying the prompt. In one embodiment of the present application, the method for obtaining preference data further includes: modifying the first prompt to obtain a second prompt, wherein the features and elements in the first prompt are consistent with the features and elements in the second prompt; inputting the second prompt into the first language model to obtain a second response corresponding to the first prompt output by the first language model; and annotating the second response according to the first annotation rule to obtain preference data corresponding to the first prompt and the second response.

[0127] For example, the prompt can be modified by changing the active sentence in the prompt to a passive sentence, changing the non-inverted sentence in the prompt to an inverted sentence, changing the affirmative sentence in the prompt to a double negative sentence, etc., while keeping the features and elements in the prompt unchanged. Since the features and elements in the prompt are unchanged, the response obtained based on the modified prompt can be used as the response corresponding to the original prompt, so that multiple responses to the same prompt can be obtained.

[0128] By changing the prompt while keeping the key information of the prompt unchanged, multiple responses to the same prompt can be obtained, which helps to improve the diversity of responses. It can be seen that by changing the prompt, the diversity of the language model output response can be improved when the language model types are limited.

[0129] It is feasible to obtain multiple responses to the same prompt by inputting a way to influence the language model's thinking. In one embodiment of the present application, the preference data acquisition method further includes: inputting a first prompt and a first instruction into a first language model, and obtaining a third response corresponding to the first prompt output by the first language model; wherein the first instruction is an instruction that influences the thinking logic of the first language model; and annotating the third response according to a first annotation rule to obtain preference data corresponding to the first prompt and the third response.

[0130] Exemplarily, the additional instruction input to the language model may be an instruction for changing the depth of thinking and / or the way of thinking of the language model, for example, the additional instruction may require the language model to generate a response after in-depth thinking. By changing the thinking logic of the language model, multiple responses to the same prompt may be obtained.

[0131] By adding instructions that influence the language model's thinking, multiple responses to the same prompt can be obtained, which helps to improve the diversity of responses. It can be seen that by adding additional instructions, the diversity of language model output responses can be improved when the language model types are limited.

[0132] It is feasible that the same prompt can be input into different models to obtain multiple responses to the same prompt. In one embodiment of the present application, the method for obtaining preference data further includes: inputting the first prompt into the second language model, obtaining a fourth response corresponding to the first prompt output by the second language model; wherein the first language model and the second language model are different language models obtained by model training on the same model (or the two models are based on the same or similar base); annotating the fourth response according to the first annotation rule, and obtaining preference data corresponding to the first prompt and the fourth response.

[0133] The diversity of responses can be increased by inputting the prompt into different models, which can usually be multiple models with the same or close model base.

[0134] Exemplarily, different model training data and / or model training methods may be used to train the same model separately to obtain different language models.

[0135] For example, the multiple models may be Baichuan models whose model identifiers are baichuan 13B and baichuan 7B respectively.

[0136] For example, the multiple models may be models obtained after fine-tuning the baichuan 13B model in different rounds.

[0137] For example, the multiple models may be models of different sizes in the qwen2 series models.

[0138] For example, the multiple models may be models obtained after performing different fine-tuning training on the qwen2 series models.

[0139] By using different models based on the same or similar bases, multiple responses to the same prompt can be obtained, which helps to increase the diversity of responses.

[0140] In one example, the electronic device inputs the prompt X into the language model, and the language model can generate a response based on the prompt X, for example, recorded as response X. The specific content of response X is as follows:

[0141] "Spring when steel flowers bloom"

[0142] Chapter 1: New Beginnings and Challenges On a sunny September morning, the sun shines through the lush leaves of the towering trees, casting shadows on the campus of a famous high school. This is a place where dreams take root, and every young heart carries the hope for a better future. Lin Ran is one of them. He carries an old brown backpack, and under his black hair, his deep eyes seem to carry countless stories.

[0143] Lin Ran's family background is quite unique - his father is a skilled blacksmith who runs a small workshop with a high reputation in the local area. Growing up in such an environment, Lin Ran developed a special feeling for steel, not only mastering the basic blacksmith skills, but also understanding the persistence and love behind the craft. However, this family background made him somewhat out of place among his classmates, who were more concerned with academic achievements and social trends rather than traditional handicrafts.

[0144] Despite this, Lin Ran remained fearless. He believed that everyone was born with a unique meaning and mission, so he decided to face the challenges of high school life in his own way. He joined the school's only blacksmith club and became one of its most active members. Whenever the topic of handicrafts was mentioned in class, Lin Ran always offered insightful opinions and suggestions, and he could even demonstrate his blacksmith skills on the spot, winning the respect of many classmates.

[0145] Chapter 2: Sparks of Youth As the semester progressed, Lin Ran began to interact with more peers of the same age. He found that everyone had their own troubles and joys, secrets and aspirations for the future. In a casual conversation, Lin Ran learned that his roommate Li Hao was struggling to receive sex education. Teenagers often avoid such topics, but Lin Ran used a mature mentality to understand and help Li Hao. Together, they discussed the importance of sex education and how to properly face and deal with these complex emotions. It was this frank communication that made the two inseparable friends.

[0146] At the same time, the school's blacksmith club is also very lively. The annual "Steel into Art" competition is coming, which is an important stage to show the skills and creativity of club members. Lin Ran and Li Hao joined forces to prepare a creative iron art work. In the process of creation, they not only strengthened their friendship, but also gradually found their respective positions and values ​​in artistic creation.

[0147] Chapter 3: Dreams Bloom Time flies, and the day of the "Steel into Art" competition finally arrives, and the whole campus is filled with tension and excitement. The diversity of the entries is breathtaking, from exquisite small household items to magnificent sculptures, each work reflects the sweat and wisdom of the creators. Among these exquisite works of art, the iron flower co-created by Lin Ran and Li Hao stands out, like a blooming steel flower, dazzling and vibrant, but exuding indomitable power. When asked about their creative inspiration, Lin Ran honestly said that it came from his father's teachings on the ductility and endurance of steel, symbolizing the colorful brilliance of youth and dreams. These words not only deeply touched every audience, but also won their works the highest award! At this moment, all their efforts have paid off.

[0148] It can be seen that there are answer contents in response X corresponding to the features and elements in prompt X.

[0149] For example, prompt X involves the element "dealing with the complex emotions of adolescence and the difficulties of sex education" of the feature "theme" of the generated text. Correspondingly, there is an answer content corresponding to this element in response X, namely, "Lin Ran learned that his roommate Li Hao was struggling to receive sex education. Teenagers often avoid such topics, but Lin Ran used a mature attitude to understand and help Li Hao. Together they discussed the importance of sex education and how to properly face and deal with these complex emotions."

[0150] For another example, prompt X involves the element "elements of the blacksmith club" which is related to the feature of the "theme" of the generated text. Correspondingly, there is an answer content corresponding to this element in response X, namely, "He joined the school's only blacksmith club and became one of its most active members. Whenever the topic of handicrafts is mentioned in class, Lin Ran always makes insightful comments and suggestions, and he can even demonstrate his blacksmith skills on the spot", and "At the same time, the school's blacksmith club is also very lively. The annual "Art of Steelmaking" competition is coming, which is an important stage for displaying the skills and creativity of club members. Lin Ran and Li Hao have teamed up to prepare a creative iron art work."

[0151] For another example, prompt X involves the element "novel" of the feature "story type" required for generation, and correspondingly, response X is an article whose story type is campus novel.

[0152] Step 105, generating a first labeling rule based on the features and elements in the first prompt; wherein the first labeling rule includes: a rule for scoring the response corresponding to the first prompt based on the features and elements in the first prompt (i.e., a scoring rule), and a rule for giving a preference label to the response corresponding to the first prompt based on the first prompt and the score of the response corresponding to the first prompt (i.e., a preference labeling rule).

[0153] By generating labeling rules based on the features and elements in the prompts, it is helpful for electronic devices to perform preference labeling according to the key points reflected in the prompts, which helps to improve the rationality and standardization of preference labeling.

[0154] Figure 1 The illustrated embodiment performs multi-layer decomposition of the task so that the prompts generated based on the decomposition results have clear features and elements, and further the labeling rules generated based on the features and elements in the improvement have objective and clear scoring standards, which helps to improve the labeling model to accurately and objectively score the response to the prompts based on the labeling rules.

[0155] In one embodiment, the scoring rules in the annotation rule template may include a correctness score and a quality score (or language score).

[0156]

[0013] Optionally, a scoring rule for the correctness score may include: scoring the correctness score based on whether there is answer content in the response that corresponds to features and elements in the first prompt.

[0157] In one example, the correctness score can use a descending system with a full score of 5 points, with 1 point deducted for each missing element. If the answer content of each element of each feature is presented in the response, 5 points are selected.

[0158] To label the correctness score, the labeling model can extract the content corresponding to the features and elements in the prompt from the response corresponding to the prompt, and then perform a correctness score based on whether the extracted content meets the features and elements in the prompt.

[0159] Exemplarily, the features in the above prompt X include the feature of "theme", and the elements in the above prompt X include the three elements of the feature of "theme", namely "dealing with the complex emotions of adolescence and the difficulties of sex education", "elements of the blacksmith club", and "campus life, family education and craft inheritance", as well as the element of "novel" including the feature of "story type".

[0160] Since the three elements of the feature "theme" all have corresponding answer content in response X, and the content of the article in response X meets the element "novel" of the feature "story type", all elements are presented in response X, so the correctness score of response X can be 5 points.

[0161] In another example, for each element of each feature, if its answer content is presented in the response, it will get one point, and if it is not presented, it will get no point. Then, according to the total number of elements, the obtained scores are normalized to get the correctness score. In this way, even if the total number of elements in different prompts is different, the normalized score can still accurately evaluate the correctness of the response.

[0162] It can be seen that the normalization process allows the labeling model to score each response based on the same scoring benchmark, regardless of the number of elements in the prompt. This helps to accurately label the responses based on the scoring results.

[0163] Compared with the correctness score, the quality score is relatively more dependent on the annotator's (such as the annotation model described below) subjective judgment of the text language.

[0164] For the quality score, the electronic device may generate quality scoring rules related to the task and the features separated from the task. For example, the decomposition result of the generated task may include features such as the theme and the role. Then, referring to Table 2 below, the scoring rules may include quality scoring rules for the role shaping ability and the depth of the theme.

[0165] Table 2

[0166]

[0167] It is feasible that the quality scoring rule may include multiple scoring requirements as shown in Table 2. In this way, the annotation model may perform quality scoring according to the response content and the specific scoring requirements.

[0168] It is feasible to refine the quality score judgment criteria into multiple dimensions such as language quality, literary quality, and plot richness to help annotators give a comprehensive quality score to reflect the language expression ability of the language model. For example, the full score of the quality score can be 5 points.

[0169] It can be seen that for an element in the prompt, the correctness score can be used to reflect whether the answer content of the element exists in the response, and the quality score can be used to reflect the quality of the answer content.

[0170] In one example, the quality score may involve seven factors, namely, rich plot, characterization, expression, theme depth, creative imagination, emotional resonance, and logical rigor. The annotator can consider each factor comprehensively and choose an appropriate score from 1 to 5 points.

[0171] It can be seen that the scoring rules in the annotation rules can be used as quantitative objective standards, which can make the annotation scores more objective, so that the preference annotation based on objective scores will not have the problem of excessive subjective preference, thereby reducing the significant noise in the annotation data caused by excessive subjective preference, thus improving the quality of preference data.

[0172] The rules for preference labeling in the labeling rules support the labeler to make subjective labels according to his or her own preferences, so as to maintain the diversity of preferences and obtain high-quality preference data.

[0173] It is feasible that the annotation model can make a final subjective ranking based on its scores for each response, according to the scores and its own preferences. The order of the ranking reflects the degree of preference of the annotation model for each response. For example, the annotation model can make an overall comparison of multiple or all responses to the same prompt, and give a final ranking based on its own subjective preferences during the comparison process.

[0174] Exemplarily, all responses to the same prompt may include responses obtained by, for example, modifying the prompt, inputting additional instructions to the language model, inputting the prompt into different models separately, and the like.

[0175] In one embodiment of the present application, the rules for preference marking of responses corresponding to the first prompt include: rules for determining the sorting results of the responses corresponding to the first prompt; wherein the sorting results are used to describe the arrangement sequence numbers among the multiple responses corresponding to the first prompt.

[0176] For multiple responses to the same prompt, the electronic device can perform preference sorting on the multiple responses according to the scores of the individual responses, so as to reflect the preference differences for the different responses through different sorting positions.

[0177] By sorting to mark preferences, the sorting results can clearly reflect the differentiated preference levels of multiple responses to the same prompt, which helps to accurately reflect subjective preferences.

[0178] Exemplarily, the annotation rules require the annotation model to comprehensively evaluate the responses based on the annotated correctness scores and quality scores during the preference annotation process, so as to prioritize the different responses to the same prompt. For example, assuming that there are two responses to the same prompt, namely, response A and response B, if the correctness score and quality score of response A are both 5 points, and the correctness score of response B is 4 points and the quality score is 3 points, then response A can be marked as being ranked before response B (i.e., more preferred to response A). If the correctness scores and quality scores of the two responses are the same or not much different overall, the annotation model can mark the ranking of the two responses according to its own preferences.

[0179] It can be seen that the labeling rules limit the labeling model to first objectively score the response according to the prompt, and then perform subjective preference labeling on the response based on the objective score. By combining the objective standardized score and the subjective preference ranking, it is possible to minimize noise while maintaining the diversity of preferences, thereby obtaining high-quality human preference data.

[0180] In one embodiment of the present application, a first labeling rule is generated based on features and elements in a first prompt, including: obtaining a labeling rule template for a target task; wherein the labeling rule template for the target task includes: a rule for scoring responses corresponding to the prompt based on features and elements in the prompt, and a rule for giving preference labels to responses based on the scores of the responses; and the features and elements in the first prompt are added to the labeling rule template to obtain the first labeling rule.

[0181] By generating instances of labeling rules according to the features and elements in a specific prompt, the electronic device can accurately label the response to the specific prompt based on the generated labeling rules, which helps to improve the efficiency of obtaining preference data.

[0182] In one embodiment, the features and elements in the first prompt may be the features and elements used to generate the first prompt.

[0183] In another embodiment, the features and elements in the first prompt may be features and elements extracted from the first prompt.

[0184] In one embodiment of the present application, obtaining a labeling rule template for a target task includes: using a rule generation model to generate a labeling rule template for the target task; wherein a rule generation sample of the rule generation model includes multiple tasks, and a label of the sample includes a labeling rule template for each of the multiple tasks.

[0185] In one embodiment, the labels of the model training samples of the rule generation model may be labeling rule templates designed by human annotators for tasks according to professional knowledge. Preferably, the human annotators may be domain experts to improve the accuracy of rule design.

[0186] In other embodiments, the labels of the model training samples of the rule generation model may be information that is designed by AIF to label rule templates, so that AI design replaces human design, and is confirmed and optimized as needed by human annotators. Preferably, the human annotators may be domain experts.

[0187] It is feasible to perform model training based on the model training samples and their labels to obtain the rule generation model. In one embodiment, model training can be performed on the LLM to obtain the rule generation model.

[0188] Since the rule design logic reflected by the labels of the model training samples of the rule generation model depends on human logic, when the rule generation model is used to label the rule templates for task design, the results of the model design can usually accurately reflect human design needs to ensure the accuracy of the rule design and significantly improve the efficiency of the rule design.

[0189] Step 106, using the labeling model, label the first response according to the first labeling rule to obtain preference data corresponding to the first prompt and the first response, wherein the model training sample of the labeling model includes multiple responses and the prompt corresponding to each of the multiple responses, and the label of the sample includes the score and preference information of each of the multiple responses.

[0190] Figure 1 In the illustrated embodiment, prompts are generated based on the multi-layer decomposition results of the task, and labeling rules are generated according to the features and elements in the prompts, so that the prompts and labeling rules match. This can better guide the labeling model to accurately and objectively score the responses to the prompts based on the labeling rules, thereby improving the preference labeling quality of the labeling model.

[0191] It is feasible that the preference information in the above labels can be provided by human annotators, so that when the annotation model annotates the responses, it has the effect of annotating the preferences based on human feedback. Figure 1 The illustrated embodiments are applicable to open generation tasks and help obtain high-quality human preference data in the field of open generation tasks.

[0192] In one embodiment, the labels of the model training samples of the annotation model may be information obtained by a human annotator performing preference annotation based on the samples. Preferably, the human annotator may be a domain expert or a non-professional, and the subjectivity of the preference data may be ensured.

[0193] In a feasible implementation, the electronic device can train the reward model based on the training data set to obtain the labeled model. Exemplarily, the reward model can use a large language model that has been pre-trained based on the transform architecture. It is feasible to use a reward model with the largest model size possible as the starting point for training, such as the Baichuan pre-trained model.

[0194] The training data set used may include multiple training data, each of which is a partially ordered data, and the data form of the partially ordered data is a triple data of a partially ordered pair (x, a, b). The partially ordered pair (x, a, b) indicates that both response a and response b are responses corresponding to prompt x, and response a is ranked before response b (i.e., response a is preferred).

[0195] It is feasible to obtain enough preference ranking data, which may include multiple prompts and preference rankings of multiple responses corresponding to each prompt. Then, according to the preference rankings of different responses to the same prompt, the obtained preference ranking data are processed to obtain enough partial order pairs (x, a, b) as training data sets.

[0196] Optionally, the preference ranking in the above preference ranking data may be provided by human annotators.

[0197] For example, if the ranking of response a1 corresponding to prompt x1 precedes the ranking of response b1 corresponding to prompt x1, and the ranking of response b1 corresponding to prompt x1 precedes the ranking of response c1 corresponding to prompt x1, then the partial order pair (x1,a1,b1), partial order pair (x1,a1,c1), and partial order pair (x1,b1,c1) can be constructed.

[0198] In order to obtain a labeled model based on the partial order pair (x, a, b), the last layer of the reward model can be changed to an MLP linear layer so that the final output of the modified reward model becomes a value, and the modified reward model is trained based on the partial order pair (x, a, b).

[0199] In the process of training the modified reward model based on the partial order pair (x, a, b), the training goal is to reduce the value of the following loss function:

[0200] L = log(σ(r(x,a)-r(x,b)))

[0201] Where L represents the loss function value, r(x,a) represents the score predicted by the modified reward model for prompt x and response a, and r(x,b) represents the score predicted by the modified reward model for prompt x and response b.

[0202] By using the value of the loss function as the training goal, the modified reward model can give as high a score as possible to the top ranked responses and as low a score as possible to the bottom ranked responses, so that the model's preferences are consistent with human preferences. After the model training of the modified reward model is completed, the trained model can be used as the annotation model.

[0203] In other embodiments, the labels of the model training samples of the annotation model may be annotated by AIF, so as to use AI feedback instead of human feedback, and the information may be confirmed by a human annotator and optimized as needed. Preferably, the human annotator may be a domain expert or a non-professional.

[0204] It is feasible to perform model training based on the model training samples and their labels to obtain the above-mentioned annotation model. In one embodiment, model training can be performed on the LLM to obtain the annotation model.

[0205] Since the preference data reflected by the labels of the model training samples of the annotation model depends on human preferences, when the annotation model is used for preference annotation, the preference data annotated by the model can usually accurately reflect human preference data to ensure the quality of the preference data, while significantly improving the efficiency of obtaining preference data.

[0206] In a feasible implementation, any multiple of the above-mentioned task decomposition model, rule generation model and annotation model may be the same model, for example, the same language model.

[0207] After obtaining the preference data, RLHF can be used to perform alignment training on the LLM based on a large amount of preference data provided by electronic devices, so that the output of the aligned trained LLM is consistent with human intentions, thereby improving the model performance of the LLM.

[0208] For example, the model identified as baichuan SFT can be selected as the initial model for reinforcement training, and the reinforcement learning method PPO is used. At the same time, the KL divergence of the baichuan SFT model is introduced as a constraint to perform alignment training on the baichuan SFT model to complete the RLHF process. KL divergence (Kullback-Leibler Divergence) is a method for measuring the difference between two probability distributions.

[0209] In one embodiment of the present application, the method for obtaining preference data further includes: performing alignment training on a third language model according to the preference data; and the first language model and the third language model have the same model manufacturer.

[0210] It is feasible to use RLHF to align the LLM with the preference data to improve the model performance of the LLM so that the LLM is better aligned with human preferences.

[0211] Language models from different vendors may differ significantly in one or more aspects, such as architecture, optimizer, positional encoding, tokenizer, activation function, and attention mechanism.

[0212] Experiments have shown that, compared with using language models from different model vendors, aligning the response preference data output by the language models from the same model vendor can improve model performance more effectively. For example, in the field of creative writing, aligning the language models can significantly improve the language model's objective writing ability for creative writing tasks.

[0213] Feasibly, the first language model and the third language model may be the same model or different models.

[0214] like Figure 2 As shown, the embodiment of the present application provides a preference data acquisition device 10, which is applied to an electronic device, including:

[0215] The task decomposition module 11 is used to perform multi-layer decomposition on the target task to obtain a decomposition result; wherein the decomposition result includes multiple task types of the target task, multiple instruction types of each task type, multiple features of each instruction type, and multiple elements of each feature;

[0216] A sampling module 12 is used to sample one or more features of various instruction types of a first task type and one or more elements of the sampled features according to the decomposition result to obtain a first sampling result; wherein the first task type is any one of the multiple task types;

[0217] A prompt generating module 13, configured to generate a first prompt of a first task type according to the first sampling result;

[0218] A response acquisition module 14, configured to input the first prompt into the first language model, and obtain a first response corresponding to the first prompt output by the first language model;

[0219] The marking rule generating module 15 is used to generate a first marking rule according to the features and elements in the first prompt; wherein the first marking rule includes: a rule for scoring the response corresponding to the first prompt according to the features and elements in the first prompt, and a rule for giving a preference marking to the response corresponding to the first prompt according to the first prompt and the score of the response corresponding to the first prompt;

[0220] The labeling module 16 is used to use the labeling model to label the first response according to the first labeling rule to obtain preference data corresponding to the first prompt and the first response, wherein the model training sample of the labeling model includes multiple responses and the prompt corresponding to each of the multiple responses, and the label of the sample includes the score and preference information of each of the multiple responses.

[0221] An embodiment of the present application provides a chip, including: a processor, which is used to execute computer program instructions stored in a memory, wherein when the computer program instructions are executed by the processor, the chip is triggered to execute the method described in any embodiment of the present application.

[0222] An embodiment of the present application provides an electronic device, which includes at least one processor, the processor and a memory are coupled, the memory is used to store computer program instructions, and the processor is used to execute the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method described in any embodiment of the present application.

[0223] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in any embodiment of the present application.

[0224] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program runs on a computer, the computer executes the method described in any embodiment of the present application.

[0225] Figure 3 A schematic diagram of a computer device provided in one embodiment of the present application. Figure 3 As shown, the computer device 20 of this embodiment includes: a processor 21 and a memory 22, the memory 22 is used to store a computer program 23 that can be run on the processor 21, and the computer program 23 is executed by the processor 21 to implement the steps in the method embodiment of the present application. To avoid repetition, they are not described one by one here. Alternatively, when the computer program 23 is executed by the processor 21, the functions of each model / unit in the device embodiment of the present application are implemented. To avoid repetition, they are not described one by one here.

[0226] The computer device 20 includes but is not limited to a processor 21 and a memory 22. Those skilled in the art will appreciate that Figure 3 It is only an example of the computer device 20 and does not constitute a limitation of the computer device 20. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0227] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0228] The memory 22 may be an internal storage unit of the computer device 20, such as a hard disk or memory of the computer device 20. The memory 22 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media (SM) card, a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 20. Further, the memory 22 may also include both an internal storage unit of the computer device 20 and an external storage device. The memory 22 is used to store the computer program 23 and other programs and data required by the computer device. The memory 22 may also be used to temporarily store data that has been output or is to be output.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0232] The integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0233] In the embodiments of the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0234] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0235] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0236] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the same or similar parts between the various embodiments in this application can be referred to each other. For example, the specific working process of the system, device and unit described in the embodiments of this application can refer to the corresponding process in the method embodiment of this application, and will not be repeated here.

[0237] The above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for obtaining preference data, characterized in that: Used in electronic equipment, including: Performing multi-layer decomposition on the target task to obtain a decomposition result; wherein the decomposition result includes multiple task types of the target task, multiple instruction types of each task type, multiple features of each instruction type, and multiple elements of each feature; According to the decomposition result, one or more features of various instruction types of a first task type and one or more elements of the sampled features are sampled to obtain a first sampling result; wherein the first task type is any one of the multiple task types; generating a first prompt of the first task type according to the first sampling result; Inputting the first prompt into a first language model, and obtaining a first response corresponding to the first prompt output by the first language model; Generate a first labeling rule according to the features and elements in the first prompt; wherein the first labeling rule includes: a rule for scoring a response corresponding to the first prompt according to the features and elements in the first prompt, and a rule for performing a preference labeling on the response corresponding to the first prompt according to the first prompt and the scores of the response corresponding to the first prompt; Using a labeling model, the first response is labeled according to the first labeling rule to obtain preference data corresponding to the first prompt and the first response, wherein a model training sample of the labeling model includes multiple responses and a prompt corresponding to each of the multiple responses, and a label of the sample includes a score and preference information of each of the multiple responses.

2. The method according to claim 1, characterized in that The multi-layer decomposition of the target task includes: Enumerate the characteristics of each instruction type to obtain a characteristic set of each instruction type; Enumerate the elements of each feature in the feature set to obtain a set of elements of each feature in the feature set; The sampling of one or more features of various instruction types of the first task type, and one or more elements of the sampled features, include: Randomly selecting one or more features from the feature set of each instruction type of the first task type; Randomly select one or more elements from the set of elements of the selected feature.

3. The method according to claim 1, characterized in that Generating a first prompt of the first task type according to the first sampling result includes: Generate a prompt of the first instruction type according to the sampled features of the first instruction type and the sampled elements of the sampled features; wherein the first instruction type is any instruction type of the first task type; The first prompt is generated according to prompts of various instruction types of the first task type.

4. The method according to claim 1, characterized in that: The rule for marking the preference of the responses corresponding to the first prompt includes: a rule for determining the ranking result of the responses corresponding to the first prompt; The sorting result is used to describe the arrangement sequence in the multiple responses corresponding to the first prompt.

5. The method according to claim 1, characterized in that: The method further comprises: Modifying the first prompt to obtain a second prompt, wherein features and elements in the first prompt are consistent with features and elements in the second prompt; Inputting the second prompt into the first language model, and obtaining a second response corresponding to the first prompt output by the first language model; The second response is labeled according to the first labeling rule to obtain preference data corresponding to the first prompt and the second response.

6. The method according to claim 1, characterized in that The method further comprises: Inputting the first prompt and the first instruction into the first language model, and obtaining a third response corresponding to the first prompt output by the first language model; wherein the first instruction is an instruction that affects the thinking logic of the first language model; The third response is labeled according to the first labeling rule to obtain preference data corresponding to the first prompt and the third response.

7. The method according to claim 1, characterized in that The method further comprises: Inputting the first prompt into a second language model, and obtaining a fourth response corresponding to the first prompt output by the second language model; wherein the first language model and the second language model are different language models obtained by training the same model; The fourth response is labeled according to the first labeling rule to obtain preference data corresponding to the first prompt and the fourth response.

8. The method according to claim 1, characterized in that The step of generating a first labeling rule according to the features and elements in the first prompt includes: Obtaining a labeling rule template for the target task; wherein the labeling rule template for the target task includes: a rule for scoring responses corresponding to prompts according to features and elements in the prompts, and a rule for labeling responses with preferences according to the scores of the responses; The features and elements in the first prompt are added to the marking rule template to obtain the first marking rule.

9. The method according to any one of claims 1 to 8, characterized in that: The target task is a writing task; The multiple task types of the writing task include generation tasks, extended writing tasks, and rewriting tasks; The multiple instruction types for each of the task types include text messages and writing requirements.

10. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Performing alignment training on a third language model according to the preference data; the first language model and the third language model have the same model manufacturer.

11. A preference data acquisition device, characterized in that: Used in electronic equipment, including: A task decomposition module, used to perform multi-layer decomposition on the target task to obtain a decomposition result; wherein the decomposition result includes multiple task types of the target task, multiple instruction types of each task type, multiple features of each instruction type, and multiple elements of each feature; a sampling module, configured to sample one or more features of various instruction types of a first task type and one or more elements of the sampled features according to the decomposition result, to obtain a first sampling result; wherein the first task type is any one of the multiple task types; a prompt generating module, configured to generate a first prompt of the first task type according to the first sampling result; a response acquisition module, configured to input the first prompt into a first language model, and obtain a first response corresponding to the first prompt output by the first language model; a labeling rule generating module, configured to generate a first labeling rule according to the features and elements in the first prompt; wherein the first labeling rule comprises: a rule for scoring a response corresponding to the first prompt according to the features and elements in the first prompt, and a rule for performing a preference labeling on a response corresponding to the first prompt according to the first prompt and the scores of the response corresponding to the first prompt; A labeling module is used to use a labeling model to label the first response according to the first labeling rule to obtain preference data corresponding to the first prompt and the first response, wherein the model training sample of the labeling model includes multiple responses and the prompt corresponding to each of the multiple responses, and the label of the sample includes the score and preference information of each of the multiple responses.

12. An electronic device, characterized in that: The electronic device comprises at least one processor, the processor is coupled to a memory, the memory is used to store computer program instructions, and the processor is used to execute computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 10.

Images