Preference learning method and device, model application method and device, equipment and storage medium

Abstract

The invention provides a preference learning method which can be applied to the technical field of artificial intelligence. The preference learning method comprises the steps that sample questions matched with preset standards are determined from a text document pre-trained by a large model, the preset standards comprise a forward standard and a reverse standard, and the sample questions are used for representing questions generated when a user interacts with the large model; inputting the plurality of sample questions into a large model to generate a plurality of question and answer pairs according to answers corresponding to the sample questions output by the large model; evaluating answers in each question and answer pair based on a forward standard by using a plurality of evaluation models to obtain an evaluation label corresponding to each question and answer pair; and training the large model by using preference data generated based on the evaluation label to obtain a target model aligned with the forward standard. The invention further provides a model application method, a preference learning device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a preference learning method, a model application method, an apparatus, a device and a storage medium. Background Art

[0002] As large models are increasingly used, their security issues are becoming increasingly prominent. During the training of large models, if the learning data contains unsafe factors, it may cause deviations in the answers output by the model. To solve this problem, related technologies manually annotate data and combine reinforcement learning with human feedback to guide model training.

[0003] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: the efficiency of manually annotating data is low, which limits the amount of high-quality data available for training, and the process of using human feedback to train the reward model involves high complexity, resulting in poor learning effect of the model on human preferences. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a preference learning method, a model application method, an apparatus, a device, a medium and a program product.

[0005] According to the first aspect of the present disclosure, a preference learning method is provided, comprising: determining sample questions matching preset standards from a text document pre-trained by a large model, wherein the preset standards include a positive standard and a negative standard, and the sample questions are used to characterize the questions generated when a user interacts with the large model; inputting a plurality of the sample questions into the large model to generate a plurality of question-answer pairs according to the answers corresponding to each of the sample questions output by the large model; utilizing a plurality of evaluation models to respectively evaluate the answers in each of the question-answer pairs based on the positive standard to obtain evaluation labels corresponding to each of the question-answer pairs; and utilizing the preference data generated based on the evaluation labels to train the large model to obtain a target model aligned with the positive standard.

[0006] According to an embodiment of the present disclosure, the method of determining sample questions matching a preset standard from a text document pre-trained by a large model includes: constructing prompt words corresponding to the positive standard and prompt words corresponding to the negative standard; utilizing the large model to extract positive sample questions corresponding to the positive standard and reverse sample questions corresponding to the reverse standard from the text document according to the prompt words.

[0007] According to an embodiment of the present disclosure, the above-mentioned multiple sample questions are input into the above-mentioned big model to generate multiple question-answer pairs according to the answers corresponding to each of the above-mentioned sample questions output by the above-mentioned big model, including: for each of the above-mentioned sample questions, the above-mentioned sample questions are repeatedly input into the above-mentioned big model multiple times to obtain a preset number of answers; according to the mapping relationship between the above-mentioned sample questions and the above-mentioned answers, the above-mentioned question-answer pairs are generated, wherein the above-mentioned question-answer pairs include the above-mentioned preset number of answers.

[0008] According to an embodiment of the present disclosure, the above-mentioned use of multiple evaluation models to evaluate the answers in each of the above-mentioned question-answer pairs based on the above-mentioned positive standards respectively, to obtain an evaluation label corresponding to each of the above-mentioned question-answer pairs, including: for each of the above-mentioned question-answer pairs, the above-mentioned question-answer pairs are respectively input into the above-mentioned multiple evaluation models to evaluate each answer in the above-mentioned question-answer pairs based on the above-mentioned positive standards, and output evaluation results; based on the evaluation results output by each of the above-mentioned evaluation models, a multi-model voting method is used to determine the evaluation label corresponding to the above-mentioned question-answer pair.

[0009] According to an embodiment of the present disclosure, the evaluation label corresponding to the question-answer pair includes an evaluation sub-label corresponding to each of the answers. The evaluation label corresponding to the question-answer pair is determined based on the evaluation results output by each of the evaluation models by using a multi-model voting method, including: for each answer in the question-answer pair, determining, among the multiple evaluation results, the number of positive evaluation results that characterize the answer as positive, and the number of negative evaluation results that characterize the answer as negative; when it is determined that the positive number is greater than the negative number, determining the evaluation sub-label of the answer as a positive label; when it is determined that the positive number is less than or equal to the negative number, determining the evaluation sub-label of the answer as a negative label.

[0010] According to an embodiment of the present disclosure, the above-mentioned large model is trained using the preference data generated based on the above-mentioned evaluation labels to obtain a target model aligned with the above-mentioned positive standard, including: marking the evaluation sub-label corresponding to each of the above-mentioned answers in the above-mentioned question-answer pairs to obtain the above-mentioned preference data; and the above-mentioned large model is trained using the above-mentioned preference data to obtain a target model aligned with the above-mentioned positive standard.

[0011] A second aspect of the present disclosure provides a model application method, including: obtaining a target question input by a user; inputting the target question into a target model, and outputting an answer that meets a positive standard in a preset standard.

[0012] The third aspect of the present disclosure provides a preference learning device, including: a question determination module, used to determine sample questions that match preset standards from a text document pre-trained by a large model, wherein the preset standards include positive standards and negative standards, and the sample questions represent questions generated when a user interacts with the large model; an answer generation module, used to input multiple sample questions into the large model to generate multiple question-answer pairs according to the answers corresponding to each sample question output by the large model; an answer evaluation module, used to evaluate the answers in each question-answer pair based on the positive standards using multiple evaluation models, and obtain evaluation labels corresponding to each question-answer pair; a model training module, used to train the large model using preference data generated based on the evaluation labels, and obtain a target model aligned with the positive standards.

[0013] The fourth aspect of the present disclosure provides a model application device, comprising: a question acquisition module, used to acquire a target question input by a user; an answer determination module, used to input the target question into a target model, and output an answer that meets a positive standard in a preset standard.

[0014] A fifth aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The sixth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] The seventh aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0017] According to the embodiments of the present disclosure, by generating question-answer pairs according to positive standards and negative standards, and using multiple evaluation models to evaluate the question-answer pairs, the evaluation labels of each question-answer pair are obtained, thereby realizing automatic labeling of data. Since the evaluation model evaluates the answer based on the positive standard, the evaluation label is determined according to the evaluation result output by the evaluation model, so that the evaluation label is determined more accurately. By integrating the evaluation model to learn the preferences of the large model, the large model can learn a large amount of automatically labeled data during the security alignment process, and the use of question-answer pairs makes the learning process faster and more efficient, effectively improving the accuracy of the large model in learning human preferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 A diagram schematically illustrates an application scenario of a preference learning method, a model application method, an apparatus, a device, a medium, and a program product according to an embodiment of the present disclosure;

[0020] Figure 2 A flowchart of a preference learning method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 3 A flowchart of a preference learning method according to another embodiment of the present disclosure is schematically shown;

[0022] Figure 4 A flowchart of large model training according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A flowchart of a model application method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 6 The structure block diagram of the preference learning device according to the embodiment of the present disclosure is schematically shown;

[0025] Figure 7 A structural block diagram of a model application device according to an embodiment of the present disclosure is schematically shown; and

[0026] Figure 8 A block diagram of an electronic device suitable for implementing a preference learning method and a model application method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0033] In the financial industry, large models are increasingly used, for example, to generate various research reports, marketing copy, or financial Q&A. However, due to the existence of hallucinations and the fact that the learned data may contain unsafe data, large models may have a large number of security issues during use, especially in the customer process. However, the overall cost of constructing and training security data is currently high, mainly due to the difficulty of data construction and annotation, the high complexity of the RLHF (Reinforcement Learning from Human Feedback) alignment algorithm, and the difficulty of convergence.

[0034] The specific process of the RLHF alignment algorithm is as follows:

[0035] (1) Initializing the model: First, you need to initialize a language model, such as a neural network-based language model (such as a recurrent neural network or a transformer model), which will be optimized to generate better text output.

[0036] (2) Interact with humans: Next, interact with humans to collect feedback. This can be done in different ways,

[0037] (3) Building a feedback model: Converting human feedback into a form that the model can understand, such as converting human demonstrations into state-action pairs, or converting human evaluations into reward signals.

[0038] (4) Reinforcement learning training: Use a reinforcement learning algorithm, such as a deep reinforcement learning algorithm, and use the feedback model as a reward signal to train the language model. The model interacts with the environment and adjusts the model parameters based on the reward signal provided by the feedback model to maximize the long-term cumulative reward.

[0039] Iterative optimization: Repeat steps 2 to 4 to collect more feedback through interaction with humans and continuously improve and optimize the language model.

[0040] Embodiments of the present disclosure provide a preference learning method, a model application method, an apparatus, a device and a storage medium, wherein the preference learning method comprises: determining sample questions that match preset standards from a text document pre-trained by a large model, wherein the preset standards include a positive standard and a negative standard, and the sample questions are used to characterize the questions generated when a user interacts with the large model; inputting multiple sample questions into the large model to generate multiple question-answer pairs based on the answers corresponding to each sample question output by the large model; using multiple evaluation models to evaluate the answers in each question-answer pair based on the positive standard, and obtaining evaluation labels corresponding to each question-answer pair; and using the preference data generated based on the evaluation labels to train the large model to obtain a target model aligned with the positive standard.

[0041] Figure 1 An application scenario diagram of a preference learning method, a model application method, an apparatus, a device, a medium, and a program product according to an embodiment of the present disclosure is schematically shown.

[0042] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0044] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0046] It should be noted that the preference learning method and model application method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the preference learning device and model application device provided in the embodiments of the present disclosure can generally be set in the server 105. The preference learning method and model application method provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the preference learning device and model application device provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0047] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0048] Figure 2 The flowchart of the preference learning method according to the embodiment of the present disclosure is schematically shown.

[0049] like Figure 2 As shown, the preference learning method of this embodiment includes operations S210 to S240.

[0050] In operation S210, sample questions matching preset standards are determined from the text document pre-trained by the large model, wherein the preset standards include positive standards and negative standards, and the sample questions are used to characterize the problems generated when the user interacts with the large model.

[0051] In operation S220, a plurality of sample questions are input into the large model to generate a plurality of question-answer pairs according to an answer corresponding to each sample question output by the large model.

[0052] In operation S230, multiple evaluation models are used to evaluate the answers in each question-answer pair based on the positive criteria to obtain evaluation labels corresponding to each question-answer pair.

[0053] In operation S240 , the large model is trained using the preference data generated based on the evaluation labels to obtain a target model aligned with the forward standard.

[0054] According to the embodiments of the present disclosure, the insecurity of the learning data during the training process of the large model may lead to deviations in the output answers. Therefore, the learning data needs to be labeled during the training process so that the large model can complete security alignment. Specifically, first based on the SFT (Supervised Fine-Tuning) learning scheme, the instruction fine-tuning data is trained to train a large model that has instruction compliance capabilities but does not have security alignment. Among them, instruction fine-tuning is to use the large model to learn the logic and method of analyzing problems from the question and answer pairs, so as to achieve similar capabilities as those shown by the data set. By means of instruction fine-tuning, a small amount of data can be used for model training, which greatly reduces the subsequent training costs of the model.

[0055] According to an embodiment of the present disclosure, a preset standard needs to be constructed before the large model is safely aligned, wherein the preset standard includes a positive standard and a negative standard. The positive standard is created based on correct values, for example: the positive standard includes positive views, while the negative standard is the opposite of the positive standard.

[0056] According to an embodiment of the present disclosure, a text document pre-trained by a large model is obtained, and the large model currently pre-trained has the ability to generate sentence text using keywords. From the text document pre-trained by the large model, sample questions that match the preset standards are determined. The sample questions are used to characterize the questions generated when the user interacts with the large model. Multiple sample questions are input into the large model to obtain the answer output by the large model for each sample question.

[0057] According to an embodiment of the present disclosure, a question-answer pair is generated based on a mapping relationship between a sample question and an answer, wherein the question-answer pair may be in the form of a sample question-answer. After obtaining the question-answer pair, the answer in the question-answer pair is evaluated using multiple evaluation models. Specifically, the question-answer pair is input into the evaluation model, and the evaluation model evaluates the answer based on a positive standard to obtain an evaluation result.

[0058] According to the embodiments of the present disclosure, the evaluation model is the optimal large model that can be obtained. In generative AI (artificial intelligence) tasks, the effect of the evaluation model is mainly based on manual evaluation, and the evaluation has problems such as poor consistency and non-disclosure and opacity. Therefore, a large model is trained using a large amount of manually annotated evaluation data (sufficiently consistent alignment) so that the large model has the ability to automatically distinguish generative effects.

[0059] According to an embodiment of the present disclosure, an evaluation label corresponding to each question-answer pair is determined according to the evaluation result output by each evaluation model, wherein the evaluation label is used to characterize whether the answer meets the positive standard. Preference data is generated based on the evaluation label, and the large model is trained using the preference data to obtain a target model aligned with the positive standard. Specifically, the RLHF alignment algorithm and the DPO (Direct Preference Optimization) algorithm can be used for learning in the process of preference learning.

[0060] According to the embodiments of the present disclosure, the role of RLHF is to provide a natural, humane, interactive learning process for machines by incorporating human feedback into the training process. This is just like the way humans learn professional knowledge from another professional. By building a bridge between humans and AI, RLHF allows AI to quickly master human experience. In RLHF, reinforcement learning is combined with human feedback, and human preferences are used as reward signals to guide the training of the model, thereby enhancing the model's understanding and satisfaction of human intentions. In the generative model, RLHF can also fully align the generated images with the text prompts.

[0061] According to the embodiments of the present disclosure, the DPO algorithm cleverly combines the reward model (reward model training) and reinforcement learning steps in reinforcement learning RLHF into one, making training faster and more efficient. At the same time, DPO has only one hyperparameter beta (used to control the KL divergence between the new model and the SFT model), making debugging easier. Among them, the KL divergence (Kullback–Leibler divergence) is used to measure the distance between two probability distributions and measure the similarity between two probability distributions. DPO can achieve a similar security alignment effect as RLHF, and has simpler requirements for data labeling.

[0062] According to the embodiments of the present disclosure, by generating question-answer pairs according to positive standards and negative standards, and using multiple evaluation models to evaluate the question-answer pairs, the evaluation labels of each question-answer pair are obtained, thereby realizing automatic labeling of data. Since the evaluation model evaluates the answer based on the positive standard, the evaluation label is determined according to the evaluation result output by the evaluation model, so that the evaluation label is determined more accurately. By integrating the evaluation model to learn the preferences of the large model, the large model can learn a large amount of automatically labeled data during the security alignment process, and the use of question-answer pairs makes the learning process faster and more efficient, effectively improving the accuracy of the large model in learning human preferences.

[0063] According to an embodiment of the present disclosure, sample questions matching preset standards are determined from a text document pre-trained by a large model, including: constructing prompt words corresponding to a positive standard and prompt words corresponding to a negative standard; utilizing the large model to extract positive sample questions corresponding to the positive standard and reverse sample questions corresponding to the reverse standard from the text document based on the prompt words.

[0064] According to an embodiment of the present disclosure, a prompt word is a text fragment input into a large model to guide the large model to generate a specific output or perform a specific task. Before constructing the prompt word, it is clear what task the large model is expected to complete or the type of output it is expected to generate. The prompt word should include sufficient contextual information so that the model can understand the specific context of the task, and the prompt word also includes keywords related to the task.

[0065] According to an embodiment of the present disclosure, multiple prompt words are constructed to cover different aspects of a task or the needs of different users. The prompt words are adjusted according to the characteristics of the pre-trained large model used, and different models may have different sensitivities to the prompt words. Specifically, the prompt words constructed corresponding to the positive standard may include keywords associated with positive viewpoints, and the prompt words constructed for the negative standard include keywords that are opposite to the positive standard.

[0066] According to an embodiment of the present disclosure, a large model is used to extract positive sample questions corresponding to the positive standard and reverse sample questions corresponding to the reverse standard from a text document according to prompt words. Extracting sample questions refers to directly identifying and extracting actual existing questions from existing text documents. These are questions that have been clearly raised in the document and do not need to be created through any generation process. The process of extracting questions involves steps such as data cleaning, question identification and question extraction, among which the extracted questions are usually closely related to the context of the document.

[0067] According to an embodiment of the present disclosure, the large model can also directly generate sample questions based on the prompt words. Generating sample questions means creating new questions through algorithms. These questions may not exist in the original document, but are generated through the creative process of the model. Specifically, generating sample questions can select appropriate strategies based on specific needs and resources, and automatically generate a large number of them in batches from pre-trained text documents. Generating sample questions can cover a wider range of topics and fields, thereby increasing the diversity of questions.

[0068] According to an embodiment of the present disclosure, multiple sample questions are input into a large model to generate multiple question-answer pairs based on the answers corresponding to each sample question output by the large model, including: for each sample question, the sample question is repeatedly input into the large model multiple times to obtain a preset number of answers; based on the mapping relationship between the sample question and the answer, a question-answer pair is generated, wherein the question-answer pair includes a preset number of answers.

[0069] According to an embodiment of the present disclosure, a sample question is input into the large model twice to obtain two answers, forming a question-answer pair: Q-A1-A2, where Q is the sample question, A1 is answer 1, and A2 is answer 2. In DPO (Direct Preference Optimization), two answers need to be generated for the same question. The core of the DPO algorithm is to use preference data for policy optimization without explicitly fitting a reward model. It first generates two answers based on a given prompt word, and compares the two answers through manual annotation to obtain a preference result. This method can construct a data set that reflects human preferences for training and optimizing models.

[0070] According to the embodiments of the present disclosure, two answers are generated for the same sample question in DPO in order to construct a data set that reflects human preferences, simplify the training objectives, optimize the model using the contrastive learning principle, and prevent the degradation of model capabilities through dynamic weights, ultimately improving the model's ability to generate answers that meet human preferences.

[0071] According to an embodiment of the present disclosure, multiple evaluation models are used to evaluate the answers in each question-answer pair based on a positive standard to obtain an evaluation label corresponding to each question-answer pair, including: for each question-answer pair, the question-answer pair is input into multiple evaluation models respectively to evaluate each answer in the question-answer pair based on the positive standard, and an evaluation result is output; based on the evaluation result output by each evaluation model, a multi-model voting method is used to determine the evaluation label corresponding to the question-answer pair.

[0072] According to an embodiment of the present disclosure, multiple excellent large models are used as evaluation models to evaluate question-answer pairs. The question-answer pairs are input into multiple evaluation models respectively, and each answer in the question-answer pair is evaluated based on a positive standard, and an evaluation result is output. The evaluation result can be a score or a level of the answer in the question-answer pair.

[0073] According to an embodiment of the present disclosure, a multi-model voting method is used to determine the evaluation label corresponding to the question-answer pair. Among them, more votes for the answer are good results, and fewer votes for the answer are bad results. After the voting is completed, the evaluation label corresponding to the question-answer pair is determined, where the evaluation label can be represented by good and bad. The method of scoring and voting based on multiple evaluation models at the same time further improves the evaluation accuracy of the large model, thereby achieving higher automation and precision.

[0074] According to an embodiment of the present disclosure, the evaluation label corresponding to the question and answer pair includes an evaluation sub-label corresponding to each answer. Based on the evaluation results output by each evaluation model, a multi-model voting method is used to determine the evaluation label corresponding to the question and answer pair, including: for each answer in the question and answer pair, determining, among multiple evaluation results, the number of positive evaluation results representing that the answer is positive, and the number of negative evaluation results representing that the answer is negative; when it is determined that the number of positive answers is greater than the number of negative answers, determining that the evaluation sub-label of the answer is a positive label; when it is determined that the number of positive answers is less than or equal to the number of negative answers, determining that the evaluation sub-label of the answer is a negative label.

[0075] According to an embodiment of the present disclosure, in the training process of DPO (Direct Preference Optimization), each answer in the question-answer pair needs to be labeled. The training data set of DPO consists of three parts: prompt (sample question), chosen (answer 1) and rejected (answer 2). Prompt is equivalent to a sample question, while chosen and rejected are two answers generated for the question, which are respectively marked as answers that meet the positive criteria and answers that do not meet the positive criteria. The goal of DPO training is to make the model more inclined to generate answers with the chosen label rather than answers with the rejected label.

[0076] According to an embodiment of the present disclosure, for two answers to the same sample question, it is necessary to determine which of the two answers better meets the positive standard through label annotation, that is, which one is chosen and which one is rejected. Specifically, determine the evaluation results for the answers output by each evaluation model, as well as the number of positives and negatives in multiple evaluation results. Using a multi-model voting method, determine that the answer with more votes is chosen, that is, the number of positives is greater than the number of negatives. The answer with fewer votes is rejected, that is, the number of positives is less than or equal to the number of negatives. Among them, the positive label can be set to good, and the negative label can be set to bad. By annotating the labels of the answers, the model is trained so that it can learn how to generate answers that are more in line with human preferences.

[0077] According to an embodiment of the present disclosure, preference data generated based on evaluation labels is used to train a large model to obtain a target model aligned with a positive standard, including: annotating the evaluation sub-label corresponding to each answer in a question-answer pair to obtain preference data; and training the large model with the preference data to obtain a target model aligned with a positive standard.

[0078] According to an embodiment of the present disclosure, the evaluation sub-label corresponding to each answer is annotated in the question-answer pair, and a good question-answer pair Q-Agood and a bad question-answer pair Q-Abad are obtained respectively. The good question-answer pairs and the bad question-answer pairs in the preference data are used to train the large model to obtain a target model aligned with the positive standard.

[0079] According to an embodiment of the present disclosure, during the training of a large model, the loss function of DPO is as follows:

[0080]

[0081] Among them, L DPO (π θ ;π ref ) represents the loss function of direct preference optimization; π θ Represents the model strategy to be optimized, with parameter θ; π ref represents the reference strategy, which is usually a trained model used to control the behavior of the large model; E(x,y w ,y l )∼D represents sampling input x from data distribution D, and the corresponding preferred output y w and the unpreferable output y l ; σ represents the logical function, which is used to map the input between 0 and 1 and calculate the probability; β represents a hyperparameter used to adjust the weight of the two probability ratios; w represents the answer that meets the positive standard; l represents the answer that does not meet the positive standard.

[0082] According to an embodiment of the present disclosure, the left half of formula (1) βlog(π θ (y w ∣x) / π ref (y w ∣x)) is the actor to be trained (a large model to be trained). Model selection chosen takes precedence over rejected. The right half βlog(π θ (y l ∣x) / π ref (y l ∣x)) The frozen reference model chooses chosen over rejected. The difference between the two can be similar to the KL divergence, ensuring that the distribution of the actor model (the model to be trained) and the distribution of the reference model (the model of the sft) will not be very different. The overall result is to make the model selection tend to meet the positive standard answer.

[0083] According to an embodiment of the present disclosure, KL divergence is used to measure the distance between two probability distributions and to measure the similarity between the two probability distributions. It is defined as follows:

[0084]

[0085] Among them, D KL (P||Q) represents the KL divergence between probability distributions P and Q, where X represents the set of all possible events; P(x) represents the probability of event x in distribution P; Q(x) is the probability of event x in distribution Q. The divergence is 0 if and only if the distributions of P and Q are exactly the same. The greater the difference between the two distributions, the greater the divergence.

[0086] According to an embodiment of the present disclosure, KL divergence is used to describe the asymmetric difference of one probability distribution relative to another probability distribution.

[0087] Figure 3 The flowchart of large model training according to an embodiment of the present disclosure is schematically shown.

[0088] like Figure 3 As shown, in the process of training a large model, there are two key components: a training language model 301, which is being optimized; and a frozen language model 302, which serves as a reference model. The labeled preference data are input into the training language model 301 and the frozen language model 302, respectively, wherein the preference data includes the labeled question-answer pairs 303 that meet the positive standard and the question-answer pairs 304 that do not meet the positive standard.

[0089] According to an embodiment of the present disclosure, the language model 301 in training and the frozen language model 302 generate scores for each question-answer pair, including scores 305 for question-answer pairs that meet the positive standard and scores 306 for question-answer pairs that do not meet the positive standard. These scores reflect the model's preference for each output and are used as a reference.

[0090] According to the embodiments of the present disclosure, the score can be calculated using the Bradley–Terry model (statistical model), which is used to model the comparative relationship between things. For example, in chess games, chess players often win and lose against each other. The Bradley–Terry model can model all professional chess players based on this win-loss information and give them an intrinsic "score". This score can reflect the level of the chess player to a certain extent and predict the probability of winning or losing a game between two chess players. The calculation formula is as follows:

[0091]

[0092] Where P(i>j) represents the probability that project i wins over project j; β i and β j is a parameter associated with items i and j, usually learned from the data through methods such as maximum likelihood estimation; e is the base of the natural logarithm.

[0093] According to an embodiment of the present disclosure, the score ratio between the score 305 of the question-answer pair that meets the positive standard and the score 306 of the question-answer pair that does not meet the positive standard is calculated in the language model 301 in training and the frozen language model 302. The score ratio 307 of the language model 301 in training and the score ratio 308 of the frozen language model 302 are input into the loss function 309 to calculate the loss between the models. According to the calculated loss, the weight of the language model 301 in training will be updated to reduce the loss and optimize the model so that it is more inclined to generate the selected output.

[0094] According to an embodiment of the present disclosure, the model's preference is optimized by directly comparing the difference between the output generated by the model and the output preferred by humans, without explicitly annotating the preference label of each output. This method can more effectively use human feedback to guide model training.

[0095] Figure 4 The flowchart of a preference learning method according to another embodiment of the present disclosure is schematically shown.

[0096] like Figure 4 As shown, the preference learning method of another embodiment includes operations S410 to S470.

[0097] In operation S410, a positive sample question corresponding to a positive standard is extracted.

[0098] In operation S420, reverse sample questions corresponding to the reverse standard are extracted.

[0099] According to an embodiment of the present disclosure, sample questions are extracted based on positive criteria and negative criteria, respectively, and specifically are extracted from pre-trained documents using a large model.

[0100] In operation S430, the answer to the positive sample question is generated twice to generate a question-answer pair.

[0101] In operation S440, an answer to the reverse sample question is generated twice to generate a question-answer pair.

[0102] According to an embodiment of the present disclosure, a sample question is input into the large model twice, the corresponding answer is output, and a corresponding question-answer pair is generated according to the mapping relationship.

[0103] In operation S450, multiple evaluation models are used to evaluate the answers in each question-answer pair based on the positive criteria to obtain an evaluation label corresponding to each question-answer pair.

[0104] According to the embodiments of the present disclosure, the method of simultaneously scoring and voting based on multiple evaluation models further improves the evaluation accuracy of large models, thereby achieving higher automation and precision.

[0105] In operation S460 , preference data is generated based on the evaluation tags.

[0106] In operation S470, the large model is trained using the preference data to obtain a target model aligned with the forward standard.

[0107] According to the embodiments of the present disclosure, by generating question-answer pairs according to positive standards and negative standards, and using multiple evaluation models to evaluate the question-answer pairs, the evaluation labels of each question-answer pair are obtained, thereby realizing automatic labeling of data. Since the evaluation model evaluates the answer based on the positive standard, the evaluation label is determined according to the evaluation result output by the evaluation model, so that the evaluation label is determined more accurately. By integrating the evaluation model to learn the preferences of the large model, the large model can learn a large amount of automatically labeled data during the security alignment process, and the use of question-answer pairs makes the learning process faster and more efficient, effectively improving the accuracy of the large model in learning human preferences.

[0108] Figure 5 The flowchart of the model application method according to the embodiment of the present disclosure is schematically shown.

[0109] like Figure 5As shown, the model application method of this embodiment includes operations S510 and S520.

[0110] In operation S510, a target question input by a user is obtained.

[0111] In operation S520, the target question is input into the target model, and an answer that meets the positive standard in the preset standard is output.

[0112] According to the embodiments of the present disclosure, when a user uses a trained target model, the user inputs the target question to be asked into the target model, and the target model analyzes the target question and outputs the corresponding answer. The output answer meets the positive standard, that is, the answer tends to be positive. By using the safely aligned target model to answer the target question, the feedback of the model is more in line with human values.

[0113] Based on the above preference learning method, the present disclosure also provides a preference learning device. Figure 6 The device is described in detail.

[0114] Figure 6 The structure block diagram of the preference learning device according to the embodiment of the present disclosure is schematically shown.

[0115] like Figure 6 As shown, the preference learning device 600 of this embodiment includes a question determination module 610 , an answer generation module 620 , an answer evaluation module 630 and a model training module 640 .

[0116] The question determination module 610 is used to determine sample questions that match preset standards from the text documents pre-trained by the large model, where the preset standards include positive standards and negative standards, and the sample questions represent the problems generated when the user interacts with the large model. In one embodiment, the question determination module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0117] The answer generation module 620 is used to input multiple sample questions into the large model to generate multiple question-answer pairs according to the answers corresponding to each sample question output by the large model. In one embodiment, the answer generation module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0118] The answer evaluation module 630 is used to evaluate the answers in each question-answer pair based on the positive criteria using multiple evaluation models to obtain an evaluation label corresponding to each question-answer pair. In one embodiment, the answer evaluation module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0119] The model training module 640 is used to train the large model using the preference data generated based on the evaluation tags to obtain a target model aligned with the positive standard. In one embodiment, the model training module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0120] According to an embodiment of the present disclosure, the question determination module 610 includes a prompt construction submodule and a question extraction submodule.

[0121] The prompt construction submodule is used to construct prompt words corresponding to the positive standard and prompt words corresponding to the negative standard.

[0122] The question extraction submodule is used to extract positive sample questions corresponding to the positive standard and reverse sample questions corresponding to the reverse standard from the text document according to the prompt words using the large model.

[0123] According to an embodiment of the present disclosure, the answer generation module 620 includes an answer determination submodule and a question and answer generation submodule.

[0124] The answer determination submodule is used to input the sample question into the large model repeatedly for each sample question to obtain a preset number of answers.

[0125] The question-answer generation submodule is used to generate question-answer pairs based on the mapping relationship between sample questions and answers, wherein the question-answer pairs include a preset number of answers.

[0126] According to an embodiment of the present disclosure, the answer evaluation module 630 includes a question and answer evaluation submodule and a label determination submodule.

[0127] The question-answer evaluation submodule is used to input each question-answer pair into multiple evaluation models respectively, so as to evaluate each answer in the question-answer pair based on the positive standard and output the evaluation result.

[0128] The label determination submodule is used to determine the evaluation label corresponding to the question-answer pair based on the evaluation results output by each evaluation model using a multi-model voting method.

[0129] According to an embodiment of the present disclosure, the tag determination submodule includes a quantity determination unit, a forward determination unit, and a reverse determination unit.

[0130] The quantity determination unit is used to determine, for each answer in the question-answer pair, the number of positive evaluation results representing that the answer is positive, and the number of negative evaluation results representing that the answer is negative, among multiple evaluation results.

[0131] The positive determination unit is used to determine that the evaluation sub-label of the answer is a positive label when it is determined that the positive number is greater than the negative number.

[0132] The reverse determination unit is used to determine that the evaluation sub-label of the answer is a reverse label when it is determined that the positive number is less than or equal to the reverse number.

[0133] According to an embodiment of the present disclosure, the model training module 640 includes a label annotation submodule and a model training submodule.

[0134] The label annotation submodule is used to annotate the evaluation sublabel corresponding to each answer in the question-answer pair to obtain preference data.

[0135] The model training submodule uses the preference data to train the large model and obtains the target model that is aligned with the positive standard.

[0136] Figure 7 The structural block diagram of the model application device according to the embodiment of the present disclosure is schematically shown.

[0137] like Figure 7 As shown, the model application device 700 of this embodiment includes a question acquisition module 710 and an answer determination module 720 .

[0138] The question acquisition module 710 is used to acquire the target question input by the user.

[0139] The answer determination module 720 is used to input the above target question into the target model and output an answer that meets the positive standard in the preset standard.

[0140] According to an embodiment of the present disclosure, any multiple modules in the question determination module 610, the answer generation module 620, the answer evaluation module 630 and the model training module 640, and the question acquisition module 710 and the answer determination module 720 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the question determination module 610, the answer generation module 620, the answer evaluation module 630 and the model training module 640, and the question acquisition module 710 and the answer determination module 720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the question determination module 610, the answer generation module 620, the answer evaluation module 630 and the model training module 640 and the question acquisition module 710 and the answer determination module 720 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0141] Figure 8 A block diagram of an electronic device suitable for implementing a preference learning method and a model application method according to an embodiment of the present disclosure is schematically shown.

[0142] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0143] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0144] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the input / output (I / O) interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage portion 808 as needed.

[0145] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0146] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0147] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the preference learning method and model application method provided by the embodiment of the present disclosure.

[0148] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0149] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0150] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0151] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0152] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0154] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A preference learning method, characterized in that: The method comprises: Determine, from the text document pre-trained by the large model, sample questions that match the preset standard, wherein the preset standard includes a positive standard and a negative standard, and the sample questions are used to characterize the problems generated when the user interacts with the large model; Inputting a plurality of the sample questions into the large model to generate a plurality of question-answer pairs according to the answers corresponding to each of the sample questions output by the large model; Using multiple evaluation models, based on the positive criteria, the answers in each of the question-answer pairs are evaluated respectively to obtain evaluation labels corresponding to each of the question-answer pairs; The large model is trained using preference data generated based on the evaluation labels to obtain a target model aligned with the positive standard.

2. The method according to claim 1, characterized in that: Determining sample questions that match preset standards from the text documents pre-trained by the large model includes: Constructing prompt words corresponding to the positive standard and prompt words corresponding to the negative standard; By using the large model, positive sample questions corresponding to the positive standard and negative sample questions corresponding to the negative standard are extracted from the text document according to the prompt words.

3. The method according to claim 1, characterized in that The step of inputting the plurality of sample questions into the large model to generate a plurality of question-answer pairs according to the answer corresponding to each of the sample questions output by the large model comprises: For each of the sample questions, the sample question is repeatedly input into the large model to obtain a preset number of answers; The question-answer pair is generated according to the mapping relationship between the sample question and the answer, wherein the question-answer pair includes the preset number of answers.

4. The method according to claim 1, characterized in that The method of using multiple evaluation models to evaluate the answers in each question-answer pair based on the positive criteria to obtain an evaluation label corresponding to each question-answer pair includes: For each of the question-answer pairs, the question-answer pairs are respectively input into a plurality of the evaluation models to evaluate each answer in the question-answer pair based on the positive criteria, and output an evaluation result; Based on the evaluation results output by each of the evaluation models, an evaluation label corresponding to the question-answer pair is determined using a multi-model voting method.

5. The method according to claim 4, characterized in that The evaluation label corresponding to the question-answer pair includes an evaluation sub-label corresponding to each of the answers, and the evaluation label corresponding to the question-answer pair is determined by a multi-model voting method based on the evaluation results output by each of the evaluation models, including: For each answer in the question-answer pair, determining, among the plurality of evaluation results, the number of positive evaluation results that characterize the answer as positive, and the number of negative evaluation results that characterize the answer as negative; In the case where it is determined that the positive number is greater than the negative number, determining the evaluation sub-label of the answer to be a positive label; In the case where it is determined that the positive quantity is less than or equal to the negative quantity, the evaluation sub-label of the answer is determined to be a negative label.

6. The method according to claim 5, characterized in that The step of training the large model using the preference data generated based on the evaluation labels to obtain a target model aligned with the positive standard includes: Annotating the evaluation sub-label corresponding to each of the answers in the question-answer pair to obtain the preference data; The large model is trained using the preference data to obtain a target model aligned with the forward standard.

7. A model application method, characterized in that: The method comprises: Get the target question input by the user; The target question is input into the target model described in any one of claims 1 to 6, and an answer that meets the positive standard in the preset standard is output.

8. A preference learning device, characterized in that: The device comprises: A question determination module, used to determine sample questions matching preset criteria from the text documents pre-trained by the large model, wherein the preset criteria include a positive criterion and a negative criterion, and the sample questions represent questions generated when the user interacts with the large model; An answer generation module, used for inputting a plurality of the sample questions into the large model, so as to generate a plurality of question-answer pairs according to the answers corresponding to each of the sample questions output by the large model; An answer evaluation module, used to evaluate the answers in each question-answer pair based on the positive criteria using multiple evaluation models to obtain evaluation labels corresponding to each question-answer pair; A model training module is used to train the large model using preference data generated based on the evaluation labels to obtain a target model aligned with the positive standard.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6 or claim 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 or claim 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 or claim 7 are implemented.

Images