Model knowledge injection method and device and electronic equipment

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a model knowledge injection method and device and electronic equipment. The method comprises the following steps: if quality scores of two answers of the same question in a specified field meet the following preference requirements: the quality score of a first answer is greater than a first set threshold value, and the quality score of a second answer is lower than a second set threshold value, taking the two answers as a preference data pair corresponding to the question, constructing a preference data set based on the preference data pair corresponding to each question; in the alignment stage, outputting token prediction distribution of each question based on the input preference data set through the current knowledge answer model; calculating a first loss and a second loss based on the token prediction distribution and the answer of each question; and if the first loss and the second loss meet a preset iteration stop condition, taking the current knowledge answer model as a target knowledge answer model, otherwise, adjusting parameters of the knowledge answer model, and continuing iterative training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, and electronic device for injecting model knowledge. Background Technology

[0002] Since large models still have significant limitations in understanding and processing domain-specific knowledge, knowledge injection techniques have been widely introduced to overcome this bottleneck. These techniques embed expert experience and domain information into the model to enhance its cognitive and reasoning abilities in specialized scenarios.

[0003] In terms of knowledge injection, the current mainstream technical approaches mainly include incremental pre-training and full parameter fine-tuning. However, incremental pre-training requires a large amount of training data and has extremely high computational resource requirements. Full parameter fine-tuning has slightly lower resource and data requirements than incremental pre-training, but it still carries the risk of the model forgetting its original capabilities. It relies on a fine-grained general data ratio to maintain performance balance, and often requires an additional alignment process after training to make the model's behavior conform to human preferences, significantly increasing training complexity and cost. Therefore, when injecting knowledge, a technical solution that can balance training cost and model performance is needed. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, and electronic device for injecting model knowledge.

[0005] On the one hand, this application provides a method for injecting model knowledge, the method including: If the quality scores of two answers to the same question in a specified domain meet the following preference requirements: the quality score of the first answer is greater than a first set threshold, the quality score of the second answer is lower than a second set threshold, and the second set threshold is not greater than the first set threshold, then the two answers are regarded as the preference data pair corresponding to the question, and a preference dataset is constructed based on the preference data pairs corresponding to each question; the quality score is obtained after evaluating the answers to the question based on the knowledge of the specified domain. During the alignment phase, the current knowledge-answering model outputs the predicted distribution of the token for each question based on the input preference dataset. The first preference probability is calculated based on the token prediction distribution of each question and the first answer corresponding to each question, and the second preference probability is calculated based on the token prediction distribution of each question and the second answer corresponding to each question. The first loss is calculated based on the first preference probability and the second preference probability of each question, and the second loss is calculated based on the first preference probability of each question. The first loss is used to reduce the probability that the knowledge answering model generates a non-preference answer to guide the knowledge answering model away from non-preference answers, where the second answer is a non-preference answer. The second loss is used to make the output of the knowledge answering model converge towards a preference answer, where the first answer is a preference answer. If the first loss and the second loss satisfy the preset iteration stopping condition, the current knowledge answering model is used as the target knowledge answering model; otherwise, the parameters of the knowledge answering model are adjusted and iterative training continues. The target knowledge answering model is used to receive user input of questions related to a specified domain and generate answers that conform to the specified domain knowledge and human preferences.

[0006] On one hand, this application provides a device for injecting model knowledge, including: The construction unit is used to construct a preference dataset based on the preference data pairs corresponding to the question if the quality scores of two answers to the same question in a specified domain meet the following preference requirements: the quality score of the first answer is greater than a first set threshold, the quality score of the second answer is lower than a second set threshold, and the second set threshold is not greater than the first set threshold. The output unit is used during the alignment phase to output the predicted distribution of the token for each question based on the input preference dataset using the current knowledge-answering model. The computational unit is used to calculate a first preference probability based on the token prediction distribution of each question and the first answer corresponding to each question, and to calculate a second preference probability based on the token prediction distribution of each question and the second answer corresponding to each question. It also calculates a first loss based on the first preference probability and the second preference probability of each question, and a second loss based on the first preference probability of each question. The first loss is used to reduce the probability that the knowledge answering model generates a non-preference answer to guide the model away from non-preference answers; the second answer is a non-preference answer. The second loss is used to make the output of the knowledge answering model converge towards a preference answer; the first answer is a preference answer. The adjustment unit is used to take the current knowledge answering model as the target knowledge answering model if the first loss and the second loss meet the preset iteration stopping condition; otherwise, it adjusts the parameters of the knowledge answering model and continues iterative training. The target knowledge answering model is used to receive user input of questions related to a specified domain and generate answers that conform to the specified domain knowledge and human preferences.

[0007] On one hand, this application provides an electronic device, including: Processor; and The memory stores computer instructions that cause the processor to perform steps of the methods provided in various alternative implementations of the knowledge injection model described above.

[0008] On the one hand, embodiments of this application provide a computer-readable storage medium storing computer instructions for causing a computer to perform the steps of the method provided in various optional implementations of any of the above-described model knowledge injection methods.

[0009] On one hand, this application provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the steps of the method provided in various optional implementations of any of the above-described model knowledge injection methods.

[0010] The model knowledge injection method in this application includes: if the quality scores of two answers to the same question in a specified domain meet the following preference requirements: the quality score of the first answer is greater than a first set threshold, and the quality score of the second answer is lower than a second set threshold, and the second set threshold is not greater than the first set threshold, then the two answers are regarded as a preference data pair corresponding to the question, and a preference dataset is constructed based on the preference data pairs corresponding to each question; the quality score is obtained after evaluating the answers to the question based on the specified domain knowledge; in the alignment stage, the current knowledge answering model outputs the token prediction distribution of each question based on the input preference dataset; the first preference probability is calculated based on the token prediction distribution of each question and the first answer corresponding to each question, and the first preference probability is calculated based on the token prediction distribution of each question. The algorithm calculates a second preference probability for each question's second answer, and a first loss based on the first and second preference probabilities for each question, and a second loss based on the first preference probability for each question. The first loss reduces the probability of the knowledge answering model generating a non-preference answer, guiding it away from non-preference answers; the second answer is considered a non-preference answer. The second loss makes the output of the knowledge answering model converge towards a preference answer; the first answer is considered a preference answer. If the first and second losses satisfy a preset iteration stopping condition, the current knowledge answering model is used as the target knowledge answering model; otherwise, the parameters of the knowledge answering model are adjusted, and iterative training continues. The target knowledge answering model receives user input on questions related to a specified domain and generates answers that conform to the specified domain knowledge and human preferences. This approach balances the training cost of model alignment training with model performance. Attached Figure Description

[0011] Figure 1 This is a flowchart of a model knowledge injection method in an embodiment of this application.

[0012] Figure 2 This is a flowchart of a method for generating preference data pairs in an embodiment of this application.

[0013] Figure 3 This is a schematic diagram of an ORPO mechanism in an embodiment of this application.

[0014] Figure 4 This is a structural block diagram of a model knowledge injection device according to an embodiment of this application.

[0015] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0016] The technical solution of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0017] In today's rapidly developing field of artificial intelligence, large language models have become a core driving force for technological progress. These models rely on deep learning technology to extract and learn complex patterns from massive amounts of data, demonstrating near-human-level capabilities in various tasks.

[0018] Nevertheless, large models still have significant limitations in understanding and processing domain-specific knowledge. To overcome this bottleneck, knowledge infusion techniques have been widely adopted, enhancing a model's cognitive and reasoning abilities in specialized scenarios by embedding expert experience and domain information within the model itself. This technique not only improves the model's performance in vertical tasks such as medical diagnosis, legal analysis, and financial forecasting, but also further expands its generalization capabilities to handle complex data patterns.

[0019] Currently, building a usable large-scale model that conforms to human preferences typically involves three core stages: pre-training, fine-tuning, and human preference alignment. In the pre-training stage, the model learns general knowledge and language representations using a large-scale corpus; in the fine-tuning stage, supervised training is conducted using instruction data to ensure the model follows user intent; and in the preference alignment stage, human feedback is used to optimize the output, making it aligned with ethical principles and value preferences.

[0020] In terms of knowledge infusion, the mainstream technical approaches currently include incremental pre-training and full parameter fine-tuning. Incremental pre-training enables the model to gradually absorb new knowledge while retaining the base knowledge, but it has significant drawbacks: on the one hand, it requires a large amount of training data, often in the gigabytes (GB) of text data; on the other hand, it places extremely high demands on computing resources, for example, training a 7B parameter model requires at least 8 graphics processing units (GPUs) and 640GB of video memory. In addition, to alleviate the problem of catastrophic forgetting, it is necessary to introduce general data mixing training, which not only further increases the data scale but also brings the burden of optimizing data ratios.

[0021] Full parameter fine-tuning is another common knowledge injection method, which typically constructs domain knowledge into question-answer pairs to fine-tune the model. Although its resource and data requirements are slightly lower than incremental pre-training, there is still a risk that the model will forget its original capabilities. It relies on a fine-grained general data allocation to maintain performance balance, and often requires an additional alignment process after training to make the model behavior conform to human preferences, which significantly increases training complexity and cost.

[0022] Therefore, existing knowledge injection methods generally face key challenges such as high dependence on data scale, large consumption of computing resources, complex training process, and difficulty in taking into account general professional capabilities. There is an urgent need for a more efficient and lightweight technical path to achieve accurate, controllable and low-cost domain knowledge fusion.

[0023] Based on the deficiencies of the aforementioned related technologies, this application provides a method, apparatus, and electronic device for model knowledge injection, aiming to balance model training cost and model performance during knowledge injection.

[0024] This application provides a method for injecting model knowledge. This method can be applied to electronic devices. This application does not limit the type of electronic device, which can be any suitable type of device, such as terminal devices and servers. This application will not elaborate further on this.

[0025] See Figure 1 The diagram shown is a flowchart of a model knowledge injection method according to an embodiment of this application. The following is a combination of... Figure 1 The method is described below, and the specific implementation process is as follows: Step 101: If the quality scores of two answers to the same question in a specified domain meet the following preference requirements: the quality score of the first answer is greater than the first set threshold, the quality score of the second answer is lower than the second set threshold, and the second set threshold is not greater than the first set threshold, then the two answers are regarded as the preference data pair corresponding to the question, and a preference dataset is constructed based on the preference data pairs corresponding to each question.

[0026] The quality score is obtained by evaluating the answers to questions based on knowledge of a specified domain. The specified domain may include, but is not limited to, at least one of the following: information and communication technology, finance and commerce, law and compliance, healthcare, industry and manufacturing, and energy and public utilities. The specified domain knowledge may be industry-specific expertise, including but not limited to: network device configuration commands in the information and communication technology field, legal provisions and case interpretations in the legal field, or clinical treatment guidelines in the healthcare field. The purpose of this application's embodiments is to efficiently inject the aforementioned domain knowledge into a pre-trained general-purpose language model, transforming it into a domain-specific model.

[0027] In practical applications, the first and second threshold values ​​can be set according to the actual application scenario, and no restrictions are imposed here.

[0028] Furthermore, preference data pairs from questions in general knowledge domains can be added to the preference dataset.

[0029] In one implementation, the number of questions in a specified domain contained in the preference dataset is obtained; the number of general domain questions is obtained based on the data ratio coefficient and the number of questions in the specified domain; and the questions with the number of general domain questions and their corresponding preference data pairs are selected from the original question-and-answer data.

[0030] The data ratio can be 1:10. In practical applications, the data ratio can be set according to the actual application scenario, and there is no restriction here.

[0031] In one implementation, raw question-and-answer data containing multiple questions and their corresponding answers is obtained; the raw question-and-answer data includes answers to multiple questions in a specified domain generated based on specified domain knowledge; questions belonging to the specified domain and having at least two answers are selected from the raw question-and-answer data; each answer to each selected question is evaluated according to the specified domain knowledge to obtain a quality score for each answer; for each question, if there is a first answer with a quality score greater than a first set threshold and a second answer with a quality score lower than a second set threshold, then a preference data pair corresponding to the question is generated based on the first and second answers to the question.

[0032] When constructing a preference dataset, a positive answer (i.e., a preferred answer) and a negative answer (i.e., a non-preferred answer) are needed. Therefore, the original question-and-answer data can be used to construct the preference dataset. The preference dataset can also be called Odds Ratio Preference Optimization (ORPO). ORPO is a large language model training algorithm that does not rely on additional reward models or complex reinforcement learning frameworks. It achieves efficient alignment between model output and human preferences by directly optimizing the odds ratio between preferred and non-preferred answers. Constructing preference dataset pairs requires two conditions: first, there must be two answers to the same question; second, these two answers must be discriminative, distinguishable by a quality score threshold.

[0033] Optionally, multiple questions can be input into a knowledge-answering model obtained through a pre-training phase or a third-party model used for knowledge question answering to obtain answers to each question. Alternatively, knowledge of a specific domain, such as documents related to a specific domain, can be obtained, and answers to questions in that specific domain can be searched based on that knowledge. Answers to each question can also be obtained through methods such as manual annotation.

[0034] The original question data contains a structured object for each question. This structured object includes the question, the answer, and some related information. This structured object can be a lightweight data interchange format (JavaScript Object Notation, JSON) object.

[0035] The following example uses a structured object corresponding to a question as an illustration: [ { "Serial Number": 0, "Original serial number": 1, First-level heading: "Security Policy" Second-level heading: "Security Policy Configuration Commands" "Third-level heading": "accelerate enhanced enable", Document content: "......" "File name": "02-Security Policy Command" "Document Content Length": 865 "Generate question-answer pairs_tags": 1, Question: What is the function of the `accelerate enhanced enable` command? Answer: "The 'accelerate enhanced enable' command is used to activate the acceleration feature of security policy rules..." Source file: "The function of command 01_processed_group_1_20241203_145041_processed.xlsx" "Extract Question Number": NaN, "Source file 2": "sorted_merged_group_1_20241203_195458.xlsx", "Duplicate marker": false, "id": 0, "response": "The 'accelerate enhanced enable' command is used to enable the hardware fast forwarding function..." “resp_eval”: “The Artificial Intelligence (AI) assistant’s answer in this example has several problems. ... Combining the scores of each dimension, the AI ​​assistant’s answer score is: 3 points.” } ] The question is: What is the function of the `accelerate enhanced enable` command? The first answer, as stated above, is: The `accelerate enhanced enable` command is used to activate the acceleration function of security policy rules... This is based on the `01 command_processed_group_1_20241203_145041_processed.xlsx` file. The second answer, or "response," is: The `accelerate enhanced enable` command is used to enable the hardware fast forwarding function... This is generated by an AI assistant, which is either a knowledge-answering model obtained during the pre-training phase or a third-party model used for knowledge question answering. The quality score of the first answer is 9, and the quality score of the second answer is 3. Both the first and second set thresholds are 7. Therefore, the first answer is set as a "chosen" (preferred answer), and the second answer is set as a "rejected" (unpreferred answer).

[0036] In one implementation, for each question, a standard answer to the question is retrieved from the specified domain knowledge, and the standard answer is used as the preferred answer. A quality score is generated based on the semantic distance between the other answers to the question and the preferred answer, and based on the semantic distance.

[0037] Optionally, preferred answers can be directly labeled by humans, generated through a domain-specific question-answering model, or retrieved from domain-specific knowledge.

[0038] In practical applications, the data format of the original problem data can be set according to the actual application scenario, and there are no restrictions here.

[0039] The following is combined Figure 2 To illustrate this, let's take generating preference data pairs corresponding to a question as an example. (See attached document.) Figure 2 The diagram shows a flowchart of a method for generating preference data pairs. The method includes the following steps: S201: Obtain raw question and answer data.

[0040] S202: Determine whether the target question in the original question-and-answer data contains two answers. If yes, execute S203; otherwise, execute S206.

[0041] S203: Based on knowledge of the specified domain, evaluate each answer to the target question and obtain a quality score for each answer.

[0042] S204: Determine whether there is a first answer with a quality score greater than the first set threshold and a second answer with a quality score lower than the second set threshold among the answers to the target question. If so, execute S205; otherwise, execute S206.

[0043] S205: Take the first answer as the preferred answer and the second answer as the non-preferred answer, and take the first answer and the second answer as the corresponding preference data pair for the target question.

[0044] S206: End process.

[0045] The preference dataset contains at least one structured object corresponding to a question; each structured object contains a preference field and a non-preference field. The value of the preference field is an ordered list containing a first data object and a second data object. The role field of the first data object is the questioner's role identifier, and the text content field of the first data object is the question. The role field of the second data object is the respondent's role identifier, and the text content field of the second data object is the first answer corresponding to the question. The value of the non-preference field is an ordered list containing a third data object and a fourth data object: the role field of the third data object is the questioner's role identifier, and the text content field of the third data object is the question itself; the role field of the fourth data object is the respondent's role identifier, and the text content field of the fourth data object is the second answer to the question.

[0046] The structured objects in the preference dataset can also be JSON objects. Optionally, the structured object for each question can also include a data source field to declare the source of the preference dataset.

[0047] Below is an example of the data format for a structured object of a question. `source` and `Airoboros` represent the data source field and its value, respectively. `role: user` indicates the role of the questioner. `role: assistant` indicates the role of the responder. `content` is the text content field. [ { "source": "Airoboros", “chosen”: [ { "content": "xxx", "role": "user" }, { "content": "xxx", “role”: “assistant” }, { "content": "xxx", "role": "user" } ] “rejected”: [ { "content": "xxx", "role": "user" }, { "content": "xxx", “role”: “assistant” } ] ] Below is a specific example; [ { "source": "security_qa", “chosen”: [ { "content": "What is the function of the 'accelerate enhanced enable' command?" "role": "user" }, { "content": "The 'accelerate enhanced enable' command is used to activate the acceleration function of security policy rules to improve the speed at which packets match security policy rules." “role”: “assistant” } ] “rejected”: [ { "content": "What is the function of the 'accelerate enhanced enable' command?" "role": "user" }, { "content": "The 'accelerate enhanced enable' command is used to enable the hardware fast forwarding function, which improves the forwarding speed of packets." “role”: “assistant” } ] ] In this way, a preference dataset can be constructed.

[0049] Step 102: In the alignment phase, the current knowledge answering model outputs the token prediction distribution for each question based on the input preference dataset.

[0050] In one implementation, during the pre-training stage, the model is first pre-trained to obtain an initial knowledge answering model. During the pre-training stage, the model learns general knowledge and language representations with the help of a large-scale corpus. Then, during the human preference alignment stage, the initial knowledge answering model is injected with specified domain knowledge based on the preference dataset.

[0051] The token prediction distribution refers to the set of all possible tokens and their corresponding probability values ​​output by the knowledge answering model at each position when generating an answer.

[0052] Step 103: Calculate the first preference probability based on the token prediction distribution of each question and the first answer corresponding to each question, calculate the second preference probability based on the token prediction distribution of each question and the second answer corresponding to each question, calculate the first loss based on the first preference probability and the second preference probability of each question, and calculate the second loss based on the first preference probability of each question.

[0053] The first loss is used to reduce the probability of the knowledge answering model generating a non-preferred answer in order to guide the knowledge answering model away from non-preferred answers, and the second answer belongs to the non-preferred answer; the second loss is used to make the output of the knowledge answering model converge towards the preferred answer, and the first answer belongs to the preferred answer.

[0054] When determining the probability of the first preference, the following steps can be performed for each question: For each token position in the first answer to the question, assuming that all tokens before that position are consistent with the corresponding tokens in the first answer to the question, the predicted probability value of the token at that position in the first answer is determined based on the token prediction distribution at that position; the first preference probability of the question is obtained based on the average of the logarithms of the predicted probability values ​​at each position.

[0055] When determining the probability of the second preference, the following steps can be performed for each question: For each problem, perform the following steps: For each token position in the second answer to the question, under the condition that all tokens before that position are consistent with the corresponding tokens in the second answer to the question, the predicted probability value of the token at that position in the second answer is obtained from the token prediction distribution at that position; The second preference probability for this problem is obtained by averaging the logarithms of the predicted probability values ​​for each location.

[0056] Optionally, the first preference probability and the second preference probability can be determined based on the average log-likelihood formula for input x and output y containing m tokens: Where x is the question, y is the answer, m is the number of tokens in y, and t is the token index. This represents the probability that the knowledge-based answering model will output y based on the input x, given the current parameter θ. When y is the first answer, Let y be the probability of the first preference, when y is the second answer. This represents the second preference probability.

[0057] When calculating the first loss based on the first preference probability and the second preference probability for each problem, the following steps can be taken for each problem: Calculate the first difference between 1 and the first preference probability of the problem, and the second difference between 1 and the second preference probability of the problem; calculate the first ratio between the first preference probability of the problem and the first difference, and the second ratio between the second preference probability of the problem and the second difference; based on the ratio between the first ratio and the second ratio, obtain the first loss corresponding to the problem.

[0058] Optionally, the following formula can be used to determine both the first ratio and the second ratio: ; Where y is the first answer, As the first ratio, when y is the second answer, This is the second ratio.

[0059] In this way, the ratio of the probability of an event (i.e., input x, output y) occurring to the probability of the event not occurring can be determined using this formula.

[0060] Optionally, the ratio between the first ratio and the second ratio The following formula can be used: ; in, As the first answer, This is the second answer. For the selected output and rejected output The output probability ratio.

[0061] Optionally, determine the first loss. When this is the case, the following formula can be used; ; When calculating the second loss based on the first preference probability for each problem, the following steps can be used: Calculate the average of the negative natural logarithms of the first preference probabilities for each question to obtain the second loss.

[0062] Optionally, calculate the second loss. When this is the case, the following formula can be used: ; Where n is the number of questions, k is the question number, V is the number of tokens in y, and i is the token number. This is the first preference probability corresponding to the k-th question.

[0063] See Figure 3 This is a schematic diagram of an ORPO mechanism. Figure 3 middle, This is the core of the ORPO algorithm. The algorithm's goal is to maximize this ratio. This means simultaneously increasing the numerator (i.e., increasing the probability of preferred responses) and decreasing the denominator (i.e., decreasing the probability of unpreferred responses), and then integrating them into a unified loss function, without the need for traditional reinforcement learning or additional reward models. Preferred responses (Chosen Responses) are high-quality responses selected by human annotators or rating models that conform to preferences. Unpreferred responses (Rejected Responses) are responses judged as low-quality and not conforming to preferences.

[0064] Strong Adaptation: ORPO provides strong rewards (positive incentives) for these types of answers, significantly increasing the likelihood (i.e., Odds) that the model generates such answers. The logarithm on the arrow indicates that this incentive process is achieved by calculating the log-odds ratio.

[0065] Weak Penalty: ORPO imposes only mild suppression on these types of responses. This means the model doesn't need to completely "forget" how to generate these responses, but simply reduces their probability of generation. This design helps maintain the model's creativity and generalization ability, avoiding over-optimization.

[0066] This allows for strong rewards for preferred answers and weak penalties for non-preferred answers. The first loss helps the model avoid incorrect responses, accelerating parameter updates and convergence. The second loss helps the model converge toward the correct answer (i.e., the preferred answer).

[0067] Step 104: If the first loss and the second loss satisfy the preset iteration stopping condition, then the current knowledge answering model is taken as the target knowledge answering model; otherwise, the parameters of the knowledge answering model are adjusted and iterative training continues.

[0068] The target knowledge answering model is used to receive user input of questions related to a specified domain and generate answers that conform to the specified domain knowledge and human preferences.

[0069] In one implementation, step 104 may be performed using the following steps: The first and second losses for each question are weighted and summed according to preset weight coefficients, and the total loss is obtained based on the weighted sum for each question. If the total loss meets the preset iteration stopping condition, the current knowledge answering model is used as the target knowledge answering model; otherwise, the parameters of the knowledge answering model are adjusted and iterative training continues.

[0070] The following formula can be used to calculate the total loss: ; in, The preset weighting coefficient can be set according to the actual application scenario; for example, it can be 0.5, and there is no restriction here.

[0071] In a specific experimental embodiment of this application, an optimization experiment was conducted on λ. The knowledge injection effect and general performance of the knowledge answering model were tested when λ was 0.2 and 0.5, respectively.

[0072] The experimental training data consists of two parts: Domain knowledge data: Contains 20 preference data pairs for a single knowledge point to be injected; General data: Approximately 2,000 preference data pairs from general domains, used to maintain the original capabilities of the model.

[0073] The experiment adopts the unified training method of the embodiments of this application, which mixes domain knowledge data with general data and performs preference alignment training in one go.

[0074] Experimental results show that when λ is 0.5, the model achieves 100% accuracy in injecting knowledge at this knowledge point, significantly outperforming the performance when λ=0.2. Furthermore, under this parameter, the model's performance on the general capability evaluation set not only does not decrease but also shows a slight improvement, demonstrating that the method in this embodiment can effectively maintain or even enhance the model's general capability while efficiently injecting knowledge.

[0075] This embodiment demonstrates that the training method provided in this application only requires a small number of domain samples mixed with general data for one training session, which can simultaneously achieve accurate knowledge injection and good capability retention, avoiding the complex process of step-by-step fine-tuning and alignment required by traditional methods.

[0076] In this embodiment, a novel preference alignment algorithm is introduced, which combines ratio-based penalty with traditional causal language modeling binding loss (e.g., cross-entropy loss). During preference optimization, the total loss function consists of a first loss, the Likelihood Odds Ratio Loss, and a second loss, the Supervised Fine-Tuning Loss. The first loss term effectively penalizes the model's assignment of high probabilities to rejected responses (i.e., non-preferred answers) by maximizing the ratio between the probabilities of popular responses (i.e., preferred answers) and unpopular responses (i.e., non-preferred answers).

[0077] The total loss function combines the supervised fine-tuning loss and the preference advantage ratio loss into a unified loss term. This method is suitable for application scenarios where a pre-trained language model needs to be adapted to a specific domain or task, and where the model's output must conform to human preferences. A prerequisite is the availability of paired preference datasets, containing preferred responses (i.e., preferred answers) and rejected responses (i.e., non-preferred answers), such as publicly available datasets like the UltraFeedback dataset or the Human-Human Preference Alignment Dataset (HH-RLHF).

[0078] Therefore, the method of this application embodiment does not require the introduction of additional reference models, reward models or the execution of multi-stage fine-tuning processes, and can thus serve as a more efficient and concise alternative to methods such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO).

[0079] Specifically, the core of a large language model is to simulate and generate content that conforms to the rules of language. Its performance is measured by the probability that the model assigns to the correct answer: the higher the probability that the model assigns to the correct label, the better its performance.

[0080] In the pre-training and fine-tuning phases, the training objective is to enable the model to accurately predict the next word in the sequence, i.e., to optimize the model parameters by maximizing the probability of the true label. However, in the human preference alignment phase, the training mechanism undergoes a significant shift: the model no longer learns only a single correct answer, but instead needs to select the answer that better aligns with human preferences from a pair of responses to the same prompt. This process relies on a preference dataset containing "preferred" and "second-best" responses (i.e., preferred and non-preferred answers). The model learns to align with human values ​​by choosing the preferred response and receiving a reward, and avoiding the second-best response and avoiding punishment. Therefore, unlike the goal of maximizing likelihood in the first two phases, alignment is essentially a comparison-based reward maximization process.

[0081] To achieve efficient knowledge injection, this application's embodiments directly use the alignment process as the primary pathway for knowledge learning. Specifically, when constructing preference data, preferred answers are set as the correct answers to be injected, conforming to domain knowledge, while non-preferred answers are set as responses generated by the original model or other unsuitable answers. Supervised fine-tuning loss and preference alignment loss are organically combined, simultaneously performing knowledge learning and preference optimization within a unified framework. This method effectively accelerates the convergence speed of domain knowledge, achieving accurate knowledge injection and capability preservation for large-scale language models with extremely low data preparation costs and computational resource consumption.

[0082] The following section, with reference to Table 1, illustrates the effectiveness of the aligned knowledge-answering model in the embodiments of this application. The following dataset can be used for evaluation: The Chinese Domain Knowledge Evaluation Dataset (A Multi-Level Multi-Discipline ChineseEvaluation Suite for Foundation Models, C-Eval) refers to a comprehensive knowledge evaluation benchmark built for Chinese scenarios, encompassing general and professional domains. It covers multiple knowledge categories, including humanities and social sciences, natural sciences, and engineering technology. It includes multiple-choice questions for knowledge comprehension and application tests, and is used to quantitatively evaluate the model's mastery of basic theories, professional concepts, and knowledge applications in the Chinese context.

[0083] The Chinese Massive Multi-task Language Understanding (CMMLU) dataset is a multi-task benchmark that focuses on Chinese language understanding capabilities. It covers task types such as language reasoning, text analysis, and professional knowledge question answering. It includes test samples for Chinese semantic understanding, logical judgment, and domain knowledge application, and is used to evaluate the model's multi-task processing capabilities and cross-domain knowledge transfer effects in Chinese scenarios.

[0084] The Massive Multitask Language Understanding (MMLU) dataset is a benchmark for evaluating multitask language understanding in English-speaking scenarios. It covers multiple knowledge domains, including natural sciences, humanities, business finance, and technology engineering. It includes test samples for English text understanding, professional knowledge question answering, and logical reasoning. It is used to quantitatively evaluate the model's ability to understand cross-domain knowledge, analyze language logic, and answer professional questions in English contexts.

[0085] Table 1 below shows the evaluation results, indicating that the model performance is comprehensively improved after training using the method of this application embodiment. Significant knowledge injection effect: On the test set of generalized knowledge points for a specified domain, the model accuracy reaches 100%, effectively verifying the feasibility of knowledge injection through ORPO alignment. Increased general capabilities without loss: Through training with mixed general data, the model's performance in multiple general benchmark evaluations is improved by 2% to 5%, indicating that the method of this application embodiment, while injecting professional knowledge, not only does not damage the original capabilities but also enhances the model's generalization ability.

[0086] Table 1 Under relevant technologies, training a usable large language model that conforms to human preferences typically involves three independent stages: pre-training, fine-tuning, and human preference alignment. However, this staged approach has the following drawbacks: First, to achieve domain knowledge injection, incremental pre-training or full-parameter fine-tuning is usually required. The former requires a large amount of domain corpus and high computing power, while the latter is prone to catastrophic forgetting. Both also require additional preference alignment, making the process complex and costly.

[0087] In order to solve the above problems, this application provides an efficient and low-cost domain knowledge injection method. Its core is that knowledge injection and preference alignment are completed simultaneously during the alignment stage, avoiding incremental pre-training and additional fine-tuning, significantly reducing data requirements and computing resource consumption, and effectively suppressing the degradation of the model's general capabilities.

[0088] Based on the same inventive concept, this application also provides a model knowledge injection device. Since the principle of the above-mentioned device and apparatus for solving the problem is similar to that of a model knowledge injection method, the implementation of the above-mentioned device can refer to the implementation of the method, and repeated details will not be elaborated further. This device can be applied to electronic devices. This application does not limit the type of electronic device; it can be any suitable type of device, such as terminal devices and servers, etc., which will not be elaborated further in this application. The device embodiment can be implemented by software, or by hardware, or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by the processor of the electronic device reading the corresponding computer program instructions from the non-volatile memory into memory and running them.

[0089] See Figure 4 The diagram shown is a structural block diagram of a model knowledge injection apparatus according to an embodiment of this application. In some embodiments, the model knowledge injection apparatus exemplified in this application includes: The construction unit 401 is used to construct a preference dataset based on the preference data pairs corresponding to the question if the quality scores of two answers to the same question in a specified domain meet the following preference requirements: the quality score of the first answer is greater than a first set threshold, the quality score of the second answer is lower than a second set threshold, and the second set threshold is not greater than the first set threshold. Output unit 402 is used to output the predicted distribution of the token for each question based on the input preference dataset by the current knowledge answering model during the alignment phase; The calculation unit 403 is used to calculate a first preference probability based on the token prediction distribution of each question and the first answer corresponding to each question, and to calculate a second preference probability based on the token prediction distribution of each question and the second answer corresponding to each question, and to calculate a first loss based on the first preference probability and the second preference probability of each question, and to calculate a second loss based on the first preference probability of each question; the first loss is used to reduce the probability that the knowledge answering model generates a non-preference answer so as to guide the knowledge answering model away from non-preference answers, and the second answer belongs to non-preference answers; the second loss is used to make the output of the knowledge answering model converge to a preference answer, and the first answer belongs to a preference answer; The adjustment unit 404 is used to take the current knowledge answering model as the target knowledge answering model if the first loss and the second loss meet the preset iteration stopping condition; otherwise, it adjusts the parameters of the knowledge answering model and continues iterative training. The target knowledge answering model is used to receive user input of questions related to a specified domain and generate answers that conform to the specified domain knowledge and human preferences.

[0090] In one embodiment, the computing unit 403 is used for: For each problem, perform the following steps: For each token position in the first answer to the question, under the condition that all tokens before that position are consistent with the corresponding tokens in the first answer to the question, the predicted probability value of the token at that position in the first answer is determined based on the token prediction distribution at that position. The first preference probability for this problem is obtained by averaging the logarithms of the predicted probability values ​​for each location.

[0091] In one embodiment, the computing unit 403 is used for: For each problem, perform the following steps: For each token position in the second answer to the question, under the condition that all tokens before that position are consistent with the corresponding tokens in the second answer to the question, the predicted probability value of the token at that position in the second answer is obtained from the token prediction distribution at that position; The second preference probability for this problem is obtained by averaging the logarithms of the predicted probability values ​​for each location.

[0092] In one embodiment, the computing unit 403 is used for: For each problem, the following steps are provided: Calculate the first difference between 1 and the first preference probability of the question, and the second difference between 1 and the second preference probability of the question; Calculate the first ratio between the first preference probability and the first difference for this problem, and the second ratio between the second preference probability and the second difference for this problem; Based on the ratio between the first ratio and the second ratio, the first loss corresponding to the problem is obtained.

[0093] In one embodiment, the computing unit 403 is used for: Calculate the average of the negative natural logarithms of the first preference probabilities for each question to obtain the second loss.

[0094] In one embodiment, the adjustment unit 404 is used to: The first and second losses for each problem are weighted and summed according to preset weight coefficients, and the total loss is obtained based on the weighted sum for each problem. If the total loss meets the preset iteration stopping condition, the current knowledge answering model is used as the target knowledge answering model; otherwise, the parameters of the knowledge answering model are adjusted, and iterative training continues.

[0095] In one implementation, the preference dataset contains a structured object corresponding to at least one question; each structured object contains preference fields and non-preference fields. The value of the preference field is an ordered list containing a first data object and a second data object. The role field value of the first data object is the questioner role identifier, the text content field value of the first data object is the question, the role field value of the second data object is the responder role identifier, and the text content field value of the second data object is the first answer corresponding to the question. The value of the non-preference field is an ordered list containing a third data object and a fourth data object: the role field of the third data object is the questioner's role identifier, and the text content field of the third data object is the question itself; the role field of the fourth data object is the respondent's role identifier, and the text content field of the fourth data object is the second answer to the question.

[0096] In one embodiment, the building unit 401 is used for: Obtain raw question-and-answer data containing multiple questions and their corresponding answers; the raw question-and-answer data includes answers to multiple questions in a specified domain, generated based on knowledge of that domain. From the raw question-and-answer data, filter out questions that belong to the specified domain and have at least two answers; Based on knowledge of the specified domain, each answer to each selected question is evaluated to obtain a quality score for each answer; For each question, if there is a first answer with a quality score greater than a first set threshold and a second answer with a quality score lower than a second set threshold, then a preference data pair corresponding to the question is generated based on the first and second answers.

[0097] In one embodiment, the construction unit 401 is further configured to: obtain the number of specified domain questions in the specified domain contained in the preference dataset; Based on the data ratio coefficient and the number of questions in the specified domain, obtain the number of questions in the general domain; From the raw question-and-answer data, we filter out the questions with the highest number of general domain questions and their corresponding preference data pairs.

[0098] The model knowledge injection method in this application includes: if the quality scores of two answers to the same question in a specified domain meet the following preference requirements: the quality score of the first answer is greater than a first set threshold, and the quality score of the second answer is lower than a second set threshold, and the second set threshold is not greater than the first set threshold, then the two answers are regarded as a preference data pair corresponding to the question, and a preference dataset is constructed based on the preference data pairs corresponding to each question; the quality score is obtained after evaluating the answers to the question based on the specified domain knowledge; in the alignment stage, the current knowledge answering model outputs the token prediction distribution of each question based on the input preference dataset; the first preference probability is calculated based on the token prediction distribution of each question and the first answer corresponding to each question, and the first preference probability is calculated based on the token prediction distribution of each question. The algorithm calculates a second preference probability for each question's second answer, and a first loss based on the first and second preference probabilities for each question, and a second loss based on the first preference probability for each question. The first loss reduces the probability of the knowledge answering model generating a non-preference answer, guiding it away from non-preference answers; the second answer is considered a non-preference answer. The second loss makes the output of the knowledge answering model converge towards a preference answer; the first answer is considered a preference answer. If the first and second losses satisfy a preset iteration stopping condition, the current knowledge answering model is used as the target knowledge answering model; otherwise, the parameters of the knowledge answering model are adjusted, and iterative training continues. The target knowledge answering model receives user input on questions related to a specified domain and generates answers that conform to the specified domain knowledge and human preferences. This approach balances the training cost of model alignment training with model performance.

[0099] In this embodiment of the application, an electronic device is also provided, including: Processor; and The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.

[0100] In this application embodiment, a computer-readable storage medium is provided, storing computer instructions for causing a computer to perform the methods of any of the above embodiments.

[0101] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the method of any of the above-described embodiments.

[0102] Figure 5 A schematic diagram of the structure of an electronic device 5000 is shown. (See also...) Figure 5As shown, the electronic device 5000 includes a processor 5010 and a memory 5020, and optionally may also include a power supply 5030, a display unit 5040, and an input unit 5050.

[0103] The processor 5010 is the control center of the electronic device 5000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 5000 by running or executing software programs and / or data stored in the memory 5020, thereby performing overall monitoring of the electronic device 5000.

[0104] In this embodiment, when the processor 5010 calls the computer program stored in the memory 5020, it executes the steps in the above embodiments.

[0105] Optionally, the processor 5010 may include one or more processing units; preferably, the processor 5010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 5010.

[0106] The memory 5020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 5000, etc. In addition, the memory 5020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0107] Electronic device 5000 also includes a power supply 5030 (such as a battery) that supplies power to various components. The power supply can be logically connected to processor 5010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0108] The display unit 5040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 5000. In this embodiment, it is mainly used to display the display interfaces of various applications in the electronic device 5000, as well as text, images, and other objects displayed on the display interfaces. The display unit 5040 may include a display panel 5041. The display panel 5041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0109] The input unit 5050 can be used to receive information such as numbers or characters input by the user. The input unit 5050 may include a touch panel 5051 and other input devices 5052. The touch panel 5051, also known as a touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 5051).

[0110] Specifically, the touch panel 5051 can detect user touch operations and the signals generated by these operations, convert them into touch point coordinates, send them to the processor 5010, and receive and execute commands from the processor 5010. Furthermore, the touch panel 5051 can be implemented using various types of touch technologies, including resistive, capacitive, infrared, and surface acoustic wave. Other input devices 5052 can include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0111] Of course, the touch panel 5051 can cover the display panel 5041. When the touch panel 5051 detects a touch operation on or near it, it transmits the information to the processor 5010 to determine the type of touch event. Subsequently, the processor 5010 provides corresponding visual output on the display panel 5041 according to the type of touch event. Although in Figure 5 In this embodiment, the touch panel 5051 and the display panel 5041 are two separate components to realize the input and output functions of the electronic device 5000. However, in some embodiments, the touch panel 5051 and the display panel 5041 can be integrated to realize the input and output functions of the electronic device 5000.

[0112] The electronic device 5000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity sensor, etc. Of course, depending on the specific application, the electronic device 5000 may also include other components such as a camera. Since these components are not the focus of this application's embodiments, therefore... Figure 5 It is not shown in the text and will not be described in detail here.

[0113] Those skilled in the art will understand that Figure 5 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0114] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

Claims

1. A method of model knowledge injection, the method comprising: The method comprises: If the quality scores of two answers to the same question in the specified field meet the following preference requirements: the quality score of the first answer is greater than a first set threshold, and the quality score of the second answer is lower than a second set threshold, the second set threshold is not greater than the first set threshold, the two answers are taken as a preference data pair corresponding to the question, and a preference data set is constructed based on the preference data pair corresponding to each question; the quality score is obtained based on the evaluation of the answer to the question by the knowledge in the specified field; In the alignment stage, the token prediction distribution of each question is output by the current knowledge answer model based on the input of the preference data set; The first preference probability is calculated based on the token prediction distribution of each question and the first answer corresponding to each question, the second preference probability is calculated based on the token prediction distribution of each question and the second answer corresponding to each question, the first loss is calculated based on the first preference probability and the second preference probability of each question, and the second loss is calculated based on the first preference probability of each question; the first loss is used to reduce the probability of the knowledge answer model generating a non-preferred answer to guide the knowledge answer model away from the non-preferred answer, and the second answer belongs to the non-preferred answer; the second loss is used to make the output of the knowledge answer model converge to the preferred answer, and the first answer belongs to the preferred answer; If the first loss and the second loss meet the preset iteration stopping condition, the current knowledge answer model is taken as a target knowledge answer model, otherwise, the parameters of the knowledge answer model are adjusted, and the iteration training is continued; the target knowledge answer model is used to receive a question related to a specified field input by a user, and generate an answer conforming to the knowledge in the specified field and conforming to human preferences.

2. The method of claim 1, wherein, The first preference probability is calculated based on the token prediction distribution of each question and the first answer corresponding to each question, comprising: The following steps are performed for each question respectively: For each token position in the first answer corresponding to the question, if all tokens before the position are consistent with the corresponding token in the first answer to the question, the prediction probability value of the token at the position in the first answer is determined based on the token prediction distribution at the position. The first preference probability of the question is obtained based on the average value of the logarithm of the prediction probability value of each position.

3. The method of claim 1, wherein, The second preference probability is calculated based on the token prediction distribution of each question and the second answer corresponding to each question, comprising: The following steps are performed for each question respectively: For each token position in the second answer corresponding to the question, if all tokens before the position are consistent with the corresponding token in the second answer to the question, the prediction probability value of the token at the position in the second answer is obtained from the token prediction distribution at the position. The second preference probability of the question is obtained based on the average value of the logarithm of the prediction probability value of each position.

4. The method according to any one of claims 1 to 3, characterized in that, The first loss is calculated based on the first preference probability and the second preference probability of each question, comprising: The following steps are performed for each question respectively: calculate a first difference between 1 and a first preference probability of the question, and a second difference between 1 and a second preference probability of the question; calculate a first ratio between the first preference probability of the question and the first difference, and a second ratio between the second preference probability of the question and the second difference; obtain a first loss corresponding to the question based on a ratio between the first ratio and the second ratio.

5. The method of claim 4, wherein, The second loss is calculated based on the first preference probability of each question, including: calculate an average of negative natural logarithms of the first preference probability of each question to obtain the second loss.

6. The method according to any one of claims 1 to 3, characterized in that, If the first loss and the second loss satisfy a preset iteration stopping condition, the current knowledge answering model is taken as a target knowledge answering model, otherwise, the parameters of the knowledge answering model are adjusted for further iteration training, including: weight and sum the first loss and the second loss of each question according to a preset weight coefficient respectively, and obtain a total loss based on the weighted sum corresponding to each question; If the total loss satisfies a preset iteration stopping condition, the current knowledge answering model is taken as a target knowledge answering model, otherwise, the parameters of the knowledge answering model are adjusted for further iteration training.

7. The method according to any one of claims 1 to 3, characterized in that, The preference data set includes at least one structured object corresponding to a question; each structured object includes a preference field and a non-preference field; The value of the preference field is an ordered list including a first data object and a second data object, the role field value of the first data object is a questioner role identifier, the text content field value of the first data object is a question, the role field value of the second data object is a responder role identifier, and the text content field value of the second data object is a first answer corresponding to the question; The value of the non-preference field is an ordered list including a third data object and a fourth data object: the role field value of the third data object is a questioner role identifier, the text content field value of the third data object is the question, the role field value of the fourth data object is a responder role identifier, and the text content field value of the fourth data object is a second answer corresponding to the question.

8. The method according to any one of claims 1 to 3, characterized in that, If the quality scores of two answers of the same question in a specified field satisfy the following preference requirements: the quality score of the first answer is greater than a first set threshold, and the quality score of the second answer is lower than a second set threshold, and the second set threshold is not greater than the first set threshold, the two answers are taken as a preference data pair corresponding to the question, and a preference data set is constructed based on the preference data pairs corresponding to each question, including: obtain original question and answer data including a plurality of questions and corresponding answers; the original question and answer data include answers of a plurality of questions in a specified field generated based on knowledge of the specified field; From the original question and answer data, questions belonging to the specified field and having at least two answers are screened out; According to the knowledge of the specified field, each answer of each question screened out is evaluated to obtain a quality score of each answer; For each question, if there is a first answer with a quality score greater than a first set threshold and a second answer with a quality score lower than a second set threshold, a preference data pair corresponding to the question is generated based on the first answer and the second answer of the question.

9. The method of claim 8, wherein, The method further comprises: obtaining a number of questions in the specified field included in the preference data set; obtaining a number of questions in the general field according to the data proportion coefficient and the number of questions in the specified field; filtering out questions in the number of questions in the general field and their corresponding preference data pairs from the original question and answer data.

10. An apparatus for model knowledge injection, the apparatus comprising: The device comprises: a construction unit configured to, if the quality scores of two answers to a same question in a specified field satisfy the following preference requirement: the quality score of a first answer is greater than a first set threshold, and the quality score of a second answer is lower than a second set threshold, which is not greater than the first set threshold, then the two answers are taken as a preference data pair corresponding to the question, and a preference data set is constructed based on the preference data pair corresponding to each question; the quality score is obtained based on the evaluation of the answer to the question by the specified field knowledge; an output unit configured to, in the alignment stage, output a token prediction distribution of each question by the current knowledge answer model based on the input preference data set; a calculation unit configured to calculate a first preference probability based on the token prediction distribution of each question and the first answer corresponding to each question, calculate a second preference probability based on the token prediction distribution of each question and the second answer corresponding to each question, calculate a first loss based on the first preference probability and the second preference probability of each question, and calculate a second loss based on the first preference probability of each question; the first loss is used to reduce the probability of the knowledge answer model generating a non-preference answer to guide the knowledge answer model away from the non-preference answer, and the second answer belongs to the non-preference answer; the second loss is used to make the output of the knowledge answer model converge to a preference answer, and the first answer belongs to the preference answer; an adjustment unit configured to, if the first loss and the second loss satisfy a preset iteration stop condition, take the current knowledge answer model as a target knowledge answer model, otherwise, adjust the parameters of the knowledge answer model and continue iteration training; the target knowledge answer model is used to receive a question related to a specified field input by a user and generate an answer conforming to the knowledge of the specified field and conforming to human preference.

11. The apparatus of claim 10, wherein, The calculation unit is configured to: for each question, perform the following steps: for each token position in the first answer corresponding to the question, if all tokens before the position are consistent with the corresponding token in the first answer of the question, determine a prediction probability value of the token at the position based on the token prediction distribution at the position. obtain the first preference probability of the question based on the average of the logarithms of the prediction probability values of each position.

12. The apparatus of claim 10, wherein, The calculation unit is configured to: for each question, perform the following steps: For each token position in the second answer corresponding to the question, a predicted probability value of a token at the position in the second answer is obtained from a token prediction distribution at the position, under a condition that all tokens before the position are consistent with the corresponding token in the second answer of the question; A second preference probability of the question is obtained based on an average value of logarithms of the predicted probability values of the positions.

13. The apparatus of any of claims 10-12, wherein, The computing unit is configured to: For each question, the following steps are performed: Calculate a first difference value between 1 and the first preference probability of the question, and a second difference value between 1 and the second preference probability of the question; Calculate a first ratio between the first preference probability of the question and the first difference value, and a second ratio between the second preference probability of the question and the second difference value; Obtain a first loss corresponding to the question based on a ratio between the first ratio and the second ratio.

14. The apparatus of claim 13, wherein, The computing unit is configured to: Calculate an average value of negative natural logarithms of the first preference probabilities of the questions to obtain the second loss.

15. The apparatus of any one of claims 10-12, wherein, The adjusting unit is configured to: Summarize the first loss and the second loss of each question by weighting according to a preset weight coefficient, and obtain a total loss based on the weighted summations corresponding to the questions; If the total loss meets a preset iteration stopping condition, the current knowledge answering model is taken as a target knowledge answering model, otherwise, the parameters of the knowledge answering model are adjusted for further iteration training.

16. The apparatus of any one of claims 10-12, wherein, The preference data set includes at least one structured object corresponding to a question; each structured object includes a preference field and a non-preference field; The value of the preference field is an ordered list including a first data object and a second data object, the role field value of the first data object is a questioner role identifier, the text content field value of the first data object is a question, the role field value of the second data object is a responder role identifier, and the text content field value of the second data object is a first answer corresponding to the question; The value of the non-preference field is an ordered list including a third data object and a fourth data object: the role field value of the third data object is a questioner role identifier, the text content field value of the third data object is the question, the role field value of the fourth data object is a responder role identifier, and the text content field value of the fourth data object is a second answer corresponding to the question.

17. The apparatus of any of claims 10-12, wherein, The constructing unit is configured to: Obtain original question and answer data including a plurality of questions and corresponding answers; the answers of the questions in the specified field are generated based on specified domain knowledge; From the original question and answer data, questions belonging to the specified field and having at least two answers are screened out; According to the specified domain knowledge, each answer to each question is evaluated to obtain a quality score of each answer; For each question, if there is a first answer with a quality score greater than a first set threshold and a second answer with a quality score lower than a second set threshold in the answers of the question, a preference data pair corresponding to the question is generated based on the first answer and the second answer of the question.

18. The apparatus of claim 17, wherein, The construction unit is further configured to: acquire a specified field question quantity of questions in the specified field included in the preference data set; obtain a general field question quantity according to a data proportionality coefficient and the specified field question quantity; screen out questions in the general field question quantity and corresponding preference data pairs from the original question and answer data.

19. An electronic device, comprising: Comprise: a processor; and a memory storing computer instructions for causing the processor to execute the method of any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer instructions are stored in the memory and are used for causing the computer to execute the method of any one of claims 1 to 9.

21. A computer program product, characterised in that, The computer readable code or the non-volatile computer readable storage medium carrying the computer readable code, when the computer readable code runs in the processor of the electronic device, the processor in the electronic device executes the method of any one of claims 1 to 9.

Images