Large language model training method and device and question and answer processing method and device

Abstract

The embodiment of the invention provides a large language model training method and device and a question and answer processing method and device, and the method comprises the steps: obtaining a plurality of samples, enabling each sample to comprise a sample question, a first answer corresponding to the sample question, and a second answer corresponding to the sample question, and according to the similarity between the lexical element of each answer in each sample and the lexical element of another answer, obtaining the lexical element of each answer; determining the weight of lexical elements in each answer, inputting the sample question into a large language model for each sample to obtain first probability distribution and second probability distribution, weighting the probability of each first lexical element in the first probability distribution according to the weight of each lexical element of the first answer, and weighting the probability of each second lexical element in the second probability distribution according to the weight of each lexical element of the second answer; and weighting the probability of each second lexical element in the second probability distribution, and adjusting model parameters in the large language model according to the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample. Based on the method, the quality of model output answers is effectively improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology and may relate to fields such as artificial intelligence and natural language processing. Specifically, this application relates to a training method for a large language model, a question-answering processing method, and an apparatus. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and natural language processing technologies, large language models (LLMs) have made significant progress in various natural language generation tasks such as machine translation, semantic parsing, text summarization, and intelligent question answering.

[0003] To ensure that the text generated by LLM can fully satisfy human language and text preferences, related technologies employ a preference alignment method based on supervised fine-tuning. This method involves manually constructing a high-quality, large-scale instruction-following dataset and directly performing supervised parameter fine-tuning on the pre-trained language model. This enables the model to better perform natural language tasks and generate answers that meet user expectations.

[0004] However, large language models trained with supervised fine-tuning lack the ability to perceive human preference data, and there is still a significant deviation between the model output and the actual human preferences. Summary of the Invention

[0005] The purpose of this application is to provide a training method, question-answering method, and apparatus for a large language model that can effectively improve the model's ability to understand and generate preferences, and promote the model's output to be deeply aligned with human expectations. To achieve this purpose, the technical solutions provided by this application are as follows: On the one hand, embodiments of this application provide a method for training a large language model, the method comprising: Multiple samples are obtained, each sample including a sample question, a first answer and a second answer corresponding to the sample question, wherein the answer quality of the first answer is higher than that of the second answer; For each sample, the weight of each word element is determined based on the similarity between each word element in each answer of that sample and each word element in another answer; wherein the weight of each word element represents the importance of that word element in its respective answer; Based on the multiple samples, the large language model to be trained is continuously trained to obtain the trained large language model: For each sample, the sample question in the sample is input into a large language model to obtain a first prediction result corresponding to the sample question; wherein, the first prediction result includes a first probability distribution and a second probability distribution, the first probability distribution includes each first word element generated by the large language model and the probability of each first word element, the second probability distribution includes each second word element generated by the large language model and the probability of each second word element, each first word element corresponds to each word element of the first answer in the sample, and each second word element corresponds to each word element of the second answer in the sample; For each sample, the probabilities of each first word in the first probability distribution are weighted according to the weight of each word in the first answer in the sample to obtain a weighted first probability distribution; the probabilities of each second word in the second probability distribution are weighted according to the weight of each word in the second answer in the sample to obtain a weighted second probability distribution. The training loss is determined based on the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample, and the model parameters in the large language model are adjusted based on the training loss.

[0006] On one hand, embodiments of this application provide a training apparatus for a large language model, the apparatus comprising: The sample acquisition module is used to acquire multiple samples, each sample including a sample question, a first answer and a second answer corresponding to the sample question, wherein the answer quality of the first answer is higher than that of the second answer. The word weight determination module is used to determine the weight of each word for each sample based on the similarity between each word in each answer of the sample and each word in another answer; wherein the weight of each word represents the importance of the word in its corresponding answer; The training module is used to continuously perform training operations on the large language model to be trained based on the multiple samples, so as to obtain the trained large language model: For each sample, the sample question in the sample is input into a large language model to obtain a first prediction result corresponding to the sample question; wherein, the first prediction result includes a first probability distribution and a second probability distribution, the first probability distribution includes each first word element generated by the large language model and the probability of each first word element, the second probability distribution includes each second word element generated by the large language model and the probability of each second word element, each first word element corresponds to each word element of the first answer in the sample, and each second word element corresponds to each word element of the second answer in the sample; For each sample, the probabilities of each first word in the first probability distribution are weighted according to the weight of each word in the first answer in the sample to obtain a weighted first probability distribution; the probabilities of each second word in the second probability distribution are weighted according to the weight of each word in the second answer in the sample to obtain a weighted second probability distribution. The training loss is determined based on the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample, and the model parameters in the large language model are adjusted based on the training loss.

[0007] Optionally, the lexical weight determination module can be used to: The trained feature extraction model is used to segment each answer in the sample into words and encode them at the word level to obtain the feature vector of each word in each answer of the sample. For each word element in each answer of the sample, the weight of the word element is determined based on the similarity between the feature vector of the word element and the feature vectors of each word element in another answer. The training module can be used for: For each first word element in the first probability distribution, from the word elements obtained by segmenting the first answer in the sample through the feature extraction model, determine each first target word element that belongs to the same word segment as the first word element. Based on the weights of the determined first target word elements, determine the weight of the first word element. Based on the weights of each first word element, weight the probabilities of each first word element to obtain the weighted first probability distribution. For each second word element in the second probability distribution, from the word elements obtained by segmenting the second answer in the sample through the feature extraction model, determine each second target word element that belongs to the same word segment as the second word element. Based on the weights of the determined second target word elements, determine the weight of the second word element. Based on the weights of each second word element, weight the probabilities of each second word element to obtain the weighted second probability distribution.

[0008] Optionally, the lexical weight determination module can be used to: For each word element in each answer of the sample, determine the similarity between that word element and each word element in another answer; For each word element in each answer of the sample, the weight of the word element is determined based on the maximum similarity among the similarity values ​​corresponding to that word element, wherein the maximum similarity value is negatively correlated with the weight.

[0009] Optionally, the lexical weight determination module can be used to: The type of a word is determined based on the maximum similarity among all similarities corresponding to that word. If the maximum similarity of the word is greater than the first threshold, the type of the word is determined to be a non-keyword; otherwise, the type of the word is a keyword. If the word element is a keyword, its weight is determined as the first weight; if the word element is not a keyword, its weight is determined as the second weight, wherein the first weight is greater than the second weight.

[0010] Optionally, the training module can also be used for: For each sample, the sample question in the sample is input into the reference large language model to obtain the second prediction result corresponding to the sample question; wherein, the second prediction result includes a third probability distribution and a fourth probability distribution, the third probability distribution includes each first word element generated by the reference large language model and the probability of each first word element, and the fourth probability distribution includes each second word element generated by the reference large language model and the probability of each second word element; For each sample, the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to that sample is determined by the following method: Based on the weights of each word element in the first answer in the sample, the probabilities of each first word element in the third probability distribution are weighted to obtain the weighted third probability distribution. Based on the weights of each word element in the second answer in the sample, the probabilities of each second word element in the fourth probability distribution are weighted to obtain the weighted fourth probability distribution. Determine the first distribution difference between the weighted first probability distribution and the weighted third probability distribution corresponding to the sample, and the second distribution difference between the weighted second probability distribution and the weighted fourth probability distribution corresponding to the sample; The difference between the first distribution difference and the second distribution difference is taken as the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to the sample.

[0011] Optionally, the first threshold is either the second threshold corresponding to the first answer or the third threshold corresponding to the second answer; The terminology weight determination module can be used for: If a word element belongs to the first answer of the sample, and the maximum similarity corresponding to the word element is greater than the second threshold, the type of the word element is determined to be a non-keyword; otherwise, the type of the word element is a keyword. If a word belongs to the second answer of the sample, and the maximum similarity of the word is greater than the third threshold, the word is determined to be a non-keyword; otherwise, the word is a keyword.

[0012] Optionally, each of the samples corresponds to its own second threshold and third threshold; For each of the samples, the second threshold corresponding to the first answer in that sample is determined in the following way: For each word element in the first answer of the sample, determine the maximum similarity corresponding to that word element; The second threshold corresponding to the first answer is determined based on the sum of the maximum similarity of each word in the first answer and the number of words in the first answer; For each of the samples, the third threshold corresponding to the second answer in that sample is determined in the following way: For each word element in the second answer of the sample, determine the maximum similarity corresponding to that word element; The third threshold corresponding to the second answer is determined based on the sum of the maximum similarity of each word in the second answer and the number of words in the second answer.

[0013] Optionally, the lexical weight determination module can be used to perform any of the following: If the word element is a keyword, its weight is determined to be the first preset weight. If the word element is not a keyword, and the maximum similarity corresponding to the word element is not greater than 0.5, the maximum similarity corresponding to the word element is used as the weight of the word element. Otherwise, the difference between the first preset weight and the maximum similarity is used as the weight of the word element. Wherein, the first preset weight is greater than 0.5. If the word is not a keyword, the weight of the word is determined to be the second preset weight; if the word is a keyword, the difference between the first threshold and the maximum similarity corresponding to the word is determined, and the sum of the second preset weight and the difference is used as the weight of the word.

[0014] On the other hand, embodiments of this application provide a question-and-answer processing method, the method comprising: Get the questions that are pending; The question is input into the trained large language model to obtain the answer to the question. The large language model is trained using the training method of the large language model provided in any embodiment of this application.

[0015] On the other hand, embodiments of this application provide a question-and-answer processing apparatus, the apparatus comprising: The question retrieval module is used to retrieve questions that need to be answered. The answer generation module is used to input the question into the trained large language model to obtain the answer to the question. The large language model is trained using the training method of the large language model provided in any embodiment of this application.

[0016] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of this application.

[0017] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.

[0018] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any optional embodiment of this application.

[0019] The beneficial effects of the technical solution provided in this application are as follows: The large language model training method provided in this application determines the weight of each word in each answer based on the similarity between the word units of each answer and the word units of another answer. Based on the weights of each word unit in the first answer, the probabilities of each first word unit in the first probability distribution output by the large language model are weighted. Based on the weights of each word unit in the second answer, the probabilities of each second word unit in the second probability distribution are weighted. The model parameters in the large language model are adjusted based on the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample, where the quality of the first answer is higher than that of the second answer. This method effectively identifies the importance of each word unit by introducing word-level similarity analysis, adjusts its contribution weight in training according to its importance, reduces training interference caused by background redundancy, enhances the key information on which human preferences depend, significantly improves the large language model's alignment ability with human preferences, and improves the quality of model output. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0021] Figure 1 A schematic diagram illustrating an implementation environment provided in an embodiment of this application; Figure 2 A flowchart illustrating a training method for a large language model provided in an embodiment of this application; Figure 3 This is a schematic diagram of the model architecture provided in the embodiments of this application; Figure 4 A schematic diagram illustrating word-level dual-queue weight alignment provided in an embodiment of this application; Figure 5 A schematic diagram illustrating training in conjunction with a reference large language model, provided for embodiments of this application; Figure 6 A flowchart illustrating a question-and-answer processing method provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a training device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a question-and-answer processing device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0023] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0024] To better understand and explain the methods provided in the embodiments of this application, some technical terms involved in the embodiments of this application will be explained and described below.

[0025] Large Language Models (LLMs) are large-scale language models that learn rich linguistic knowledge and patterns through pre-training on large-scale text data. They can generate coherent and logical text and demonstrate powerful capabilities and flexibility in various natural language processing tasks, such as text classification, language translation, and question answering systems.

[0026] Token: The smallest semantic unit into which a text sequence is divided.

[0027] Bidirectional Encoder Representation from Transformers (BERT) is a pre-trained model based on the Transformer architecture. Through bidirectional masking language models and next-sentence prediction tasks, the model learns the left and right context information of each token in the text simultaneously, thereby generating a more comprehensive semantic representation.

[0028] Out of Vocabulary (OOV): Vocabulary in natural language processing that is not included in a predefined vocabulary.

[0029] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes a terminal 10 and a server 20. The terminal 10 and the server 20 can be connected directly or indirectly via wired or wireless communication, which is not limited herein.

[0030] In one possible implementation, terminal 10 can be a smartphone, tablet, laptop, desktop computer, smart voice interaction device (e.g., smart speaker), wearable electronic device (e.g., smartwatch), in-vehicle terminal, smart home appliance (e.g., smart TV), AR / VR device, etc., but is not limited to these.

[0031] In one possible implementation, server 20 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0032] In one possible implementation, server 20 undertakes the primary computing task, and terminal 10 undertakes the secondary computing task; alternatively, server 20 undertakes the secondary computing task, and terminal 10 undertakes the primary computing task; or, server 20 and terminal 10 collaborate on computing using a distributed computing architecture. Those skilled in the art will understand that the computer program involved in this application can be deployed and executed on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network; this application does not impose any limitations in this regard.

[0033] The large language model training method provided in this application embodiment is executed by an electronic device. Optionally, the electronic device is a terminal 10 or a server 20. The following description uses a server 20 as an example. The terminal 10 is used to acquire a training dataset (containing a large number of samples, each sample including a sample question, a first answer, and a second answer corresponding to the sample question). The terminal 10 sends the training dataset to the server 20 via a network connection. The server 20 receives and stores the training dataset sent by the terminal 10, and trains the large language model based on this training dataset.

[0034] Once the large language model has been trained, it can be deployed to server 20. Terminal 10 receives the question / query text to be answered and sends it to server 20 via a network connection. Server 20 generates the corresponding answer based on the received question using the large language model. Server 20 then sends the answer to terminal 10 via the network connection and displays it on terminal 10.

[0035] It should be noted that, in the optional embodiments of this application, the relevant data, such as object information (e.g., collected sample data), requires the permission or consent of the object when the embodiments of this application are applied to specific products or technologies. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0036] The technical solutions of this application and their effects are described below through several embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0037] This application provides a method for training a large language model, which can be executed by any electronic device, such as a server or terminal.

[0038] Figure 2 This is a flowchart illustrating the training method for a large language model provided in an embodiment of this application, as shown below. Figure 2 As shown, the method may include the following steps S110-S130, wherein: Step S110: Obtain multiple samples.

[0039] The training method for the large language model provided in this application embodiment, also known as the Preference-Aware Optimization (PAO) method, is used to train the large language model for preference alignment, improve the ability of the large language model (LLM) to align with human preferences, and make the output of the large language model more in line with human preferences.

[0040] Each sample includes a sample question, a first answer, and a second answer. The first answer aligns with human preferences, while the second answer does not. The quality of the first answer is higher than that of the second answer.

[0041] For example, in a text summarization task, suppose the given question is: Please summarize the following: "A research institution recently announced a major breakthrough in the semiconductor field, developing a new generation of high-performance computing chips. This chip adopts an innovative architecture design, achieving a computing speed 1.8 times faster than similar products while reducing energy consumption by 40%. Test data shows that in complex algorithm processing tasks, the chip reduces task completion time to 58% of traditional chips and improves heat dissipation efficiency by 35%. Multiple patents have been applied for for the core technology. However, analysis indicates that current mass production costs are high, and large-scale application may face price challenges."

[0042] The generated abstract 1 reads, "A new computing chip achieves a 1.8-fold increase in computing speed and a 40% reduction in energy consumption. Tests show that the processing time for complex tasks is reduced by 58%, and heat dissipation efficiency is improved by 35%, but the high mass production cost may affect large-scale applications." The generated abstract 2 reads, "The newly developed chip has better performance and is more energy-efficient, with faster processing speeds, but its cost is higher."

[0043] Because Abstract 1 accurately summarizes the above content using quantitative indicators, while Abstract 2 lacks detailed information and is overly simplified, it fails to meet human requirements for information completeness and precise expression. Therefore, Abstract 1 is the first answer that conforms to human preferences, and Abstract 2 is the second answer that does not conform to human preferences.

[0044] This application does not limit the source of samples. For example, existing datasets used for preference training (such as Reddit TL;DR or Anthropic-HH) can be used, or samples can be constructed based on user historical query data. When constructing samples based on user historical query data, the user query can be used as the sample question, and the type of answer can be determined based on the user's feedback. For example, the answer selected by the user can be used as the first answer, and the unselected answer as the second answer; or the answer with valid user feedback can be used as the first answer, and the answer with invalid user feedback as the second answer.

[0045] Step S120: For each sample, determine the weight of each word based on the similarity between each word in each answer of the sample and each word in another answer, wherein the weight of each word character represents the importance of the word character in its respective answer.

[0046] Whether a model's output aligns with human preferences often depends on a few key pieces of information. However, in existing preference alignment methods, a large amount of non-critical background information is treated equally with key information during training. This causes the model to be disturbed by background noise during training, making it difficult to truly understand human judgment criteria for the value of information and affecting the final alignment quality.

[0047] The training method provided in this application distinguishes the importance of each word in the preferred answer (first answer) and the non-preferred answer (second answer), thereby suppressing the interference of background words (words with lower importance) on the optimization direction of the model during the training process, so that the model focuses on words that truly reflect human preferences.

[0048] Understandably, in preference alignment tasks, if a word appears in both preferred and non-preferred answers, it indicates that the word belongs to the general content of answer generation and does not depend on human preference selection; therefore, it can be considered a non-keyword / background word with lower importance. If a word appears only in either preferred or non-preferred answers, it is often related to human preference selection and belongs to a keyword / preference target word with higher importance. Based on this, in this embodiment, the importance of words can be distinguished according to the similarity between words in two answers.

[0049] Specifically, for each sample, each answer in the sample is segmented into word units and encoded at the word unit level to obtain the feature vector of each word unit in each answer of the sample; for each word unit in each answer of the sample, the weight of the word unit is determined based on the (semantic) similarity between the feature vector of the word unit and the feature vectors of each word unit in another answer.

[0050] Optionally, when determining the weight of a word element based on similarity, for each word element in each answer of the sample, the (feature) similarity between that word element and each word element in another answer can be determined; for each word element in each answer of the sample, the weight of that word element is determined according to the maximum similarity among the similarities corresponding to that word element, wherein the maximum similarity corresponding to the word element is negatively correlated with the weight of the word element, the larger the maximum similarity corresponding to the word element, the lower the weight of the word element.

[0051] Optionally, when determining the weight of a word element based on the maximum similarity corresponding to that word element, the type of the word element can be determined according to the maximum similarity among the various similarities corresponding to that word element. The type of word element includes keywords and non-keywords. If the word element is a keyword, the weight of the word element is determined as the first weight. If the word element is a non-keyword, the weight of the word element is determined as the second weight. The first weight is greater than the second weight.

[0052] Specifically, for each word element in each answer of the sample, if the maximum similarity corresponding to the word element is greater than the first threshold, it means that the word element exists in both answers, and the type of the word element can be determined as non-keyword; otherwise, it means that the word element only appears in a single answer, and the type of the word element is keyword.

[0053] Optionally, different threshold standards can be set for the first answer and the second answer. The first threshold is either the second threshold corresponding to the first answer or the third threshold corresponding to the second answer. If the maximum similarity of the word element to the first answer of the sample is greater than the second threshold, the word element is determined to be a non-keyword; otherwise, the word element is determined to be a keyword. If the word element to the second answer of the sample is greater than the third threshold, the word element is determined to be a non-keyword; otherwise, the word element is determined to be a keyword.

[0054] In this embodiment, key words and background words are accurately distinguished based on the similarity between word elements, so that the model training can focus more on key information that reflects user preferences and reduce interference from invalid content.

[0055] Optionally, the second and third thresholds can be fixed thresholds set based on prior knowledge of the task, such as setting the second threshold to 0.5 and the third threshold to 0.4.

[0056] Furthermore, considering the differences in semantic similarity between preferred and non-preferred answers across different tasks, the threshold can be dynamically adjusted based on the similarity of each word unit in the answer to better suit different task scenarios. As an alternative approach, the second and third thresholds can be determined based on the average of the maximum similarity of each word unit in the answer.

[0057] In this embodiment, a dynamic threshold calculation method that integrates the semantic similarity of preference answer terms is adopted. Compared with a fixed threshold strategy, this method can significantly improve the adaptability to the semantic complexity of different preference samples, thereby optimizing the recognition accuracy and robustness.

[0058] As an optional implementation, for the first answer obtained... Second answer Each word is encoded at the word level to obtain the corresponding feature vector. and Then, the semantic similarity between lexical units is calculated based on the cosine similarity function, which can be expressed as:

[0059] in, , Indicate the answer The number of lexical units in the answer, where i and j represent the positions of the lexical units in the corresponding answers.

[0060] For each answer, calculate the corresponding threshold based on the maximum similarity of each word element in that answer. :

[0061] Finally, for each word in each answer, the maximum similarity corresponding to that word is compared with the threshold corresponding to that answer to determine whether the word belongs to a keyword or a non-keyword.

[0062] For example, suppose the word segments obtained from the first answer are A1A2A3, and the word segments obtained from the second answer are B1B2, with a first weight of 1 and a second weight of 0. The similarity between each word segment in the first answer and each word segment in the second answer is shown in the table below:

[0063] Specifically, based on the maximum similarity of each word in the first answer, the second threshold corresponding to the first answer is determined to be (0.7+0.1+0.1) / 3=0.3; based on the maximum similarity of each word in the second answer, the third threshold corresponding to the second answer is determined to be (0.7+0.1) / 2=0.4.

[0064] The maximum similarity of word A1 is 0.7, which is greater than the second threshold of 0.3. The maximum similarity of word A2 is 0.05, which is less than the second threshold of 0.3. The maximum similarity of word A3 is 0.1, which is less than the second threshold of 0.3. Therefore, word A1 in the first answer is a non-keyword and its weight is 0. Words A2 and A3 are keywords and their weights are 1.

[0065] The maximum similarity of word B1 is 0.7, which is greater than the third threshold of 0.4. The maximum similarity of word B2 is 0.1, which is less than the third threshold of 0.4. Therefore, word B1 in the second answer is a non-keyword and its weight is 0. Word B2 is a keyword and its weight is 1.

[0066] In this application embodiment, any strategy can be used to determine the weights corresponding to different types of lexical units (keywords / non-keywords): Strategy 1: Set the first weight corresponding to the keyword and the second weight corresponding to the non-keyword to a fixed value, and the first weight is greater than the second weight.

[0067] Optionally, the weights corresponding to background words can be reset to zero, while the weights of keyword words remain unchanged. That is, the second weight for non-keyword words is 0, and the first weight for keyword words is 1.

[0068] This strategy can be expressed as:

[0069] Strategy 2: Background Word Weakening Strategy (WB) – Maintain the primary weight of keywords while reducing the secondary weight of non-keyword keywords. Optionally, the background word weights can be reduced proportionally to the similarity, and a weight cap can be set to prevent outliers in negative similarity.

[0070] Specifically, for each word element in the answer, if the word element is a keyword, the weight of the word element is determined as the first preset weight (i.e., the first weight); if the word element is not a keyword, if the maximum similarity corresponding to the word element is not greater than 0.5, the maximum similarity corresponding to the word element is used as the weight of the word element; otherwise, the difference between the first preset weight and the maximum similarity is used as the weight of the word element; wherein, the first preset weight is greater than 0.5.

[0071] When the first preset weight is set to 1, the strategy can be expressed as:

[0072] Strategy 3: Keyword Meta Enhancement Strategy (ET), keeping the second weight of non-keywords unchanged, and increasing the first weight of keywords. Specifically, for each word element in the answer, if the word element is not a keyword, the weight of the word element is determined as the second preset weight (i.e., the second weight); if the word element is a keyword, the difference between the first threshold and the maximum similarity corresponding to the word element is determined, and the sum of the second preset weight and the difference is used as the weight of the word element.

[0073] Optionally, the first threshold is either the second threshold corresponding to the first answer or the third threshold corresponding to the second answer. When determining the difference between the first threshold and the maximum similarity corresponding to the word, if the word is a word in the first answer, then the difference between the second threshold and the maximum similarity corresponding to the word is determined; if the word is a word in the second answer, then the difference between the third threshold and the maximum similarity corresponding to the word is determined.

[0074] When the second preset weight is 1, the strategy can be expressed as:

[0075] In this embodiment, a weight strategy adjustment mechanism is used to adjust the weights of keywords / non-keywords, which can effectively enhance the relevance and controllability of the training signal and guide the model to better learn preference expressions.

[0076] Step S130: Based on multiple samples, continuously perform training operations on the large language model to be trained to obtain the trained large language model.

[0077] The training termination conditions and loss function of the large language model can be configured according to requirements. For example, training termination conditions may include, but are not limited to, reaching a preset number of training iterations, loss function convergence (e.g., training loss is less than a preset value, or the training loss is less than a preset value for multiple consecutive iterations), and the model's test metrics meeting preset metrics. The model's training loss represents the difference between the probabilities of predicting preferred and unpreferred answers. By using the model's training loss function and employing the gradient descent algorithm to train the model, the probability of outputting preferred answers can be increased, and the probability of outputting unpreferred answers can be decreased, thereby obtaining a well-trained large language model that meets the needs of practical applications.

[0078] This application does not impose any restrictions on the model architecture of large language models. For example, it can adopt large basic language models (Large Language Model Meta AI, LLaMA), chat general language models (ChatGLM), etc.

[0079] The training operation may include the following steps S1301-S1303.

[0080] Step S1301: For each sample, input the sample question in the sample into the large language model to obtain the first prediction result corresponding to the sample question.

[0081] The first and second answers of this sample can also be used as labels input into the large language model. The first prediction result includes a first probability distribution and a second probability distribution. The first probability distribution includes each first word element generated by the large language model and its probability. The second probability distribution includes each second word element generated by the large language model and its probability. Each first word element corresponds to each word element of the first answer in this sample, and each second word element corresponds to each word element of the second answer in this sample.

[0082] Step S1302: For each sample, according to the weight of each word element of the first answer in the sample, the probability of each first word element in the first probability distribution is weighted to obtain the weighted first probability distribution; according to the weight of each word element of the second answer in the sample, the probability of each second word element in the second probability distribution is weighted to obtain the weighted second probability distribution.

[0083] Step S1303: Determine the training loss based on the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample, and adjust the model parameters in the large language model based on the training loss.

[0084] The training loss is used to increase the probability of outputting preferred answers and suppress the probability of outputting unpreferred answers.

[0085] Optionally, the loss function can be expressed as:

[0086] in, This represents the sequence of first-word weights, including the weights of each first-word element in the first answer. This represents the first probability distribution, including the probability of each first word element in the first answer output by the large language model; This represents the second word weight sequence, including the weights of each second word in the second answer. This represents the second probability distribution, which includes the probability of each second word in the second answer output by the large language model.

[0087] Figure 3 This is a schematic diagram of the model architecture provided in an embodiment of this application. The training samples include sample question x and preferred answer. and non-preferred answers Based on preferred answers Each word in the non-preference answer The similarity between various lexical units in the figure determines the type of each lexical unit (keyword lexical unit / background lexical unit). of , and The background terminology is shown in the diagram, and the rest are keyword terms. of and The first term is the background term, and the rest are the keyword terms. The weight of each term is determined according to its type. The keyword term has the first weight, and the background term has the second weight. The first weight is greater than the second weight.

[0088] The sample question x is input into the large language model to be trained, resulting in a first probability distribution and a second probability distribution generated by the large language model. The first probability distribution includes the preference output. The first word element and its generation probability are given in the second probability distribution, which includes the non-preferred output. The algorithm calculates the generation probability of each second word element and the first answer (the actual preferred answer) based on the generation probability of each first word element (predicted preferred answer) and the first answer. It also calculates the cross-entropy loss of the preferred answer based on the weights of each word element in the first answer. We weight the cross-entropy loss of the preferred answer to obtain the weighted preference loss. Based on the generation probability of each second word element (predicting the unbiased answer) and the second answer (the actual unbiased answer), calculate the cross-entropy loss of the unbiased answer, and then calculate the loss based on the weights of each word element in the second answer. The cross-entropy loss of non-preferred answers is weighted to determine the weighted non-preferred loss. The training loss is determined based on the difference between the weighted preference loss and the weighted non-preference loss. This is done by continuously training the model so that it can increase the probability of outputting preferred answers and decrease the probability of outputting unpreferred answers.

[0089] based on Figure 2The training method for the large language model, as shown, determines the weight of each word in each answer based on the similarity between word units of each answer and word units of another answer. Based on the weights of each word unit in the first answer, the probabilities of each first word unit in the first probability distribution output by the large language model are weighted. Similarly, based on the weights of each word unit in the second answer, the probabilities of each second word unit in the second probability distribution are weighted. Finally, the model parameters in the large language model are adjusted based on the difference between the weighted first probability distribution and the weighted second probability distribution for each sample, where the quality of the first answer is higher than that of the second answer. This method, by introducing word-level similarity analysis, effectively identifies the importance of each word unit and adjusts its contribution weight in training according to its importance. This alleviates training interference caused by background redundancy, enhances the key information upon which human preferences rely, significantly improves the large language model's alignment ability with human preferences, and improves the quality of the model's output.

[0090] Furthermore, the embodiments of this application do not require the introduction of additional supervision signals or reward models, which greatly reduces training costs and deployment complexity. They are applicable to various language alignment scenarios and have good versatility and scalability.

[0091] Optionally, in step S120, for each sample, the trained feature extraction model can be used to segment each answer of the sample into words and perform word-level encoding to obtain the feature vector of each word in each answer of the sample; for each word in each answer of the sample, the weight of the word is determined based on the similarity between the feature vector of the word and the feature vector of each word in another answer.

[0092] If the granularity of the word units divided by the feature extraction model is the same as that of the word units generated by the large language model, then each first word unit generated by the large language model is the word unit divided by the first answer, and each second word unit generated by the large language model is the word unit divided by the second answer.

[0093] If the lexicalization mechanisms of the feature extraction model and the large language model differ, resulting in different granularities of the segmented lexical units, then cross-model lexical unit weight alignment is required. This involves aligning the weights of each lexical unit determined by the feature extraction model to the weights corresponding to the lexical units generated by the large language model. For example, suppose the feature extraction model segments the first answer into lexical units including "Do", "n't", "you", and "agree", while the large language model generates the first lexical units as "Don", "'", "t", "you", and "agree". Since the two models segment the word "Don't" differently, the weights of the determined lexical units "Do" and "n't" need to be converted into the weights of the lexical units "Don", "'", and "t".

[0094] This application does not impose any restrictions on the model architecture of the feature extraction model, as long as it has the ability to segment words and encode words at the word level. For example, the feature extraction model can adopt the BERT model, the Word2Vec model, etc.

[0095] Among them, BERT is preferred as a feature extraction model because its deep bidirectional Transformer architecture can generate lexical embeddings with rich contextual information, and its WordPiece segmentation algorithm can decompose lexical units into semantically meaningful sub-word units, effectively solving the problem of out-of-vocabulary (OOV) recognition. In addition, the lexical embedding dimension of BERT matches the input space of the Large Language Model (LLM) without the need for additional dimensionality transformation operations.

[0096] Optionally, when weighting the (generation) probability of the corresponding word element in the first prediction result based on the weight of each word element in the answer, for each first word element in the first probability distribution, from the word elements obtained by segmenting the first answer in the sample through the feature extraction model, determine each first target word element belonging to the same word segment as the first word element. Based on the weight of each determined first target word element, determine the weight of the first word element. Based on the weight of each first word element, weight the (generation) probability of each first word element to obtain the weighted first probability distribution. For each second word element in the second probability distribution, from the word elements obtained by segmenting the second answer in the sample through the feature extraction model, determine each second target word element belonging to the same word segment as the second word element. Based on the weight of each determined second target word element, determine the weight of the second word element. Based on the weight of each second word element, weight the (generation) probability of each second word element to obtain the weighted second probability distribution.

[0097] Continuing with the example of word units divided by the feature extraction model including “Do”, “n’t”, “you”, and “agree”, and the first word units generated by the large language model being “Don”, “’”, “t”, “you”, and “agree”, for the first word unit “Don”, we can determine the word units “Do” and “n’t” that belong to the same word segmentation as this word unit from the word units divided by the feature extraction model, and use the average weight of word units “Do” and “n’t” as the weight of the first word unit “Don”.

[0098] Optionally, when aligning word weights across models, a word-level dual-queue alignment algorithm can be used. Specifically, the words representing the answer segmented by the feature extraction model can be sequentially input into the first queue, and the words representing the corresponding answer generated by the large language model can be sequentially input into the second queue. If the words in the two queues match, the weights of the matching words in the second queue are determined based on the weights of the matching words in the first queue.

[0099] like Figure 4 As shown, assuming the first answer is "Don't...you love...", the word sequence obtained by using the BERT model is: "Do", "n't"... "you", "love"..., and the word sequence generated by the large language model LLM is: "Don", "'", "t"... "you", "love"....

[0100] Establish a BERT word queue and an LLM word queue. Add the first word from the BERT word sequence and the first word from the LLM word sequence to their respective queues. Determine if the words in the two queues match. If they match, set the weight of the LLM word to the weight of the corresponding BERT word and dequeue the word with the determined weight. If they don't match, enqueue the next word with the shorter character sequence until a match is found. When multiple words in the BERT word queue match words in the LLM word queue, calculate the average weight of each word in the BERT word queue and use this average weight as the weight of each word in the LLM word queue.

[0101] exist Figure 4 In the BERT queue, "Do" and "n't" match "Don", "'", and "t" in the LLM lexical queue. The weight of the lexical "Do" is... The weight of the word "n't" is Then the word units “Don”, “'”, and “t” have the same weight, which is 1. .

[0102] In this embodiment of the application, to address the issue of different lexicalization mechanisms between the feature extraction model and the large language model, a lexical-level dual-queue alignment algorithm can be adopted to achieve seamless transfer of lexical weights between heterogeneous models, thus ensuring training consistency.

[0103] In this embodiment, a reference large language model can be further introduced to determine the training loss based on the output difference between the currently trained large language model and the reference large language model.

[0104] The training operation further includes: for each sample, inputting the sample question from that sample into a reference large language model to obtain a second prediction result corresponding to that sample question; wherein the second prediction result includes a third probability distribution and a fourth probability distribution, the third probability distribution including each first word element generated by the reference large language model and its probability, and the fourth probability distribution including each second word element generated by the reference large language model and its probability. The reference large language model is a pre-trained large language model, which is also the large language model to be trained in this application, and possesses basic question-answering generation capabilities.

[0105] When calculating the training loss, for each sample, the probabilities of each first word in the third probability distribution corresponding to the sample are weighted according to the weights of each word in the first answer, resulting in a weighted third probability distribution. Similarly, the probabilities of each second word in the fourth probability distribution corresponding to the sample are weighted according to the weights of each word in the second answer, resulting in a weighted fourth probability distribution. The first distribution difference between the weighted first and third probability distributions corresponding to the sample, and the second distribution difference between the weighted second and fourth probability distributions corresponding to the sample, are determined. The difference between the determined first and second distribution differences is used as the difference between the weighted first and second probability distributions corresponding to the sample. The training loss is determined based on the differences between the weighted first and second distribution differences for each sample. The first distribution difference includes the probability differences of each first word, and the second distribution difference includes the probability differences of each second word.

[0106] Optionally, the training loss function can be expressed as:

[0107] in, This represents the sequence of first-word weights, including the weights of each first-word element in the first answer. This represents the first probability distribution, including the probability of each first word element in the first answer output by the large language model. This represents the third probability distribution, which includes the probability of each first word in the first answer output by the reference large language model; This represents the second word weight sequence, including the weights of each second word in the second answer. This represents the second probability distribution, which includes the probability of each second word element in the second answer output by the large language model. This represents the fourth probability distribution, which includes the probability of each second word in the second answer output by the reference large language model.

[0108] Figure 5 This is a schematic diagram illustrating training in conjunction with a reference large language model, provided in an embodiment of this application. The sample question x is input into both the currently trained large language model and the reference large language model, respectively, to obtain the first probability distribution predicted by the currently trained large language model. Second probability distribution And the third probability distribution predicted by the reference large language model. and the fourth probability distribution According to the first word weight sequence For the first probability distribution and the third probability distribution The differences between them are weighted to obtain the weighted first distribution difference. According to the second word weight sequence For the second probability distribution and the fourth probability distribution The differences between them are weighted to obtain the weighted second distribution difference. Based on the weighted first distribution difference Difference between the second and second distributions Determine training loss Based on training loss Adjust the model parameters in the currently trained large language model.

[0109] It should be noted that the training method for the large language model provided in this application embodiment can be applied independently to the stage of fine-tuning the pre-trained large language model, or it can be used in combination with existing preference alignment techniques, such as Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF), to further enhance the focus on key information on which human preferences depend, reduce the interference of background redundant information on training, and improve training efficiency and the accuracy of model output.

[0110] This application also provides a question-answering processing method, which can use a large language model trained in any embodiment of this application to perform question-answering processing. The method can be executed by any electronic device, such as a server or a terminal.

[0111] Figure 6 This is a flowchart illustrating the question-and-answer processing method provided in the embodiments of this application, such as... Figure 6 As shown, the method may include the following steps S210-S220, wherein: Step S210: Obtain the questions to be answered; Step S220: Input the question into the trained large language model to obtain the answer to the question.

[0112] The large language model trained in this application can theoretically be applied to natural language processing tasks in any scenario, improving the alignment of model output with human preferences. For example, it can be applied to script-based intelligent question-answering systems, enhancing the model's ability to summarize script summaries in multi-turn dialogue scenarios related to scripts, making them more in line with business needs; it can also be applied to content generation platforms (such as news, story, and advertising creation), improving the model's understanding of key semantics in creation tasks, making the generated content more in line with human preferences. The training process of the large language model can be found above, and will not be repeated here.

[0113] The questions to be answered can be query text for the corresponding task scenario. For example, for a summary generation task, the question includes the original text data to be summarized and the task instructions; for a translation task, the question includes the text to be translated and the task instructions.

[0114] In this application embodiment, the large language model used in question-and-answer processing can be the one described above. Figure 2 The specific implementation of the scheme and optional embodiments shown can be referred to the above. Figure 2 The specific details of the illustrated scheme and its optional embodiments will not be repeated here.

[0115] Based on and Figure 2 The training method for the large language model shown follows the same principle. This application provides a training apparatus for a large language model, such as... Figure 7 As shown, the training device 300 may include: a sample acquisition module 310, a word weight determination module 320, and a training module 330, wherein: The sample acquisition module 310 is used to acquire multiple samples, each sample including a sample question, a first answer and a second answer corresponding to the sample question, wherein the answer quality of the first answer is higher than that of the second answer. The word weight determination module 320 is used to determine the weight of each word element for each sample based on the similarity between each word element in each answer of the sample and each word element in another answer; wherein the weight of each word element represents the importance of the word element in its respective answer; Training module 330 is used to continuously perform training operations on the large language model to be trained based on the multiple samples, so as to obtain the trained large language model: For each sample, the sample question in the sample is input into a large language model to obtain a first prediction result corresponding to the sample question; wherein, the first prediction result includes a first probability distribution and a second probability distribution, the first probability distribution includes each first word element generated by the large language model and the probability of each first word element, the second probability distribution includes each second word element generated by the large language model and the probability of each second word element, each first word element corresponds to each word element of the first answer in the sample, and each second word element corresponds to each word element of the second answer in the sample; For each sample, the probabilities of each first word in the first probability distribution are weighted according to the weight of each word in the first answer in the sample to obtain a weighted first probability distribution; the probabilities of each second word in the second probability distribution are weighted according to the weight of each word in the second answer in the sample to obtain a weighted second probability distribution. The training loss is determined based on the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample, and the model parameters in the large language model are adjusted based on the training loss.

[0116] Optionally, the lexical weight determination module 320 can be used to: The trained feature extraction model is used to segment each answer in the sample into words and encode them at the word level to obtain the feature vector of each word in each answer of the sample. For each word element in each answer of the sample, the weight of the word element is determined based on the similarity between the feature vector of the word element and the feature vectors of each word element in another answer. The training module 330 can be used for: For each first word element in the first probability distribution, from the word elements obtained by segmenting the first answer in the sample through the feature extraction model, determine each first target word element that belongs to the same word segment as the first word element. Based on the weights of the determined first target word elements, determine the weight of the first word element. Based on the weights of each first word element, weight the probabilities of each first word element to obtain the weighted first probability distribution. For each second word element in the second probability distribution, from the word elements obtained by segmenting the second answer in the sample through the feature extraction model, determine each second target word element that belongs to the same word segment as the second word element. Based on the weights of the determined second target word elements, determine the weight of the second word element. Based on the weights of each second word element, weight the probabilities of each second word element to obtain the weighted second probability distribution.

[0117] Optionally, the lexical weight determination module 320 can be used to: For each word element in each answer of the sample, determine the similarity between that word element and each word element in another answer; For each word element in each answer of the sample, the weight of the word element is determined based on the maximum similarity among the similarity values ​​corresponding to that word element, wherein the maximum similarity value is negatively correlated with the weight.

[0118] Optionally, the lexical weight determination module 320 can be used to: The type of a word is determined based on the maximum similarity among all similarities corresponding to that word. If the maximum similarity of the word is greater than the first threshold, the type of the word is determined to be a non-keyword; otherwise, the type of the word is a keyword. If the word element is a keyword, its weight is determined as the first weight; if the word element is not a keyword, its weight is determined as the second weight, wherein the first weight is greater than the second weight.

[0119] Optionally, the training module 330 can also be used for: For each sample, the sample question in the sample is input into the reference large language model to obtain the second prediction result corresponding to the sample question; wherein, the second prediction result includes a third probability distribution and a fourth probability distribution, the third probability distribution includes each first word element generated by the reference large language model and the probability of each first word element, and the fourth probability distribution includes each second word element generated by the reference large language model and the probability of each second word element; For each sample, the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to that sample is determined by the following method: Based on the weights of each word element in the first answer in the sample, the probabilities of each first word element in the third probability distribution are weighted to obtain the weighted third probability distribution. Based on the weights of each word element in the second answer in the sample, the probabilities of each second word element in the fourth probability distribution are weighted to obtain the weighted fourth probability distribution. Determine the first distribution difference between the weighted first probability distribution and the weighted third probability distribution corresponding to the sample, and the second distribution difference between the weighted second probability distribution and the weighted fourth probability distribution corresponding to the sample; The difference between the first distribution difference and the second distribution difference is taken as the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to the sample.

[0120] Optionally, the first threshold is either the second threshold corresponding to the first answer or the third threshold corresponding to the second answer; The lexical weight determination module 320 can be used for: If a word element belongs to the first answer of the sample, and the maximum similarity corresponding to the word element is greater than the second threshold, the type of the word element is determined to be a non-keyword; otherwise, the type of the word element is a keyword. If a word belongs to the second answer of the sample, and the maximum similarity of the word is greater than the third threshold, the word is determined to be a non-keyword; otherwise, the word is a keyword.

[0121] Optionally, each of the samples corresponds to its own second threshold and third threshold; For each of the samples, the second threshold corresponding to the first answer in that sample is determined in the following way: For each word element in the first answer of the sample, determine the maximum similarity corresponding to that word element; The second threshold corresponding to the first answer is determined based on the sum of the maximum similarity of each word in the first answer and the number of words in the first answer; For each of the samples, the third threshold corresponding to the second answer in that sample is determined in the following way: For each word element in the second answer of the sample, determine the maximum similarity corresponding to that word element; The third threshold corresponding to the second answer is determined based on the sum of the maximum similarity of each word in the second answer and the number of words in the second answer.

[0122] Optionally, the lexical weight determination module 320 can be used to perform any of the following: If the word element is a keyword, its weight is determined to be the first preset weight. If the word element is not a keyword, and the maximum similarity corresponding to the word element is not greater than 0.5, the maximum similarity corresponding to the word element is used as the weight of the word element. Otherwise, the difference between the first preset weight and the maximum similarity is used as the weight of the word element. Wherein, the first preset weight is greater than 0.5. If the word is not a keyword, the weight of the word is determined to be the second preset weight; if the word is a keyword, the difference between the first threshold and the maximum similarity corresponding to the word is determined, and the sum of the second preset weight and the difference is used as the weight of the word.

[0123] Based on and Figure 6 The question-and-answer processing method shown in this application follows the same principle. This application provides a question-and-answer processing device, such as... Figure 8 As shown, the question-and-answer processing device 400 may include: a question acquisition module 410 and an answer generation module 420, wherein: The question retrieval module is used to retrieve questions that need to be answered. The answer generation module is used to input the question into the trained large language model to obtain the answer to the question. The large language model is trained using the training method of the large language model provided in the embodiments of this application.

[0124] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0125] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, it can implement the method in any optional embodiment of this application.

[0126] Figure 9 A schematic diagram of the structure of an electronic device to which an embodiment of the present invention applies is shown, such as... Figure 9 As shown, the electronic device can be a server or a terminal, and it can be used to implement the methods provided in any embodiment of the present invention.

[0127] like Figure 9 As shown, the electronic device 2000 may primarily include at least one processor 2001. Figure 9 The diagram shows components such as a memory 2002, a communication module 2003, and an input / output interface 2004. Optionally, these components can be connected and communicate with each other via a bus 2005. It should be noted that... Figure 9 The structure of the electronic device 2000 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of this application are applicable.

[0128] The memory 2002 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 2001, and can also include programs for implementing other functions or services. The memory 2002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0129] Processor 2001 is connected to memory 2002 via bus 2005, and implements corresponding functions by calling application programs stored in memory 2002. Processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0130] Electronic device 2000 can connect to a network via communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. Communication module 2003 may include wired network interfaces and / or wireless network interfaces, meaning the communication module may include at least one of wired or wireless communication modules.

[0131] Electronic device 2000 can connect to required input / output devices, such as keyboards and display devices, via input / output interface 2004. Electronic device 2000 itself may have a display device, and other display devices can also be connected externally via interface 2004. Optionally, storage devices, such as hard drives, can also be connected via interface 2004 to store data from electronic device 2000, retrieve data from storage device, or store data from storage device into memory 2002. It is understood that input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to input / output interface 2004 can be a component of electronic device 2000 or an external device connected to electronic device 2000 when needed.

[0132] The bus 2005 used to connect the components may include a pathway for transmitting information between the components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 2005 can be divided into address bus, data bus, control bus, etc.

[0133] Optionally, for the solution provided in the embodiments of the present invention, the memory 2002 can be used to store a computer program that executes the solution of the present invention, and the processor 2001 runs the computer program. When the processor 2001 runs the computer program, it implements the operation of the method or apparatus provided in the embodiments of the present invention.

[0134] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0135] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0136] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0137] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0138] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A training method for a large language model, characterized in that, include: Multiple samples are obtained, each sample including a sample question, a first answer and a second answer corresponding to the sample question, wherein the answer quality of the first answer is higher than that of the second answer; For each sample, the weight of each word element is determined based on the similarity between each word element in each answer of that sample and each word element in another answer; wherein the weight of each word element represents the importance of that word element in its respective answer; Based on the multiple samples, the large language model to be trained is continuously trained to obtain the trained large language model: For each sample, the sample question in the sample is input into a large language model to obtain a first prediction result corresponding to the sample question; wherein, the first prediction result includes a first probability distribution and a second probability distribution, the first probability distribution includes each first word element generated by the large language model and the probability of each first word element, the second probability distribution includes each second word element generated by the large language model and the probability of each second word element, each first word element corresponds to each word element of the first answer in the sample, and each second word element corresponds to each word element of the second answer in the sample; For each sample, the probabilities of each first word in the first probability distribution are weighted according to the weight of each word in the first answer in the sample to obtain a weighted first probability distribution; the probabilities of each second word in the second probability distribution are weighted according to the weight of each word in the second answer in the sample to obtain a weighted second probability distribution. The training loss is determined based on the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample, and the model parameters in the large language model are adjusted based on the training loss.

2. The method according to claim 1, characterized in that, For each sample, determining the weight of each word element based on the similarity between each word element in each answer of that sample and each word element in another answer includes: The trained feature extraction model is used to segment each answer in the sample into words and encode them at the word level to obtain the feature vector of each word in each answer of the sample. For each word element in each answer of the sample, the weight of the word element is determined based on the similarity between the feature vector of the word element and the feature vectors of each word element in another answer. For each sample, the probability of each first word in the first probability distribution is weighted according to the weight of each word in the first answer in the sample to obtain a weighted first probability distribution; the probability of each second word in the second probability distribution is weighted according to the weight of each word in the second answer in the sample to obtain a weighted second probability distribution, including: For each first word element in the first probability distribution, from the word elements obtained by segmenting the first answer in the sample through the feature extraction model, determine each first target word element that belongs to the same word segment as the first word element. Based on the weights of the determined first target word elements, determine the weight of the first word element. Based on the weights of each first word element, weight the probabilities of each first word element to obtain the weighted first probability distribution. For each second word element in the second probability distribution, from the word elements obtained by segmenting the second answer in the sample through the feature extraction model, determine each second target word element that belongs to the same word segment as the second word element. Based on the weights of the determined second target word elements, determine the weight of the second word element. Based on the weights of each second word element, weight the probabilities of each second word element to obtain the weighted second probability distribution.

3. The method according to claim 1 or 2, characterized in that, The step of determining the weight of each word element based on the similarity between each word element in each answer of the sample and each word element in another answer includes: For each word element in each answer of the sample, determine the similarity between that word element and each word element in another answer; For each word element in each answer of the sample, the weight of the word element is determined based on the maximum similarity among the similarity values ​​corresponding to that word element, wherein the maximum similarity value is negatively correlated with the weight.

4. The method according to claim 3, characterized in that, For each word element in each answer of the sample, the step of determining the weight of the word element based on the maximum similarity among the similarity scores corresponding to that word element includes: The type of a word is determined based on the maximum similarity among all similarities corresponding to that word. If the maximum similarity of the word is greater than the first threshold, the type of the word is determined to be a non-keyword; otherwise, the type of the word is a keyword. If the word element is a keyword, its weight is determined as the first weight; if the word element is not a keyword, its weight is determined as the second weight, wherein the first weight is greater than the second weight.

5. The method according to claim 1, characterized in that, The training operation also includes: For each sample, the sample question in the sample is input into the reference large language model to obtain the second prediction result corresponding to the sample question; wherein, the second prediction result includes a third probability distribution and a fourth probability distribution, the third probability distribution includes each first word element generated by the reference large language model and the probability of each first word element, and the fourth probability distribution includes each second word element generated by the reference large language model and the probability of each second word element; For each sample, the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to that sample is determined by the following method: Based on the weights of each word element in the first answer in the sample, the probabilities of each first word element in the third probability distribution are weighted to obtain the weighted third probability distribution. Based on the weights of each word element in the second answer in the sample, the probabilities of each second word element in the fourth probability distribution are weighted to obtain the weighted fourth probability distribution. Determine the first distribution difference between the weighted first probability distribution and the weighted third probability distribution corresponding to the sample, and the second distribution difference between the weighted second probability distribution and the weighted fourth probability distribution corresponding to the sample; The difference between the first distribution difference and the second distribution difference is taken as the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to the sample.

6. The method according to claim 4, characterized in that, The first threshold is either the second threshold corresponding to the first answer or the third threshold corresponding to the second answer; If the maximum similarity corresponding to the word element is greater than the first threshold, the word element is determined to be a non-keyword; otherwise, the word element is determined to be a keyword. This includes: If a word element belongs to the first answer of the sample, and the maximum similarity corresponding to the word element is greater than the second threshold, the type of the word element is determined to be a non-keyword; otherwise, the type of the word element is a keyword. If a word belongs to the second answer of the sample, and the maximum similarity of the word is greater than the third threshold, the word is determined to be a non-keyword; otherwise, the word is a keyword.

7. The method according to claim 6, characterized in that, Each sample corresponds to its own second threshold and third threshold; For each of the samples, the second threshold corresponding to the first answer in that sample is determined in the following way: For each word element in the first answer of the sample, determine the maximum similarity corresponding to that word element; The second threshold corresponding to the first answer is determined based on the sum of the maximum similarity of each word in the first answer and the number of words in the first answer; For each of the samples, the third threshold corresponding to the second answer in that sample is determined in the following way: For each word element in the second answer of the sample, determine the maximum similarity corresponding to that word element; The third threshold corresponding to the second answer is determined based on the sum of the maximum similarity of each word in the second answer and the number of words in the second answer.

8. A question-and-answer processing method, characterized in that, include: Get the questions that are pending; The question is input into the trained large language model to obtain the answer to the question, wherein the large language model is trained using the method described in any one of claims 1 to 7.

9. A training device for a large language model, characterized in that, The device includes: The sample acquisition module is used to acquire multiple samples, each sample including a sample question, a first answer and a second answer corresponding to the sample question, wherein the answer quality of the first answer is higher than that of the second answer. The word weight determination module is used to determine the weight of each word for each sample based on the similarity between each word in each answer of the sample and each word in another answer; wherein the weight of each word represents the importance of the word in its corresponding answer; The training module is used to continuously perform training operations on the large language model to be trained based on the multiple samples, so as to obtain the trained large language model: For each sample, the sample question in the sample is input into a large language model to obtain a first prediction result corresponding to the sample question; wherein, the first prediction result includes a first probability distribution and a second probability distribution, the first probability distribution includes each first word element generated by the large language model and the probability of each first word element, the second probability distribution includes each second word element generated by the large language model and the probability of each second word element, each first word element corresponds to each word element of the first answer in the sample, and each second word element corresponds to each word element of the second answer in the sample; For each sample, the probabilities of each first word in the first probability distribution are weighted according to the weight of each word in the first answer in the sample to obtain a weighted first probability distribution; the probabilities of each second word in the second probability distribution are weighted according to the weight of each word in the second answer in the sample to obtain a weighted second probability distribution. The training loss is determined based on the difference between the weighted first probability distribution and the weighted second probability distribution corresponding to each sample, and the model parameters in the large language model are adjusted based on the training loss.

10. A question-and-answer processing device, characterized in that, The device includes: The question retrieval module is used to retrieve questions that need to be answered. The answer generation module is used to input the question into a trained large language model to obtain the answer to the question, wherein the large language model is trained using the method described in any one of claims 1 to 7.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1 to 7 or 8.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7 or 8.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7 or 8.

Images