A language model comparison method and device based on preferences

Abstract

The application provides a preferred language model comparison method and device, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining to-be-processed data containing multiple sub-data, performing semantic truncation processing on each sub-data to obtain truncated data; generating first data and second data with different positions of answers based on answer difference data between first answers and second answers and the truncated data; using a preference prediction model to perform prediction processing on the first data and the second data, so that corresponding preference probabilities can be obtained; and finally, using the predicted preference probabilities, a comparison result of the model is obtained. The semantic truncation processing can ensure that the truncated data carries higher effective information, the different positions can eliminate the influence of the positions, and the preference prediction model can be used to realize accurate model comparison.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a method and apparatus for comparing language models based on preferences. Background Technology

[0002] In the field of artificial intelligence, preference refers to the degree to which users like the content generated by a model, and it is an important basis for evaluating and comparing models. Human evaluation is the gold standard for directly obtaining human preferences, but it also suffers from problems such as high cost and low efficiency. With the continuous improvement of the capabilities of large language models, direct preference prediction based on large language models has emerged. This technology can use a pre-trained large language model to simulate a human judge and automatically judge preferences for the answers output by different models.

[0003] However, in this approach, large language models may not base their preference judgments on the quality of the answers themselves, but rather tend to select answers from fixed positions. Furthermore, due to the inherent limitations of large language models, they struggle to capture effective information when processing long texts, leading to inaccurate preference judgments and difficulty in obtaining accurate model comparison results. In addition, training large language models requires a large amount of manually labeled preference data, resulting in high costs, data sparsity, and, since most of the data is in a single language, insufficient generalization ability of the model. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to provide a preference-based language model comparison method and apparatus, which can effectively reduce the impact of content location on prediction and retain core information in the data to improve the accuracy of model comparison.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] On one hand, the present invention provides a preference-based language model comparison processing method, comprising:

[0007] Acquire data to be processed, which includes multiple sub-data, including a prompt to be processed, a first answer generated by a first language model based on the prompt to be processed, and a second answer generated by a second language model based on the prompt to be processed.

[0008] For each of the sub-data, based on the length of the sub-data, the length of the data to be processed, and the length limit of the data to be processed by the preference prediction model, the sub-data is semantically truncated to obtain truncated data, which includes truncated prompts, a first truncated response, and a second truncated response.

[0009] Based on the answer difference data between the first answer and the second answer and the truncated data, first data and second data are generated, wherein the positions of the first truncated answer and the second truncated answer in the first data are different from those in the second data;

[0010] The preference prediction model is used to predict the first data and the second data to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model.

[0011] Based on the first preference probability and the second preference probability, the comparison result of the first language model and the second language model is determined.

[0012] On the other hand, the present invention also provides a preference-based language model comparison device, comprising:

[0013] The acquisition module is used to acquire data to be processed, which includes multiple sub-data, including a prompt to be processed, a first answer generated by a first language model based on the prompt to be processed, and a second answer generated by a second language model based on the prompt to be processed.

[0014] The truncation module is used to perform semantic truncation processing on each of the sub-data based on the length of the sub-data, the length of the data to be processed, and the length limit of the data to be processed by the preference prediction model, to obtain truncated data, wherein the truncated data includes truncation prompts, a first truncated response, and a second truncated response.

[0015] A generation module is used to generate first data and second data based on the answer difference data between the first answer and the second answer and the truncated data, wherein the positions of the first truncated answer and the second truncated answer in the first data are different from those in the second data.

[0016] The prediction module is used to use a preference prediction model to predict the first data and the second data, and to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model.

[0017] The comparison module is used to determine the comparison result of the first language model and the second language model based on the first preference probability and the second preference probability.

[0018] On the other hand, the present invention also provides an electronic device including a processor and a memory, the memory storing a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the preference-based language model comparison methods provided by the present invention.

[0019] On the other hand, the present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform steps in any of the preference-based language model comparison methods provided by the present invention.

[0020] On the other hand, the present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in any of the preference-based language model comparison methods provided by the present invention.

[0021] The beneficial effects of the technical solution provided by this invention include at least the following:

[0022] This invention can acquire data to be processed containing multiple sub-data. The sub-data is semantically truncated according to its length, the length of the data to be processed, and the length limit imposed by the preference prediction model. This ensures that the truncated data carries a high level of effective information. First data and second data are then generated based on the truncated data and answer difference data. Introducing answer difference data helps the model accurately understand the differences between the two answers. Furthermore, the positions of the two answers in the first data differ from those in the second data, reducing the impact of answer position on the result. After processing by the preference prediction model, the first preference probability of the first language model and the second preference probability of the second language model are obtained. Finally, the comparison result is directly obtained based on the first and second preference probabilities, enabling accurate model comparison. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of the preference-based language model comparison method provided in this embodiment of the invention;

[0025] Figure 2 This is a flowchart illustrating the preference-based language model comparison method provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the semantic truncation processing provided in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram illustrating the generation of first and second data provided in an embodiment of the present invention;

[0028] Figure 5This is a schematic diagram of the structure of the preference-based language model comparison device provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] It is understood that in specific embodiments of the present invention, data involving user information and related data requires user permission or consent, and the collection, use and processing of such data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0032] This invention provides a preference-based language model comparison method, which can be found in the following embodiments. Figure 1 The diagram illustrates an application scenario for a preference-based language model comparison method. This application scenario can include a terminal 101 and a server 102, which can exchange data via a network. The terminal 101 can have a question-and-answer related application installed. The terminal 101 can be a mobile phone, tablet, smart Bluetooth device, computer, large screen, robot, or other similar device; the server 102 can be a single server or a server cluster consisting of multiple servers.

[0033] Users can send data to be processed to server 102 through terminal 101. The data to be processed includes multiple sub-data, including a prompt to be processed, a first answer generated by a first language model based on the prompt to be processed, and a second answer generated by a second language model based on the prompt to be processed.

[0034] For each sub-data set, server 102 performs semantic truncation based on the sub-data set's length, the length of the data to be processed, and the length limit imposed by the preference prediction model, resulting in truncated data. The truncated data includes truncation prompts, a first truncated response, and a second truncated response. Based on the response difference data between the first and second responses and the truncated data, server 102 generates first data and second data, where the positions of the first and second truncated responses in the first data differ from those in the second data. The preference prediction model is then used to predict the first and second data, yielding a first preference probability corresponding to the first language model and a second preference probability corresponding to the second language model. Based on the first and second preference probabilities, the comparison result between the first and second language models is determined.

[0035] Finally, server 102 can send the comparison results to terminal 101 so that they can be displayed to the user through terminal 101.

[0036] In this embodiment, a preference-based language model comparison method is provided, such as... Figure 2 As shown, the specific process of this preference-based language model comparison method can be as follows:

[0037] S110, Obtain the data to be processed.

[0038] Data to be processed refers to the data that needs to be processed. Data to be processed may include multiple sub-data, which may include prompts to be processed, a first answer generated by the first language model based on the prompts to be processed, and a second answer generated by the second language model based on the prompts to be processed.

[0039] Here, "pending processing prompts" can be understood as descriptions related to tasks that the model needs to perform, and "first language model" and "second language model" refer to two models that need to be compared. These two models can be different versions of the same model, or they can be two completely different models.

[0040] The first answer is the output of the first language model after processing the prompt, and the second answer is the output of the second language model after processing the prompt. The prompt, the first answer, and the second answer can all be referred to as sub-data. The data to be processed typically includes the prompt, the first answer, and the second answer.

[0041] The data to be processed can be the user's pre-input of the prompts to be processed into the first language model and the second language model respectively, to obtain the first answer and the second answer, and the user combines the prompts to be processed, the first answer and the second answer into the data to be processed.

[0042] Of course, users can also directly provide data to be processed, inputting the data into the first language model and the second language model respectively. The server directly receives the first answer output by the first language model and the second answer output by the second language model, and automatically combines the first answer, the second answer and the prompt to be processed into the data to be processed.

[0043] S120. For each of the sub-data, based on the length of the sub-data, the length of the data to be processed, and the length limit of the data to be processed by the preference prediction model, the sub-data is semantically truncated to obtain truncated data.

[0044] The data to be processed contains multiple sub-data. For each sub-data, the length of each sub-data can be obtained, and the length of the data to be processed is the sum of the lengths of all sub-data. In this embodiment of the invention, the length of the sub-data and the length of the data to be processed can be measured by the number of characters. For example, if a sub-data contains 10 characters, then the length of the sub-data is 10.

[0045] The prediction model is based on comparing the output effects of two models using preference comparisons. The prediction model can be pre-trained using a large amount of preference data. Understandably, the length of the model's input data is not unlimited; typically, the length of the input data cannot exceed the model's maximum length limit. In this embodiment of the invention, the input data of the prediction model includes the data to be processed, and a corresponding length limit can be set for the data to be processed. This length limit should be less than the maximum length limit of the prediction model. The specific length can be set according to actual needs or experience, and is not specifically limited here.

[0046] If the length of the data to be processed exceeds the length limit, the data needs to be truncated. Specifically, for each sub-data, semantic truncation is performed based on the length of the sub-data, the length of the data to be processed, and the length limit imposed by the prediction model. This yields the truncated data, which may include truncation prompts, a first truncated response, and a second truncated response.

[0047] The truncated suggestion is the data obtained after semantically truncating the suggestion to be processed. The first truncated answer is the data obtained after semantically truncating the first answer. The second truncated answer is the data obtained after semantically truncating the second answer. The truncated suggestion, the first truncated answer, and the second truncated answer can constitute the truncated data.

[0048] As one implementation method, to avoid missing important content in the truncated sub-data, the available length of each sub-data can be calculated, and different methods can be used to truncate the sub-data according to its type, prioritizing the retention of important content. Specifically, during semantic truncation processing, for each sub-data, the ratio of its length to the length of the data to be processed can be multiplied by the specified length limit to calculate the available length of the sub-data. If the length of the sub-data is greater than its available length and the sub-data is a prompt to be processed, the prompt to be processed is truncated based on its available length and a preset rule to obtain a truncated prompt. If the length of the sub-data is greater than its available length and the sub-data is answer data, candidate clauses are determined from the answer data using its available length, and a truncated answer corresponding to the answer data is generated based on the candidate clauses. The answer data includes a first answer and a second answer, and the truncated answer includes a first truncated answer and a second truncated answer.

[0049] See also Figure 3 This diagram illustrates semantic truncation processing. For each sub-data element, its available length can be calculated using its length, the length of the data to be processed, and the length limit. For example, the length ratio of the sub-data element to the data to be processed can be calculated; that is, the length ratio is obtained by dividing the sub-data element's length by the length of the data to be processed, and then multiplying the length ratio by the length limit to obtain the available length of the sub-data element. Figure 3 If the length of the pending prompt is denoted as s1, and the available length of the pending prompt is denoted as s2; the length of the first answer is denoted as s3, then the available length of the first answer is denoted as s4; and the length of the second answer is denoted as s5, then the available length of the second answer is denoted as s6.

[0050] The sub-data may include pending prompts, first answers, and second answers. As one implementation, if the sub-data is a pending prompt and the length of the pending prompt is greater than its corresponding available length, i.e., s1>s2, the available length of data can be directly extracted from the pending prompt as the truncated prompt word.

[0051] As another implementation, since the pending prompt usually contains the task's objective, in order to avoid ambiguity of the task objective, the pending prompt can be truncated based on the available length of the pending prompt and preset rules to obtain a truncated prompt.

[0052] Optionally, when obtaining a truncated prompt by truncating the pending prompt, the available length of the pending prompt may be divided into a first length and a second length, the sum of the first length and the second length being the available length; the content of the first length of the pending prompt is taken as the first truncated content; the content of the second length of the pending prompt is taken as the second truncated content; and the first truncated content and the second truncated content are concatenated to obtain the truncated prompt.

[0053] The beginning of a pending prompt typically includes the problem background, while the end typically includes a concluding requirement; these two parts should be retained first. Specifically, the available length s2 of the pending prompt can be divided into a first length s21 and a second length s22. The first length s1 can be considered the length of the content to be retained in the beginning of the pending prompt, and the second length s2 can be considered the length of the content to be retained in the end of the pending prompt. The sum of the first and second lengths is the available length of the pending prompt, which can be set according to actual needs. In this embodiment of the invention, the first and second lengths can be equal.

[0054] Take the first length of the content before the prompt to be processed as the first truncated content, take the second length of the content after the prompt to be processed as the second truncated content, and then concatenate the first truncated content and the second truncated content to obtain the truncated prompt.

[0055] If the sub-data is response data and the response data exceeds its corresponding available length, then the response data needs to be truncated. Truncating the response data can be done by using the available length of the response data to determine candidate clauses, and then using these candidate clauses to generate the truncated response corresponding to the response data. Wherein, the response data includes a first response and a second response, then the truncated response includes the corresponding first truncated response and the second truncated response.

[0056] Optionally, when semantically truncating the answer data to obtain a truncated answer, the answer data may be split into multiple clauses, wherein each clause has a clause number that represents the order of the clause in the answer data; the semantic similarity between each clause and the prompt to be processed is calculated; based on the semantic similarity and the available length of the answer data, candidate clauses are determined from the multiple clauses; and the candidate clauses are sorted according to the clause number to obtain the truncated answer.

[0057] For the response data, it can be broken down into a series of clauses according to punctuation marks, and each clause can be assigned a corresponding clause number. The clause number indicates the order of the clauses in the response data. For example, the first clause in the response data could be numbered 1, the second clause 2, and so on. For example, the response data could be: To reduce carbon emissions, the following measures are recommended: 1. Promote renewable energy (such as solar and wind power) to reduce dependence on fossil fuels; 2. Implement carbon tax policies to curb high-carbon emission behaviors through economic levers; 3. Develop carbon capture technologies to offset industrial emissions. Furthermore, international cooperation should be strengthened to develop unified emission reduction standards. The split clauses can be: Clause 1: "Promote renewable energy (such as solar and wind power) to reduce dependence on fossil fuels" (ID=1); Clause 2: "Implement carbon tax policies to curb high carbon emission behavior through economic levers" (ID=2); Clause 3: "Develop carbon capture technology to offset industrial emissions" (ID=3); Clause 4: "Strengthen international cooperation and formulate unified emission reduction standards" (ID=4), where ID is the clause number.

[0058] To prioritize retaining clauses with high semantic relevance to the prompt to be processed, the semantic similarity between each clause and the prompt can be calculated. Then, based on the semantic similarity and the available length of the answer data, candidate clauses are determined from multiple clauses. Optionally, all clauses can be sorted in descending order of semantic similarity to obtain a clause sequence. An estimated length is obtained by adding the length of a specified clause to the current length of all candidate clauses in the candidate list, where the specified clause is the clause that appears first in the current clause sequence. If the estimated length is not greater than the available length of the answer data, the specified clause is added to the candidate list as a candidate clause and removed from the clause sequence until the estimated length is greater than the available length of the answer data, thus obtaining the candidate clauses in the candidate list.

[0059] It should be noted that higher semantic similarity indicates a higher relevance between the clause and the prompt to be processed. Therefore, clauses can be directly arranged in descending order of semantic similarity to obtain a clause sequence. A candidate list is pre-set to store candidate clauses. When determining candidate clauses, the clause at the beginning of the current clause sequence can be used as the specified clause. Then, the sum of the lengths of all candidate clauses in the current candidate list and the length of the specified clause is calculated as the estimated length. If the estimated length is not greater than the available length of the answer data, it can be considered that adding the specified clause will still not exceed the available length of the answer data, and the specified clause can be directly moved from the clause sequence to the candidate list, i.e., it is used as a candidate clause. The above process is repeated until the estimated length is greater than the available length of the answer data, indicating that adding the specified clause will exceed the available length. At this point, all candidate clauses in the candidate list can be directly obtained.

[0060] After identifying candidate clauses, they can be sorted based on their clause numbers to restore the natural word order and logical coherence of the text. For example, candidate clauses can be arranged in ascending order of their clause numbers, and then the sorted clauses can be joined together to form a first and second truncated response that retains the core information while meeting the length limit.

[0061] The aforementioned truncation methods for pending prompts and response data ensure that, while meeting the length limit, the data retains crucial contextual information and semantically relevant content essential for preference judgment. This provides higher-quality input for subsequent preference prediction models, improving prediction accuracy and consequently enhancing model comparison accuracy.

[0062] S130. Based on the answer difference data between the first answer and the second answer and the truncated data, generate the first data and the second data.

[0063] The first and second answers are responses obtained by different models for the same prompt. These two answers typically differ in some ways. To help the model more accurately capture these specific differences, answer difference data between the first and second answers can be introduced as auxiliary input. This answer difference data can include both similarity differences and a difference summary.

[0064] By concatenating the difference data from the responses and the truncated data, and adjusting the positions of the first and second truncated responses in the concatenated data, we can obtain the first and second data sets. In other words, the content of the first and second data sets is actually the same; only the positions of the first and second truncated responses differ, thus reducing the impact of the response positions on the prediction results.

[0065] Optionally, when generating the first and second data based on the answer difference data and the truncated data, the following steps can be taken: calculating the average similarity between all candidate clauses of the first answer and the prompt to be processed to obtain a first similarity; calculating the average similarity between all candidate clauses of the second answer and the prompt to be processed to obtain a second similarity; identifying the differences between the first and second answers to obtain a difference summary; using the first similarity, the second similarity, and the difference summary as answer difference data; and combining the truncated prompt, the first truncated answer, the second truncated answer, and the answer difference data in a specified order to generate the first and second data.

[0066] For example, see Figure 4 The diagram illustrates the generation of the first and second data. As described above, by truncating the first and second responses respectively, candidate clauses corresponding to the first and second responses were obtained. The similarity difference can be directly calculated using the candidate clauses corresponding to the two responses.

[0067] For each candidate clause corresponding to the first answer, the semantic similarity between each candidate clause and the prompt to be processed can be calculated, resulting in multiple first candidate similarities; the average of these multiple first candidate similarities is then calculated to obtain the first similarity. Similarly, for each candidate clause corresponding to the second answer, the semantic similarity between each candidate clause and the prompt to be processed can be calculated, resulting in multiple second candidate similarities; the average of these multiple second candidate similarities is then calculated to obtain the second similarity. The first similarity and the second similarity represent the similarity difference.

[0068] By identifying the differences between the first and second answers, a difference summary can be obtained. For example, a large language model can be used to identify key content in both answers, which may include key arguments, core entities, key values, etc., and then a short text summary can be generated to describe the differences between them. Examples include, "The first answer mentions price information, which the second answer does not," or "Argument x in the first answer contradicts argument y in the second answer." To avoid overly lengthy summaries, the conciseness of the difference summary can be ensured by controlling the maximum number of tokens in the output length.

[0069] The response difference data can include the first similarity, the second similarity, and the difference summary. At this point, you have obtained the response difference data, truncation prompts, the first truncated response, and the second truncated response. Combining these four parts in the specified order will give you the first data and the second data.

[0070] For example, the specified order can include a first order and a second order. The first order is truncated prompt, first truncated answer, second truncated answer, and answer difference data; the second order is truncated prompt, second truncated answer, first truncated answer, and answer difference data. Concatenating the data according to the first order yields the first data, and concatenating it according to the second order yields the second data. The first and second orders can be set according to actual needs, as long as the positions of the first and second answers in the first data are different from those in the second data, to reduce the interference of position on the prediction results.

[0071] S140. Use the preference prediction model to predict the first data and the second data to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model.

[0072] The preference prediction model is trained in advance based on a large amount of preference data. This model can be used to predict and compare two answers to determine the preference probability corresponding to each answer. The higher the preference probability, the more the answer matches the user's preference and the more acceptable it is to the user. In this case, the model that outputs that answer performs better. Thus, by comparing the preference probabilities corresponding to two answers, two language models can be compared.

[0073] When the first set of data is input into the preference prediction model, the model outputs a first prediction probability for the first language model and a second prediction probability for the second language model based on the first set of data. When the second set of data is input into the preference prediction model, the model outputs a third prediction probability for the first language model and a fourth prediction probability for the second language model based on the second set of data.

[0074] The first preference probability of the first language model is obtained by taking a weighted average of the first and third prediction probabilities corresponding to the first language model. Similarly, the second preference probability of the second language model is obtained by taking a weighted average of the second and fourth prediction probabilities corresponding to the second language model.

[0075] Before inputting the first and second data into the preference prediction model, a large amount of preference data is needed to train the base model to obtain the preference prediction model. Optionally, the preference prediction model can be obtained as follows: The base model is fine-tuned using real preference data to obtain a base prediction model. The target loss function of the fine-tuning process includes cross-entropy loss, entropy regularization, and comparison loss. The base model includes a base attention layer and a linear classification head. The real preference data includes real preference labels. Based on the prompts for the sample to be predicted and multiple candidate models, multiple prediction preference data are constructed. The prediction preference data includes prediction preference labels. The discrimination ability of the base prediction model is first enhanced using the prediction preference data to obtain an intermediate prediction model. The language ability of the intermediate prediction model is second enhanced using multilingual real preference data to obtain the preference prediction model. The learning rate of the second enhancement process on the base attention layer is lower than that of the first enhancement process, while the learning rate on the linear classification head is higher than that of the first enhancement process.

[0076] Preference data typically includes two responses from two models to the same prompt, along with preference labels. These labels are used to identify the response that is more aligned with human preferences. Based on the source of the preference labels, preference data can be divided into true preference data and predicted preference data. True preference data may contain true preference labels, which are annotated by humans; predicted preference data may contain predicted preference labels, which are predicted by the base prediction model.

[0077] The base model is obtained by modifying a pre-trained Large Language Model (LLM). The pre-trained LLM can be a high-performance, general-purpose large model suitable for fine-tuning in downstream tasks, serving as its backbone. To enable preference judgment, one or more fully connected layers can be added above the output representation of the last transformer block at the top of the LLM, forming a non-linear classification head using a non-linear activation function. The output layer dimension of this non-linear classification head can be set to n, corresponding to n output nodes, each representing the preference probability of one of the n models. The preference probability can be understood as the winning probability of each model in the comparison of the n models. n can be set according to actual needs; in this embodiment, n can be set to 2. Theoretically, the values ​​of these two preference probabilities should be between 0 and 1, and their sum should be close to 1. Modifying the general-purpose large language model in this way yields the base model, which can include a transformer block (basic attention layer) and a linear classification head.

[0078] Real preference data can be obtained from various public websites or through manual annotation. Fine-tuning the base model using real preference data yields a basic prediction model. Specifically, when fine-tuning the base model using real preference data, multiple real preference data sets can be obtained. These real preference data sets include sample prompts, a first sample answer generated by a first sample model based on the sample prompts, a second sample answer generated by a second sample model based on the sample prompts, sample answer difference data between the first and second sample answers, and real preference labels. Semantic truncation is performed on each real preference data set, and the positions of the first and second sample answers are swapped according to a specified probability, resulting in multiple preference data sets to be used. The base model is then used to predict these preference data sets, yielding the first sample preference probability of the first sample model. A target loss function is calculated based on the first sample preference probability and the real preference labels. When the target loss function converges, the basic prediction model is obtained.

[0079] The true preference data may include sample prompts, a first sample answer, a second sample answer, sample answer difference data, and true preference labels. The first sample answer is generated by the first sample model based on the sample prompts, and the second sample answer is generated by the second sample model based on the sample prompts. The sample answer difference data is generated in the same way as the aforementioned answer difference data, and can be referred to the corresponding description in the previous embodiments, which will not be repeated here.

[0080] For genuine preference data, semantic truncation processing is required as described above. The specific processing procedure can be found in the corresponding descriptions in the aforementioned embodiments. To ensure the model learns the quality of the content itself, rather than relying on the order of responses, the positions of the first and second sample responses can be swapped with a certain probability, thereby significantly reducing positional bias. The specified probability can be set according to actual needs and is not specifically limited here.

[0081] The true preference data after semantic truncation and exchange processing can be recorded as the preference data to be used. The preference data to be used is input into the basic model, which performs prediction processing and outputs the first sample preference probability corresponding to the first sample model.

[0082] The basic model is organized strictly according to the format of system prompts and user preference data. System prompts are a key component, customizable as needed, and aim to clearly instruct the basic model to act as an impartial judge, providing specific guidance on how to handle checks and directing the model to focus on content quality itself. In this embodiment, system prompts may include the following:

[0083] Act as an impartial evaluator to assess two AI responses to the user's query.Compare their quality based on:

[0084] • Instruction adherence

[0085] Question relevance

[0086] •Accuracy and depth

[0087] Creative problem-solving

[0088] Level of detail

[0089] Overall helpfulness

[0090] Analyze both responses objectively,disregarding presentation order,response length,and assistant identities.Provide a concise comparisonhighlighting key strengths / weaknesses,then conclude with your verdict using[[A]]or[[B]]for the superior response.

[0091] The base model outputs the first sample preference probability corresponding to the first sample answer and the second sample preference probability corresponding to the second sample answer. Using the first sample preference probability and the true preference label, a target loss function can be constructed, which may include cross-entropy loss, entropy regularization, and comparison loss.

[0092] The cross-entropy loss is the fundamental part of the objective loss function, ensuring that the basic prediction model can perform basic classification. Assuming there are N preference data points to be used in a training batch, the cross-entropy loss can be expressed as:

[0093]

[0094] Among them, L CEThe cross-entropy loss function is represented; y represents the true preference label, y∈{0,1}, 1 represents the first sample model winning, 0 represents the second sample model winning; p represents the winning probability of the first sample model predicted by the base model, i.e. the first sample preference probability; N represents the number of preference data to be used in the current training batch.

[0095] Entropy measures the uncertainty of a probability distribution. By penalizing high entropy—that is, when the model is uncertain about the prediction result and the probability distribution is relatively flat—we can encourage the model to generate sharper probability outputs with more pronounced peaks. Therefore, the objective loss function can also include entropy regularization, a penalty term related to the entropy of the model's output probability distribution. Entropy regularization can be expressed as follows:

[0096]

[0097] Among them, L entropy Entropy regularization is represented by p; p represents the winning probability of the first sample model predicted by the base model, i.e., the first sample preference probability.

[0098] Comparison loss originates from the match-matching model in statistics and is suitable for handling preference selection problems. It does not directly optimize the score or probability of a single answer, but rather focuses on optimizing for a significantly higher score for the winner than the loser. Specifically, it tends to maximize the sigmoid function that maximizes the difference between the scores. Compared to cross-entropy, comparison loss focuses more directly on the relative ranking relationship between sample pairs, helping the model to form a clearer boundary near the decision boundary, especially effectively distinguishing samples of similar quality that are difficult to judge, thereby improving the robustness and ranking consistency of the model. Comparison loss can be expressed by the following formula:

[0099] L BT =-logσ(s) pos -s neg );

[0100] Among them, s pos The sum of the first-sample preference probabilities of the data with a true preference label of 1 (the first-sample model wins) among the N unused preference data in the current training batch; s neg σ represents the sum of the first sample preference probabilities of the data whose true preference label is 0 (the second sample model wins) among N preference data to be used; σ is the Sigmoid function.

[0101] Therefore, the objective loss function can be expressed as:

[0102] L = L CE +λ1*L entropy +λ2*L BT ;

[0103] Where L represents the target loss function; L CE Characterizing cross-entropy loss; L entropy Characterization of entropy regularization; L BT The loss is represented by λ1. The initial value of λ1 can be set to a small value to avoid overly inhibiting model exploration. If there is a large amount of ambiguous data in the preference data to be used, such as two answers having similar quality, this parameter can be gradually increased. In this embodiment, the initial value of λ1 can be set to 0.1, and subsequently increased to 0.3. The initial value of λ2 can be set to a small value to emphasize the ability to distinguish data. If the validation set shows that the model performs poorly on difficult data, such as data predicted with low confidence, it can be gradually increased. In this embodiment, the initial value of λ2 can be set to 0.5, and subsequently gradually increased to 1.0.

[0104] When the target loss function converges, the basic prediction model can be obtained. This basic prediction model has the ability to make preliminary preference judgments and has a certain resistance to some biases.

[0105] Due to the limited amount and high cost of real preference data, to expand the dataset and improve the model's generalization ability and coverage of various scenarios, predicted preference data can be constructed based on the prompts for the samples to be predicted and multiple candidate models. The prompts for the samples to be predicted can include prompts in different language types to improve the model's multilingual capabilities. The predicted preference data can include predicted preference labels. In other words, the composition of the predicted preference data is consistent with that of the real preference data; the difference is that the real preference labels are obtained through user annotation, while the predicted preference labels are obtained through model prediction.

[0106] The sample to be predicted refers to existing large-scale, real, unlabeled user dialogue or question data, which can be obtained from existing publicly available datasets. To facilitate subsequent processing, the question data can be deduplicated and divided into English datasets and other multilingual datasets according to language type.

[0107] Multiple candidate models refer to multiple existing models that cover different developers, model sizes, etc., to ensure sufficient differences in capability and style. Predictive preference data can be constructed using the sample prompt to be predicted and the multiple candidate models. Optionally, for each candidate sample prompt, a first candidate model and a second candidate model can be determined from the multiple candidate models; the first candidate model and the second candidate model can be used to process the candidate sample prompt to obtain a first candidate sample answer and a second candidate sample answer; the candidate sample prompt, the first candidate sample answer, and the second candidate sample answer can be input into the base prediction model to obtain a first candidate preference probability and a second candidate preference probability; if the absolute value of the difference between the first candidate preference probability and the second candidate preference probability is greater than a specified value, a predicted preference label can be determined based on the first candidate preference probability and the second candidate preference probability; the candidate sample prompt, the first candidate sample answer, the second candidate sample answer, and the predicted preference label can be used as the predicted preference data.

[0108] To increase data diversity, each prediction prompt can be rewritten to generate multiple new prompts with similar semantics but different expressions, thus obtaining multiple candidate prompts. The rewriting process can utilize an existing large language model; the prediction prompt is input into the large language model, which outputs multiple rewritten new prompts. These new prompts, along with the prediction prompt, are then used as candidate prompts.

[0109] For each candidate sample suggestion, a first candidate model and a second candidate model can be determined from the candidate models. The first and second candidate models can be two models with significant differences among the candidate models, and the selection rules for the first and second candidate models can be set according to actual needs. For example, in this embodiment of the invention, when determining the first and second candidate models from multiple candidate models, the following steps can be taken: for each candidate model, obtain the model size and model source corresponding to the candidate model; based on the model size and a preset size range, determine the capability level of each candidate model; randomly select one candidate model from the candidate models as the first candidate model, and obtain the first capability level and first model source corresponding to the first candidate model; and determine any candidate model among the remaining candidate models excluding the first candidate model whose capability level is not the first capability level and whose model source is not the first model source as the second candidate model.

[0110] For each candidate model, its corresponding model size and source can be obtained. Model size mainly refers to the parameter scale of the model, such as 10B, 1.5B, etc. Model source can include the model's developer, architecture, etc. Based on the model size, the capabilities of each candidate model can be roughly classified. Generally, the larger the parameter scale of a model, the stronger its capability. In this embodiment of the invention, preset size ranges corresponding to different capability levels can be pre-set, and the capability level of each candidate model is determined based on the preset size range in which the candidate model falls. The preset size range can be set according to actual needs. For example, a parameter scale of no more than 10B is defined as a weak model; a parameter model with a parameter scale greater than 10B but not greater than 100B is defined as a medium model; and a parameter scale greater than 100B is defined as a strong model.

[0111] Then, the manufacturer and architecture of each candidate model can be used as the source label. For each candidate sample suggestion, one can be randomly selected from multiple candidate models as the first candidate model, and the capability level corresponding to the first candidate model can be obtained as the first capability level. The source label of the first candidate model can also be obtained as the first model source. Then, from the remaining candidate models excluding the first candidate model, the candidate models whose capability level is not the first capability level and whose model source is not the first model source can be used as the second candidate models.

[0112] Optionally, the optional capability levels can be determined based on the first capability level. For example, if the first capability level is a medium model, then the optional capability levels are weak and strong models; conversely, if the first capability level is a strong model, then the optional capability levels are weak and medium models. Candidate models in the optional capability levels are filtered based on the source of the first model, eliminating those whose source differs from the first model's source. For ease of description, the remaining candidate models can be designated as intermediate models. Then, one is randomly selected from the intermediate models as the second candidate model. The first and second candidate models selected in this way differ in model size and source, and their responses to the same prompt may exhibit more significant quality or style differences. This results in a more informative model that can help the base prediction model learn from training samples with different styles, leading to a more accurate preference prediction model.

[0113] The candidate sample prompts are input into the first candidate model and the second candidate model respectively, so that the first candidate model and the second candidate model can process them to obtain the first candidate sample answer output by the first candidate model and the second candidate sample answer output by the second candidate model.

[0114] By combining the candidate sample prompts, the first candidate sample responses, and the second candidate sample responses in the manner described above for generating the first and second data, corresponding input data can be obtained. This input data is then fed into the basic prediction model to obtain the corresponding probability output by the model. This probability indicates which model's response better matches the user's preference, thus yielding the predicted preference label.

[0115] Since the basic prediction model is not a completely accurate model, its judgment on some data may be inaccurate. To improve the quality of the predicted preference data, low-confidence comparison results can be filtered out. According to the aforementioned description of the basic prediction model obtained through fine-tuning, the basic prediction model can predict the probability of the first candidate preference corresponding to the first candidate sample's answer, and the probability of the second candidate preference corresponding to the second candidate sample's answer. The absolute value of the difference between these two candidate preference probabilities is calculated. If this absolute value is not greater than a specified value, it indicates that the basic prediction model's judgment of the two answers is very close, with low discrimination. Such samples can be considered fuzzy, may contain noise, and need to be discarded. The specified value can be set according to actual needs and is not specifically limited here. In this embodiment of the invention, the specified value can be set to 0.03.

[0116] If the absolute value is greater than a specified value, the discrimination is considered high, and the predicted preference label can be determined based on the probability of the first candidate preference and the probability of the second candidate preference. Similarly, the constructed input data can be divided into first input data and second input data. Similar to the first data and second data mentioned above, the predicted preference label can be determined based on the corresponding preference probability by fine-tuning the basic prediction model.

[0117] Optionally, to further improve the reliability of the predicted preference labels, multiple basic prediction models with different architectures or training batches can be pre-trained. These two basic prediction models are used to predict the same input data to obtain the predicted preference labels. Only the data with consistent preference labels between the two basic models are retained, thereby achieving cross-validation of the results and ensuring the reliability and accuracy of the predicted preference labels.

[0118] The candidate sample prompts, the first candidate sample answer, the second candidate sample answer, and the predicted preference label are used as the predicted preference data.

[0119] It should be noted that the process of constructing the predicted preference data described above can be performed offline in batches using optimized LLM inference frameworks and server engines. These frameworks support optimization techniques such as request batch processing, concurrent inference, and key-value caching, which can greatly improve inference throughput and the efficiency of constructing predicted preference data.

[0120] By using the predicted preference data to enhance the discrimination ability of the basic prediction model, an intermediate prediction model can be obtained. In the aforementioned predicted preference data, each predicted preference data point has an absolute value of the difference in preference probabilities. Since a larger absolute value indicates that the two candidate sample answers in the predicted preference data are easier to distinguish and have a higher discrimination, and a smaller absolute value indicates that the two candidate sample answers are harder to distinguish and have a lower discrimination, multiple predicted preference data points can be graded based on this absolute value, thus obtaining predicted preference data points with multiple levels of discrimination.

[0121] Optionally, if the absolute value is greater than a first specified value, it can be identified as low-difficulty data; if the absolute value is greater than a second specified value but not greater than the first specified value, it can be identified as medium-difficulty data, where the second specified value is less than the first specified value; if the absolute value is not greater than the second specified value, it can be identified as high-difficulty data. The first and second specified values ​​can be set according to actual needs. In this embodiment, the first specified value can be set to 0.8, and the second specified value can be set to 0.5. Then, the predicted preference data for each difficulty level are proportionally balanced to ensure that the final predicted preference data used for training includes data of each difficulty level, thus ensuring that the model can learn to process data of various difficulties and improve the discrimination and judgment ability of the preference prediction model.

[0122] When further fine-tuning the base prediction model, to avoid the inherent noise in the prediction preference data interfering with the model learning process and the problem of insufficient generalization ability in other languages ​​due to overfitting to a single language, a multi-stage fine-tuning process can be performed using both prediction preference data and true preference data. First, the discrimination ability of the base prediction model is enhanced using the prediction preference data to obtain an intermediate prediction model; then, the language ability of the intermediate prediction model is enhanced using multilingual true preference data to obtain the final preference prediction model. It is understood that the multilingual true preference data used here and the true preference data obtained by fine-tuning the base model can be different.

[0123] When obtaining the intermediate prediction model, the generated prediction preference data can be used to further fine-tune the basic prediction model. This allows the basic prediction model to learn general preference judgment patterns, strengthening its discriminative ability, and also exposing it to various different answer styles. Specifically, when fine-tuning the basic prediction model, it can be trained step by step according to the aforementioned difficulty levels, in the order of low, medium, and high difficulty, to obtain the intermediate prediction model.

[0124] The intermediate prediction model already possesses basic preference discrimination capabilities. A second enhancement of its language ability can be performed using some real preference data. For example, data from multilingual real preference datasets that do not include the specified language can be used to fine-tune the intermediate prediction model, ensuring that, under accurate supervision, the model's preference judgment accuracy in the specified language aligns with human preferences. Building on this, further fine-tuning can be performed using other real preference data besides the specified language, as well as a small amount of unused real preference data from the specified language, to improve the model's generalization ability in non-specified languages. This allows it to accurately judge the quality of responses in different language environments, thus obtaining the final preference prediction model.

[0125] The process of fine-tuning the prediction model from the basic model to the preference prediction model involved three stages. The first stage involved fine-tuning to obtain an intermediate prediction model. The second stage involved fine-tuning the intermediate prediction model using real preference data in a specified language. The third stage involved fine-tuning the final preference prediction model based on the second stage, using other languages ​​and a small amount of specified real preference data. The specified language can be set according to actual needs, such as English or Chinese.

[0126] Because the model may forget previously learned knowledge during multi-stage fine-tuning—a phenomenon known as catastrophic forgetting—different learning rates can be used at different training stages to mitigate these issues. For example, in stages 2 and 3 using multilingual real preference data, a smaller learning rate can be used for the base attention layer than in stage 1, while a smaller learning rate can be used for the linear classification head than in stage 1.

[0127] For example, the learning rate configuration for each stage in this invention can be found in Table 1:

[0128] Table 1

[0129] Basic attention layer Linear classification head Phase 1 1e-4 1e-4 Phase 2 2e-5 2e-4 Phase 3 2e-5 2e-4

[0130] During the fine-tuning phases 2 and 3, most parameters at the bottom layer of the model can be completely frozen (except for the last 8 Transformer blocks and the classification head). Only a few top layers and the final classification head are fine-tuned. This more strongly protects the core representation of the model from being compromised, allowing the model to focus on learning higher-level knowledge related to specific preference judgment patterns and language. By fine-tuning the basic prediction model in this way across multiple stages, a preference prediction model with good performance and accurate preference prediction can be obtained.

[0131] By fine-tuning guidance, optimizing the loss function, and employing a multi-stage fine-tuning strategy, the model's sensitivity to non-content factors such as answer position and length can be effectively suppressed. This makes the preference prediction model's predictions more reflective of the quality of the answer content itself, resulting in more accurate model comparisons and a better alignment with human preferences. Constructing and quality-controlling predicted preference data allows for the generation of large-scale training data at low cost and high efficiency. Furthermore, staged fine-tuning ensures the preference prediction model maintains its accuracy across multiple language environments. Using this preference prediction model for model comparisons improves the accuracy of the comparisons.

[0132] S150. Based on the first preference probability and the second preference probability, determine the comparison result of the first language model and the second language model.

[0133] The first preference probability can be understood as the probability that the first answer output by the first language model conforms to human preferences, and the second preference probability can be understood as the probability that the second answer output by the second language model conforms to human preferences. By comparing the first preference probability and the second preference probability, the comparison result of the first language model and the second language model can be obtained. For example, if the first preference probability is greater than the second preference probability, then the first language model is considered superior to the second language model. The preference-based language model comparison method provided in this embodiment of the invention can be used in various scenarios where it is necessary to compare the output effect and quality of models. For example, for a global intelligent customer service assistant agent, after the developer updates and iterates it, the agent before the update can be used as the first language model, and the agent after the update can be used as the second language model. By automatically calling the preference prediction model, the answer effect of the two versions can be compared. To ensure the accuracy of the comparison, multiple sets of different data to be processed can be provided. For each set of data to be processed, the preference prediction model can output the first preference probability and the second preference probability. The average of the first preference probability and the average of the second preference probability corresponding to multiple sets of data to be processed are compared to realize the comparison of the response effects of the old and new versions, and thus obtain the comparison results. This can greatly shorten the traditional testing time. Moreover, since the preference prediction model is trained in the above way, it can accurately compare the responses and output accurate comparison results.

[0134] Optionally, a preset threshold can be set in advance. When the average preference probability of the new version is less than the average preference probability of the old version, and the difference between the two is greater than the preset threshold, an alarm can be set directly.

[0135] The preference-based language model comparison scheme provided in this invention can be applied to various scenarios involving comparison of model output quality and effectiveness. For example, taking the iteration of new and old versions of a smart assistant as an example, the scheme provided in this invention can efficiently and accurately compare the response effects of two language models based on preferences using a preference prediction model, thereby obtaining the comparison results. This can replace the traditional method of testing two models, making it more efficient and accurate.

[0136] The preference-based model comparison method provided in this invention can train a preference prediction model through a specific training method. This preference prediction model is applicable to multilingual environments and can accurately predict preferences. It uses this preference prediction model to predict the preference probabilities of a first language model and a second language model, and determines the final comparison result based on the corresponding preference probabilities, making it more reliable and accurate. To better implement the above method, this invention also provides a preference-based language model comparison device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0137] For example, in this embodiment, the method of the present invention will be described in detail by taking the example of a preference-based language model comparison device specifically integrated into a server.

[0138] For example, such as Figure 5 As shown, the preference-based language model comparison device 200 may include an acquisition module 210, a truncation module 220, a generation module 230, a prediction module 240, and a comparison module 250, as follows:

[0139] The acquisition module 210 is used to acquire data to be processed, the data to be processed includes multiple sub-data, the sub-data includes a prompt to be processed, a first answer generated by a first language model based on the prompt to be processed, and a second answer generated by a second language model based on the prompt to be processed.

[0140] The truncation module 220 is used to perform semantic truncation processing on each of the sub-data based on the length of the sub-data, the length of the data to be processed, and the length limit of the data to be processed by the preference prediction model, to obtain truncated data, wherein the truncated data includes truncation prompts, a first truncated response, and a second truncated response.

[0141] The generation module 230 is used to generate first data and second data based on the answer difference data between the first answer and the second answer and the truncated data, wherein the positions of the first truncated answer and the second truncated answer in the first data are different from those in the second data.

[0142] Prediction module 240 is used to perform prediction processing on the first data and the second data using a preference prediction model to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model.

[0143] The comparison module 250 is used to determine the comparison result of the first language model and the second language model based on the first preference probability and the second preference probability.

[0144] In some embodiments, the truncation module 220 is specifically used for:

[0145] For each sub-data, the ratio of the length of the sub-data to the length of the data to be processed is multiplied by the limit length to calculate the available length of the sub-data;

[0146] If the length of the sub-data is greater than the available length of the sub-data and the sub-data is a pending prompt, the pending prompt is truncated based on the available length of the pending prompt and a preset rule to obtain a truncated prompt;

[0147] If the length of the sub-data is greater than the available length of the sub-data and the sub-data is answer data, candidate clauses are determined from the answer data using the available length of the answer data, and a truncated answer corresponding to the answer data is generated based on the candidate clauses. The answer data includes a first answer and a second answer, and the truncated answer includes a first truncated answer and a second truncated answer.

[0148] In some embodiments, the truncation module 220 is specifically used for:

[0149] The available length of the prompt to be processed is divided into a first length and a second length, and the sum of the first length and the second length is the available length;

[0150] The first length of the content before the prompt to be processed is taken as the first truncated content;

[0151] The second length of the content following the prompt to be processed is taken as the second truncated content;

[0152] By concatenating the first truncated content and the second truncated content, a truncation prompt is obtained.

[0153] In some embodiments, the truncation module 220 is specifically used for:

[0154] The answer data is split into multiple clauses, wherein each clause has a clause number that represents the order of the clauses in the answer data;

[0155] Calculate the semantic similarity between each clause and the prompt to be processed;

[0156] Based on the semantic similarity and the available length of the answer data, candidate clauses are determined from the plurality of clauses;

[0157] The candidate clauses are sorted according to their clause numbers to obtain a truncated answer.

[0158] In some embodiments, the generation module 230 is specifically used for:

[0159] Calculate the average similarity between all candidate clauses of the first answer and the prompt to be processed to obtain the first similarity;

[0160] Calculate the average similarity between all candidate clauses of the second answer and the prompt to be processed to obtain the second similarity;

[0161] Identify the differences between the first and second answers to obtain a difference summary;

[0162] The first similarity, the second similarity, and the difference summary are used as the answer difference data;

[0163] The truncated prompt, the first truncated answer, the second truncated answer, and the answer difference data are combined in a specified order to generate the first data and the second data.

[0164] In some embodiments, the preference-based language model comparison device 200 further includes a training module. Before using a preference prediction model to perform prediction processing on the first data and the second data to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model, the training module is specifically used for:

[0165] The base model is fine-tuned using real preference data to obtain a base prediction model. The target loss function of the fine-tuning process includes cross-entropy loss, entropy regularization, and comparison loss. The base model includes a base attention layer and a linear classification head. The real preference data includes real preference labels.

[0166] Based on the sample to be predicted and multiple candidate models, predictive preference data is constructed, which includes predictive preference labels.

[0167] The discrimination ability of the basic prediction model is enhanced by the prediction preference data to obtain an intermediate prediction model.

[0168] The language ability of the intermediate prediction model is enhanced by using multilingual real preference data to obtain a preference prediction model. The learning rate of the second enhancement process on the basic attention layer is less than that of the first enhancement process, but the learning rate on the linear classification head is greater than that of the first enhancement process.

[0169] In some embodiments, the training module is specifically used for:

[0170] Acquire multiple real preference data, which include sample prompts, a first sample answer generated by a first sample model based on the sample prompts, a second answer generated by a second sample model based on the sample prompts, sample answer difference data between the first sample answer and the second sample answer, and real preference labels;

[0171] Semantic truncation is performed on each of the real preference data, and the positions of the first sample answer and the second sample answer are swapped according to a specified probability to obtain multiple preference data to be used;

[0172] The basic model is used to predict the preference data to be used, and the first sample preference probability of the first sample model is obtained.

[0173] The target loss function is calculated based on the first sample preference probability and the true preference label;

[0174] When the target loss function converges, the basic prediction model is obtained.

[0175] In some embodiments, the training module is specifically used for:

[0176] For each of the predicted sample prompts, the predicted sample prompts are rewritten to obtain multiple candidate sample prompts;

[0177] For each candidate sample suggestion, a first candidate model and a second candidate model are determined from the plurality of candidate models;

[0178] The candidate sample prompts are processed using the first candidate model and the second candidate model respectively to obtain the first candidate sample answer and the second candidate sample answer;

[0179] The candidate sample prompts, the first candidate sample responses, and the second candidate sample responses are input into the basic prediction model to obtain the first candidate preference probability and the second candidate preference probability.

[0180] If the absolute value of the difference between the first candidate preference probability and the second candidate preference probability is greater than a specified value, the predicted preference label is determined based on the first candidate preference probability and the second candidate preference probability.

[0181] The candidate sample prompts, the first candidate sample responses, the second candidate sample responses, and the predicted preference labels are used as the predicted preference data.

[0182] In some embodiments, the training module is specifically used for:

[0183] For each candidate model, obtain the model size and model source corresponding to the candidate model;

[0184] Based on the model size and the preset size range, the capability level of each candidate model is determined;

[0185] Randomly select one candidate model from the candidate models as the first candidate model, and obtain the first capability level and the first model source corresponding to the first candidate model;

[0186] Any candidate model among the remaining candidate models excluding the first candidate model whose capability level is not the first capability level and whose model source is not the first model source is determined as the second candidate model.

[0187] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0188] As can be seen from the above, the preference-based language model comparison device of this embodiment can acquire data to be processed containing multiple sub-data. According to the length of the sub-data, the length of the data to be processed, and the restriction length of the preference prediction model, the sub-data is semantically truncated to ensure that the truncated data carries a high amount of effective information. Then, based on the truncated data and the answer difference data, the first data and the second data are generated. Introducing the answer difference data can help the model accurately understand the difference between the two answers. Moreover, the positions of the two answers in the first data are different from those in the second data, which can reduce the influence of the position of the answer on the result. After processing by the preference prediction model, the first preference probability of the first language model and the second preference probability of the second language model can be obtained. Finally, the comparison result is obtained directly based on the first preference probability and the second preference probability, which can accurately realize the model comparison.

[0189] This invention also provides an electronic device, which can be a terminal, a server, or other similar devices. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0190] In some embodiments, the preference-based language model comparison device can also be integrated into multiple electronic devices. For example, the preference-based language model comparison device can be integrated into multiple servers, and the preference-based language model comparison method of the present invention can be implemented by multiple servers.

[0191] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 6As shown, it illustrates a structural schematic diagram of the electronic device involved in an embodiment of the present invention, specifically:

[0192] The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will understand that... Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0193] The processor 310 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 320, and by calling data stored in the memory 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 310.

[0194] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0195] The electronic device also includes a power supply 330 that supplies power to the various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 330 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0196] The electronic device may also include an input module 340, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0197] The electronic device may also include a communication module 350. In some embodiments, the communication module 350 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 350 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0198] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 310 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 320 according to the following instructions, and the processor 310 runs the applications stored in the memory 320, thereby implementing the steps in the methods of the various embodiments of the present invention.

[0199] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0200] As can be seen from the above, the electronic device provided in this embodiment of the invention can acquire data to be processed containing multiple sub-data. According to the length of the sub-data, the length of the data to be processed, and the length limit of the preference prediction model to be processed, the sub-data is semantically truncated to ensure that the truncated data carries a high amount of effective information. Then, based on the truncated data and the answer difference data, first data and second data are generated. Introducing the answer difference data can help the model accurately understand the difference between the two answers. Moreover, the positions of the two answers in the first data are different from those in the second data, which can reduce the influence of the position of the answer on the result. After processing by the preference prediction model, the first preference probability of the first language model and the second preference probability of the second language model can be obtained. Finally, the comparison result is obtained directly based on the first preference probability and the second preference probability, which can accurately realize the model comparison.

[0201] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0202] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions which can be loaded by a processor to execute steps in any of the preference-based language model comparison methods provided in embodiments of the present invention.

[0203] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0204] According to one aspect of the present invention, a computer program product or computer program is provided, comprising a computer program / instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the methods provided in various optional implementations of the preference-based language model comparison aspect or preference prediction model training or preference prediction data construction aspect provided in the above embodiments.

[0205] Since the instructions stored in the storage medium can execute the steps in any of the preference-based language model comparison methods provided in the embodiments of the present invention, the beneficial effects that any of the preference-based language model comparison methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0206] The foregoing has provided a detailed description of a preference-based language model comparison method and apparatus provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A preference-based language model comparison method, characterized in that, The method includes: Acquire data to be processed, which includes multiple sub-data, including a prompt to be processed, a first answer generated by a first language model based on the prompt to be processed, and a second answer generated by a second language model based on the prompt to be processed. For each of the sub-data, based on the length of the sub-data, the length of the data to be processed, and the length limit of the data to be processed by the preference prediction model, the sub-data is semantically truncated to obtain truncated data, which includes truncated prompts, a first truncated response, and a second truncated response. Based on the answer difference data between the first answer and the second answer and the truncated data, first data and second data are generated, wherein the positions of the first truncated answer and the second truncated answer in the first data are different from those in the second data; The preference prediction model is used to predict the first data and the second data to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model. Based on the first preference probability and the second preference probability, the comparison result of the first language model and the second language model is determined.

2. The method according to claim 1, characterized in that, For each of the sub-data items, based on the length of the sub-data item, the length of the data to be processed, and the length limit of the data to be processed by the preference prediction model, semantic truncation is performed on the sub-data item to obtain truncated data, including: For each sub-data, the ratio of the length of the sub-data to the length of the data to be processed is multiplied by the limit length to calculate the available length of the sub-data; If the length of the sub-data is greater than the available length of the sub-data and the sub-data is a pending prompt, the pending prompt is truncated based on the available length of the pending prompt and a preset rule to obtain a truncated prompt; If the length of the sub-data is greater than the available length of the sub-data and the sub-data is answer data, candidate clauses are determined from the answer data using the available length of the answer data, and a truncated answer corresponding to the answer data is generated based on the candidate clauses. The answer data includes a first answer and a second answer, and the truncated answer includes a first truncated answer and a second truncated answer.

3. The method according to claim 2, characterized in that, The step of truncating the pending prompt based on its available length and a preset rule to obtain a truncated prompt includes: The available length of the prompt to be processed is divided into a first length and a second length, and the sum of the first length and the second length is the available length; The first length of the content before the prompt to be processed is taken as the first truncated content; The second length of the content following the prompt to be processed is taken as the second truncated content; By concatenating the first truncated content and the second truncated content, a truncation prompt is obtained.

4. The method according to claim 2, characterized in that, The step of determining candidate clauses from the answer data using the available length of the answer data, and generating a truncated answer corresponding to the answer data based on the candidate clauses, includes: The answer data is split into multiple clauses, wherein each clause has a clause number that represents the order of the clauses in the answer data; Calculate the semantic similarity between each clause and the prompt to be processed; Based on the semantic similarity and the available length of the answer data, candidate clauses are determined from the plurality of clauses; The candidate clauses are sorted according to their clause numbers to obtain a truncated answer.

5. The method according to claim 2, characterized in that, The process of generating first data and second data based on the answer difference data between the first and second answers and the truncated data includes: Calculate the average similarity between all candidate clauses of the first answer and the prompt to be processed to obtain the first similarity; Calculate the average similarity between all candidate clauses of the second answer and the prompt to be processed to obtain the second similarity; Identify the differences between the first and second answers to obtain a difference summary; The first similarity, the second similarity, and the difference summary are used as the answer difference data; The truncated prompt, the first truncated answer, the second truncated answer, and the answer difference data are combined in a specified order to generate the first data and the second data.

6. The method according to claim 1, characterized in that, Before using the preference prediction model to predict the first data and the second data to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model, the method further includes: The base model is fine-tuned using real preference data to obtain a base prediction model. The target loss function of the fine-tuning process includes cross-entropy loss, entropy regularization, and comparison loss. The base model includes a base attention layer and a linear classification head. The real preference data includes real preference labels. Based on the sample to be predicted and multiple candidate models, predictive preference data is constructed, which includes predictive preference labels. The discrimination ability of the basic prediction model is enhanced by the prediction preference data to obtain an intermediate prediction model. The language ability of the intermediate prediction model is enhanced by using multilingual real preference data to obtain a preference prediction model. The learning rate of the second enhancement process on the basic attention layer is less than that of the first enhancement process, but the learning rate on the linear classification head is greater than that of the first enhancement process.

7. The method according to claim 6, characterized in that, The process of fine-tuning the basic model using real preference data to obtain the basic prediction model includes: Acquire multiple real preference data, which include sample prompts, a first sample answer generated by a first sample model based on the sample prompts, a second answer generated by a second sample model based on the sample prompts, sample answer difference data between the first sample answer and the second sample answer, and real preference labels; Semantic truncation is performed on each of the real preference data, and the positions of the first sample answer and the second sample answer are swapped according to a specified probability to obtain multiple preference data to be used; The basic model is used to predict the preference data to be used, and the first sample preference probability of the first sample model is obtained. The target loss function is calculated based on the first sample preference probability and the true preference label; When the target loss function converges, the basic prediction model is obtained.

8. The method according to claim 6, characterized in that, The process of constructing prediction preference data based on the sample to be predicted and multiple candidate models includes: For each of the predicted sample prompts, the predicted sample prompts are rewritten to obtain multiple candidate sample prompts; For each candidate sample suggestion, a first candidate model and a second candidate model are determined from the plurality of candidate models; The candidate sample prompts are processed using the first candidate model and the second candidate model respectively to obtain the first candidate sample answer and the second candidate sample answer; The candidate sample prompts, the first candidate sample responses, and the second candidate sample responses are input into the basic prediction model to obtain the first candidate preference probability and the second candidate preference probability. If the absolute value of the difference between the first candidate preference probability and the second candidate preference probability is greater than a specified value, the predicted preference label is determined based on the first candidate preference probability and the second candidate preference probability. The candidate sample prompts, the first candidate sample responses, the second candidate sample responses, and the predicted preference labels are used as the predicted preference data.

9. The method according to claim 8, characterized in that, The step of determining the first candidate model and the second candidate model from the plurality of candidate models includes: For each candidate model, obtain the model size and model source corresponding to the candidate model; Based on the model size and the preset size range, the capability level of each candidate model is determined; Randomly select one candidate model from the candidate models as the first candidate model, and obtain the first capability level and the first model source corresponding to the first candidate model; Any candidate model among the remaining candidate models excluding the first candidate model whose capability level is not the first capability level and whose model source is not the first model source is determined as the second candidate model.

10. A preference-based language model comparison device, the device being used to implement the method as described in any one of claims 1-9, characterized in that, The device includes: The acquisition module is used to acquire data to be processed, which includes multiple sub-data, including a prompt to be processed, a first answer generated by a first language model based on the prompt to be processed, and a second answer generated by a second language model based on the prompt to be processed. The truncation module is used to perform semantic truncation processing on each of the sub-data based on the length of the sub-data, the length of the data to be processed, and the length limit of the data to be processed by the preference prediction model, to obtain truncated data, wherein the truncated data includes truncation prompts, a first truncated response, and a second truncated response. A generation module is used to generate first data and second data based on the answer difference data between the first answer and the second answer and the truncated data, wherein the positions of the first truncated answer and the second truncated answer in the first data are different from those in the second data. The prediction module is used to use a preference prediction model to predict the first data and the second data, and to obtain the first preference probability corresponding to the first language model and the second preference probability of the second language model. The comparison module is used to determine the comparison result of the first language model and the second language model based on the first preference probability and the second preference probability.

Images