Language model evaluation method, device, equipment, medium and product

Abstract

Disclosed are a method, apparatus, device, medium and product for language model evaluation, and relate to the fields of data processing technology, large model technology and large language model technology. The method includes inputting each question into multiple language models to be evaluated round by round to obtain multiple answers in each round. The method also includes obtaining multiple disordered answers in each round by performing disorder processing on multiple answers in each round of multiple language models to be evaluated. The method also includes evaluating multiple disordered answers in each round of multiple language models to be evaluated by an evaluation language model to obtain multiple scores in each round. The method also includes determining the total performance score of each language model to be evaluated based on the multiple scores of each round of multiple language models to be evaluated. In this way, the question-answering performance of each language model to be evaluated can be observed round by round, and the order of the answers in each round is disrupted, which can prevent the evaluation model used for scoring from generating bias.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of computers, and in particular to methods and apparatuses, devices, media, and products for language model evaluation. Background Art

[0002] In recent years, language models have developed rapidly and have become an influential and highly-watched technology. Large Language Model (LLM), as a representative of language models, is widely used for its powerful natural language understanding and generalization capabilities. One of the core functions of language models is to significantly simplify the previously complex and cumbersome knowledge acquisition process. Users can overcome the high knowledge barriers of the past and achieve functions equivalent to those difficult-to-use professional tools, thereby reducing the entry threshold and learning costs of knowledge.

[0003] The evaluation of the effect and capability of the language model is undoubtedly very important. By evaluating the model, we can predict its performance in actual applications and determine whether it can meet expectations and meet needs. With a scientific and comprehensive evaluation process, we can timely and accurately identify various problems in the language model that may cause performance degradation, and accordingly formulate and implement optimization strategies in a targeted manner to achieve optimal adjustment of the model and ensure that the model can run stably and efficiently in various application scenarios. Summary of the invention

[0004] An embodiment of the present disclosure provides a solution for language model evaluation.

[0005] In a first aspect of the present disclosure, a method for language model evaluation is provided, the method comprising obtaining multiple answers for each round of multiple language models to be evaluated by inputting each question in a question set into multiple language models to be evaluated in rounds. The method also comprises obtaining multiple random answers for each round of multiple language models to be evaluated by performing random processing on multiple answers for each round of multiple language models to be evaluated. The method also comprises evaluating multiple random answers for each round of multiple language models to be evaluated by an evaluation language model to obtain multiple scores for each round of multiple language models to be evaluated, and the evaluation language model is different from the multiple language models to be evaluated. The method also comprises determining the total performance score of each language model to be evaluated in the multiple language models to be evaluated based on the multiple scores for each round of the multiple language models to be evaluated.

[0006] In a second aspect of the present disclosure, a device for language model evaluation is provided, the device comprising an answer acquisition module, configured to obtain multiple answers for each round of multiple language models to be evaluated by inputting each question in a question set into multiple language models to be evaluated round by round. The device also comprises a shuffling module, configured to obtain multiple shuffled answers for each round of multiple language models to be evaluated by performing shuffling processing on multiple answers for each round of multiple language models to be evaluated. The device also comprises a first scoring module, configured to evaluate multiple shuffled answers for each round of multiple language models to be evaluated by an evaluation language model to obtain multiple scores for each round of multiple language models to be evaluated, the evaluation language model being different from the multiple language models to be evaluated. The device also comprises a second scoring module, configured to determine the total performance score of each language model to be evaluated in the multiple language models to be evaluated based on the multiple scores for each round of the multiple language models to be evaluated.

[0007] According to a third aspect of the present disclosure, an electronic device is provided. The computing device includes a processor and a memory, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor executes a method or process according to an embodiment of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a machine-readable storage medium is provided, wherein machine-executable instructions are stored on the machine-readable storage medium, and when the machine-executable instructions are executed by a processor, the processor is enabled to execute a method or process according to an embodiment of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable storage medium and includes a computer program, which, when executed by a processor of a computer, causes the processor to perform a method or process according to an embodiment of the present disclosure.

[0010] Please note that the summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the present disclosure will become more clearly understood by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a diagram schematically illustrating an example environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0013] Figure 2 is a flowchart schematically illustrating a method for language model evaluation according to an embodiment of the present disclosure;

[0014] Figure 3 A diagram schematically illustrates an example of an evaluation process based on a dual-evaluation language model according to an embodiment of the present disclosure;

[0015] Figure 4 A diagram schematically illustrates an example of an evaluation and annotation process according to an embodiment of the present disclosure;

[0016] Figure 5 A diagram schematically illustrates an example of a tuning process for a language model according to an embodiment of the present disclosure;

[0017] Figure 6 is a diagram schematically illustrating an apparatus for language model evaluation according to an embodiment of the present disclosure;

[0018] Figure 7 is a schematic block diagram of an example device that may be used to implement embodiments according to the present disclosure.

[0019] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly indicated to be different.

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] As mentioned above, it is necessary to carry out performance evaluation work on language models. One of the most common and time-consuming parts is to evaluate and compare the output results of multiple language models at the same time. By way of example, conversational robots (such as chat assistants, smart butlers, etc.) developed and built based on language models are common in people's daily lives. Conversational robots can be configured to respond to natural language commands or commands of users in the form of text or voice, and output response information in a corresponding form. For example, a conversational robot can communicate with users in natural language, reply to users' voice commands, etc. The performance of the language model is related to the quality of natural language interaction products such as the above-mentioned conversational robots.

[0027] Different language models often have different architectural designs, algorithmic logic, etc. By way of example, among multiple language models with different configurations, some models may focus on deep mining of semantic understanding, while some models may pay more attention to the fluency and diversity of generated text. For example, a language model may have a high accuracy rate when processing professional field texts, but its output content is a bit stiff. Another language model has fluent and easy-to-read output content, but there may be content that is irrelevant to the problem or degraded content that is not allowed to appear. For the same output problem, referencing and comparing the output results of multiple language models at the same time, and giving labels and ranking relationships to the output of each language model can help to gain insight into the current status of the model, thereby helping to improve the model or select the appropriate model to develop natural language interaction products.

[0028] Currently, some language model evaluation schemes rely heavily on manual work. However, during the manual evaluation of language models, the evaluator may subtly discover the model rules from historical evaluation cases, thereby developing a preference or bias for a specific model. Manual language model evaluation also has the defects of being inefficient and prone to errors. When the evaluator changes, the evaluation results lose stability. In addition, these language model evaluation schemes may not be able to make intuitive and detailed horizontal comparisons, and the display of results lacks diversity and interactivity.

[0029] In order to solve at least some of the above and other potential problems, an embodiment of the present disclosure proposes a language model evaluation scheme. The scheme includes inputting each question in the question set into multiple language models to be evaluated in rounds, respectively, to obtain multiple answers for each round of multiple language models to be evaluated. The scheme also includes obtaining multiple random answers for each round of multiple language models to be evaluated by performing random processing on multiple answers for each round of multiple language models to be evaluated. The scheme also includes evaluating multiple random answers for each round of multiple language models to be evaluated by an evaluation language model to obtain multiple scores for each round of multiple language models to be evaluated, and the evaluation language model is different from the multiple language models to be evaluated. The scheme also includes determining the total performance score of each language model to be evaluated in the multiple language models to be evaluated based on the multiple scores for each round of the multiple language models to be evaluated.

[0030] According to the language model evaluation scheme of the embodiment of the present disclosure, the question-answering performance of each language model to be evaluated can be observed round by round. During each round of answer evaluation, the order of output answers is disrupted, thereby preventing the evaluation model configured for scoring from generating preferences or biases, and ensuring the objectivity and fairness of the evaluation results. In addition, the high-quality output of each language model to be evaluated and the corresponding questions will be used to further tune the model and promote dynamic model enhancement.

[0031] Reference below Figures 1 to 7 It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0032] Figure 1 1 is a schematic diagram schematically illustrating an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1 As shown in , the example environment 100 includes a knowledge base 110, a computing system 120 including computing nodes (eg, locally arranged computing nodes 121 and 122 and a computing node 123 arranged in the cloud), and an evaluation result 130. Figure 1The implementation environment of the embodiment of the present disclosure is described by taking a hybrid computing system as an example. It should be understood that this is only an exemplary and non-restrictive description, and other different types of computing systems are also feasible, such as distributed or centralized computing systems. The appropriate computing environment configuration can be selected according to actual use needs.

[0033] exist Figure 1 In the figure, only limited components and exemplary connection relationships are shown. It should be understood that this is for the purpose of ease of explanation and ease of illustration and is not intended to limit the scope of the present disclosure, and other different components may also exist. For example, a display component and an input component, etc. In an exemplary and non-limiting manner, the evaluation result 130 of the language model to be evaluated, for example, output by the computing system 120, can be displayed on the display component, and the user's instructions and annotations, etc. can be input through the input component.

[0034] According to an embodiment of the present disclosure, the knowledge base 110 can store a question set. The question set includes multiple questions, and these questions cannot have problems such as unclear, subjective, ambiguous, unanswerable or wrongly labeled, because such degradation problems will cause distortion of the evaluation results. For example, if the language model gives a "wrong" answer to an extremely ambiguous question, then what should be questioned is the question itself, not the reasoning ability of the model. In some embodiments, the question set can be associated with a predetermined indication field, for example, the questions included in the question set are inquiries or questions in the direction of medical or education.

[0035] As described above, at least one computing node included in the computing system 120 can perform processing and operations corresponding to the language model evaluation according to the embodiment of the present disclosure. In an exemplary and non-restrictive manner, computing nodes 121 and 122 can be arranged locally at the user, and computing node 123 can be arranged in the cloud. The cloud can indicate a service model built based on cloud or distributed technology. In this mode, computing resources, storage resources, etc. are coupled together through a network to form a schedulable and scalable resource collection. At least a portion of these resources can be dynamically accessed or allocated to complete tasks without the need to locally own or manage these physical resources.

[0036] A computing node refers to a computing resource or computing power, and may be a device with computing power. For example, a computing node may be provided with a processor and a memory, etc., or may be equipped with a dedicated accelerator (such as a graphics processing unit (GPU)). In addition, a computing node may store and maintain data. In addition, the model involved in the language model evaluation scheme according to an embodiment of the present disclosure (for example, a plurality of language models to be evaluated and an evaluation language model for evaluating the plurality of language models to be evaluated) may be a model deployed on at least one of the computing nodes, and may also be a callable external model.

[0037] like Figure 1 As exemplarily shown in FIG. 1 , local computing nodes 121 and 122 and computing node 123 at the cloud can be interconnected via a network for communication. Figure 1 Taking the architecture in the example, computing nodes 121-123 can communicate via a network to achieve, for example, data synchronization or sharing between nodes. Multiple computing nodes in the example environment 100 can process computing tasks in parallel. In addition, multiple computing nodes in the example environment 100 can have redundancy and fault tolerance mechanisms to ensure reliable execution of computing tasks. When a computing node fails, the system can automatically migrate the job to other nodes that are working normally to ensure the continuity and availability of the task.

[0038] Examples of computing nodes may include supercomputers, personal computers, laptop computers, vehicle-mounted computing devices, mobile devices (such as smart phones, tablet computers, etc.), wearable electronic devices, multimedia devices, personal digital assistants (PDAs), smart home devices, consumer electronic products, or a combination of any one or more of the above devices, etc. It should be understood that the computing nodes described herein are merely exemplary and non-restrictive, and for example, other different types of computing nodes may also be used.

[0039] Combined with the above Figure 1 Describes an example environment in which the methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 The method 200 for language model evaluation according to an embodiment of the present disclosure is described. Through the method 200, the order of answers in each round is disrupted, which can prevent the evaluation model used for scoring from generating bias, and high-quality output will be used to further tune the model.

[0040] Figure 22 is a flow chart schematically illustrating a method 200 for language model evaluation according to an embodiment of the present disclosure. At 210, multiple answers of each round of multiple language models to be evaluated are obtained by inputting each question in the question set into multiple language models to be evaluated round by round. In one round, one question in the multiple questions is input into multiple language models to be evaluated respectively to obtain multiple answers corresponding to the question from multiple language models to be evaluated respectively. Multiple rounds are iterated until the last question of the multiple questions is input into multiple language models to be evaluated respectively.

[0041] For example, in the first round, the first question among the multiple questions is input into the language model A to be evaluated, the language model B to be evaluated, and the language model C to be evaluated, respectively, to obtain the answer to the first question (i.e., answer A1) of the language model A to be evaluated, the answer to the first question (i.e., answer B1) of the language model B to be evaluated, and the answer to the first question (i.e., answer C1) of the language model C to be evaluated. The answer acquisition process will be iterated until all questions are input into the position.

[0042] At 220, multiple random answers for each round of multiple language models to be evaluated are obtained by performing random processing on multiple answers for each round of multiple language models to be evaluated. In one round, the order of multiple answers from multiple language models to be evaluated is disrupted to prevent the evaluator from discovering the model regularity. For example, in the first round, the order of the obtained answers A1, B1, and C1 is changed to B1, A1, and C1. It should be understood that the random processing of each round is different (e.g., random). For example, the random results of the first round are B1, A1, and C1, while the random results of the second round are C2, B2, and A2. In some embodiments, random processing of multiple answers for each round can be achieved by introducing random numbers.

[0043] At 230, the evaluation language model evaluates the multiple out-of-order answers of each round of the multiple language models to be evaluated to obtain multiple scores of each round of the multiple language models to be evaluated, and the evaluation language model is different from the multiple language models to be evaluated. According to an embodiment of the present disclosure, the evaluation language model can be a fully trained and available language model, and is configured to evaluate the multiple answers of each round of the multiple language models to be evaluated. For example, for the multiple out-of-order answers B1, A1 and C1 of the first round, the evaluation language model gives corresponding scores S_B1, S_A1 and S_C1 respectively. The evaluation language model evaluates the multiple out-of-order answers from the multiple language models to be evaluated round by round until the scoring of the multiple out-of-order answers of the last round is completed.

[0044] In some embodiments, the evaluation language model used to evaluate out-of-order answers can be trained based on world knowledge, common sense, etc., or can be trained based on business line knowledge, specific domain knowledge, etc. Based on the degree of training of the evaluation language model on the domain knowledge of the predetermined domain and / or the accuracy of the response to the query or question of the predetermined domain, the domain adaptability of the evaluation language model for the predetermined domain can be determined. In addition, the scoring process through the evaluation language model can be based on the answer similarity, answer degradation, answer generation efficiency, etc. of each answer in each round relative to the true value answer of the question in that round. In the following, the evaluation process according to the embodiments of the present disclosure will be further described in detail.

[0045] At 240, based on the multiple scores of each round of the multiple language models to be evaluated, the total performance score of each language model to be evaluated in the multiple language models to be evaluated is determined. After obtaining the multiple scores of each round, the question-answering performances of the multiple language models to be evaluated in each round are integrated to determine the total performance score of each language model to be evaluated in the multiple language models to be evaluated. In the following, the above-mentioned various processes according to the embodiments of the present disclosure will be further described in detail.

[0046] According to the method 200 for evaluating a language model of an embodiment of the present disclosure, the question-answering performance of each language model to be evaluated can be observed round by round. During each round of answer evaluation, the order of output answers is disrupted, thereby preventing the evaluation model configured for scoring from generating preferences or biases, and ensuring the objectivity and fairness of the evaluation results. In addition, the high-quality output of each language model to be evaluated and the corresponding questions will be used to further tune the model and promote dynamic enhancement of the model.

[0047] Figure 3 A diagram schematically illustrates an example 300 of an evaluation process based on a dual evaluation language model according to an embodiment of the present disclosure. As described above, a fully trained evaluation language model is used to evaluate multiple out-of-order answers for each round of multiple language models to be evaluated to replace manual work. According to an embodiment of the present disclosure, two such evaluation language models can be selected to evaluate multiple out-of-order answers for each round of multiple language models to be evaluated. In this way, the evaluation results output by the two evaluation language models can verify each other. It should be understood that the embodiment of the present disclosure can also use evaluation experts and evaluation language models to perform evaluations separately, and further compare the differences between the evaluation results of the two to achieve verification.

[0048] According to an embodiment of the present disclosure, the first evaluation language model 310 scores the multiple random answers of each round of the multiple language models to be evaluated to obtain the first multiple scores 315 of each round of the multiple language models to be evaluated, and the second evaluation language model 320 scores the multiple random answers of each round of the multiple language models to be evaluated to obtain the second multiple scores 325 of each round of the multiple language models to be evaluated. After obtaining the first multiple scores 315 and the second multiple scores 325, the difference between the first multiple scores 315 and the second multiple scores 325 can be determined.

[0049] For ease of illustration and understanding, Figure 3 The description and explanation are based on a limited number of language models to be evaluated and a limited number of rounds. It should be understood that the embodiments of the present disclosure may also include a larger number of language models to be evaluated and iterate more rounds. Figure 3 As shown, for the first round (i.e., the first question in the question set), the answers output by the multiple language models to be evaluated are randomly ordered as B1, A1, and C1, and the first evaluation language model 310 can score the multiple random answers in the first round as S1_B1, S1_A1, and S1_C1, and the second evaluation language model 320 can score the multiple random answers in the first round as S2_B1, S2_A1, and S2_C1. For the second round (i.e., the second question in the question set), the answers output by the multiple language models to be evaluated are randomly ordered as C2, B2, and A2, and the first evaluation language model 310 can score the multiple random answers in the second round as S1_C2, S1_B2, and S1_A2, and the second evaluation language model 320 can score the multiple random answers in the second round as S2_C2, S2_B2, and S2_A2.

[0050] It should be understood that the shuffling process is intended to avoid the bias that may exist in the evaluation language model (e.g., the first evaluation language model 310 and the second evaluation language model 320) during the evaluation process. After the evaluation language model completes the evaluation, the correspondence between the multiple answers of each round and the multiple language models to be evaluated can be determined (e.g., based on the naming tags), or the original order of the shuffled answers can be restored. In some embodiments, a sequence recovery process is performed on the multiple shuffled scores of each round to determine that each of the multiple shuffled scores of each round belongs to a corresponding language model to be evaluated in the multiple language models to be evaluated. The sequence recovery process may include the inverse process of the above-mentioned shuffling process that introduces random numbers.

[0051] According to an embodiment of the present disclosure, in order to determine the difference between the first plurality of scores 315 and the second plurality of scores 325, the difference measure of the round can be obtained by calculating the difference between the corresponding scores of each round and averaging them. The difference between the first plurality of scores 315 and the second plurality of scores 325 can be indicated by combining the difference measures of multiple rounds. For example, for the second round, the difference between S1_C2 and S2_C2, the difference between S1_B2 and S2_B2, and the difference between S1_A2 and S2_A2 are calculated and averaged to obtain the difference measure of the round. If the difference measure of a round is too large, it can be indicated that the evaluation of multiple answers to the round is invalid.

[0052] As described above, the question set may correspond to a predetermined domain, and the evaluation language model (e.g., the first evaluation language model 310 and the second evaluation language model 320) may have domain adaptability for the predetermined domain. According to an embodiment of the present disclosure, the first evaluation language model 310 and the second evaluation language model 320 may be assigned a first weight and a second weight, respectively, based on their domain adaptability relative to the predetermined domain. In the case where it is determined that the difference between the first plurality of scores 315 and the second plurality of scores 325 is less than or equal to the difference threshold, that is, the evaluation results of the two evaluation language models are verified to be valid, the performance total score of each of the multiple language models to be evaluated may be calculated based on the first plurality of scores 315 and the first weight and the second plurality of scores 325 and the second weight, for example, by weighted averaging. In addition to weighted averaging, a voting mechanism or other data fusion algorithm may also be used. If the difference between the first plurality of scores 315 and the second plurality of scores 325 is greater than the difference threshold, the evaluation results of the two evaluation language models are verified to be invalid.

[0053] Figure 4 A diagram schematically illustrates an example 400 of an evaluation annotation process according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, during the evaluation, for multiple out-of-order answers in each round of multiple language models to be evaluated, the answer degradation and evaluation opinions for each out-of-order answer are identified. In addition to scoring, the evaluation language model can identify whether each answer has degradation, and can determine the degradation level of the identified degradation. In addition, the evaluation language model can generate evaluation opinions such as natural language descriptions for each answer.

[0054] According to an embodiment of the present disclosure, in response to a corresponding round of multiple rounds being selected by a user, questions and multiple answers of the corresponding rounds of multiple language models to be evaluated, as well as multiple scores, may be displayed to the user. Figure 4As shown in , a user such as a reviewer can select a corresponding round (or a total score column) in box 410, and the scores of multiple answers in this round, answer degradation annotations, and evaluation opinions can be presented to the user. In addition, the questions in this round and multiple answers from multiple language models to be evaluated can also be presented to the user.

[0055] Figure 5 A diagram schematically illustrates an example 500 of a tuning process for a language model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, among the multiple answers in each round of the multiple language models to be evaluated (e.g., the first evaluation language model 502, the second evaluation language model 504, etc.), a target answer whose answer satisfaction meets a satisfaction threshold can be determined. The evaluation language model 506 can be adjusted based on the target answer and the associated question.

[0056] like Figure 5 As shown in , among the output answers of the first evaluation language model 502, the second evaluation language model 504, etc., at least one answer that satisfies the answer satisfaction and meets the satisfaction threshold is determined as the target answer, and this process is based on, for example, the accuracy of the answer, whether it contains degradation information, etc. These target answers are attached with corresponding questions to form target question-answer pairs, such as Figure 5 The first question-answer pair, the second question-answer pair, etc. shown in . The target question-answer pair will be stored in the knowledge base and can be used to further tune the evaluation language model. Alternatively, it can also be used to further tune multiple language models to be evaluated. It should be understood that for ease of understanding and illustration, Figure 5 Only a limited number of models and question-answer pairs are shown in FIG. , which is merely exemplary and not limiting.

[0057] Figure 6 Schematically illustrates a device 600 for language model evaluation according to an embodiment of the present disclosure, and the device 600 may include multiple units or modules for performing the steps or actions in the method or process discussed above. Figure 6As shown in , the device 600 includes: an answer acquisition module 610, which is configured to obtain multiple answers for each round of the multiple language models to be evaluated by inputting each question in the question set into multiple language models to be evaluated round by round; a shuffling module 620, which is configured to obtain multiple shuffled answers for each round of the multiple language models to be evaluated by performing shuffling processing on the multiple answers for each round of the multiple language models to be evaluated; a first scoring module 630, which is configured to evaluate the multiple shuffled answers for each round of the multiple language models to be evaluated by an evaluation language model to obtain multiple scores for each round of the multiple language models to be evaluated, and the evaluation language model is different from the multiple language models to be evaluated; and a second scoring module 640, which is configured to determine the total performance score of each language model to be evaluated in the multiple language models to be evaluated based on the multiple scores for each round of the multiple language models to be evaluated.

[0058] In some embodiments, wherein the multiple scores of each round of the multiple language models to be evaluated are multiple out-of-order scores, the apparatus 600 further includes a sequence recovery module configured to perform sequence recovery processing on the multiple out-of-order scores of each round to determine that each out-of-order score of the multiple out-of-order scores of each round belongs to a corresponding language model to be evaluated in the multiple language models to be evaluated.

[0059] In some embodiments, wherein the evaluation language model comprises a first evaluation language model and a second evaluation language model, the first scoring module is further configured to: score the multiple out-of-order answers of each round of the multiple language models to be evaluated by the first evaluation language model to obtain a first plurality of scores of each round of the multiple language models to be evaluated; score the multiple out-of-order answers of each round of the multiple language models to be evaluated by the second evaluation language model to obtain a second plurality of scores of each round of the multiple language models to be evaluated; and determine the difference between the first plurality of scores and the second plurality of scores.

[0060] In some embodiments, the question set is associated with a predetermined knowledge domain, and the first evaluation language model and the second evaluation language model are assigned a first weight and a second weight, respectively, based on domain adaptability relative to the predetermined knowledge domain.

[0061] In some embodiments, the second scoring module 640 is further configured to: in response to determining that the difference between the first multiple scores and the second multiple scores is less than or equal to a difference threshold, calculate the total performance score based on the first multiple scores and the first weights and the second multiple scores and the second weights.

[0062] In some embodiments, the apparatus 600 further comprises an identification module configured to identify, during the evaluation, for the multiple out-of-order answers of each round of the multiple language models to be evaluated, the answer degradation and evaluation opinion for each out-of-order answer.

[0063] In some embodiments, the apparatus 600 further comprises a display module configured to display questions and multiple answers, as well as multiple scores of the corresponding rounds of the multiple language models to be evaluated to the user in response to a corresponding round of the multiple rounds being selected by the user.

[0064] In some embodiments, the device 600 also includes a tuning module, which is configured to determine a target answer whose answer satisfaction meets a satisfaction threshold among the multiple answers in each round of the multiple language models to be evaluated; and adjust the evaluation language model based on the target answer and the corresponding question.

[0065] In some embodiments, the multiple true answers to the question include: a first true answer in the form of an SQL statement of the first type, the first true answer corresponding to the first predicted answer and coming from the first SQ source, the first SQ source being configured to provide annotated SQL statements of the first type; a second true answer in the form of an visualization URL, the second true answer corresponding to the second predicted answer and coming from the visualization URL source, the visualization URL source being configured to provide annotated visualization URLs; a third true answer in the form of an SQL statement of the second type, the third true answer corresponding to the third predicted answer and coming from the second SQ source, the second SQ source being configured to provide Providing annotated SQL statements of the second type; an answer in the form of a fourth true answer of the SQL execution result, the fourth true answer corresponding to the fourth predicted answer and coming from the SQL execution result source, the SQL execution result source being configured to provide annotated SQL execution results; an answer in the form of a fifth true answer of the recall field, the fifth true answer corresponding to the fifth predicted answer and coming from the recall field source, the recall field source being configured to provide annotated recall fields; and an answer in the form of a sixth true answer of the URL execution result, the sixth true answer corresponding to the sixth predicted answer and coming from the URL execution result source, the URL execution result source being configured to provide annotated URL execution results.

[0066] Figure 7 FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a coprocessor.

[0067] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0068] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0069] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0070] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0071] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0072] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0073] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0075] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0076] Various embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the various embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

1. A method for evaluating a language model, comprising: By inputting each question in the question set into the multiple language models to be evaluated round by round, multiple answers for each round of the multiple language models to be evaluated are obtained; Obtaining multiple out-of-order answers for each round of the multiple language models to be evaluated by performing out-of-order processing on the multiple answers for each round of the multiple language models to be evaluated; The multiple out-of-order answers of each round of the multiple language models to be evaluated are evaluated by an evaluation language model to obtain multiple scores of each round of the multiple language models to be evaluated, wherein the evaluation language model is different from the multiple language models to be evaluated; as well as determining a total performance score of each of the multiple language models to be evaluated based on the multiple scores of each round of the multiple language models to be evaluated, The evaluation language model includes a first evaluation language model and a second evaluation language model, and the evaluation includes: Scoring the plurality of out-of-order answers of each round of the plurality of language models to be evaluated by the first evaluation language model to obtain a plurality of first scores of each round of the plurality of language models to be evaluated; Scoring the plurality of out-of-order answers of each round of the plurality of language models to be evaluated by the second evaluation language model to obtain a second plurality of scores of each round of the plurality of language models to be evaluated; and A dissimilarity between the first plurality of scores and the second plurality of scores is determined.

2. The method according to claim 1, wherein the multiple scores of each round of the multiple language models to be evaluated are multiple out-of-order scores, and the method further comprises: An order recovery process is performed on the multiple random scores of each round to determine that each random score of the multiple random scores of each round belongs to a corresponding language model to be evaluated among the multiple language models to be evaluated.

3. The method according to claim 1, wherein: The set of questions is associated with a predetermined knowledge area, And the first evaluation language model and the second evaluation language model are assigned a first weight and a second weight, respectively, based on domain adaptability with respect to the predetermined knowledge domain.

4. The method of claim 3, wherein determining the overall performance score comprises: In response to determining that the difference between the first plurality of scores and the second plurality of scores is less than or equal to a difference threshold, the performance total score is calculated based on the first plurality of scores and the first weights and the second plurality of scores and the second weights.

5. The method according to claim 1, further comprising: During the evaluation, for the multiple out-of-order answers of each round of the multiple language models to be evaluated, the answer degradation and evaluation opinion for each out-of-order answer are identified.

6. The method according to claim 1, further comprising: In response to a corresponding round of the plurality of rounds being selected by a user, questions and a plurality of answers of the corresponding round of the plurality of language models to be evaluated, and a plurality of scores are displayed to the user.

7. The method according to claim 1, further comprising: Determine, among the multiple answers in each round of the multiple language models to be evaluated, a target answer whose answer satisfaction meets a satisfaction threshold; as well as Based on the target answer and the corresponding question, the evaluation language model is adjusted.

8. A language model evaluation device, comprising: An answer acquisition module is configured to obtain multiple answers of each round of the multiple language models to be evaluated by inputting each question in the question set into the multiple language models to be evaluated round by round; A shuffling module configured to obtain a plurality of shuffled answers for each round of the plurality of language models to be evaluated by performing shuffling processing on the plurality of answers for each round of the plurality of language models to be evaluated; A first scoring module is configured to evaluate the multiple out-of-order answers of each round of the multiple language models to be evaluated by an evaluation language model to obtain multiple scores of each round of the multiple language models to be evaluated, wherein the evaluation language model is different from the multiple language models to be evaluated; as well as A second scoring module is configured to determine a total performance score of each of the multiple language models to be evaluated based on the multiple scores of each round of the multiple language models to be evaluated, The evaluation language model includes a first evaluation language model and a second evaluation language model, and the first scoring module is further configured as follows: Scoring the plurality of out-of-order answers of each round of the plurality of language models to be evaluated by the first evaluation language model to obtain a plurality of first scores of each round of the plurality of language models to be evaluated; Scoring the plurality of disordered answers of each round of the plurality of language models to be evaluated by the second evaluation language model to obtain a second plurality of scores of each round of the plurality of language models to be evaluated; as well as A dissimilarity between the first plurality of scores and the second plurality of scores is determined.

9. An electronic device, comprising: processor; as well as A memory coupled to the processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 7.

10. A machine-readable storage medium having machine-executable instructions stored thereon, wherein the machine-executable instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program which, when executed by a processor of a computer, causes the processor to perform the method according to any one of claims 1 to 7.

Images