Language model evaluation method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a language model evaluation method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of obtaining at least one question answer pair in a preset vertical field; the question-answer pairs comprise questions and answers corresponding to the questions; performing anthropomorphic processing on the target question through a natural language processing model to obtain at least one question text of the target question in different question modes; the target question is a question in any question answer pair; questioning a question answering model corresponding to a preset vertical field based on the at least one question text, and obtaining at least one reply correspondingly output by the question answering model; and performing quality evaluation on the question answering model according to the answer corresponding to the target question and the at least one reply to obtain a quality evaluation result of the question answering model. Therefore, the efficiency and accuracy of evaluating the question answering model can be effectively improved, and meanwhile, the question answering model can be comprehensively evaluated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a language model evaluation method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the popularization of large artificial intelligence models, different companies or enterprises have begun to develop question-solving models in vertical fields, and each question-solving model has a unique knowledge base and functions.

[0003] In order to verify the accuracy and robustness of the question answering model, it is necessary to evaluate the question answering model. However, the existing language model evaluation methods have shortcomings such as low evaluation efficiency and accuracy of the question answering model, and the inability to comprehensively evaluate the question answering model. Summary of the invention

[0004] The purpose of this application is to provide a language model evaluation method, device, electronic device and computer-readable storage medium, which can effectively improve the efficiency and accuracy of question answering model evaluation, and at the same time realize comprehensive evaluation of question answering model.

[0005] To achieve the above objectives:

[0006] In a first aspect, an embodiment of the present application provides a language model evaluation method, comprising:

[0007] Obtain at least one question-answer pair in a preset vertical field; the question-answer pair includes a question and an answer corresponding to the question;

[0008] Anthropomorphize the target question through a natural language processing model to obtain at least one question text of the target question in different questioning modes; the target question is a question in any of the question-answer pairs;

[0009] Based on the at least one question text, a question is asked to the question answering model corresponding to the preset vertical field, and at least one reply corresponding to the output of the question answering model is obtained;

[0010] The quality of the question answering model is evaluated according to the answer corresponding to the target question and the at least one reply to obtain a quality evaluation result of the question answering model.

[0011] Optionally, obtaining at least one question-answer pair in a preset vertical field includes:

[0012] A preset knowledge base in a preset vertical field is identified by using a regular expression to obtain at least one question-answer pair in the preset vertical field; the preset knowledge base is constructed based on preset knowledge points and different questions in the preset vertical field.

[0013] Optionally, performing quality assessment on the question answering model according to the answer corresponding to the target question and the at least one reply to obtain a quality assessment result of the question answering model includes:

[0014] Based on the pre-trained model, semantic understanding and similarity comparison are performed on the answer corresponding to the target question and the at least one reply to obtain a preset quality assessment index value for each reply;

[0015] A quality assessment result of the question answering model is generated based on the preset quality assessment index value of each of the responses.

[0016] Optionally, the performing semantic understanding and similarity comparison on the answer corresponding to the target question and the at least one reply based on the pre-trained model to obtain a preset quality assessment index value for each reply includes:

[0017] Performing word segmentation processing on the answer corresponding to the target question and the at least one reply respectively, to obtain at least one first word block constituting the answer corresponding to the target question and at least one second word block constituting each of the replies;

[0018] Performing semantic understanding processing on the at least one first word block and the at least one second word block respectively through a pre-trained model to obtain vector representations of each of the first word blocks and vector representations of each of the second word blocks respectively containing contextual semantic information;

[0019] The similarity between the vector representation of each of the first word blocks and the vector representation of each of the second word blocks is calculated, and a preset quality assessment index value of each of the replies is determined based on the similarity.

[0020] Optionally, the calculating the similarity between the vector representation of each of the first word chunks and the vector representation of each of the second word chunks, and determining a preset quality assessment index value of each of the replies according to the similarity, includes:

[0021] Calculate the cosine similarity between the vector representation of each of the first word chunks and the vector representation of each of the second word chunks to obtain a similarity matrix between the answer corresponding to the target question and the target response; the target response is any of the responses;

[0022] Based on the similarity matrix, the maximum similarity scores of the answer corresponding to the target question and the target reply are respectively accumulated and normalized to obtain a preset quality evaluation index value of the target reply.

[0023] Optionally, the preset quality assessment indicators include recall rate, precision rate and F1 score.

[0024] Optionally, the quality assessment result includes a quality score; and generating the quality assessment result of the question answering model according to the preset quality assessment index value of each of the responses includes:

[0025] Determine the variance of the recall rate, the variance of the precision rate, the variance of the F1 score, and the average recall rate, the average precision rate, and the average F1 score according to the preset quality assessment indicator value of each of the responses;

[0026] The variance of the recall rate, the variance of the precision rate, the variance of the F1 score, the average recall rate, the average precision rate and the average F1 score are respectively summed with the corresponding preset weights to determine the quality score of the question answering model.

[0027] In a second aspect, an embodiment of the present application provides a language model evaluation device, the device comprising: an acquisition module, a processing module, a questioning module and an evaluation module; wherein:

[0028] The acquisition module is used to acquire at least one question-answer pair in a preset vertical field; the question-answer pair includes a question and an answer corresponding to the question;

[0029] The processing module is used to perform anthropomorphic processing on the target question through a natural language processing model to obtain at least one question text of the target question in different questioning modes; the target question is a question in any of the question-answer pairs;

[0030] The questioning module is used to ask a question to the question answering model corresponding to the preset vertical field based on the at least one question text, and obtain at least one reply outputted by the question answering model;

[0031] The evaluation module is used to perform a quality evaluation on the question answering model according to the answer corresponding to the target question and the at least one reply, so as to obtain a quality evaluation result of the question answering model.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing a computer program, wherein when the processor runs the computer program, the above-mentioned language model evaluation method is implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned language model evaluation method is implemented.

[0034] The language model evaluation method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application include: obtaining at least one question-answer pair in a preset vertical field; the question-answer pair includes a question and an answer corresponding to the question; anthropomorphizing the target question through a natural language processing model to obtain at least one question text of the target question under different questioning methods; the target question is the question in any of the question-answer pairs; based on the at least one question text, asking a question to a question answering model corresponding to the preset vertical field to obtain at least one reply corresponding to the output of the question answering model; performing quality evaluation on the question answering model according to the answer corresponding to the target question and the at least one reply to obtain a quality evaluation result of the question answering model. In this way, after extracting at least one question-answer pair from a preset vertical field, a natural language processing model is used to generate at least one question text of the question in the question-answer pair under different questioning methods, that is, multiple questions with semantics similar to the question in the question-answer pair, so as to achieve exponential growth of test questions and ensure full coverage of potential problem scenarios of the model. Then, based on the response of the question answering model in the preset vertical field to at least one question text and the answer in the question-answer pair, the quality of the question answering model is evaluated, and a quality evaluation result that can quantitatively analyze the performance of the model is obtained, which can effectively improve the efficiency and accuracy of the question answering model evaluation, and at the same time, a comprehensive evaluation of the question answering model can be achieved. At the same time, through automated evaluation operations, manual intervention can be reduced, labor costs can be reduced, and human errors can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of a language model evaluation method provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a specific flow chart of a language model evaluation method provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of the structure of a language model evaluation device provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0039] In the figure: processor 310, memory 311, network interface 312, bus system 313. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0041] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0042] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0043] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0044] It should be noted that, in this article, step codes such as S101, S102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.

[0045] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0047] See also Figure 1 , is a language model evaluation method provided in an embodiment of the present application. The language model evaluation method can be executed by a language model evaluation device provided in an embodiment of the present application. The language model evaluation device can be implemented in software and / or hardware, such as an electronic device such as a vehicle-mounted terminal or a computer. In this embodiment, the execution subject of the language model evaluation method is an electronic device as an example. The language model evaluation method provided in this embodiment includes:

[0048] Step S101: Obtain at least one question-answer pair in a preset vertical field; the question-answer pair includes a question and an answer corresponding to the question.

[0049] Among them, the preset vertical fields can be set according to actual needs, and can specifically be sub-fields within various industries such as the vehicle field, tax field, financial field, financial field, and financial field. Each question-answer pair includes a question and the answer corresponding to the question, wherein the answer can be considered as a reference or standard answer corresponding to the question. Taking the preset vertical field as the vehicle field as an example, the question-answer pair can be "How to turn on the comfort mode" and "Click the button on the upper left corner of the screen", etc. Here, the method of obtaining at least one question-answer pair in the preset vertical field can be directly selecting a set question-answer pair from the preset vertical field, or extracting the question-answer pair from a preset knowledge base in the preset vertical field.

[0050] In one embodiment, obtaining at least one question-answer pair in a preset vertical field includes:

[0051] The preset knowledge base in the preset vertical field is identified by regular expressions to obtain at least one question-answer pair in the preset vertical field; the preset knowledge base is constructed based on preset knowledge points and different questions in the preset vertical field.

[0052] It can be understood that since the preset knowledge base is constructed based on the preset knowledge points and different questions in the preset vertical field, and the questions are actually related to the knowledge points, the preset knowledge base in the preset vertical field can be identified by regular expressions to obtain at least one question-answer pair in the preset vertical field. It should be noted that in actual applications, questions and titles can be extracted from multiple knowledge base documents in the preset vertical field, and the corresponding answers can be generated in combination with the explanations and processing steps in the knowledge base. Then, the questions and answers are saved one by one to generate a database corresponding to the questions and answers, and then the question-answer pairs can be directly extracted from the database. In this way, the automatic extraction of question-answer pairs can be achieved, further improving the efficiency and accuracy of the evaluation of the question-answering model.

[0053] Step S102: anthropomorphize the target question through a natural language processing model to obtain at least one question text of the target question in different questioning modes; the target question is a question in any question-answer pair.

[0054] Specifically, for the question in each question-answer pair, namely the target question, the target question is anthropomorphized through a natural language processing model to change the expression form of the target question and obtain at least one question text of the target question under different questioning methods.

[0055] Among them, the natural language processing model can also be called a general large model, which can simulate the way users ask questions in different situations, thereby generating multiple similar questions with the same semantics as the target question. In practical applications, the natural language processing model can specifically be ChatGPT (Chat Generative Pre-trained Transformer), or ChatGLM (ChatGenerative Language Model), or Baichuan (Baichuan Large Model), or ChatGLM2 (ChatGenerative Language Model Version2), etc., which are not specifically limited here. For example, taking the target question as "How to turn on the comfort mode" as an example, the target question is anthropomorphized by the natural language processing model, and question texts such as "How to adjust to the comfort mode" and "How to enter the comfort mode" can be obtained.

[0056] It can be understood that by anthropomorphizing the target question through the natural language processing model, the expression form of the target question can be changed, and at least one question text of the target question under different questioning methods can be obtained, so that the generated question text is closer to the diversity of natural language, thereby expanding the coverage of questions, ensuring that the question answering model can accurately understand and handle various variant questions in practical applications, and improving the robustness and adaptability of the model. It should be noted that each question text in this embodiment can be understood as a question, and the question is semantically similar or identical to the target question.

[0057] Step S103: Ask a question to a question answering model corresponding to a preset vertical field based on at least one question text, and obtain at least one reply outputted by the question answering model.

[0058] Specifically, the interface of the question answering model corresponding to the preset vertical field is called, and for at least one question text under different questioning methods for each target question, the at least one question text is input into the question answering model corresponding to the preset vertical field, so as to ask questions to the question answering model corresponding to the preset vertical field, and obtain at least one reply output by the question answering model corresponding to the at least one question text.

[0059] Here, the question answering model can also be called a private large model, which is a model whose performance is to be evaluated and is used to answer questions to be answered in a preset vertical field. For each question text, the question answering model will output a response.

[0060] Step S104: Perform a quality assessment on the question answering model according to the answer and at least one reply corresponding to the target question to obtain a quality assessment result of the question answering model.

[0061] Specifically, according to the answer corresponding to each target question and at least one reply corresponding to at least one question text of each target question under different questioning methods, a quality assessment is performed on the question answering model in a preset vertical field to obtain a quality assessment result of the question answering model.

[0062] It can be understood that when the answer corresponding to the target question is known, since at least one reply is generated by the question answering model by answering at least one semantically similar question text obtained by anthropomorphizing the target question with the natural language processing model, the answer corresponding to the target question can be regarded as a reference answer corresponding to the at least one question text, and thus the quality of the question answering model can be evaluated based on the answer corresponding to the target question and the at least one reply to obtain a quality evaluation result of the question answering model.

[0063] Among them, the quality assessment result of the question answering model is used to characterize the performance of the question answering model, and can be specifically represented or quantified through indicators such as the quality score of the question answering model.

[0064] In one embodiment, the quality of the question answering model is evaluated according to the answer and at least one reply corresponding to the target question to obtain the quality evaluation result of the question answering model, including:

[0065] Based on the pre-trained model, semantic understanding and similarity comparison are performed on the answer corresponding to the target question and at least one reply to obtain a preset quality assessment index value for each reply;

[0066] The quality assessment results of the question answering model are generated based on the preset quality assessment indicator values ​​of each response.

[0067] Specifically, for each target question, for at least one reply corresponding to at least one question text under different questioning methods, a semantic understanding and similarity comparison are performed on the answer corresponding to the target question and the at least one reply based on the pre-trained model to obtain a preset quality assessment index value for each reply, and then based on the preset quality assessment index value for each reply, a quality assessment result of the question answering model is generated.

[0068] Among them, the pre-trained model has the functions of semantic understanding and similarity comparison of text. In this embodiment, the pre-trained model is taken as an existing BERT model as an example. The BERT model is a pre-trained language model that understands text through a bidirectional Transformer encoder, thereby achieving remarkable results in various natural language processing tasks.

[0069] It can be understood that since at least one reply is generated based on the question answering model to answer the question text with semantics similar to the target question, in theory, the semantics of the at least one reply and the answer corresponding to the target question should be the same or close. Therefore, the answer corresponding to the target question and at least one reply can be semantically understood and compared for similarity based on the pre-trained model to obtain a preset quality evaluation index value for each reply. Among them, the preset quality evaluation index value can be used to characterize the semantic matching degree between the reply based on the question answering model to answer the question text with semantics similar to the target question and the answer corresponding to the target question. After obtaining the preset quality evaluation index value of each reply, the preset quality evaluation index value of each reply can be statistically analyzed to determine the quality evaluation result of the question answering model, that is, to determine whether the question answering model can accurately understand and answer the question. In this way, the efficiency and accuracy of the question answering model evaluation can be further improved, and the comprehensive evaluation of the question answering model can be achieved.

[0070] In one embodiment, semantic understanding and similarity comparison are performed on the answer corresponding to the target question and at least one reply based on the pre-trained model to obtain a preset quality assessment index value for each reply, including:

[0071] Performing word segmentation processing on the answer corresponding to the target question and at least one reply respectively, to obtain at least one first word block constituting the answer corresponding to the target question and at least one second word block constituting each reply;

[0072] Performing semantic understanding processing on at least one first word block and at least one second word block respectively through the pre-trained model to obtain vector representations of each first word block and each second word block respectively containing contextual semantic information;

[0073] The similarity between the vector representation of each first word block and the vector representation of each second word block is calculated, and a preset quality assessment index value of each reply is determined according to the similarity.

[0074] Specifically, for at least one reply corresponding to at least one question text of each target question under different questioning methods, the answer and at least one reply corresponding to the target question are firstly segmented to obtain at least one first word block constituting the answer corresponding to the target question and at least one second word block constituting each reply; then, semantic understanding processing is performed on the at least one first word block and the at least one second word block through a pre-trained model to obtain vector representations of each first word block respectively containing contextual semantic information and vector representations of each second word block respectively containing contextual semantic information; then, the similarity between the vector representation of each first word block and the vector representation of each second word block is calculated, that is, the similarity between each reply and the answer corresponding to the target question is calculated, and the preset quality assessment index value of each reply is determined according to the calculated similarity.

[0075] Among them, the pre-trained model has strong context understanding and deep representation capabilities, and can accurately obtain the deep semantic information of each reply, which can facilitate the accurate calculation of the similarity of the vector representation of different word blocks. In this way, semantic understanding and similarity comparison through the pre-trained model can reduce the influence of subjective judgment and improve the consistency and reliability of the preset quality evaluation index values ​​of each reply.

[0076] In one embodiment, calculating the similarity between the vector representation of each first word block and the vector representation of each second word block, and determining the preset quality evaluation index value of each reply according to the similarity, includes:

[0077] Calculate the cosine similarity between the vector representation of each first word block and the vector representation of each second word block to obtain a similarity matrix between the answer corresponding to the target question and the target response; the target response is any response;

[0078] Based on the similarity matrix, the maximum similarity scores of the answer to the target question and the target response are accumulated and normalized to obtain the preset quality evaluation index value of the target response.

[0079] Specifically, for any reply, namely the target reply, the cosine similarity between the vector representation of at least one first word block that constitutes the answer corresponding to the target question and the vector representation of at least one second word block that constitutes the target reply is calculated, and based on the cosine similarity between the vector representation of each first word block and the vector representation of each second word block, a similarity matrix of the answer corresponding to the target question and the target reply is constructed; then, based on the similarity matrix, the maximum similarity scores of the answer corresponding to the target question and the target reply are respectively accumulated, and then normalized to obtain a preset quality assessment index value of the target reply.

[0080] In this way, by calculating the similarity matrix between the different word blocks that make up the answer and the reply and determining the preset quality evaluation index value of the reply, it is possible to accurately evaluate the matching degree between each reply and the corresponding answer, which can further improve the accuracy of the evaluation of the question answering model.

[0081] In one embodiment, the preset quality assessment indicators include recall, precision and F1 score. The recall rate indicates the proportion of the content of the answer corresponding to the question and the reply that matches. The precision rate indicates the proportion of the content in the reply that matches the answer corresponding to the question. The F1 score is the harmonic mean of the recall rate and the precision rate.

[0082] In one embodiment, the quality assessment result includes a quality score; and the quality assessment result of the question answering model is generated according to the preset quality assessment index value of each response, including:

[0083] According to the preset quality assessment indicator values ​​of each response, the variance of the recall rate, the variance of the precision rate, the variance of the F1 score, and the average recall rate, the average precision rate, and the average F1 score are determined;

[0084] The variance of the recall rate, the variance of the precision rate, the variance of the F1 score, the average recall rate, the average precision rate and the average F1 score are respectively summed with the corresponding preset weights to determine the quality score of the question answering model.

[0085] Specifically, based on the recall rate, precision and F1 score of each response, calculate and determine the variance of the recall rate, the variance of the precision rate, the variance of the F1 score, and the average recall rate, the average precision rate and the average F1 score of all responses; then, the variance of the recall rate, the variance of the precision rate, the variance of the F1 score, the average recall rate, the average precision rate and the average F1 score are respectively summed with the corresponding preset weights to determine the quality score of the question answering model.

[0086] Among them, the preset weights corresponding to the variance of recall rate, the variance of precision rate, the variance of F1 score, the average recall rate, the average precision rate and the average F1 score can be set based on actual needs, and the sum of the preset weights corresponding to the variance of recall rate, the variance of precision rate, the variance of F1 score, the average recall rate, the average precision rate and the average F1 score is 1. In this way, by quantifying the quality evaluation results of the question answering model with specific indicators, it is convenient to know the performance of the question answering model, and then it is convenient to optimize the question answering model.

[0087] It should be noted that each step in the language model evaluation method provided in this embodiment can be implemented through an automated script to reduce manual intervention.

[0088] In summary, in the language model evaluation method provided by the above embodiment, after extracting at least one question-answer pair from a preset vertical field, a natural language processing model is used to generate at least one question text of the question in the question-answer pair under different questioning methods, that is, multiple questions with semantics similar to the question in the question-answer pair, so as to achieve exponential growth of test questions and ensure comprehensive coverage of potential problem scenarios of the model. Then, based on the response of the question answering model in the preset vertical field to at least one question text and the answer in the question-answer pair, the quality of the question answering model is evaluated, and a quality evaluation result that can be used to quantitatively analyze the performance of the model is obtained, which can effectively improve the efficiency and accuracy of the evaluation of the question answering model, and at the same time, a comprehensive evaluation of the question answering model can be achieved. In addition, through automated evaluation operations, manual intervention can be reduced, labor costs can be reduced, and human errors can be reduced.

[0089] Based on the same inventive concept as the aforementioned embodiment, the aforementioned embodiment is described in detail below through a specific example. In this example, the aforementioned question answering model is taken as a private large model, and the pre-trained model includes a preprocessing model and a text evaluation model.

[0090] With the popularity of AI big models, different companies have begun to develop private big models in vertical fields, each of which has a unique knowledge base and functions. In order to verify the accuracy and robustness of these models, they usually need to be tested. However, traditional testing methods usually rely on manual input verification results, which has the following shortcomings:

[0091] 1) There are many resources in the private knowledge base, and the manual testing method is slow and inefficient, which makes it difficult to meet the needs of rapid iteration of model development and limits the speed and frequency of model optimization.

[0092] 2) Traditional testing methods make it difficult to conduct quantitative analysis of test results, lack clear indicators to evaluate model performance, and make it difficult to provide targeted improvement suggestions.

[0093] 3) There is a lack of diversity in the way questions are asked about private large models.

[0094] 4) The knowledge base and functions of private large models will be constantly updated, and manual testing methods need to be retested when the model is updated, increasing maintenance costs and workload.

[0095] 5) Manual testing methods may not be able to fully cover all functions and scenarios of the model, and may easily miss some potential problems, resulting in poor performance of the model in practical applications.

[0096] In view of the above problems, this embodiment provides a language model evaluation method, such as Figure 2 As shown, it mainly includes the following processes:

[0097] Step S201: extract questions and answers from knowledge base documents through regular expression matching.

[0098] Here, regular expressions can be used to extract questions and titles from multiple knowledge base documents of a private large model, and corresponding answers can be generated in combination with the explanations and processing steps in the knowledge base. The questions and answers are then saved one by one to generate a database corresponding to the questions and answers.

[0099] Step S202: anthropomorphize the extracted questions using a natural language processing model to generate diversified question texts.

[0100] It can be understood that by changing the expression form of the question to make it closer to the diversity of natural language, the test coverage can be expanded to ensure that the private large model can accurately understand and handle various variant problems in practical applications, and improve the robustness and adaptability of the model. After reading the question from the summarized knowledge base, the interface of the natural language processing model is called to generate multiple anthropomorphic questions with the same semantics, that is, to simulate the user's questioning methods in different situations and generate questions in various scenarios, such as short sentences, long sentences, questions with modal particles, and questions with emotional colors, fuzzy questions, colloquial expressions, industry terms, regional expressions, multi-step questions, and multiple options.

[0101] Step S203: Ask questions to the private big model based on the question text and obtain a reply from the private big model.

[0102] Specifically, all the questions extracted and generated (including question texts) are called through the interface of the private big model to obtain the replies of the private big model, and these replies are written into the question answer library.

[0103] Step S204: Based on the preprocessing model and the text evaluation model, the reply of the private large model and the extracted answer are compared in terms of semantics and similarity to obtain a comparison result.

[0104] Specifically, the performance of the private big model is evaluated by comparing the semantics and similarity of the private big model's replies with the extracted answers, thereby determining whether the private big model can accurately understand and answer the questions, and then the quality of the private big model can be scored.

[0105] First, based on the powerful context understanding and deep representation capabilities of the preprocessing model in the pretrained model, the replies of the private large model and the answers extracted from the knowledge base can be processed for context understanding. Here, these answers and replies are first divided into tokens, and then these tokens are converted into word vectors. Subsequently, these word vectors are processed by the preprocessing model to generate context-related representations of each token, thereby obtaining deep semantic information of the answers and replies.

[0106] Next, the text evaluation model is used to calculate the similarity between the generated text (i.e., the reply) and the reference text (i.e., the answer). Here, the text evaluation model evaluates the similarity of the text by calculating R (Recall), P (Precision), and F (F1 Score). All three values ​​are in the range of [0,1], where 1 represents a perfect match and 0 represents a complete mismatch. R is used to evaluate the proportion of content in the generated text that matches the reference text, P is used to evaluate the proportion of content in the generated text that matches the reference text, and F represents the harmonic mean of P and R, which is used to balance precision and recall and provide a single performance measurement indicator.

[0107] Step S205: Obtain a score for the private large model based on the comparison result.

[0108] Here, after obtaining the comparison results of all replies and answers of the private big model, a set of weight parameters are used to comprehensively calculate various evaluation indicators to obtain the score S of the private big model. The calculation formula is as follows:

[0109] S=α·Pavg+β·Ravg+γ·Favg+δ·Var(P)+∈·Var(R)+ζ·Var(F)

[0110] Among them, Pavg is the average precision value of all replies and answers, Ravg is the average recall value of all replies and answers, Favg is the average F1 score value of all replies and answers, Var(P) is the variance of the precision value, Var(R) is the variance of the recall value, Var(F) is the variance of the F1 score value, α, β, γ, δ, ∈, ζ are the weights of different indicators, and satisfy α+β+γ+δ+∈+ζ=1.

[0111] In summary, the language model evaluation method provided in the above embodiment has the following advantages: 1) The whole process uses automated scripts, that is, automated scripts are used from question-answer pair extraction, question derivation and semantic comparison, which reduces manual intervention and improves test efficiency and data extraction accuracy. At the same time, by executing the automated script, not only can the newly added knowledge base be tested, but also the knowledge already stored can be quickly regressed, the impact of the newly added knowledge on the existing knowledge can be quickly discovered, and timely feedback can be provided for processing. 2) The natural language processing model is used to generate multiple questions with semantics similar to the original question, so that the number of test questions can be increased exponentially, ensuring that the potential problem scenarios of the model are fully covered, solving the problem of insufficient diversity of manually input questions in traditional methods, and improving the coverage of test questions. 3) The pre-trained model is used for semantic comparison, and the influence of subjective judgment is reduced through an objective scoring mechanism, thereby improving the consistency and reliability of the evaluation results. 4) The semantic comparison method is service-oriented, so that it can be easily called, which improves the test efficiency and synergy. 5) Quantify the evaluation results, provide specific scores and indicators through semantic comparison, quantify the evaluation results, evaluate the performance of the model through clear indicators, and provide clear improvement directions and effect evaluation.

[0112] Based on the same inventive concept as the above embodiments, the embodiment of the present invention provides a language model evaluation device, such as Figure 3 As shown, it includes: an acquisition module, a processing module, a questioning module and an evaluation module; wherein,

[0113] An acquisition module, used to acquire at least one question-answer pair in a preset vertical field; the question-answer pair includes a question and an answer corresponding to the question;

[0114] A processing module, used for anthropomorphizing a target question through a natural language processing model to obtain at least one question text of the target question in different questioning modes; the target question is a question in any question-answer pair;

[0115] A questioning module, used to ask a question to a question answering model corresponding to a preset vertical field based on at least one question text, and obtain at least one reply outputted by the question answering model;

[0116] The evaluation module is used to perform a quality evaluation on the question answering model according to the answer corresponding to the target question and at least one reply, and obtain a quality evaluation result of the question answering model.

[0117] In one embodiment, the acquisition module is specifically used to:

[0118] The preset knowledge base in the preset vertical field is identified by regular expressions to obtain at least one question-answer pair in the preset vertical field; the preset knowledge base is constructed based on preset knowledge points and different questions in the preset vertical field.

[0119] In one embodiment, the evaluation module is specifically configured to:

[0120] Based on the pre-trained model, semantic understanding and similarity comparison are performed on the answer corresponding to the target question and the at least one reply to obtain a preset quality assessment index value for each reply;

[0121] A quality assessment result of the question answering model is generated based on the preset quality assessment index value of each of the responses.

[0122] In one embodiment, the evaluation module is specifically configured to:

[0123] Performing word segmentation processing on the answer corresponding to the target question and the at least one reply respectively, to obtain at least one first word block constituting the answer corresponding to the target question and at least one second word block constituting each of the replies;

[0124] Performing semantic understanding processing on the at least one first word block and the at least one second word block respectively through a pre-trained model to obtain vector representations of each of the first word blocks and vector representations of each of the second word blocks respectively containing contextual semantic information;

[0125] The similarity between the vector representation of each of the first word blocks and the vector representation of each of the second word blocks is calculated, and a preset quality assessment index value of each of the replies is determined based on the similarity.

[0126] In one embodiment, the evaluation module is specifically configured to:

[0127] Calculate the cosine similarity between the vector representation of each of the first word chunks and the vector representation of each of the second word chunks to obtain a similarity matrix between the answer corresponding to the target question and the target response; the target response is any of the responses;

[0128] Based on the similarity matrix, the maximum similarity scores of the answer corresponding to the target question and the target reply are respectively accumulated and normalized to obtain a preset quality evaluation index value of the target reply.

[0129] In one embodiment, the preset quality assessment indicators include recall, precision and F1 score.

[0130] In one embodiment, the quality assessment result includes a quality score; an assessment module, specifically configured to:

[0131] Determine the variance of the recall rate, the variance of the precision rate, the variance of the F1 score, and the average recall rate, the average precision rate, and the average F1 score according to the preset quality assessment indicator value of each of the responses;

[0132] The variance of the recall rate, the variance of the precision rate, the variance of the F1 score, the average recall rate, the average precision rate and the average F1 score are respectively summed with the corresponding preset weights to determine the quality score of the question answering model.

[0133] For the specific definition of the language model evaluation device, please refer to the definition of the language model evaluation method above, which will not be repeated here. Each module in the above-mentioned language model evaluation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor of the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0134] Based on the same inventive concept as the above embodiments, an embodiment of the present invention provides an electronic device, such as Figure 4 As shown, the electronic device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 4 The processor 310 shown in the figure is not used to indicate that the number of the processor 310 is one, but is only used to indicate the positional relationship of the processor 310 relative to other devices. In actual applications, the number of the processor 310 may be one or more; similarly, Figure 4 The memory 311 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 311 relative to other devices. In practical applications, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the above-mentioned language model evaluation method is implemented.

[0135] The electronic device may also include: at least one network interface 312. The various components in the electronic device are coupled together via a bus system 313. It is understood that the bus system 313 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 313 is not used in the following examples. Figure 4 Various buses are labeled as bus system 313.

[0136] The memory 311 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 311 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0137] The memory 311 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device. Examples of these data include: any computer program used to operate on the electronic device, such as an operating system and an application; contact data; phone book data; messages; pictures; videos, etc. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program may include various applications, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. Here, the program that implements the method of the embodiment of the present invention may be included in the application program.

[0138] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer storage medium, in which a computer program is stored. The computer storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it may also be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer storage medium is executed by the processor, the above-mentioned language model evaluation method is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 1 The description of the illustrated embodiment will not be repeated here.

[0139] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0141] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A language model evaluation method, characterized in that: include: Obtain at least one question-answer pair in a preset vertical field; the question-answer pair includes a question and an answer corresponding to the question; Anthropomorphize the target question through a natural language processing model to obtain at least one question text of the target question in different questioning modes; The target question is a question in any of the question-answer pairs; Based on the at least one question text, a question is asked to the question answering model corresponding to the preset vertical field, and at least one reply corresponding to the output of the question answering model is obtained; The quality of the question answering model is evaluated according to the answer corresponding to the target question and the at least one reply to obtain a quality evaluation result of the question answering model.

2. The method according to claim 1, characterized in that The obtaining of at least one question-answer pair in a preset vertical field includes: A preset knowledge base in a preset vertical field is identified by using a regular expression to obtain at least one question-answer pair in the preset vertical field; the preset knowledge base is constructed based on preset knowledge points and different questions in the preset vertical field.

3. The method according to claim 1, characterized in that The step of performing a quality assessment on the question answering model according to the answer corresponding to the target question and the at least one reply to obtain a quality assessment result of the question answering model includes: Based on the pre-trained model, semantic understanding and similarity comparison are performed on the answer corresponding to the target question and the at least one reply to obtain a preset quality assessment index value for each reply; A quality assessment result of the question answering model is generated based on the preset quality assessment index value of each of the responses.

4. The method according to claim 3, characterized in that: The performing semantic understanding and similarity comparison on the answer corresponding to the target question and the at least one reply based on the pre-trained model to obtain a preset quality assessment index value for each reply includes: Performing word segmentation processing on the answer corresponding to the target question and the at least one reply respectively, to obtain at least one first word block constituting the answer corresponding to the target question and at least one second word block constituting each of the replies; Performing semantic understanding processing on the at least one first word block and the at least one second word block respectively through a pre-trained model to obtain vector representations of each of the first word blocks and vector representations of each of the second word blocks respectively containing contextual semantic information; The similarity between the vector representation of each of the first word blocks and the vector representation of each of the second word blocks is calculated, and a preset quality assessment index value of each of the replies is determined based on the similarity.

5. The method according to claim 4, characterized in that The calculating the similarity between the vector representation of each first word block and the vector representation of each second word block, and determining the preset quality assessment index value of each reply according to the similarity, includes: Calculate the cosine similarity between the vector representation of each of the first word chunks and the vector representation of each of the second word chunks to obtain a similarity matrix between the answer corresponding to the target question and the target response; the target response is any of the responses; Based on the similarity matrix, the maximum similarity scores of the answer corresponding to the target question and the target reply are respectively accumulated and normalized to obtain a preset quality evaluation index value of the target reply.

6. The method according to any one of claims 3 to 5, characterized in that The preset quality evaluation indicators include recall rate, precision rate and F1 score.

7. The method according to claim 6, characterized in that The quality assessment result includes a quality score; the quality assessment result of the question answering model is generated according to the preset quality assessment index value of each of the responses, including: Determine the variance of the recall rate, the variance of the precision rate, the variance of the F1 score, and the average recall rate, the average precision rate, and the average F1 score according to the preset quality assessment indicator value of each of the responses; The variance of the recall rate, the variance of the precision rate, the variance of the F1 score, the average recall rate, the average precision rate and the average F1 score are respectively summed with the corresponding preset weights to determine the quality score of the question answering model.

8. A language model evaluation device, characterized in that: The device comprises: an acquisition module, a processing module, a questioning module and an evaluation module; wherein, The acquisition module is used to acquire at least one question-answer pair in a preset vertical field; the question-answer pair includes a question and an answer corresponding to the question; The processing module is used to perform anthropomorphic processing on the target question through a natural language processing model to obtain at least one question text of the target question in different questioning modes; the target question is a question in any of the question-answer pairs; The questioning module is used to ask a question to the question answering model corresponding to the preset vertical field based on the at least one question text, and obtain at least one reply outputted by the question answering model; The evaluation module is used to perform a quality evaluation on the question answering model according to the answer corresponding to the target question and the at least one reply, so as to obtain a quality evaluation result of the question answering model.

9. An electronic device, characterized in that: include: A processor and a memory storing a computer program, wherein when the processor runs the computer program, the language model evaluation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the language model evaluation method according to any one of claims 1 to 7 is implemented.

Images