Question and answer model evaluation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of model evaluation, and provides a question and answer model evaluation method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining an evaluation data set corresponding to an original question and answer model and an evaluation data set corresponding to a comparison question and answer model; inputting the evaluation data sets of the two into a trained evaluation model for formatting processing to obtain corresponding formatted data; performing forward propagation processing on the corresponding formatted data based on at least one preset result label to obtain an original question and answer score and a comparison question and answer score corresponding to each preset result label; respectively carrying out normalization processing on the target result scores to obtain respective corresponding target result scores; and determining evaluation result labels of the original question and answer model and the comparison question and answer model according to the target result score, and sending the evaluation result labels to the target terminal device for display, thereby improving the generalization ability and stability of the model, and ensuring the diversity and accuracy of the data.

Description

Technical Field

[0001] The present disclosure relates to the field of model evaluation technology, and in particular to a question-answering model evaluation method, device, electronic device, and readable storage medium. Background Art

[0002] Question-answering model evaluation methods are widely used in the field of artificial intelligence. Traditional question-answering model evaluation methods mostly rely on manually annotated datasets, and evaluate the accuracy of the model by comparing the model output with the manually annotated answers. However, this method has some problems in practical applications. Manually annotated datasets are often limited in scale and it is difficult to fully cover all application scenarios of the question-answering system, resulting in biased evaluation results. Secondly, the manual annotation process is time-consuming and labor-intensive, and the annotation quality is limited by the professional level and experience of the annotator, which further affects the reliability of the evaluation results. In addition, traditional evaluation methods usually focus on the accuracy of the model output, while ignoring the fluency and relevance of the model in the process of generating answers.

[0003] Under normal circumstances, the accuracy of the evaluation can be improved by expanding the size of the data set and introducing richer annotation information, or the performance of the question-answering model can be evaluated using large-scale unlabeled data through unsupervised or weakly supervised methods. However, the accuracy and reliability of the evaluation results are not high, and the existing evaluation methods focus on the static results of the model output, but lack the evaluation of the dynamic generation process of the model, resulting in the evaluation results being unable to fully reflect the performance of the model.

[0004] It can be seen that the existing technology has the problem of low accuracy and reliability of question-answering model evaluation results due to the single evaluation method and ignoring the dynamic generation process of the model. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide a question-answering model evaluation method, device, electronic device and readable storage medium to solve the problem in the prior art that the accuracy and reliability of the question-answering model evaluation results are low due to the single evaluation method and ignoring the dynamic generation process of the model.

[0006] According to a first aspect of an embodiment of the present disclosure, a question-answering model evaluation method is provided, including: obtaining an evaluation data set corresponding to an original question-answering model and an evaluation data set corresponding to a comparative question-answering model; inputting the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model into a trained evaluation model for formatting processing, thereby obtaining formatted data corresponding to the original question-answering model and formatted data corresponding to the comparative question-answering model, wherein the evaluation model is obtained by training a large language model based on multiple training dialogue data, true labels corresponding to each training dialogue data, and training dialogue enhancement data; performing forward propagation processing on the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model based on at least one preset result label, thereby obtaining an original question-answering score corresponding to each preset result label and a comparative question-answering score corresponding to each preset result label; performing normalization processing on the original question-answering score corresponding to each preset result label and the comparative question-answering score corresponding to each preset result label, thereby obtaining a target result score corresponding to the original question-answering model and a target result score corresponding to the comparative question-answering model; determining evaluation result labels of the original question-answering model and the comparative question-answering model according to the target result score corresponding to the original question-answering model and the target result score corresponding to the comparative question-answering model, and sending the evaluation result labels to a target terminal device for display.

[0007] According to a second aspect of an embodiment of the present disclosure, a question-answering model evaluation device is provided, including: an acquisition module for acquiring an evaluation data set corresponding to an original question-answering model and an evaluation data set corresponding to a comparative question-answering model; a first processing module for inputting the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model into a trained evaluation model for formatting, thereby obtaining formatted data corresponding to the original question-answering model and formatted data corresponding to the comparative question-answering model, wherein the evaluation model is obtained by training a large language model based on a multi-stage data enhancement training strategy; a second processing module for processing the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model based on at least one preset result label. The corresponding formatted data are forward propagated respectively to obtain the original question and answer scores corresponding to each preset result label and the comparative question and answer scores corresponding to each preset result label; the third processing module is used to normalize the original question and answer scores corresponding to each preset result label and the comparative question and answer scores corresponding to each preset result label, respectively, to obtain the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparative question and answer model; the determination module is used to determine the evaluation result labels of the original question and answer model and the comparative question and answer model according to the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparative question and answer model, and send the evaluation result labels to the target terminal device for display.

[0008] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] Compared with the prior art, the embodiments of the present disclosure have the following beneficial effects: by respectively acquiring an evaluation data set corresponding to the original question-answering model and an evaluation data set corresponding to the comparative question-answering model, and inputting the acquired two evaluation data sets into the trained evaluation model for formatting, the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model are obtained, and then the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model can be forward-propagated through at least one preset result label to obtain the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label, and then the original question-answering scores corresponding to the preset result labels can be respectively calculated. The comparison question and answer scores corresponding to the preset result labels are normalized to obtain the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparison question and answer model. The evaluation result labels of the original question and answer model and the comparison question and answer model are determined by score comparison based on the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparison question and answer model, and the evaluation result labels are sent to the current user graphical interface of the target terminal device for display, thereby improving the accuracy and efficiency of the question and answer model evaluation, enhancing the formatting processing capability of the evaluation data, improving the accuracy of the score calculation and normalization processing based on multiple preset result labels, and improving the reliability of the evaluation results by forward propagation processing and normalization processing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure; Figure 2 It is a flowchart of a question-answering model evaluation method provided by an embodiment of the present disclosure; Figure 3is a structural schematic diagram of a question-answering model evaluation device provided by an embodiment of the present disclosure; Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0014] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0015] A question-answering model evaluation method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0016] Figure 1 Schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include terminal devices 1, 2 and 3, a server 4 and a network 5.

[0017] Terminal devices 1, 2 and 3 can be hardware or software. When terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with display screens and supporting communication with server 4, including but not limited to smart phones, tablet computers, laptop portable computers and desktop computers, etc.; when terminal devices 1, 2 and 3 are software, they can be installed in the above electronic devices. Terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure are not limited to this. Furthermore, various applications can be installed on terminal devices 1, 2 and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0018] The server 4 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 4 may be a single server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.

[0019] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2, and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2, and 3, or it can be a single software or software module that provides various services for the terminal devices 1, 2, and 3, and the embodiments of the present disclosure are not limited to this.

[0020] The network 5 may be a wired network connected by coaxial cable, twisted pair, and optical fiber, or a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.

[0021] The user can establish a communication connection with the server 4 via the terminal devices 1, 2 and 3 via the network 5 to receive or send information, etc. Specifically, the server 4 can obtain the evaluation data set corresponding to the original question and answer model and the evaluation data set corresponding to the comparative question and answer model respectively via the terminal devices 1, 2 and 3, and input the obtained two evaluation data sets into the trained evaluation model for formatting processing to obtain the formatted data corresponding to the original question and answer model and the formatted data corresponding to the comparative question and answer model, and then forward propagate the formatted data corresponding to the original question and answer model and the formatted data corresponding to the comparative question and answer model respectively through at least one preset result label, and obtain the original question and answer scores corresponding to each preset result label and the comparative question and answer scores corresponding to each preset result label, and then normalize the original question and answer scores corresponding to the preset result labels and the comparative question and answer scores corresponding to the preset result labels respectively, and obtain the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparative question and answer model, and determine the evaluation result labels of the original question and answer model and the comparative question and answer model based on the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparative question and answer model through score comparison, and send the evaluation result labels to the current user graphical interface of the target terminal device for display.

[0022] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure are not limited to this.

[0023] Figure 2 It is a flowchart of a question-answering model evaluation method provided in an embodiment of the present disclosure. Figure 2 The question answering model evaluation method can be Figure 1 The server executes. Figure 2As shown, the question answering model evaluation method includes: Step 201, obtaining an evaluation data set corresponding to the original question-answering model and an evaluation data set corresponding to the comparative question-answering model.

[0024] Specifically, data sets related to question-answering model evaluation can be obtained from the historical question-answering data of the original question-answering model and the historical question-answering data of the comparative question-answering model, respectively, to ensure that the format, content, and annotation method of the data sets meet the evaluation requirements of the respective models, thereby enhancing the pertinence and applicability of the data sets, improving the accuracy and reliability of the evaluation, and improving the overall efficiency and quality of question-answering model evaluation.

[0025] Among them, the original question and answer model can be the initial question and answer model to be evaluated, which can be a 7b-level large language model or a 9b-level large language model, etc., which is not limited here. The evaluation data set corresponding to the original question and answer model can be a plurality of questions configured for evaluating the original question and answer model and the reference answers or evaluation criteria corresponding to each question, etc., which are not limited here. The evaluation data set corresponding to the original question and answer model can be obtained by manual annotation, automatic machine generation, or extraction from a configured knowledge base, etc., which is not limited here.

[0026] The comparative question and answer model can be another question and answer system or model used for comparative evaluation with the original question and answer model. It can be a level 7b large language model or a level 9b large language model, etc., which is not limited here. The evaluation data set corresponding to the comparative question and answer model can be a data set configured for evaluating the comparative question and answer model.

[0027] For example, an evaluation dataset may be obtained from a 7b-level large language model as the evaluation dataset corresponding to the original question-answering model, and an evaluation dataset may be obtained from a 9b-level large language model as the evaluation dataset corresponding to the comparative question-answering model.

[0028] Step 202, input the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model into the trained evaluation model for formatting, and obtain the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model. The evaluation model is obtained by training a large language model based on multiple training dialogue data, the real labels corresponding to each training dialogue data, and the training dialogue enhancement data.

[0029] Specifically, the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model can be input into a pre-trained evaluation model. The evaluation model can be obtained by training a large language model based on a large amount of training dialogue data, the real labels corresponding to each training dialogue data, and the training dialogue enhanced data generated by data enhancement. The evaluation model can format the input data set, including but not limited to data cleaning, format unification, and / or feature extraction steps, and then obtain the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model, thereby enhancing the readability and comparability of the data, improving the ability of the evaluation model to process complex dialogue data, and improving the efficiency and accuracy of question-answering model evaluation.

[0030] The evaluation model may be a machine learning model for evaluating the performance of the question-answering model, and the evaluation model may evaluate the accuracy and reliability of the question-answering model by analyzing the difference between the output of the question-answering model and the true label. The formatted data corresponding to the original question-answering model may be a data set with unified format and features converted from the original question-answering model evaluation data set after formatting by the evaluation model. The formatted data corresponding to the comparison question-answering model may be a data set converted from the comparison question-answering model evaluation data set after formatting by the evaluation model.

[0031] The training conversation data may be the original conversation data used to train the evaluation model. The training conversation data may include multiple questions and corresponding answers or feedback. The training conversation data may also be used to train the evaluation model to recognize and understand the information in the conversation. The true labels corresponding to the training conversation data may be the correct answers or true categories corresponding to the training conversation data, which are not limited here. The true labels corresponding to the training conversation data may be used to evaluate the accuracy of the model during the training process.

[0032] The training dialogue enhancement data may be data generated by transforming, expanding, and / or combining the original training dialogue data. The training dialogue enhancement data may be used to increase the diversity and richness of the training data, thereby improving the generalization ability of the evaluation model. The training dialogue enhancement data may be generated by adding noise to the data, replacing vocabulary, and / or changing the sentence structure, etc., which are not limited here.

[0033] A large language model can be a deep learning model with language understanding and generation capabilities. A large language model can contain a large number of parameters and network structures. A large language model can be used to process and understand large-scale natural language texts.

[0034] For example, in evaluating an intelligent customer service system, the evaluation data set corresponding to the original question-and-answer model can be the collected conversation data between users and the intelligent customer service system, and the evaluation data set corresponding to the comparative question-and-answer model can be the conversation data of another intelligent customer service system, and both can be input into a pre-trained evaluation model. The evaluation model can format the evaluation data set corresponding to the original question-and-answer model and the evaluation data set corresponding to the comparative question-and-answer model, respectively, extract key information and features, and generate formatted data corresponding to the original question-and-answer model and formatted data corresponding to the comparative question-and-answer model.

[0035] Step 203, based on at least one preset result label, forward propagation processing is performed on the formatted data corresponding to the original question and answer model and the formatted data corresponding to the comparative question and answer model, respectively, to obtain the original question and answer scores corresponding to each preset result label and the comparative question and answer scores corresponding to each preset result label.

[0036] Specifically, at least one preset result label can be used as the basis for classification or evaluation, and the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model can be forward-propagated separately, and the features of each input data can be matched and calculated according to the preset result label to obtain the original question-answering score and comparative question-answering score corresponding to each preset result label. The original question-answering score corresponding to the preset result label can be used to characterize the probability or prediction accuracy of the original question-answering model on the corresponding result label, and the comparative question-answering score corresponding to the preset result label can be used to characterize the probability or prediction accuracy of the comparative question-answering model on the corresponding result label, thereby enhancing the precision and interpretability of the evaluation results, improving the automation and intelligence level of the evaluation process, and improving the accuracy and comprehensiveness of the question-answering model evaluation.

[0037] The preset result labels can be classifications or evaluation criteria pre-configured according to evaluation requirements, and the preset result labels can be used to characterize different results or categories output by the question-answering model. The preset result labels include, but are not limited to, answer options, sentiment tendencies (such as positive, negative, neutral), information types (such as factuality, opinion), etc., which are not limited here.

[0038] The original question-answering score corresponding to the preset result label can be the prediction accuracy or confidence score of each evaluation data in the evaluation data set corresponding to the original question-answering model on the label under the condition of having the preset result label. The comparison question-answering score can be 0.2, 0.4, or 0.7, etc., which is not limited here. The original question-answering score corresponding to the preset result label can be used to characterize the probability of the original question-answering model on different result labels.

[0039] The comparison question-answering score corresponding to the preset result label can be the prediction accuracy or confidence score of each evaluation data in the evaluation data set corresponding to the comparison question-answering model on the corresponding label under the condition of having the preset result label. The comparison question-answering score can be 0.1, 0.2, or 0.6, etc., which is not limited here. The comparison question-answering score corresponding to the preset result label can be used to compare with the original question-answering score to evaluate the performance difference between the two question-answering models on the same task.

[0040] For example, in evaluating the application effect of the deep learning-based intelligent question-answering system in the field of education, the preset result labels can be "correct answer", "partially correct", "wrong answer", and "irrelevant answer", etc. The formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model can be respectively input into the trained classifier for forward propagation processing. The classifier can match and score the answers to each input data according to the preset result labels, and obtain the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label.

[0041] In step 204, the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label are normalized to obtain the target result scores corresponding to the original question-answering model and the target result scores corresponding to the comparative question-answering model.

[0042] Specifically, the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label can be normalized respectively. The normalization process can be to scale the scores to between 0 and 1 through the maximum and minimum normalization functions to facilitate cross-label and cross-model comparisons, thereby obtaining the target result scores corresponding to the original question-answering model and the target result scores corresponding to the comparative question-answering model, thereby enhancing the comparability and interpretability of the scores, improving the accuracy and consistency of the evaluation results, and enhancing the practicality and guiding significance of the question-answering model evaluation. Normalization allows the scores between different labels and models to be directly compared, which helps to understand the performance differences and advantages of the models.

[0043] The target result score corresponding to the original question-answering model can be a comprehensive reflection of the scores of the original question-answering model on all preset result labels after normalization, and the target result score corresponding to the original question-answering model can be 0.2, 0.77, or 0.9, etc., which are not limited here. The target result score corresponding to the original question-answering model can be used to characterize the overall performance of the original question-answering model in different aspects, and can also be used to evaluate the accuracy and reliability of the model.

[0044] The target result score corresponding to the comparison question-answering model can be a comprehensive reflection of the score of the comparison question-answering model on all preset result labels. The target result score corresponding to the comparison question-answering model can be 0.7, 0.83, or 0.91, etc., which is not limited here. The target result score corresponding to the comparison question-answering model can be used to compare with the target result score of the original question-answering model to evaluate the performance difference between the two models on the same task.

[0045] For example, in evaluating the application effects of two deep learning-based intelligent question-answering systems in the field of medical health, the preset result labels can be "accurate diagnosis", "reasonable explanation", and "wrong suggestions". The original question-answering scores corresponding to the above preset result labels and the comparison question-answering scores corresponding to the preset result labels can be normalized through the maximum and minimum normalization function, and the target result scores corresponding to the two intelligent question-answering systems are 0.2 and 0.7 respectively.

[0046] Step 205, determine the evaluation result labels of the original question and answer model and the comparative question and answer model according to the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparative question and answer model, and send the evaluation result labels to the target terminal device for display.

[0047] Specifically, the evaluation result labels of the original question-answering model and the comparative question-answering model can be determined by comparing and analyzing the size and distribution of the target result scores corresponding to the original question-answering model and the target result scores corresponding to the comparative question-answering model. The evaluation result label can be a brief description or classification, which can be used to summarize the overall performance of the model on the evaluation task. The evaluation result label can then be sent to the target terminal device via a network or other communication method. The target terminal device includes but is not limited to a smart phone, tablet computer, or computer, so that users or relevant personnel can view and understand the evaluation results of the model, thereby enhancing the intuitiveness and ease of use of the evaluation results, improving the transparency and credibility of the evaluation process, and enhancing the practical value and decision-making support capabilities of the question-answering model evaluation.

[0048] Among them, the evaluation result label can be a brief description or classification of the overall performance of the original question-answering model and the comparative question-answering model on the evaluation task. The evaluation result label can be a qualitative expression, such as "the original question-answering model is better than the comparative question-answering model", "the comparative question-answering model is better than the original question-answering model", and / or "the original question-answering model and the comparative question-answering model have the same performance", etc., which are not limited here, or a quantitative expression, such as a specific score range or level, which are not limited here. The evaluation result label can be obtained based on the comparison and analysis of the target result scores, as well as the specific requirements and standards of the evaluation task.

[0049] The target terminal device may be a device for receiving and displaying the evaluation result label, including but not limited to a smart phone, a tablet computer, or a computer and other electronic devices with network communication and display functions. The target terminal device may receive the evaluation result label from the evaluation model and display the evaluation result label on the target graphical user interface for the user or relevant personnel to view and understand the evaluation result of the model.

[0050] For example, when evaluating the application effects of two intelligent question-answering systems in the field of customer service, the corresponding evaluation result labels can be determined based on the target result scores of the original question-answering model and the comparative question-answering model. If the target result scores of the original question-answering model in key indicators such as "customer satisfaction" and "problem resolution rate" are higher than those of the comparative question-answering model, we can label the evaluation result as "the original question-answering model is better than the comparative question-answering model." The evaluation result label can then be sent to the customer service department's smartphone or computer for relevant personnel to view.

[0051] According to the technical solution provided by the embodiments of the present disclosure, by respectively acquiring an evaluation data set corresponding to the original question-answering model and an evaluation data set corresponding to the comparative question-answering model, and inputting the acquired two evaluation data sets into the trained evaluation model for formatting processing, the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model are obtained, and then the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model can be forward-propagated through at least one preset result label to obtain the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label, and then the original question-answering scores corresponding to the preset result labels and the comparative question-answering scores corresponding to the preset result labels can be respectively forward-propagated. The comparison question and answer scores corresponding to the result labels are normalized to obtain the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparison question and answer model. The evaluation result labels of the original question and answer model and the comparison question and answer model are determined by score comparison based on the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparison question and answer model. The evaluation result labels are sent to the current user graphical interface of the target terminal device for display, thereby improving the accuracy and efficiency of the question and answer model evaluation, enhancing the formatting processing capability of the evaluation data, improving the accuracy of the score calculation and normalization processing based on multiple preset result labels, and improving the reliability of the evaluation results by forward propagation processing and normalization processing data.

[0052] In some embodiments, before obtaining the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model, it also includes: obtaining multiple training conversation data and the true labels corresponding to each training conversation data; inputting each training conversation data into the evaluation model, and performing forward propagation processing based on at least one preset classification head to obtain the category data corresponding to each training conversation data; determining the cross entropy loss corresponding to the evaluation model according to the category data corresponding to each training conversation data and the true label corresponding to each training conversation data; and updating the parameters in the evaluation model according to the cross entropy loss corresponding to the evaluation model through a loop iteration.

[0053] Specifically, multiple training dialogue data can be obtained. The training dialogue data can be the historical dialogue content of the question-answering model. Each training dialogue data is configured with a corresponding true label. The true label can be used to characterize the correct answer or classification of the training dialogue data. Each training dialogue data can be input into the evaluation model. The evaluation model can contain at least one preset classification head, which can be used to map the input data to different categories or labels. Then, through forward propagation processing, the category data corresponding to each training dialogue data can be obtained, that is, the label predicted by the evaluation model. The cross entropy loss corresponding to the evaluation model can be calculated by comparing the category data predicted by the evaluation model and the true label, and then the parameters in the evaluation model can be adjusted through a cyclic iteration method so that the cross entropy loss is gradually reduced, thereby optimizing the performance of the evaluation model.

[0054] The preset classification head may be a part of the evaluation model used to map input data to different categories or labels. The preset classification head may be a classifier based on a neural network, obtained through training and learning.

[0055] The category data corresponding to the training conversation data may be a label or category obtained after the evaluation model predicts the training conversation data. The category data corresponding to the training conversation data may be used to characterize the result obtained by the evaluation model after learning the features of the training conversation data, and the category data corresponding to the training conversation data may be used to compare with the true label to calculate the cross entropy loss corresponding to the evaluation model.

[0056] The cross entropy loss corresponding to the evaluation model can be the degree of difference between the category data predicted by the evaluation model and the true label. The cross entropy loss corresponding to the evaluation model can be obtained by calculating the cross entropy between the predicted probability distribution and the true probability distribution, and the smaller the cross entropy loss, the more accurate the prediction of the evaluation model and the smaller the difference with the true label.

[0057] For example, 55k conversation data can be obtained as training conversation data through the large language model system, and the correct category of each conversation data can be marked as the true label. The training conversation data can then be input into the pre-trained 9b-level large language model and forward propagated through a preset classification head. The preset classification head can include two linear layers and a GELU activation function, and the output dimension is configured to be 3, corresponding to three categories: "the original question-and-answer model wins", "the comparative question-and-answer model wins", and "the two are tied". Then, the category data predicted by the model can be obtained, and the cross-entropy loss between the category data predicted by the 9b-level large language model and the true label can be calculated, and the parameters in the 9b-level large language model can be updated through a loop iteration to reduce the cross-entropy loss.

[0058] According to the technical solution provided by the embodiment of the present disclosure, by obtaining multiple training dialogue data, each training dialogue data is configured with a corresponding real label, each training dialogue data can be input into the evaluation model, and the evaluation model can contain at least one preset classification head, and then through forward propagation processing, the category data corresponding to each training dialogue data can be obtained, and then through a cyclic iteration, the parameters in the evaluation model can be adjusted so that the cross entropy loss gradually decreases, thereby enhancing the accuracy and generalization ability of the evaluation model, improving the adaptability of the evaluation model when processing different dialogue data, and improving the reliability and effectiveness of the question-answering model evaluation. Through a large amount of training data and the optimization of cross entropy loss, the model can better learn the characteristics of the dialogue data, so that it can more accurately predict and classify the dialogue content.

[0059] In some embodiments, the above-mentioned question-answering model evaluation method also includes: obtaining multiple training conversation enhancement data; inputting each training conversation enhancement data into the evaluation model for forward propagation processing to obtain category data corresponding to each training conversation enhancement data; performing label mapping processing on each training conversation enhancement data based on the category data corresponding to each training conversation enhancement data to obtain a training label corresponding to each training conversation enhancement data.

[0060] Specifically, multiple training conversation enhancement data can be obtained, and the training conversation enhancement data can be input into the evaluation model for forward propagation processing to obtain category data corresponding to each training conversation enhancement data, that is, the prediction result of the evaluation model for the training conversation enhancement data. The category data corresponding to each training conversation enhancement data can be mapped to the corresponding training conversation enhancement data as a training label based on a preset mapping relationship, and the training labels corresponding to the training conversation enhancement data can be used for training or evaluation of the evaluation model.

[0061] The category data corresponding to the training dialogue enhancement data may be a label or category obtained by the evaluation model predicting the training dialogue enhancement data. The category data corresponding to the training dialogue enhancement data may be obtained by inputting the training dialogue enhancement data into the evaluation model for forward propagation processing, and the category data corresponding to the training dialogue enhancement data may be used for label mapping processing.

[0062] For example, multiple training conversation enhancement data can be obtained, and each training conversation enhancement data can be forward propagated through the evaluation model to obtain category data corresponding to each training conversation enhancement data. The category data can be a qualitative category of each training conversation enhancement data, or a predicted probability of each training conversation enhancement data. Then, the category data can be mapped to the corresponding training conversation enhancement data through a preset mapping relationship as a training label corresponding to the training conversation enhancement data.

[0063] According to the technical solution provided by the embodiment of the present disclosure, by obtaining multiple training dialogue enhancement data, the training dialogue enhancement data can be input into the evaluation model for forward propagation processing to obtain the category data corresponding to each training dialogue enhancement data, and the category data corresponding to each training dialogue enhancement data can be mapped as a training label to the corresponding training dialogue enhancement data based on a pre-set mapping relationship, thereby enhancing the adaptability of the evaluation model to data changes, improving the accuracy and robustness of the evaluation model, and improving the comprehensiveness and effectiveness of the question-answering model evaluation results. By introducing the training dialogue enhancement data, the evaluation model can learn more diverse data features, so that it is more flexible and accurate when processing actual dialogues.

[0064] In some embodiments, the above-mentioned question-answering model evaluation method also includes: inputting each training conversation enhancement data into the evaluation model, performing forward propagation processing based on each preset classification head to obtain training category data corresponding to each training conversation enhancement data; determining the training cross entropy loss corresponding to the evaluation model according to the training category data corresponding to each training conversation enhancement data and the training label corresponding to each training conversation enhancement data; and updating the parameters in the evaluation model according to the training cross entropy loss corresponding to the evaluation model through a loop iteration method.

[0065] Specifically, each training dialogue enhancement data can be input into the evaluation model, and forward propagation processing can be performed based on the classification head preset in the evaluation model to obtain the training category data corresponding to each training dialogue enhancement data, that is, the prediction result of the evaluation model for each training dialogue enhancement data, and the training category data corresponding to the training dialogue enhancement data and the training labels corresponding to the training dialogue enhancement data can be compared, and the training cross entropy loss corresponding to the evaluation model can be calculated. Then, the parameters in the evaluation model can be updated according to the calculated training cross entropy loss through a cyclic iteration.

[0066] The training category data corresponding to the training dialogue enhancement data may be a label or category obtained by processing the training dialogue enhancement data by the evaluation model, which is not limited here. The training category data corresponding to the training dialogue enhancement data may be obtained by inputting the training dialogue enhancement data into the evaluation model through forward propagation processing, and the training category data corresponding to the training dialogue enhancement data may be used to characterize the evaluation model's ability to understand and classify the training dialogue enhancement data.

[0067] The training cross entropy loss corresponding to the evaluation model can be the degree of difference between the category data predicted by the evaluation model on the training set and the true label. The training cross entropy loss corresponding to the evaluation model can be obtained by calculating the cross entropy between the predicted probability distribution and the true probability distribution. The smaller the training cross entropy loss, the more accurate the prediction of the evaluation model on the training set and the smaller the difference with the true label.

[0068] For example, each training conversation enhancement data can be forward propagated through the evaluation model, and the training category data corresponding to each training conversation enhancement data can be obtained based on a preset classification head. The training cross entropy loss corresponding to the evaluation model can be calculated through the training category data corresponding to the training conversation enhancement data and the training labels corresponding to the training conversation enhancement data. Then, the parameters in the evaluation model can be updated through a cyclic fall method.

[0069] According to the technical solution provided by the embodiment of the present disclosure, by inputting each training dialogue enhancement data into the evaluation model, and performing forward propagation processing based on the classification head preset in the evaluation model, the training category data corresponding to each training dialogue enhancement data can be obtained, the training category data corresponding to the training dialogue enhancement data and the training labels corresponding to the training dialogue enhancement data can be compared, and the training cross entropy loss corresponding to the evaluation model can be calculated. Then, the parameters in the evaluation model can be updated according to the calculated training cross entropy loss in a cyclic iteration manner, thereby enhancing the evaluation model's ability to process complex dialogue data, improving the generalization ability of the evaluation model, and improving the accuracy and reliability of the question-answering model evaluation results.

[0070] In some embodiments, based on at least one preset result label, forward propagation processing is performed on the formatted data corresponding to the original question and answer model and the formatted data corresponding to the comparative question and answer model, respectively, to obtain the original question and answer scores corresponding to each preset result label and the comparative question and answer scores corresponding to each preset result label, including: vectorizing the formatted data corresponding to the original question and answer model to obtain the feature vector corresponding to the original question and answer model; performing neural network propagation processing on the feature vector corresponding to the original question and answer model to obtain the original question and answer scores corresponding to each preset result label; vectorizing the formatted data corresponding to the comparative question and answer model to obtain the feature vector corresponding to the comparative question and answer model; performing neural network propagation processing on the feature vector corresponding to the comparative question and answer model to obtain the comparative question and answer scores corresponding to each preset result label.

[0071] Specifically, the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model can be vectorized, and the text data can be converted into a feature vector that can be processed by a neural network, so as to obtain a feature vector corresponding to the original question-answering model and a feature vector corresponding to the comparative question-answering model. The vectorization process may involve processing steps such as text segmentation or word embedding, which are not limited here, to capture the semantic information in the text, and the feature vector corresponding to the original question-answering model and the feature vector corresponding to the comparative question-answering model can be respectively input into the neural network of the evaluation model for propagation processing, and the feature vector can be calculated through multiple hidden layers to obtain the original question-answering score corresponding to each preset result label and the comparative question-answering score corresponding to each preset result label. The original question-answering score and the comparative question-answering score can be used to characterize the prediction probability of the evaluation model for each preset result label.

[0072] Among them, the feature vector corresponding to the original question-answering model can be a feature representation obtained by vectorizing the formatted data corresponding to the original question-answering model. The feature vector corresponding to the original question-answering model can be obtained through text segmentation or word embedding, etc., which is not limited here. The feature vector corresponding to the original question-answering model can be used to capture semantic information in the text and can be used for subsequent neural network propagation processing. The feature vector corresponding to the comparative question-answering model can refer to the feature representation obtained by vectorizing the formatted data corresponding to the comparative question-answering model. The feature vector corresponding to the comparative question-answering model can be calculated through multiple hidden layers in the neural network to obtain the comparative question-answering scores corresponding to each preset result label.

[0073] For example, the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model can be vectorized respectively through word embedding to obtain the feature vector corresponding to the original question-answering model and the feature vector corresponding to the comparative question-answering model. Then, the feature vector corresponding to the original question-answering model and the feature vector corresponding to the comparative question-answering model can be propagated through neural network through multiple hidden layers to obtain the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label.

[0074] According to the technical solution provided by the embodiment of the present disclosure, by vectorizing the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model, the text data is converted into a feature vector that can be processed by a neural network, and the feature vector corresponding to the original question-answering model and the feature vector corresponding to the comparative question-answering model are obtained. The feature vector corresponding to the original question-answering model and the feature vector corresponding to the comparative question-answering model can be respectively input into the neural network of the evaluation model for propagation processing. The feature vector can be calculated through multiple hidden layers to obtain the original question-answering score corresponding to each preset result label and the comparative question-answering score corresponding to each preset result label, thereby enhancing the accuracy and objectivity of the evaluation model, improving the degree of automation of the evaluation process, and improving the performance of the evaluation model in practical applications. Through vectorization processing and neural network propagation processing, the semantic features of the text data can be captured more accurately, and the question-answering model can be objectively evaluated based on these features, realizing the automation of the evaluation process, reducing manual intervention, and improving evaluation efficiency.

[0075] In some embodiments, before obtaining a plurality of training dialogue enhancement data, it also includes: obtaining a plurality of generalized question training data; performing answer text generation processing on each generalized question training data through a multimodal large language model to obtain answer text data corresponding to each generalized question training data; performing score evaluation processing on the answer text data corresponding to each generalized question training data based on a generative pre-trained converter to obtain an evaluation score corresponding to each answer text data; and determining each training dialogue enhancement data according to a preset evaluation score threshold and the evaluation score corresponding to each answer text data.

[0076] Specifically, multiple generalized question training data can be obtained. The generalized question training data can be used to characterize a variety of different questioning methods and scenarios, and can be used to improve the generalization ability of the evaluation model. The generalized question training data can be processed by a multimodal large language model (such as a generative pre-trained converter, a bidirectional encoder representation model combining information such as images) to generate answer texts to generate answer text data corresponding to each question. The answer text data can contain multimodal information such as text information, images, and sounds, which are not limited here. Then, the obtained answer text data can be scored and evaluated by a generative pre-trained converter, and an evaluation score can be assigned to each answer text data. The evaluation score can be used to characterize the accuracy and fluency of the answer text data. The answer text data can be screened according to a preset evaluation score threshold to determine each training dialogue enhancement data.

[0077] Among them, the generalized question training data may be question data containing multiple questioning methods and scenarios. The multimodal large language model may be a language model for processing multiple modal information (such as text, image, sound, etc.) and generating corresponding answers. The answer text data corresponding to the generalized question training data may be the answer text corresponding to each question generated by the multimodal large language model based on the generalized question training data. The generative pre-trained converter may be a pre-trained model with text generation and evaluation capabilities, and the generative pre-trained converter may perform score evaluation processing on the generated answer text data. The evaluation score corresponding to the answer text data may be the score assigned to the answer text data by the generative pre-trained converter, and the evaluation score corresponding to the answer text data may be used to characterize the accuracy and fluency of the answer. The preset evaluation score threshold may be a threshold determined based on experience or experiment, and the preset evaluation score threshold may be used to screen the answer text data to ensure the quality of the training dialogue enhancement data.

[0078] For example, preference data sets, user dialogue data sets, code writing data sets, and math problem data sets can be obtained as generalized problem training data in different directions, and then the answer text generation processing can be performed on each generalized problem training data through the 7b-level multimodal large language model to obtain the answer text data corresponding to each generalized problem training data. The answer text data corresponding to each generalized problem training data can be score evaluated through a generative pre-trained converter to obtain the evaluation score corresponding to each answer text data. The evaluation score threshold can be pre-set to 0.8, and the answer text data with an evaluation score greater than or equal to 0.8 can be screened as training dialogue enhancement data.

[0079] According to the technical solution provided by the embodiment of the present disclosure, by acquiring multiple generalized question training data, the generalized question training data can be processed by a multimodal large language model to generate answer texts, and answer text data corresponding to each question can be generated. Then, the obtained answer text data can be processed by a generative pre-trained converter to perform score evaluation, and the answer text data can be screened according to a preset evaluation score threshold to determine each training dialogue enhancement data, thereby enhancing the evaluation model's ability to handle generalized questions, improving the accuracy and diversity of answer generation, and improving the user experience of the evaluation model in practical applications.

[0080] In some embodiments, the evaluation data set corresponding to the original question and answer model and the evaluation data set corresponding to the comparative question and answer model are input into the trained evaluation model for formatting processing to obtain formatted data corresponding to the original question and answer model and formatted data corresponding to the comparative question and answer model, including: inputting the evaluation data set corresponding to the original question and answer model into the trained evaluation model for data cleaning processing to obtain each screening data corresponding to the original question and answer model; standardizing each screening data corresponding to the original question and answer model based on a preset prompt template to obtain the formatted data corresponding to the original question and answer model; inputting the evaluation data set corresponding to the comparative question and answer model into the trained evaluation model for data cleaning processing to obtain each screening data corresponding to the comparative question and answer model; standardizing each screening data corresponding to the comparative question and answer model based on a preset prompt template to obtain the formatted data corresponding to the comparative question and answer model.

[0081] Specifically, the evaluation data set corresponding to the original question and answer model and the evaluation data set corresponding to the comparative question and answer model can be respectively input into the trained evaluation models for data cleaning processing, so as to obtain the various filtered data corresponding to the original question and answer model and the various filtered data corresponding to the comparative question and answer model. The filtered data corresponding to the original question and answer model and the filtered data corresponding to the comparative question and answer model can be standardized based on the preset prompt template, including but not limited to unified data format and / or standardized text expression, etc., to obtain the formatted data corresponding to the original question and answer model and the formatted data corresponding to the comparative question and answer model.

[0082] The filtered data corresponding to the original question-answering model may be valid data obtained by filtering out the evaluation data set corresponding to the original question-answering model through data cleaning. The filtered data corresponding to the comparative question-answering model may be valid data obtained by filtering out the evaluation data set corresponding to the comparative question-answering model after data cleaning.

[0083] For example, the evaluation data set corresponding to the original question and answer model and the evaluation data set corresponding to the comparative question and answer model can be respectively input into the trained evaluation model for data cleaning processing to obtain the filtered data corresponding to the original question and answer model and the filtered data corresponding to the comparative question and answer model. The above-mentioned filtered data can be standardized based on a preset prompt template to obtain the formatted data corresponding to the original question and answer model and the formatted data corresponding to the comparative question and answer model.

[0084] According to the technical solution provided by the embodiments of the present disclosure, the evaluation data set corresponding to the original question and answer model and the evaluation data set corresponding to the comparative question and answer model are respectively input into the trained evaluation models to perform data cleaning processing, thereby obtaining the respective filtered data corresponding to the original question and answer model and the respective filtered data corresponding to the comparative question and answer model. The filtered data corresponding to the original question and answer model and the filtered data corresponding to the comparative question and answer model can be standardized based on a preset prompt template to obtain formatted data corresponding to the original question and answer model and formatted data corresponding to the comparative question and answer model, thereby enhancing the accuracy and consistency of the evaluation data, improving the reliability and efficiency of the evaluation process, and improving the accuracy and objectivity of the evaluation of the evaluation model.

[0085] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0086] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0087] Figure 3 Schematic diagram of a question-answering model evaluation device provided by an embodiment of the present disclosure. Figure 3 As shown, the question-answering model evaluation device includes: An acquisition module 301 is used to acquire an evaluation data set corresponding to the original question-answering model and an evaluation data set corresponding to the comparative question-answering model; The first processing module 302 is used to input the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model into the trained evaluation model for formatting, thereby obtaining formatted data corresponding to the original question-answering model and formatted data corresponding to the comparative question-answering model, wherein the evaluation model is obtained by training a large language model based on a multi-stage data enhancement training strategy; The second processing module 303 is used to perform forward propagation processing on the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model based on at least one preset result label, to obtain the original question-answering score corresponding to each preset result label and the comparative question-answering score corresponding to each preset result label; The third processing module 304 is used to normalize the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label, respectively, to obtain the target result scores corresponding to the original question-answering model and the target result scores corresponding to the comparative question-answering model; Determination module 305 is used to determine the evaluation result labels of the original question and answer model and the comparative question and answer model based on the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparative question and answer model, and send the evaluation result labels to the target terminal device for display.

[0088] According to the technical solution provided by the embodiments of the present disclosure, by respectively acquiring an evaluation data set corresponding to the original question-answering model and an evaluation data set corresponding to the comparative question-answering model, and inputting the acquired two evaluation data sets into the trained evaluation model for formatting processing, the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model are obtained, and then the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model can be forward-propagated through at least one preset result label to obtain the original question-answering scores corresponding to each preset result label and the comparative question-answering scores corresponding to each preset result label, and then the original question-answering scores corresponding to the preset result labels and the comparative question-answering scores corresponding to the preset result labels can be respectively forward-propagated. The comparison question and answer scores corresponding to the result labels are normalized to obtain the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparison question and answer model. The evaluation result labels of the original question and answer model and the comparison question and answer model are determined by score comparison based on the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparison question and answer model. The evaluation result labels are sent to the current user graphical interface of the target terminal device for display, thereby improving the accuracy and efficiency of the question and answer model evaluation, enhancing the formatting processing capability of the evaluation data, improving the accuracy of the score calculation and normalization processing based on multiple preset result labels, and improving the reliability of the evaluation results by forward propagation processing and normalization processing data.

[0089] In some embodiments, the question-answering model evaluation device is also used to obtain multiple training conversation data and true labels corresponding to each training conversation data; input each training conversation data into the evaluation model, perform forward propagation processing based on at least one preset classification head, and obtain category data corresponding to each training conversation data; determine the cross entropy loss corresponding to the evaluation model according to the category data corresponding to each training conversation data and the true label corresponding to each training conversation data; and update the parameters in the evaluation model according to the cross entropy loss corresponding to the evaluation model through a loop iteration method.

[0090] In some embodiments, the question-answering model evaluation device is also used to obtain multiple training dialogue enhancement data; input each training dialogue enhancement data into the evaluation model for forward propagation processing to obtain category data corresponding to each training dialogue enhancement data; perform label mapping processing on each training dialogue enhancement data based on the category data corresponding to each training dialogue enhancement data to obtain a training label corresponding to each training dialogue enhancement data.

[0091] In some embodiments, the question-answering model evaluation device is also used to input each training conversation enhancement data into the evaluation model, perform forward propagation processing based on each preset classification head, and obtain training category data corresponding to each training conversation enhancement data; determine the training cross entropy loss corresponding to the evaluation model according to the training category data corresponding to each training conversation enhancement data and the training label corresponding to each training conversation enhancement data; and update the parameters in the evaluation model according to the training cross entropy loss corresponding to the evaluation model through a cyclic iteration method.

[0092] In some embodiments, the second processing module 303 is specifically used to vectorize the formatted data corresponding to the original question and answer model to obtain the feature vector corresponding to the original question and answer model; perform neural network propagation processing on the feature vector corresponding to the original question and answer model to obtain the original question and answer scores corresponding to each preset result label; perform vectorization processing on the formatted data corresponding to the comparative question and answer model to obtain the feature vector corresponding to the comparative question and answer model; perform neural network propagation processing on the feature vector corresponding to the comparative question and answer model to obtain the comparative question and answer scores corresponding to each preset result label.

[0093] In some embodiments, the question-answering model evaluation device is also used to obtain multiple generalized question training data; perform answer text generation processing on each generalized question training data through a multimodal large language model to obtain answer text data corresponding to each generalized question training data; perform score evaluation processing on the answer text data corresponding to each generalized question training data based on a generative pre-trained converter to obtain an evaluation score corresponding to each answer text data; determine each training dialogue enhancement data based on a preset evaluation score threshold and the evaluation score corresponding to each answer text data.

[0094] In some embodiments, the first processing module 302 is specifically used to input the evaluation data set corresponding to the original question and answer model into the trained evaluation model for data cleaning processing to obtain the various screening data corresponding to the original question and answer model; standardize the various screening data corresponding to the original question and answer model based on a preset prompt template to obtain the formatted data corresponding to the original question and answer model; input the evaluation data set corresponding to the comparative question and answer model into the trained evaluation model for data cleaning processing to obtain the various screening data corresponding to the comparative question and answer model; standardize the various screening data corresponding to the comparative question and answer model based on the preset prompt template to obtain the formatted data corresponding to the comparative question and answer model.

[0095] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0096] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0097] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 The electronic device 4 is merely an example and does not limit the electronic device 4 , and may include more or less components than those shown in the figure, or different components.

[0098] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0099] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. The memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0100] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0102] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A question-answering model evaluation method, characterized in that: include: Obtain the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model; Inputting the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model into the trained evaluation model for formatting, thereby obtaining formatted data corresponding to the original question-answering model and formatted data corresponding to the comparative question-answering model, wherein the evaluation model is obtained by training a large language model based on a plurality of training dialogue data, true labels corresponding to each of the training dialogue data, and training dialogue enhancement data; Based on at least one preset result label, forward propagation processing is performed on the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model, respectively, to obtain the original question-answering scores corresponding to each of the preset result labels and the comparative question-answering scores corresponding to each of the preset result labels; Normalizing the original question-answering scores corresponding to each of the preset result labels and the comparative question-answering scores corresponding to each of the preset result labels to obtain the target result scores corresponding to the original question-answering model and the target result scores corresponding to the comparative question-answering model; Based on the target result score corresponding to the original question and answer model and the target result score corresponding to the comparative question and answer model, the evaluation result labels of the original question and answer model and the comparative question and answer model are determined, and the evaluation result labels are sent to the target terminal device for display.

2. The question-answering model evaluation method according to claim 1, characterized in that: Before obtaining the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model, the method further includes: Obtaining a plurality of the training dialogue data and a true label corresponding to each of the training dialogue data; Inputting each of the training dialogue data into the evaluation model, performing forward propagation processing based on at least one preset classification head, and obtaining category data corresponding to each of the training dialogue data; Determining the cross entropy loss corresponding to the evaluation model according to the category data corresponding to each of the training conversation data and the true label corresponding to each of the training conversation data; The parameters in the evaluation model are updated according to the cross entropy loss corresponding to the evaluation model in a loop iterative manner.

3. The question-answering model evaluation method according to claim 2, characterized in that: The method further comprises: Acquire a plurality of the training dialogue enhancement data; Inputting each of the training dialogue enhancement data into the evaluation model for forward propagation processing to obtain category data corresponding to each of the training dialogue enhancement data; Label mapping processing is performed on each of the training dialogue enhancement data based on the category data corresponding to each of the training dialogue enhancement data to obtain a training label corresponding to each of the training dialogue enhancement data.

4. The question-answering model evaluation method according to claim 3, characterized in that: The method further comprises: Inputting each of the training dialogue enhancement data into the evaluation model, performing forward propagation processing based on each of the preset classification heads, and obtaining training category data corresponding to each of the training dialogue enhancement data; Determining the training cross entropy loss corresponding to the evaluation model according to the training category data corresponding to each of the training dialogue enhancement data and the training label corresponding to each of the training dialogue enhancement data; The parameters in the evaluation model are updated according to the training cross entropy loss corresponding to the evaluation model in a loop iterative manner.

5. The question-answering model evaluation method according to claim 1, characterized in that: The method of performing forward propagation processing on the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model based on at least one preset result label to obtain the original question-answering score corresponding to each preset result label and the comparative question-answering score corresponding to each preset result label includes: Performing vectorization processing on the formatted data corresponding to the original question-answering model to obtain a feature vector corresponding to the original question-answering model; Performing neural network propagation processing on the feature vector corresponding to the original question-answering model to obtain the original question-answering score corresponding to each of the preset result labels; Vectorizing the formatted data corresponding to the comparative question-answering model to obtain a feature vector corresponding to the comparative question-answering model; The feature vector corresponding to the comparative question-answering model is processed by neural network propagation to obtain the comparative question-answering scores corresponding to each of the preset result labels.

6. The question-answering model evaluation method according to claim 3, characterized in that: Before acquiring the plurality of training dialogue enhancement data, the method further includes: Obtain multiple generalization problem training data; Performing answer text generation processing on each of the generalized question training data through a multimodal large language model to obtain answer text data corresponding to each of the generalized question training data; Based on the generative pre-trained converter, score evaluation processing is performed on the answer text data corresponding to each of the generalized question training data to obtain an evaluation score corresponding to each of the answer text data; Each of the training dialogue enhancement data is determined according to a preset evaluation score threshold and an evaluation score corresponding to each of the answer text data.

7. The question-answering model evaluation method according to claim 1, characterized in that: The step of inputting the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model into the trained evaluation model for formatting, and obtaining the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model, comprises: Inputting the evaluation data set corresponding to the original question-answering model into the trained evaluation model for data cleaning processing to obtain each screening data corresponding to the original question-answering model; Based on a preset prompt template, each screening data corresponding to the original question-answering model is standardized to obtain formatted data corresponding to the original question-answering model; Inputting the evaluation data set corresponding to the comparative question-answering model into the trained evaluation model for data cleaning processing to obtain each screening data corresponding to the comparative question-answering model; Based on the preset prompt template, each screening data corresponding to the comparative question and answer model is standardized to obtain formatted data corresponding to the comparative question and answer model.

8. A question-answering model evaluation device, characterized in that: include: An acquisition module is used to acquire an evaluation data set corresponding to the original question-answering model and an evaluation data set corresponding to the comparative question-answering model; A first processing module is used to input the evaluation data set corresponding to the original question-answering model and the evaluation data set corresponding to the comparative question-answering model into the trained evaluation model for formatting, so as to obtain formatted data corresponding to the original question-answering model and formatted data corresponding to the comparative question-answering model, wherein the evaluation model is obtained by training a large language model based on a multi-stage data enhancement training strategy; A second processing module is used to perform forward propagation processing on the formatted data corresponding to the original question-answering model and the formatted data corresponding to the comparative question-answering model based on at least one preset result label, to obtain the original question-answering score corresponding to each of the preset result labels and the comparative question-answering score corresponding to each of the preset result labels; A third processing module is used to normalize the original question-answering scores corresponding to each of the preset result labels and the comparative question-answering scores corresponding to each of the preset result labels, respectively, to obtain the target result scores corresponding to the original question-answering model and the target result scores corresponding to the comparative question-answering model; A determination module is used to determine the evaluation result labels of the original question and answer model and the comparative question and answer model based on the target result scores corresponding to the original question and answer model and the target result scores corresponding to the comparative question and answer model, and send the evaluation result labels to the target terminal device for display.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images