Model confidence evaluation method and device, equipment, medium and program product

Abstract

The invention relates to the technical field of computers, in particular to a model confidence evaluation method and device, equipment, a medium and a program product, and the method comprises the steps: obtaining an information evaluation set; inputting test questions in the information evaluation set into the first model and the second model respectively to obtain to-be-evaluated answers output by the first model and reference answers output by the second model; inputting the reference answers and standard answers of the test questions in the information evaluation set into a pre-trained evaluation model, and outputting first deviation information between the reference answers and the standard answers; when determining that the test question and the standard answer meet a preset confidence condition based on the first deviation information, inputting the to-be-evaluated answer and the standard answer into an evaluation model, and outputting second deviation information between the to-be-evaluated answer and the standard answer; generating a confidence evaluation report of the first model according to the second deviation information; according to the scheme, the confidence test of the test questions and the standard answers is realized, and the universality and the mobility of the evaluation method are improved through the arrangement of the evaluation model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, equipment, medium and program product for evaluating model confidence. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models have become a hot topic in today's science and technology field. Before large language models are open to users, they must undergo rigorous confidence assessment to ensure the accuracy of the output results.

[0003] At present, the evaluation process of large language models is mostly coupled with the training process. This evaluation method will not only lead to incorrect output results due to the inaccuracy of training data, but also cause the evaluation of large language models to be neither universal nor transferable due to the tight coupling between the evaluation and training processes. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a model confidence evaluation method, device, equipment, medium and program product.

[0005] In a first aspect, the present invention provides a method for evaluating model confidence, comprising:

[0006] Obtain an information evaluation set, which includes test questions and standard answers; input the test questions into a first model to be evaluated and a second model that has participated in the evaluation and reached a preset evaluation standard, respectively, to obtain the answer to be evaluated output by the first model and the reference answer output by the second model; input the reference answer and the standard answer into a pre-trained evaluation model, and output first deviation information between the reference answer and the standard answer; when it is determined based on the first deviation information that the test question and the standard answer meet a preset confidence condition, input the answer to be evaluated and the standard answer into the evaluation model, and output second deviation information between the answer to be evaluated and the standard answer; generate a confidence evaluation report for the first model based on the second deviation information.

[0007] In a second aspect, the present invention provides a device for evaluating model confidence, comprising:

[0008] An acquisition module is used to obtain an information evaluation set, which includes test questions and standard answers; a first output module is used to input the test questions into a first model to be evaluated and a second model that has participated in the evaluation and reached a preset evaluation standard, respectively, to obtain the answer to be evaluated output by the first model and the reference answer output by the second model; a second output module is used to input the reference answer and the standard answer into a pre-trained evaluation model, and output first deviation information between the reference answer and the standard answer; a third output module is used to input the answer to be evaluated and the standard answer into the evaluation model when it is determined based on the first deviation information that the test question and the standard answer meet a preset confidence condition, and output second deviation information between the answer to be evaluated and the standard answer; a generation module is used to generate a confidence evaluation report for the first model based on the second deviation information.

[0009] In a third aspect, the present invention provides a computer device, comprising:

[0010] The memory and the processor are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to implement the model confidence evaluation method corresponding to the first aspect of the invention.

[0011] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the model confidence evaluation method corresponding to the first aspect of the invention.

[0012] In a fifth aspect, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for evaluating model confidence corresponding to the first aspect of the invention.

[0013] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0014] The model confidence evaluation method provided in the present embodiment obtains an information evaluation set; inputs the test questions in the information evaluation set into the first model and the second model respectively, and obtains the answer to be evaluated output by the first model and the reference answer output by the second model; inputs the reference answer and the standard answer corresponding to the test question in the information evaluation set into the pre-trained evaluation model, and outputs the first deviation information between the reference answer and the standard answer; when it is determined based on the first deviation information that the test question and the standard answer meet the preset confidence condition, inputs the answer to be evaluated and the standard answer into the evaluation model, and outputs the second deviation information between the answer to be evaluated and the standard answer; generates a confidence evaluation report of the first model according to the second deviation information; this scheme realizes the confidence test of the test question and the standard answer through the first deviation information between the reference answer and the standard answer, and on this basis, uses the pre-trained evaluation model to evaluate the first model, so that the evaluation process and the training process are completely decoupled, and the setting of the evaluation model improves the universality and transferability of the evaluation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 A schematic diagram of a flow chart of a method for evaluating model confidence provided by an embodiment of the present invention;

[0018] FIG. 2( a ) is a schematic diagram of a test question provided by an embodiment of the present invention;

[0019] FIG2( b ) is a schematic diagram of a standard answer provided by an embodiment of the present invention;

[0020] FIG2( c ) is a schematic diagram of a reference answer 1 provided in an embodiment of the present invention;

[0021] FIG2( d ) is a schematic diagram of a reference answer 2 provided in an embodiment of the present invention;

[0022] FIG2( e ) is a schematic diagram of a reference answer 3 provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a flow chart of a method for obtaining an information evaluation set provided in an embodiment of the present invention;

[0024] Figure 4A schematic diagram of a source text provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of source text corresponding to each segment provided in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of the evaluation data corresponding to each segment provided in an embodiment of the present invention;

[0027] Figure 7 A structural connection diagram of a device for evaluating model confidence provided by an embodiment of the present invention;

[0028] Figure 8 A structural connection diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0031] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0032] With the rapid development of artificial intelligence technology, large language models have become a hot topic in today's science and technology field. Before a large language model is open to users, it must undergo a strict confidence evaluation. The evaluation of the confidence of a large language model can be a comprehensive evaluation of the output results, or it can be an evaluation that focuses on a certain aspect. For example, the current application of large language models in intelligent question answering is very common, and its output results will directly affect users. In particular, when the output results do not conform to social values, it will have a very bad impact. Therefore, before a large language model is put into use, it is particularly important to conduct a confidence evaluation of its values.

[0033] At present, the evaluation process of large language models is mostly coupled with the training process. For example, during the training process, the validation set is used to evaluate the confidence of the large language model in real time. In this case, the evaluation process has actually become a part of the training process. Moreover, most training data relies on manual annotation, and the questions and answers in the validation set are also manually designed. Therefore, this evaluation method will not only lead to incorrect output results due to the inaccuracy of training data and validation data, but also the evaluation of large language models is not universal and transferable due to the tight coupling of the evaluation and training process.

[0034] To solve the above problems, the present invention provides a method for evaluating model confidence. Through the first deviation information between the reference answer and the standard answer, the confidence test of the test question and the standard answer is realized. On this basis, the first model is evaluated using a pre-trained evaluation model, so that the evaluation process is completely decoupled from the training process. The setting of the evaluation model improves the universality and transferability of the evaluation model.

[0035] According to an embodiment of the present invention, an embodiment of a method for evaluating model confidence is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] In this embodiment, a model confidence evaluation method is provided, which can be used in smart terminals such as mobile phones, tablet computers and PCs. Figure 1 is a flow chart of a method for evaluating model confidence according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0037] S101, obtaining an information evaluation set, the information evaluation set including test questions and standard answers.

[0038] The information evaluation set is a set of evaluation data, which is used to perform confidence evaluation on the pre-trained model. The confidence evaluation can be a comprehensive evaluation of the model output results, or an evaluation of the model output results in a certain aspect. The information evaluation set includes at least one set of evaluation data, and the evaluation data includes test questions and standard answers, and the test questions correspond to the standard answers. The test questions can be subjective questions or objective questions. This embodiment does not limit the number and type of test questions.

[0039] Specifically, at least one set of evaluation data is selected from the constructed evaluation database, and all the selected evaluation data form an information evaluation set. In order to make the evaluation more accurate and comprehensive, when selecting the evaluation data, evaluation data corresponding to a plurality of text categories can be selected, and the text category can be understood as the category to which the content corresponding to the test question belongs. It should also be noted that the evaluation data in the evaluation database can be generated by manual annotation or by an intelligent model, and this embodiment does not limit the construction process of the evaluation database.

[0040] S102, input the test questions into the first model to be evaluated and the second model that has participated in the evaluation and reached the preset evaluation standard, respectively, to obtain the answer to be evaluated output by the first model and the reference answer output by the second model.

[0041] Among them, the first model and the second model are similar models with the same functions. The difference between the two is that the first model is a model to be evaluated that has completed training but has not yet participated in the evaluation, and the second model is a model that has participated in the evaluation and reached the preset evaluation standard. The preset evaluation standard can be understood as a criterion for judging whether the model can be put into application. The criterion includes but is not limited to accuracy. A model that reaches the preset evaluation standard indicates that it can be put into application, otherwise it cannot. The preset evaluation standard can be determined by a technician in this field according to actual conditions. In addition, it should be noted that the number of the second model can be one or more. Exemplarily, for example, the first model and the second model are both large language models, the first model is a large language model to be evaluated, and there are multiple second models, namely Qwen (Tongyi Qianwen), ChatGLM (a human-computer dialogue model based on Transformer), and ChatGPT (Chat Generative Pre-trained Transformer).

[0042] Specifically, each test question in the information evaluation set is input into the first model to be evaluated and the second model that has participated in the evaluation and reached the preset evaluation standard, and the answer to be evaluated output by the first model and the reference answer output by the second model are obtained. It can be understood that one test question corresponds to one answer to be evaluated, at least one reference answer and one standard answer, and the number of reference answers is consistent with the number of the second model.

[0043] S103, input the reference answer and the standard answer into the pre-trained evaluation model, and output the first deviation information between the reference answer and the standard answer.

[0044] Among them, the evaluation model can be an LLM 7B base, or a model such as llama3 and transformer. The LLM7B base refers to the infrastructure or starting model of a large language model (LLM) with 7 billion parameters. llama3 is an auto-regressive language model. Transformer is a complex neural network model built by multiple stacked Transformer layers. After the evaluation model is built, the evaluation model is trained using a pre-built training set to obtain a pre-trained evaluation model. It should be noted that the content of the training data in the training set is related to the evaluation direction of the evaluation model. If the evaluation model is used to evaluate the value of the large language model, then the training data in the training set can be paragraph texts with complete semantic information in official articles and the label information corresponding to each paragraph text. Official articles can be obtained from official media newspapers, official websites, and various books promoted by the official. The label information can be obtained by processing the official articles by a small natural language processing model.

[0045] Specifically, for any test question, the reference answer and standard answer corresponding to the test question are input into the pre-trained evaluation model. When there is one reference answer, the first deviation information between the reference answer and the standard answer is directly calculated; when there are multiple reference answers, the first deviation information between each of the multiple reference answers and the standard answer is calculated separately. The first deviation information can be understood as the degree of deviation between the reference answer and the standard answer, which can be expressed as a percentage. The higher the percentage, the greater the difference between the reference answer and the standard answer, and the lower the percentage, the closer the reference answer is to the standard answer. When the test question is an objective question, there are only two situations where the reference answer and the standard answer are consistent or inconsistent. Therefore, the value of the first deviation information is 0% or 100%. When the test question is a subjective question, the value of the first deviation information is between 0% and 100%.

[0046] For example, FIG2(a) is a test question, FIG2(b) is the standard answer corresponding to the test question in FIG2(a), and FIG2(c), FIG2(d) and FIG2(e) are reference answers output by the three second models, respectively. Reference answer 1 and the standard answer are input into the pre-trained evaluation model, and the first deviation information between the output reference answer 1 and the standard answer is 0%, indicating that the two are semantically consistent; reference answer 2 and the standard answer are input into the pre-trained evaluation model, and the first deviation information between the output reference answer 2 and the standard answer is 100%, indicating that the two are semantically completely different; reference answer 3 and the standard answer are input into the pre-trained evaluation model, and the first deviation information between the output reference answer 3 and the standard answer is 0%, indicating that the two are semantically consistent.

[0047] S104, when it is determined based on the first deviation information that the test question and the standard answer meet the preset confidence condition, the answer to be evaluated and the standard answer are input into the evaluation model, and the second deviation information between the answer to be evaluated and the standard answer is output.

[0048] Specifically, the test questions and standard answers are generated manually or intelligently, so the settings of the test questions and standard answers cannot be guaranteed to be completely correct. When the test questions and standard answers are set incorrectly, if the confidence of the first model is evaluated through the second deviation information between the answer to be evaluated and the standard answer, the evaluation result will be wrong. Since the second model is a model that has participated in the evaluation and reached the preset evaluation standard, the reference answer output by the second model has a higher credibility. Therefore, this embodiment verifies whether the test questions and standard answers meet the preset confidence conditions through the first deviation information between the reference answer and the standard answer. When it is determined that the test questions and standard answers meet the preset confidence conditions, it means that the test questions and standard answers are set reasonably. At this time, the answer to be evaluated and the standard answer are input into the evaluation model, and the second deviation information between the answer to be evaluated and the standard answer is output.

[0049] In one example, determining whether a test question and a standard answer meet a preset confidence condition based on the first deviation information includes: when there is one item of first deviation information, comparing the first deviation information with a preset deviation threshold, and when the first deviation information is less than or equal to the preset deviation threshold, determining that the test question and the standard answer meet the confidence condition; or, when there are multiple items of first deviation information, comparing each item of first deviation information with the preset deviation threshold, and when the number of first deviation information that is less than or equal to the preset deviation threshold is greater than or equal to the preset number threshold, determining that the test question and the standard answer meet the confidence condition.

[0050] Specifically, judging whether the test question and the standard answer meet the preset confidence condition through the first deviation information includes two situations. The first situation is that the number of reference answers is 1, that is, there is only one first deviation information. At this time, it is only necessary to compare the first deviation information with the preset deviation threshold. When the first deviation information is greater than the preset deviation threshold, it means that there is a large difference between the reference answer and the standard answer, and the test question and the standard answer do not meet the preset confidence condition, and the test question needs to be replaced; when the first deviation information is less than or equal to the preset deviation threshold, it means that the reference answer and the standard answer have a high semantic similarity, and the test question and the standard answer meet the preset confidence condition, and the subsequent steps can be completed through the current test question and the standard answer. The preset deviation threshold can be understood as the maximum value of the deviation when the reference answer and the standard answer are semantically similar.

[0051] The second case is that there are multiple reference answers, that is, there are multiple first deviation information. At this time, each first deviation information needs to be compared with the preset deviation threshold. When the number of first deviation information less than or equal to the preset deviation threshold is greater than or equal to the preset number threshold, it means that the deviation degree between most reference answers and standard answers is within an acceptable range, that is, the test question and the standard answer meet the confidence condition; when the number of first deviation information less than or equal to the preset deviation threshold is less than the preset number threshold, it means that the deviation degree between most reference answers and standard answers exceeds the acceptable range, that is, the test question and the standard answer do not meet the confidence condition. For example, in one embodiment, there are 5 reference answers, the first deviation information between reference answer a and the standard answer is 90%, the first deviation information between reference answer b and the standard answer is 70%, the first deviation information between reference answer c and the standard answer is 75%, the first deviation information between reference answer d and the standard answer is 40%, and the first deviation information between reference answer e and the standard answer is 69%. The preset deviation threshold is 40%, and the preset number threshold is 80% of the number of reference answers (i.e. 4). By comparison, it can be seen that there is only one first deviation information that is less than or equal to the preset deviation threshold, 1<4, therefore, it is determined that the test question and the standard answer do not meet the preset confidence condition.

[0052] S105: Generate a confidence evaluation report of the first model according to the second deviation information.

[0053] Specifically, a confidence evaluation report is formed by the second deviation information of all test questions to determine whether the first model meets the preset evaluation criteria based on the evaluation report, and then determine whether the first model can be put into use. For example, when all the second deviation information is less than or equal to the preset deviation threshold, it is determined that the first model meets the preset evaluation criteria and can be put into use. When there is at least one test question with second deviation information higher than the preset deviation threshold, it is determined that the first model does not meet the preset evaluation criteria and cannot be put into use temporarily. Targeted training of the first model is required until the second deviation information of all test questions is less than or equal to the preset deviation threshold, and the first model can be put into use.

[0054] The model confidence evaluation method provided in the present embodiment obtains an information evaluation set; inputs the test questions in the information evaluation set into the first model and the second model respectively, and obtains the answer to be evaluated output by the first model and the reference answer output by the second model; inputs the reference answer and the standard answer corresponding to the test question in the information evaluation set into the pre-trained evaluation model, and outputs the first deviation information between the reference answer and the standard answer; when it is determined based on the first deviation information that the test question and the standard answer meet the preset confidence condition, inputs the answer to be evaluated and the standard answer into the evaluation model, and outputs the second deviation information between the answer to be evaluated and the standard answer; generates a confidence evaluation report of the first model according to the second deviation information; this scheme realizes the confidence test of the test question and the standard answer through the first deviation information between the reference answer and the standard answer, and on this basis, uses the pre-trained evaluation model to evaluate the first model, so that the evaluation process and the training process are completely decoupled, and the setting of the evaluation model improves the universality and transferability of the evaluation model.

[0055] In an optional embodiment, the method of obtaining the information evaluation set is optimized, such as Figure 3 As shown, obtaining the information evaluation set includes the following steps:

[0056] S301, obtaining source text.

[0057] The source text is the original text, such as text obtained from official websites, newspapers, and political books and publications. Official websites include but are not limited to official websites of national or local units at all levels, and official newspapers include but are not limited to People's Daily, Beijing Youth Daily, and Guangming Daily. The original text can be understood as policy documents, news reports, and comments obtained from official websites, official media newspapers, and political books and publications.

[0058] Specifically, the source texts are periodically obtained from official websites, official media newspapers, political books and publications, etc. This embodiment does not limit the acquisition cycle and the number of source texts, for example, the acquisition cycle can be days, and the number of source texts can be one or more. Figure 4What is shown is a source text obtained from People's Daily Online.

[0059] S302, according to the preset processing task, the source text is structured to obtain structured data corresponding to the source text, and the structured data includes text fragments, fragment entities, fragment viewpoints, emotional tendencies and text categories.

[0060] Among them, the preset processing tasks include but are not limited to text segmentation, entity recognition, opinion extraction, sentiment analysis (positive / negative) and content clustering.

[0061] Specifically, a small model capable of natural language processing is called to perform corresponding structural processing on the source text according to the preset processing task to obtain structured data corresponding to the source text. Figure 4 The source text shown in the figure is segmented to obtain Figure 5 The multiple segments shown in the figure are as follows; a small model is called to perform entity recognition on each segmented segment to obtain the entity corresponding to each segment (i.e., segment entity); opinion extraction is performed on each segment to obtain the segment opinion corresponding to each segment; sentiment analysis is performed on each segment to obtain the sentiment tendency corresponding to each segment; content clustering is performed on each segment to obtain the text category corresponding to each segment. It can be understood that after the source text is structured, at least one set of structured data will be obtained, and the number of groups of structured data corresponds to the segments into which the source text is segmented. Each set of structured data includes text segments, segment entities, segment opinions, sentiment tendencies, and text categories.

[0062] S303, generating evaluation data according to the text fragments, fragment entities, fragment viewpoints and sentiment tendencies, the evaluation data including test questions and standard answers.

[0063] Specifically, through the template method or preset algorithm, the text fragments, fragment entities, fragment viewpoints and emotional tendencies in each set of structured data are semantically analyzed and processed to obtain at least one set of evaluation data corresponding to each set of structured data, and each set of evaluation questions includes test questions and standard answers. Figure 6 Test question 1 shown uses the template method, according to Figure 5 The structured data corresponding to the first fragment in the test question 2 is generated by using the template method. Figure 5 The structured data corresponding to the second fragment in the test question 3 is generated by using the template method. Figure 5 The structured data corresponding to the third fragment in is generated.

[0064] S304, classifying the evaluation data according to the text category, and constructing an evaluation database based on the classified evaluation data.

[0065] Specifically, all obtained evaluation data are classified according to the text category of the segment to which the test question belongs, and an evaluation database is constructed based on the classified evaluation data.

[0066] S305: Select evaluation data corresponding to each text category from the evaluation database to form an information evaluation set.

[0067] Specifically, in order to ensure the comprehensiveness and accuracy of the evaluation, at least one set of evaluation data is selected from the evaluation data corresponding to each text category in the evaluation database, and the selected evaluation data constitute an evaluation data set.

[0068] The construction of the evaluation database in this embodiment is free from reliance on manual annotation. The evaluation data fully reflects my country's national conditions, national laws / regulations and the value concepts of official media newspapers, and provides a high-quality information evaluation set, so that the evaluation results are more in line with the currently advocated value viewpoints.

[0069] In an optional embodiment, after S302, the process further includes: classifying the segment contents according to the text categories; and constructing a training database according to the classified segment contents.

[0070] Exemplarily, after the source text is structured to obtain structured data, the fragment content is classified according to the text category, and a training database is constructed based on the classified fragment content. The training database constructed here is used to provide training data for the first model that does not meet the preset evaluation standard, and the training data provided here is used to conduct targeted training on the deficiencies of the first model.

[0071] In an optional embodiment, after S105, it also includes: when there is second deviation information in the confidence evaluation report that is greater than a preset deviation threshold, determining the text category of the test question corresponding to the second deviation information as the text category to be trained; selecting at least one group of training data corresponding to the text category to be trained from the training database to form a training set; and using the training set to train the first model to obtain the trained first model.

[0072] Exemplarily, when there is at least one second deviation information greater than a preset deviation threshold in the confidence evaluation report of the first model, it means that the first model has not reached the preset evaluation standard and cannot be put into use. Therefore, it is necessary to train the deficiencies of the first model. The specific training method is to obtain the text category of the test question corresponding to the second deviation information greater than the preset deviation threshold, and determine the text category as the text category to be trained. Then, at least one group of training data corresponding to the text category to be trained is selected from the training database, and a training set is formed by the selected at least one group of training data. The first model is targetedly trained using the training set to obtain the trained first model. This embodiment uses a training set composed of training data corresponding to the text category to be trained to perform targeted training on the first model, which is conducive to making up for the deficiencies of the first model and improving the accuracy of the output results.

[0073] In an optional embodiment, after training the first model using the training set to obtain the trained first model, the method also includes: selecting evaluation data corresponding to the text category to be trained from the evaluation database; evaluating the trained first model using the evaluation data corresponding to the text category to be trained, until the second deviation information in the confidence evaluation report is less than or equal to the preset deviation threshold, and then ending the evaluation operation.

[0074] Exemplarily, after obtaining the trained first model, it is necessary to re-evaluate the confidence of the first model. Specifically, the evaluation data corresponding to the text category to be trained is selected from the evaluation database, and the evaluation data corresponding to the text category to be trained is used to re-evaluate the trained first model according to steps S101 to S105, until all the second deviation information in the regenerated confidence evaluation report is less than or equal to the preset deviation threshold, indicating that the first model meets the preset evaluation standard, at which time the evaluation operation can be terminated and the first model can be put into use.

[0075] This embodiment provides a device for evaluating model confidence. Figure 7 As shown, including:

[0076] The acquisition module 701 is used to acquire an information evaluation set, which includes test questions and standard answers.

[0077] The first output module 702 is used to input the test questions into the first model to be evaluated and the second model that has participated in the evaluation and reached the preset evaluation standard, and obtain the answer to be evaluated output by the first model and the reference answer output by the second model.

[0078] The second output module 703 is used to input the reference answer and the standard answer into the pre-trained evaluation model, and output the first deviation information between the reference answer and the standard answer.

[0079] The third output module 704 is used to input the answer to be evaluated and the standard answer into the evaluation model and output the second deviation information between the answer to be evaluated and the standard answer when it is determined based on the first deviation information that the test question and the standard answer meet the preset confidence condition.

[0080] The generating module 705 is used to generate a confidence evaluation report of the first model according to the second deviation information.

[0081] In some optional embodiments, the acquisition module 701 includes:

[0082] The first acquisition submodule is used to acquire the source text; the processing submodule is used to perform structured processing on the source text according to the preset processing task to obtain the structured data corresponding to the source text, and the structured data includes text fragments, fragment entities, fragment viewpoints, emotional tendencies and text categories; the generation submodule is used to generate evaluation data according to the text fragments, fragment entities, fragment viewpoints and emotional tendencies, and the evaluation data includes test questions and standard answers; the first construction submodule is used to classify the evaluation data according to the text category, and construct an evaluation database according to the classified evaluation data; the generation submodule is used to select the evaluation data corresponding to each text category from the evaluation database to form an information evaluation set.

[0083] In some optional embodiments, the acquisition module 701 further includes:

[0084] The classification submodule is used to classify the fragment content according to the text category; the second construction submodule is used to construct a training database based on the classified fragment content.

[0085] In some optional embodiments, the device further comprises:

[0086] A first comparison module is used to compare the first deviation information with a preset deviation threshold when there is one first deviation information, and determine that the test question and the standard answer meet the confidence condition when the first deviation information is less than or equal to the preset deviation threshold; or, a second comparison module is used to compare each first deviation information with the preset deviation threshold when there are multiple first deviation information, and determine that the test question and the standard answer meet the confidence condition when the number of first deviation information less than or equal to the preset deviation threshold is greater than or equal to the preset number threshold.

[0087] In some optional embodiments, the device further comprises:

[0088] A determination module is used to determine the text category of the test question corresponding to the second deviation information as the text category to be trained when the second deviation information in the confidence evaluation report is greater than a preset deviation threshold; a first selection module is used to select at least one group of training data corresponding to the text category to be trained from the training database to form a training set; and a training module is used to train the first model using the training set to obtain the trained first model.

[0089] In some optional embodiments, the device further comprises:

[0090] The second selection module is used to select evaluation data corresponding to the text category to be trained from the evaluation database; the evaluation module is used to evaluate the trained first model using the evaluation data corresponding to the text category to be trained, until the second deviation information in the confidence evaluation report is less than or equal to the preset deviation threshold, and then the evaluation operation is terminated.

[0091] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0092] The model confidence evaluation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0093] The embodiment of the present invention also provides a computer device having the above Figure 7 The evaluation means of model confidence is shown.

[0094] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8A processor 10 is taken as an example.

[0095] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0096] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0097] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0099] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0100] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0101] In addition to the above-mentioned computer device and computer-readable storage medium, the embodiments of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the sound source localization method provided by any embodiment of the present application.

[0102] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating model confidence, characterized in that: include: Obtaining an information evaluation set, wherein the information evaluation set includes test questions and standard answers; Input the test questions into the first model to be evaluated and the second model that has participated in the evaluation and reached the preset evaluation standard, respectively, to obtain the answer to be evaluated output by the first model and the reference answer output by the second model; Inputting the reference answer and the standard answer into a pre-trained evaluation model, and outputting first deviation information between the reference answer and the standard answer; When it is determined based on the first deviation information that the test question and the standard answer meet a preset confidence condition, the answer to be evaluated and the standard answer are input into the evaluation model, and second deviation information between the answer to be evaluated and the standard answer is output; A confidence evaluation report of the first model is generated according to the second deviation information.

2. The method according to claim 1, characterized in that The obtaining of information evaluation set includes: Get the source text; According to the preset processing task, the source text is structured to obtain structured data corresponding to the source text, wherein the structured data includes text fragments, fragment entities, fragment viewpoints, emotional tendencies and text categories; Generate evaluation data according to the text segment, the segment entity, the segment viewpoint and the sentiment tendency, wherein the evaluation data includes test questions and standard answers; Classifying the evaluation data according to the text categories, and constructing an evaluation database based on the classified evaluation data; The evaluation data corresponding to each of the text categories is selected from the evaluation database to form the information evaluation set.

3. The method according to claim 2, characterized in that After the source text is structured according to the preset processing task to obtain structured data corresponding to the source text, the method further includes: Classifying the content of the fragment according to the text category; A training database is constructed based on the classified fragment contents.

4. The method according to claim 1, characterized in that: The determining, based on the first deviation information, that the test question and the standard answer meet a preset confidence condition includes: When there is one of the first deviation information, comparing the first deviation information with a preset deviation threshold, and when the first deviation information is less than or equal to the preset deviation threshold, determining that the test question and the standard answer meet the confidence condition; Alternatively, when there are multiple pieces of the first deviation information, each piece of the first deviation information is compared with a preset deviation threshold. When the number of first deviation information that is less than or equal to the preset deviation threshold is greater than or equal to a preset number threshold, it is determined that the test question and the standard answer meet the confidence condition.

5. The method according to claim 3, characterized in that: After generating the confidence evaluation report of the first model according to the second deviation information, the method further includes: When the second deviation information in the confidence evaluation report is greater than a preset deviation threshold, determining the text category of the test question corresponding to the second deviation information as the text category to be trained; Selecting at least one group of training data corresponding to the text category to be trained from the training database to form a training set; The first model is trained using the training set to obtain a trained first model.

6. The method according to claim 5, characterized in that After the first model is trained using the training set to obtain the trained first model, the method further includes: Selecting evaluation data corresponding to the text category to be trained from the evaluation database; The trained first model is evaluated using the evaluation data corresponding to the text category to be trained, and the evaluation operation is terminated when the second deviation information in the confidence evaluation report is less than or equal to the preset deviation threshold.

7. A device for evaluating model confidence, characterized in that: include: An acquisition module, used to acquire an information evaluation set, wherein the information evaluation set includes test questions and standard answers; A first output module, used to input the test questions into a first model to be evaluated and a second model that has participated in the evaluation and reached a preset evaluation standard, respectively, to obtain the answer to be evaluated output by the first model and the reference answer output by the second model; A second output module, used for inputting the reference answer and the standard answer into a pre-trained evaluation model, and outputting first deviation information between the reference answer and the standard answer; A third output module, configured to input the answer to be evaluated and the standard answer into the evaluation model and output second deviation information between the answer to be evaluated and the standard answer when it is determined based on the first deviation information that the test question and the standard answer meet a preset confidence condition; A generation module is used to generate a confidence evaluation report of the first model based on the second deviation information.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model confidence evaluation method described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the model confidence evaluation method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model confidence evaluation method according to any one of claims 1 to 6 are implemented.

Images