A method, system and device for government affair large model detection

Abstract

The application discloses a method, system and device for government affair large model detection, comprising obtaining a first answer set through a target to-be-tested model according to a first query text set; obtaining a second answer set according to the first answer set; obtaining a second query text set corresponding to the second answer set according to the second answer set; obtaining a second Prompt set corresponding to the second answer set according to the second query text set; obtaining a third answer set through the target to-be-tested model according to the second Prompt set; and obtaining an evaluation result of the target to-be-tested model according to the first answer set, the second answer set and the third answer set. At least the problem that the existing government affair large model evaluation method cannot evaluate the understanding ability of the query text and the evaluation result cannot provide help for the optimization of the government affair large model is solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence, in particular to a method, system and device for government affair large model detection. BACKGROUND

[0002] Text generation tasks in the government field usually need to deal with complex background information, legal norms and policy requirements, involving accurate understanding and reasonable expression of a large amount of knowledge. At the same time, strict filtering of negative information, protection of data privacy and security, and ensuring that the generated content meets certain standardized requirements make the requirements of the government field for text generation more demanding and special. Therefore, in this field, the text generation technology must have high knowledge understanding ability and strict content control ability to meet the actual application requirements.

[0003] Traditional text generation methods can realize basic text creation in some scenarios, but due to their limitations in processing specific domain knowledge, understanding government language and policy content, they often cannot meet the high requirements of the government field for text quality and credibility. Traditional methods often cannot avoid content logic errors, non-standard sentences, information omissions and other problems, which directly affect the quality and usability of the generated text. In addition, with the huge and complex data involved in the government field, existing technologies face great challenges in understanding and processing this information.

[0004] The government large model has high requirements for correctly quoting government files or data in the knowledge base to output accurate answers, and since users using the government large model may lack experience in asking questions to the AI model, the query text used is less standardized, and existing government large model evaluation methods generally only evaluate the comprehensive accuracy of the government large model without evaluating the understanding ability of the query text, resulting in evaluation results that cannot help optimize the government large model. SUMMARY

[0005] The application provides a method, system and device for government large model detection, which provides a detection scheme for the characteristics of the government large model database, and at least solves the problem that existing government large model evaluation methods generally only evaluate the comprehensive accuracy of the government large model without evaluating the understanding ability of the query text, and the evaluation results cannot help optimize the government large model.

[0006] In a first aspect, the application provides a method for government large model detection, comprising:

[0007] obtaining a first answer set through a target to-be-tested model according to a first query text set;

[0008] According to the first answer set, a second answer set is obtained, and the second answer set is configured to include answers in the first answer set that do not match a standard answer;

[0009] According to the second answer set, a second query text set corresponding to the second answer set is obtained, and the second query text set is configured to include query texts in the first query text set that correspond to answers of the second answer set;

[0010] According to the second query text set, a second Prompt set corresponding to the second answer set is obtained, and the second Prompt set is configured to include Prompts generated according to query texts in the second query text set;

[0011] According to the second Prompt set, a third answer set is obtained by the target to-be-tested model;

[0012] According to the first answer set, the second answer set, and the third answer set, an evaluation result of the target to-be-tested model is obtained.

[0013] Optionally, the obtaining, according to the first answer set, of a second answer set includes:

[0014] According to the first answer set, each answer in the first answer set and a query text corresponding to each answer are obtained;

[0015] According to each answer in the first answer set and a query text corresponding to each answer, it is determined whether the each answer matches a standard answer;

[0016] An answer that does not match the standard answer in the each answer is added to the second answer set.

[0017] Optionally, before the step of obtaining, according to the second query text set, a second Prompt set corresponding to the second answer set, the method further includes:

[0018] According to the target to-be-tested model, a knowledge graph of a knowledge base accessed by the target to-be-tested model is obtained;

[0019] According to the knowledge graph, a Prompt template is constructed based on entities, relationships, and attributes in the knowledge graph.

[0020] Optionally, the obtaining, according to the second query text set, of a second Prompt set corresponding to the second answer set includes:

[0021] According to the second query text set, a query text in the second query text set is obtained;

[0022] parsing the query text to obtain a key parameter corresponding to the query text;

[0023] obtaining a Prompt corresponding to the query text according to the key parameter and the Prompt template;

[0024] obtaining a second Prompt set according to the Prompts corresponding to all query texts in the second query text set.

[0025] Optionally, the obtaining of the evaluation result of the target to-be-tested model according to the first answer set, the second answer set and the third answer set comprises:

[0026] obtaining a proportion of answers in the second answer set to answers in the first answer set according to the first answer set and the second answer set;

[0027] obtaining a first evaluation parameter according to the proportion;

[0028] obtaining a similarity between answers in the third answer set and answers in the second answer set according to the second answer set and the third answer set;

[0029] obtaining a second evaluation parameter according to the similarity;

[0030] obtaining a correctness rate of answers in the third answer set according to the third answer set;

[0031] obtaining a third evaluation parameter according to the correctness rate;

[0032] obtaining the evaluation result of the target to-be-tested model according to the first evaluation parameter, the second evaluation parameter and the third evaluation parameter.

[0033] Optionally, the obtaining of the similarity between answers in the third answer set and answers in the second answer set according to the second answer set and the third answer set comprises:

[0034] obtaining a second word vector of the second answer set and a third word vector of the third answer set according to the second answer set and the third answer set;

[0035] obtaining the similarity between answers in the third answer set and answers in the second answer set according to the second word vector and the third word vector.

[0036] Optionally, the evaluation result of the target to-be-tested model comprises a comprehensive evaluation result, a query text semantic understanding evaluation result and a reasoning accuracy rate evaluation result.

[0037] The evaluation result of the target to-be-tested model is obtained according to the first evaluation parameter, the second evaluation parameter and the third evaluation parameter, and includes:

[0038] The comprehensive evaluation result of the target to-be-tested model is obtained according to the first evaluation parameter and a preset index system;

[0039] The query text semantic understanding evaluation result of the target to-be-tested model is obtained according to the second evaluation parameter and a preset index system;

[0040] The inference accuracy evaluation result of the target to-be-tested model is obtained according to the third evaluation parameter and a preset index system.

[0041] Optionally, the evaluation result of the target to-be-tested model is obtained according to the first answer set, the second answer set and the third answer set, and includes:

[0042] The proportion of answers in the second answer set to answers in the first answer set is obtained according to the first answer set and the second answer set;

[0043] The similarity of answers in the third answer set to answers in the second answer set is obtained according to the second answer set and the third answer set;

[0044] The correctness of answers in the third answer set is obtained according to the third answer set;

[0045] The evaluation result of the target to-be-tested model is calculated according to the following formula P :

[0046] ;

[0047] wherein, is a preset comprehensive accuracy parameter, is a preset semantic understanding parameter, is a preset inference correctness parameter, is the proportion of answers in the second answer set to answers in the first answer set, is the similarity of answers in the third answer set to answers in the second answer set, is the correctness of answers in the third answer set.

[0048] In still another aspect, the present application provides a system for government affairs large model detection, including:

[0049] A model management platform is configured to: obtain a first answer set by a target to-be-tested model according to a first query text set;

[0050] The model evaluation platform is configured to: obtain a second answer set according to the first answer set, the second answer set being configured to include answers in the first answer set that do not match standard answers; obtain a second query text set corresponding to the second answer set according to the second answer set, the second query text set being configured to include query texts in the first query text set that correspond to answers of the second answer set; obtain a second Prompt set corresponding to the second answer set according to the second query text set, the second Prompt set being configured to include Prompts generated according to query texts in the second query text set; obtain a third answer set by the target to-be-tested model according to the second Prompt set; and obtain an evaluation result of the target to-be-tested model according to the first answer set, the second answer set and the third answer set.

[0051] In still another aspect, the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method for government affair big model detection according to any of the embodiments of the first aspect.

[0052] In still another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and a processor executes the computer program to implement the method for government affair big model detection according to any of the embodiments of the first aspect.

[0053] Compared with the prior art, the present application has at least the following beneficial effects:

[0054] A method, system and device for government affairs large model detection, including obtaining a first answer set from a target to-be-tested model according to a first query text set; obtaining a second answer set according to the first answer set, the second answer set being configured to include answers in the first answer set that do not match standard answers; obtaining a second query text set corresponding to the second answer set according to the second answer set, the second query text set being configured to include query texts in the first query text set corresponding to the answers of the second answer set; obtaining a second Prompt set corresponding to the second answer set according to the second query text set, the second Prompt set being configured to include Prompts generated according to the query texts in the second query text set; obtaining a third answer set from the target to-be-tested model according to the second Prompt set; and obtaining an evaluation result of the target to-be-tested model according to the first answer set, the second answer set and the third answer set. At least the existing government affairs large model evaluation method generally only evaluates the comprehensive accuracy of the government affairs large model, does not evaluate the understanding ability of the query text, and the evaluation result cannot provide help for the optimization of the government affairs large model, thereby improving the model detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual proportions.

[0056] Figure 1 A flowchart of a method for government affairs large model detection in the present application;

[0057] Figure 2 An architecture diagram of a system for government affairs large model detection in the present application;

[0058] Figure 3 A structural diagram of an electronic device in the present application.

[0059] Markings in the drawings:

[0060] 110 - model management platform, 120 - model evaluation platform,

[0061] 101 - processor, 102 - communication bus, 103 - network interface, 104 - user interface, 105 - memory.

[0062] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0063] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0064] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0065] Embodiment 1

[0066] Figure 1 A flowchart of a method for government affairs large model detection provided by the present application is shown. As shown in the figure, a method for government affairs large model detection specifically includes the following steps S1-S6: Figure 1

[0067] S1, according to the first query text set, obtaining a first answer set through the target to-be-tested model.

[0068] Specifically, the first answer set is configured to include answers obtained by asking the target to-be-tested model through the query text in the first query text set.

[0069] Optionally, the query text in the first query text set includes one or more of standard query text, dialect query text, or other semantically similar query text obtained from the standard query text.

[0070] Optionally, the first answer set further includes a data table recording the mapping relationship between the query text and the answer.

[0071] Optionally, the target to-be-tested model can access a specified knowledge base, such as a test knowledge base.

[0072] S2, obtaining a second answer set according to the first answer set.

[0073] ​Specifically, the second answer set is configured to include answers in the first answer set that do not match the standard answer.

[0074] Optionally, whether each answer in the first answer set matches the standard answer is determined by keyword or key parameter comparison between the answer and the standard answer.

[0075] Optionally, the data table recording the mapping relationship between the query text and the standard answer is further included.

[0076] S3, obtaining a second query text set corresponding to the second answer set according to the second answer set.

[0077] Specifically, the second query text set is configured to include query texts in the first query text set corresponding to answers in the second answer set.

[0078] Optionally, the query texts corresponding to the answers in the second answer set are obtained through the data table recording the mapping relationship between the query text and the answer.

[0079] S4, obtaining a second Prompt set corresponding to the second answer set according to the second query text set.

[0080] Specifically, the second Prompt set is configured to include Prompts generated according to query texts in the second query text set.

[0081] Optionally, the Prompt corresponding to the query text can be obtained through a preset rule such as a preset Prompt template and / or Prompt mapping relationship, and the keyword or key parameter of the query text, and the Prompt is semantically similar to the corresponding query text.

[0082] Optionally, the second Prompt set corresponding to the second answer set can be obtained by manually writing or manually correcting the automatically generated Prompt.

[0083] S5, obtaining a third answer set through the target to-be-tested model according to the second Prompt set.

[0084] Specifically, the third answer set is configured to include answers obtained by asking the target to-be-tested model through the Prompt in the second Prompt set.

[0085] S6, obtaining an evaluation result of the target to-be-tested model according to the first answer set, the second answer set and the third answer set.

[0086] Optionally, according to the proportion of the second answer set in the first answer set, the comprehensive accuracy of the target to-be-tested model can be obtained, according to the similarity between the second answer set and the third answer set, the understanding ability of the target to-be-tested model for the query text can be obtained, and according to the accuracy of the third answer set, the actual accuracy of the target to-be-tested model after optimizing the query text can be obtained.

[0087] Optionally, a plurality of groups of first answer sets, second answer sets and third answer sets can be obtained using different first query texts, and evaluation results of the target to-be-tested model can be obtained according to the plurality of groups of first answer sets, second answer sets and third answer sets.

[0088] By comparing the answers corresponding to the original query statement and the answers corresponding to the original query statement after optimization, the performance of the government big model is evaluated from multiple angles, at least the problem that the existing government big model evaluation method generally only evaluates the comprehensive accuracy of the government big model and does not evaluate the understanding ability of the query text, and the evaluation result cannot help the optimization of the government big model is solved.

[0089] Embodiment 2

[0090] Based on embodiment 1, a method for detecting a government big model includes:

[0091] S1, obtaining a first answer set by a target to-be-tested model according to a first query text set.

[0092] Specifically, the first answer set is configured to include answers obtained by querying the target to-be-tested model with query texts in the first query text set.

[0093] Optionally, the query texts in the first query text set include one or more of standard query texts, dialect query texts or other semantically similar query texts obtained according to standard query texts.

[0094] Optionally, the first answer set further includes a data table recording the mapping relationship between the query text and the answer.

[0095] Optionally, the target to-be-tested model can access a specified knowledge base, such as a test knowledge base.

[0096] Optionally, the target to-be-tested model can further include one or more of a semantic recognition model for parsing the query text, a Prompt template or a Prompt mapping, etc.

[0097] S2, obtaining a second answer set according to the first answer set.

[0098] Specifically, the second answer set is configured to include answers in the first answer set that do not match the standard answer.

[0099] Optionally, whether the answer is consistent with the standard answer is determined by comparing the keywords or key parameters of each answer in the first answer set with the standard answer.

[0100] Optionally, the data table recording the mapping relationship between the query text and the standard answer is further included.

[0101] Optionally, the second answer set is obtained according to the first answer set, comprising:

[0102] According to the first answer set, each answer in the first answer set and the query text corresponding to each answer are obtained.

[0103] According to the first answer set, each answer in the first answer set and the query text corresponding to each answer are obtained.

[0104] The answer inconsistent with the standard answer is added to the second answer set.

[0105] Specifically, the query text belongs to the first query text set.

[0106] Optionally, in the step of obtaining each answer corresponding to the query text according to the first answer set, the data table recording the mapping relationship between the query text and the answer is used to obtain the query text corresponding to each answer.

[0107] For example:

[0108] The answer 1 is "the latest population of a certain place is 1 million", and the corresponding query text is "how much is the latest population of a certain place";

[0109] The answer 2 is "the latest land area of a certain place is 1200 square kilometers", and the corresponding query text is "how much is the latest land area of a certain place";

[0110] In the step of determining whether each answer is consistent with the standard answer according to the first answer set and the query text corresponding to each answer, each answer needs to be compared with the pre-defined standard answer to see if they match.

[0111] The standard answer is a pre-defined, verified or generally accepted answer.

[0112] For example, the standard answer is:

[0113] The standard answer 1 is "the latest population of a certain place is 120 million";

[0114] The standard answer 2 is "the latest land area of a certain place is 1200 square kilometers";

[0115] In the step of adding the answers inconsistent with the standard answer to the second answer set, we will compare each answer in the "first answer set" with the standard answer one by one:

[0116] Example:

[0117] Answer 1 (the latest population of a certain place is 1 million) is inconsistent with the standard answer 1 (the latest population of a certain place is 120 million);

[0118] Answer 2 (the latest land area of a certain place is 1200 square kilometers) is consistent with the standard answer 2 (the latest land area of a certain place is 1200 square kilometers);

[0119] Add the answers inconsistent with the standard answer to the second answer set:

[0120] In the above example, answer 1 is inconsistent with the standard answer, so answer 1 is added to the second answer set.

[0121] S3, according to the second answer set, obtain the second query text set corresponding to the second answer set.

[0122] Specifically, the second query text set is configured to include the query texts in the first query text set corresponding to the answers of the second answer set.

[0123] Optionally, the query texts corresponding to the answers of the second answer set are obtained through a data table recording the mapping relationship between the query texts and the answers.

[0124] S4, according to the second query text set, obtain the second Prompt set corresponding to the second answer set.

[0125] Specifically, the second Prompt set is configured to include the prompts generated according to the query texts in the second query text set.

[0126] Optionally, the prompts corresponding to the query texts can be obtained through preset rules such as preset prompt templates and / or prompt mapping relationships, and keywords or key parameters of the query texts, and the prompts are semantically similar to the corresponding query texts.

[0127] Optionally, the second Prompt set corresponding to the second answer set can be obtained by manually writing or manually correcting the automatically generated prompts.

[0128] Optionally, before the step of obtaining the second Prompt set corresponding to the second answer set according to the second query text set, it further includes:

[0129] According to the target to-be-tested model, obtain the knowledge graph of the knowledge base accessed by the target to-be-tested model;

[0130] According to the knowledge graph, Prompt templates are constructed based on entities, relationships and attributes in the knowledge graph.

[0131] Optionally, according to the knowledge graph, Prompt templates are constructed based on entities, relationships and attributes in the knowledge graph, comprising:

[0132] A Prompt is established, and a question placeholder is added in the Prompt;

[0133] According to the entities, relationships and attributes in the knowledge graph, the classification of knowledge is obtained;

[0134] According to the classification of the entity, a knowledge placeholder corresponding to the classification of the entity is added in the Prompt to obtain a Prompt template.

[0135] Specifically, a Prompt template is as follows:

[0136] Question: How ** / ** is something, etc.

[0137] Knowledge: answers are obtained according to files related to **, application procedures of **, application materials of **, and application acceptance agencies of **. Among them, ** in the question is a key parameter or keyword of the query text, and ** in the knowledge is a key parameter, keyword or key parameter, keyword in the knowledge graph corresponding to the entity, relationship or attribute of the query text.

[0138] Optionally, the method for generating a Prompt according to a query text set comprises:

[0139] According to the query text, the query text is parsed, and the key parameters or keywords of the query text are extracted, and then the key parameters or keywords of the query text are added to the question placeholder of the Prompt template;

[0140] According to the query text, the query text is parsed, and the key parameters or keywords of the query text are extracted, and then according to the category of the key parameters or keywords of the query text and the query result in the knowledge graph, the key parameters or keywords of the query text are added to the knowledge placeholder of the Prompt template;

[0141] According to the Prompt template after adding the key parameters or keywords, a Prompt corresponding to the query text is obtained.

[0142] Optionally, after the step of according to the query text, parsing the query text, extracting the key parameters or keywords of the query text, and adding the key parameters or keywords of the query text to the question placeholder of the Prompt template, the method further comprises judging the similarity between the text corresponding to the question and the query text, and when the similarity is lower than a preset threshold, an error is reported.

[0143] Specifically, if the query text is "how to apply for a car purchase subsidy";

[0144] First, the query text is parsed to obtain the keyword "apply for a car purchase subsidy"; according to the keyword, the keyword of the query text is added to the question placeholder of the Prompt template to obtain:

[0145] Question: How to apply for a car purchase subsidy;

[0146] According to the category after the keyword of the query text, according to the query result in the knowledge graph according to the keyword or the category after the keyword of the query text, the information of the car purchase subsidy, the "car purchase subsidy guide document" and the like is obtained;

[0147] The keyword or keyword of the query text is added to the knowledge placeholder of the Prompt template to obtain:

[0148] Knowledge: According to the files related to the "car purchase subsidy guide document", the application process of the car purchase subsidy, the application materials of the car purchase subsidy, the receiving agency of the car purchase subsidy application, etc. to obtain the answer.

[0149] Finally, the Prompt corresponding to the query text "how to apply for a car purchase subsidy" is:

[0150] Question: How to apply for a car purchase subsidy;

[0151] Knowledge: According to the files related to the "car purchase subsidy guide document", the application process of the car purchase subsidy, the application materials of the car purchase subsidy, the receiving agency of the car purchase subsidy application, etc. to obtain the answer.

[0152] Optionally, according to the second query text set, a second Prompt set corresponding to the second answer set is obtained, including:

[0153] According to the second query text set, the query text in the second query text set is obtained;

[0154] The query text is parsed to obtain the key parameters corresponding to the query text;

[0155] According to the key parameters and the Prompt template, the Prompt corresponding to the query text is obtained;

[0156] According to the Prompt corresponding to all query texts in the second query text set, the second Prompt set is obtained.

[0157] S5, according to the second Prompt set, through the target to be tested model, the third answer set is obtained.

[0158] Specifically, the third answer set is configured to include answers obtained by asking the target to-be-tested model through the prompts in the second prompt set.

[0159] S6, according to the first answer set, the second answer set and the third answer set, obtaining the evaluation result of the target to-be-tested model.

[0160] Optionally, according to the proportion of the second answer set in the first answer set, the comprehensive accuracy of the target to-be-tested model can be obtained, according to the similarity between the second answer set and the third answer set, the understanding ability of the target to-be-tested model for the query text can be obtained, and according to the accuracy of the third answer set, the actual accuracy of the target to-be-tested model after optimizing the query text can be obtained.

[0161] Optionally, when the target to-be-tested model includes one or more of a semantic recognition model, a prompt template or a prompt mapping for parsing the query text, according to the similarity between the second answer set and the third answer set, the performance of the semantic recognition model, the prompt template or the prompt mapping can be obtained.

[0162] Optionally, a plurality of groups of first answer sets, second answer sets and third answer sets can be obtained using different first query texts, and the evaluation result of the target to-be-tested model can be obtained according to the plurality of groups of first answer sets, second answer sets and third answer sets.

[0163] Optionally, according to the first answer set, the second answer set and the third answer set, the evaluation result of the target to-be-tested model is obtained, including:

[0164] According to the first answer set and the second answer set, the proportion of the answers in the second answer set to the answers in the first answer set is obtained;

[0165] According to the proportion, the first evaluation parameter is obtained;

[0166] According to the second answer set and the third answer set, the similarity between the answers in the third answer set and the answers in the second answer set is obtained;

[0167] According to the similarity, the second evaluation parameter is obtained;

[0168] According to the third answer set, the correctness of the answers in the third answer set is obtained;

[0169] According to the correctness, the third evaluation parameter is obtained;

[0170] According to the first evaluation parameter, the second evaluation parameter and the third evaluation parameter, the evaluation result of the target to-be-tested model is obtained.

[0171] Optionally, the evaluation result of the target to-be-tested model is:

[0172] (1)

[0173] wherein, is a preset comprehensive accuracy parameter, is a preset semantic understanding parameter, is a preset inference accuracy parameter, is a proportion of answers in the second answer set to answers in the first answer set, is a similarity of answers in the third answer set to answers in the second answer set, is an accuracy of answers in the third answer set.

[0174] Optionally, a value of the first evaluation parameter can be 50, a value of the second evaluation parameter can be 30, a value of the third evaluation parameter can be 20.

[0175] Optionally, considering that the comprehensive accuracy of the government big model is above 80% at present, in order to make the evaluation result more uniform, a value of the first evaluation parameter can be 300, a value of the second evaluation parameter can be 180, a value of the third evaluation parameter can be 120.

[0176] Optionally, the first evaluation parameter is: , the second evaluation parameter is: , and the third evaluation parameter is: .

[0177] By using the above scheme, the comprehensive accuracy of the government big model, the semantic understanding ability and the inference accuracy under the standard query statement are comprehensively considered, so as to accurately evaluate the government big model.

[0178] Optionally, , and can be obtained by an expert method or an index system method.

[0179] Optionally, according to the second answer set and the third answer set, the similarity of answers in the third answer set to answers in the second answer set is obtained, including:

[0180] According to the second answer set and the third answer set, a second word vector of the second answer set and a third word vector of the third answer set are obtained;

[0181] According to the second word vector and the third word vector, the similarity of answers in the third answer set to answers in the second answer set is obtained.

[0182] Specifically, the answers in the second answer set corresponding to the same query statement are defined as T 2 and the answers in the third answer set are defined as T 3The words in the text are embedded as vectors respectively v 2 and v 3,

[0183] The similarity between the answers in the third answer set and the answers in the second answer set can be calculated according to the following formula (2):

[0184] ; (2)

[0185] Wherein, represents the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the similarity between the answers in the second answer set and the answers in the third answer set , the number of word vectors in the text.

[0186] Optionally, the evaluation result of the target to-be-tested model includes a comprehensive evaluation result, a query text semantic understanding evaluation result, and a reasoning accuracy evaluation result.

[0187] Optionally, the evaluation result of the target to-be-tested model is obtained according to the first evaluation parameter, the second evaluation parameter, and the third evaluation parameter, including:

[0188] The comprehensive evaluation result of the target to-be-tested model is obtained according to the first evaluation parameter and the preset index system;

[0189] The query text semantic understanding evaluation result of the target to-be-tested model is obtained according to the second evaluation parameter and the preset index system;

[0190] The reasoning accuracy evaluation result of the target to-be-tested model is obtained according to the third evaluation parameter and the preset index system.

[0191] Further, the optimization direction of the target to-be-tested model can be obtained according to the first evaluation parameter, the second evaluation parameter, the third evaluation parameter, and the preset index system.

[0192] Optionally, for the comprehensive evaluation result: in a case where the first evaluation parameter is greater than or equal to a preset first threshold value, the comprehensive evaluation result of the target to-be-tested model is a first grade; in a case where the first evaluation parameter is less than the preset first threshold value and greater than or equal to a preset second threshold value, the comprehensive evaluation result of the target to-be-tested model is a second grade; in a case where the first evaluation parameter is less than the preset second threshold value and greater than or equal to a preset third threshold value, the comprehensive evaluation result of the target to-be-tested model is a third grade; and in a case where the first evaluation parameter is less than the preset third threshold value, the comprehensive evaluation result of the target to-be-tested model is a fourth grade.

[0193] Optionally, for the semantic understanding evaluation result: in a case where the second evaluation parameter is not less than a preset second threshold value and the second evaluation parameter is not less than a preset fourth threshold value, the semantic understanding evaluation result of the target to-be-tested model is a first grade; in a case where the second evaluation parameter is not less than the preset second threshold value, the second evaluation parameter is less than the preset fourth threshold value, and the second evaluation parameter is not less than a preset fifth threshold value, the semantic understanding evaluation result of the target to-be-tested model is a second grade; in a case where the second evaluation parameter is less than the preset second threshold value and the second evaluation parameter is not less than the preset fifth threshold value, the semantic understanding evaluation result of the target to-be-tested model is a third grade; and in a case where the second evaluation parameter is less than the preset second threshold value and the second evaluation parameter is less than the preset fifth threshold value, the semantic understanding evaluation result of the target to-be-tested model is a fourth grade.

[0194] Optionally, for the inference accuracy evaluation result: in a case where the third evaluation parameter is not less than a preset second threshold value and the third evaluation parameter is not less than a preset sixth threshold value, the inference accuracy evaluation result of the target to-be-tested model is a first grade; in a case where the third evaluation parameter is not less than the preset second threshold value, the second evaluation parameter is less than the preset sixth threshold value, and the second evaluation parameter is not less than a preset fifth threshold value, the inference accuracy evaluation result of the target to-be-tested model is a second grade; in a case where the third evaluation parameter is less than the preset second threshold value and the second evaluation parameter is not less than a preset seventh threshold value, the inference accuracy evaluation result of the target to-be-tested model is a third grade; and in a case where the third evaluation parameter is less than the preset second threshold value and the second evaluation parameter is less than the preset seventh threshold value, the inference accuracy evaluation result of the target to-be-tested model is a fourth grade.

[0195] By using the above evaluation scheme, the defects of the target to-be-tested model can be obtained according to the evaluation result of the target to-be-tested model, and targeted optimization can be performed.

[0196] Embodiment 3

[0197] Reference Figure 2 Fig. 1 is a schematic diagram of an architecture of a system 100 for detecting a government affairs large model according to an embodiment of the present application. Figure 2 As shown in Fig. 1, the system 100 for detecting the government affairs large model according to the embodiment of the present application includes a model management platform 110 and a model evaluation platform 120.

[0198] Specifically, the model management platform is configured 110 to obtain a first answer set from the target to-be-tested model according to a first query text set.

[0199] Specifically, the model evaluation platform 120 is configured to obtain a second answer set according to the first answer set, the second answer set being configured to include answers in the first answer set that do not match standard answers; obtain a second query text set corresponding to the second answer set according to the second answer set, the second query text set being configured to include query texts in the first query text set that correspond to answers in the second answer set; obtain a second Prompt set corresponding to the second answer set according to the second query text set, the second Prompt set being configured to include Prompts generated according to query texts in the second query text set; obtain a third answer set from the target to-be-tested model according to the second Prompt set; and obtain an evaluation result of the target to-be-tested model according to the first answer set, the second answer set, and the third answer set.

[0200] Optionally, obtaining the second answer set according to the first answer set includes:

[0201] obtaining each answer in the first answer set and a query text corresponding to each answer according to the first answer set;

[0202] determining whether each answer matches a standard answer according to each answer in the first answer set and the query text corresponding to each answer;

[0203] adding answers in each answer that do not match the standard answer to the second answer set.

[0204] Optionally, before the step of obtaining the second Prompt set corresponding to the second answer set according to the second query text set, the method further includes:

[0205] obtaining a knowledge graph of a knowledge base accessed by the target to-be-tested model according to the target to-be-tested model;

[0206] constructing a Prompt template based on entities, relationships, and attributes in the knowledge graph according to the knowledge graph.

[0207] Optionally, obtaining the second Prompt set corresponding to the second answer set according to the second query text set includes:

[0208] obtaining query texts in the second query text set according to the second query text set;

[0209] obtaining a key parameter corresponding to the query text by analyzing the query text;

[0210] obtaining a Prompt corresponding to the query text according to the key parameter and the Prompt template;

[0211] obtain a second prompt set according to the prompts corresponding to all query texts in the second query text set.

[0212] Optionally, an evaluation result of the target to-be-tested model is obtained according to the first answer set, the second answer set and the third answer set, including:

[0213] a proportion of answers in the second answer set to answers in the first answer set is obtained according to the first answer set and the second answer set;

[0214] a first evaluation parameter is obtained according to the proportion;

[0215] a similarity of answers in the third answer set to answers in the second answer set is obtained according to the second answer set and the third answer set;

[0216] a second evaluation parameter is obtained according to the similarity;

[0217] a correctness rate of answers in the third answer set is obtained according to the third answer set;

[0218] a third evaluation parameter is obtained according to the correctness rate;

[0219] the evaluation result of the target to-be-tested model is obtained according to the first evaluation parameter, the second evaluation parameter and the third evaluation parameter.

[0220] Optionally, the similarity of answers in the third answer set to answers in the second answer set is obtained according to the second answer set and the third answer set, including:

[0221] a second word vector of the second answer set and a third word vector of the third answer set are obtained according to the second answer set and the third answer set;

[0222] the similarity of answers in the third answer set to answers in the second answer set is obtained according to the second word vector and the third word vector.

[0223] Optionally, the evaluation result of the target to-be-tested model is obtained according to the first evaluation parameter, the second evaluation parameter and the third evaluation parameter, including:

[0224] a comprehensive evaluation result of the target to-be-tested model is obtained according to the first evaluation parameter and a preset index system;

[0225] a query text semantic understanding evaluation result of the target to-be-tested model is obtained according to the second evaluation parameter and the preset index system;

[0226] a reasoning accuracy evaluation result of the target to-be-tested model is obtained according to the third evaluation parameter and the preset index system.

[0227] Embodiment 4

[0228] The embodiment provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method for government affair large model detection according to any one of the above embodiments.

[0229] Specifically, as shown in Figure 3 , Figure 3 The device is an electronic device, which can comprise a processor 101, for example, a central processing unit (CPU), a communication bus 102, a user interface 104, a network interface 103 and a memory 105. The communication bus 102 is used to realize the connection and communication among the components. The user interface 104 can comprise a display and an input unit such as a keyboard. Optionally, the user interface 104 can further comprise a standard wired interface and a wireless interface. The network interface 103 can optionally comprise a standard wired interface and a wireless interface (for example, a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 105 can be a storage device independent of the aforementioned processor 101. The memory 105 can be a high-speed random access memory (RAM) memory or a stable non-volatile memory (NVM), for example, at least one disk memory. The processor 101 can be a general-purpose processor, including a central processing unit, a network processor and the like. The processor 101 can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0230] Those skilled in the art can understand that Figure 3 The structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can comprise more or fewer components than those shown in the figure, or some components can be combined, or different components can be arranged.

[0231] As shown in Figure 3 , the memory 105 as a storage medium can comprise an operating system, a network communication module, a user interface module and an application program for implementing the method for government affair large model detection.

[0232] In Figure 3The electronic device shown, the network interface 103 is mainly used for data communication with the network server; the user interface 104 is mainly used for data interaction with the user; the processor 101 and the memory 105 in the application can be arranged in the electronic device, and the electronic device calls the application program stored in the memory 105 for realizing a kind of method for the detection of government affairs big model to realize the above-mentioned method by processor 101.

[0233] Embodiment 5

[0234] The embodiment provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and a processor executes the computer program to realize the method for the detection of government affairs big model described in any of the above embodiments.

[0235] In some embodiments, the computer-readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc or CD-ROM memory, etc.; it can also be various devices including one or any combination of the above storage devices. The computer can be various computing devices including smart terminals and servers.

[0236] In the above embodiments of the present disclosure, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0237] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be realized by other ways. Among them, the device embodiments described above are only schematic, for example, the division of units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0238] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0239] In addition, the functional units in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or software functional unit.

[0240] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable nonvolatile storage medium. Based on such understanding, the technical solutions of the present disclosure, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a nonvolatile storage medium, including a number of instructions to make a device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods in the various embodiments of the present disclosure. The aforementioned nonvolatile storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0241] The above is only the preferred embodiment of the present disclosure, and it should be pointed out that for those skilled in the art, without departing from the principles of the present disclosure, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present disclosure.

Claims

1. A method for government affair large model detection, characterized in that, The method comprises the following steps: obtaining a first answer set according to a first query text set through a target to-be-tested model; obtaining a second answer set according to the first answer set, wherein the second answer set is configured to include answers in the first answer set that do not match standard answers; obtaining a second query text set corresponding to the second answer set according to the second answer set, wherein the second query text set is configured to include query texts in the first query text set that correspond to answers in the second answer set; obtaining a second Prompt set corresponding to the second answer set according to the second query text set, wherein the second Prompt set is configured to include Prompts generated according to query texts in the second query text set; obtaining a third answer set according to the second Prompt set through the target to-be-tested model; obtaining an evaluation result of the target to-be-tested model according to the first answer set, the second answer set and the third answer set; Before the step of obtaining a second Prompt set corresponding to the second answer set according to the second query text set, the method further comprises the following steps: obtaining a knowledge graph of a knowledge base accessed by the target to-be-tested model according to the target to-be-tested model; establishing a Prompt, adding a question placeholder in the Prompt, obtaining a classification of knowledge according to entities, relationships and attributes in the knowledge graph, adding a knowledge placeholder corresponding to the classification of the entity in the Prompt according to the classification of the entity, and obtaining a Prompt template; the step of obtaining a second Prompt set corresponding to the second answer set according to the second query text set comprises the following steps: obtaining query texts in the second query text set according to the second query text set; parsing the query texts to obtain key parameters corresponding to the query texts; adding the key parameters or key words of the query texts to the question placeholder of the Prompt template, adding the key parameters or key words of the query texts to the knowledge placeholder of the Prompt template according to the categories of the key parameters or key words of the query texts and the query results in the knowledge graph, and obtaining a Prompt corresponding to the query texts according to the Prompt template after adding the key parameters or key words; obtaining the second Prompt set according to the Prompts corresponding to all query texts in the second query text set.

2. The method for government affair large model detection according to claim 1, characterized in that, the step of obtaining a second answer set according to the first answer set comprises the following steps: obtaining each answer in the first answer set and query texts corresponding to each answer according to the first answer set; determining whether each answer in the first answer set matches a standard answer according to each answer in the first answer set and query texts corresponding to each answer; adding answers in the first answer set that do not match the standard answer to the second answer set.

3. The method for government affair large model detection according to claim 1, characterized in that, the step of obtaining an evaluation result of the target to-be-tested model according to the first answer set, the second answer set and the third answer set comprises the following steps: obtaining a proportion of answers in the second answer set to answers in the first answer set according to the first answer set and the second answer set; obtaining a first evaluation parameter according to the proportion; According to the second answer set and the third answer set, a similarity between answers in the third answer set and answers in the second answer set is obtained; According to the similarity, a second evaluation parameter is obtained; According to the third answer set, a correctness of answers in the third answer set is obtained; According to the correctness, a third evaluation parameter is obtained; According to the first evaluation parameter, the second evaluation parameter and the third evaluation parameter, an evaluation result of the target model under test is obtained.

4. The method for government affair large model detection according to claim 3, characterized in that, According to the second answer set and the third answer set, a similarity between answers in the third answer set and answers in the second answer set is obtained, including: According to the second answer set and the third answer set, a second word vector of the second answer set and a third word vector of the third answer set are obtained; According to the second word vector and the third word vector, a similarity between answers in the third answer set and answers in the second answer set is obtained.

5. The method for government affair large model detection according to claim 3, characterized in that, The evaluation result of the target model under test includes a comprehensive evaluation result, a query text semantic understanding evaluation result and a reasoning accuracy evaluation result; According to the first evaluation parameter, the second evaluation parameter and the third evaluation parameter, an evaluation result of the target model under test is obtained, including: According to the first evaluation parameter and a preset index system, a comprehensive evaluation result of the target model under test is obtained; According to the second evaluation parameter and a preset index system, a query text semantic understanding evaluation result of the target model under test is obtained; According to the third evaluation parameter and a preset index system, a reasoning accuracy evaluation result of the target model under test is obtained.

6. The method for government-oriented large model detection according to claim 1, wherein, According to the first answer set, the second answer set and the third answer set, an evaluation result of the target model under test is obtained, including: According to the first answer set and the second answer set, a proportion of answers in the second answer set to answers in the first answer set is obtained; According to the second answer set and the third answer set, a similarity between answers in the third answer set and answers in the second answer set is obtained; According to the third answer set, a correctness of answers in the third answer set is obtained; The evaluation result of the target model to be tested is calculated according to the following formula P : ； wherein, is a preset comprehensive accuracy parameter, is a preset semantic understanding parameter, is a preset inference accuracy parameter, is a proportion of answers in the second answer set to answers in the first answer set, is a similarity of answers in the third answer set to answers in the second answer set, is a correctness rate of answers in the third answer set.

7. A system for government affair large model detection, characterized in that, including: The model management platform is configured to: according to a first query text set, obtain a first answer set through a target model under test; The model evaluation platform is configured to: according to the first answer set, obtain a second answer set, the second answer set is configured to include answers in the first answer set that do not match standard answers; According to the second answer set, a second query text set corresponding to the second answer set is obtained, the second query text set is configured to include query texts in the first query text set corresponding to answers in the second answer set; According to the second query text set, a second Prompt set corresponding to the second answer set is obtained, the second Prompt set is configured to include a Prompt generated according to a query text in the second query text set; according to the second Prompt set, a third answer set is obtained through the target to-be-tested model; and according to the first answer set, the second answer set and the third answer set, an evaluation result of the target to-be-tested model is obtained. Before the step of obtaining, according to the second query text set, a second Prompt set corresponding to the second answer set, the method further includes: According to the target to-be-tested model, a knowledge graph of a knowledge base accessed by the target to-be-tested model is obtained. A Prompt is established, and a question placeholder is added in the Prompt; according to entities, relationships and attributes in the knowledge graph, a classification of knowledge is obtained; according to the classification of the entities, a knowledge placeholder corresponding to the classification of the entities is added in the Prompt, and a Prompt template is obtained. The second Prompt set corresponding to the second answer set is obtained according to the second query text set, and the second Prompt set includes: According to the second query text set, a query text in the second query text set is obtained. The query text is parsed to obtain a key parameter corresponding to the query text. The key parameter or keyword of the query text is added to the question placeholder of the Prompt template; according to the category of the key parameter or keyword of the query text and the query result in the knowledge graph, the key parameter or keyword of the query text is added to the knowledge placeholder of the Prompt template; and according to the Prompt template after the key parameter or keyword is added, a Prompt corresponding to the query text is obtained. The second Prompt set is obtained according to the Prompts corresponding to all query texts in the second query text set.

8. An electronic device, comprising: The device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method for government affairs large model detection according to any one of claims 1-6.

Images