Test method and device based on large model

Abstract

The invention relates to the technical field of data processing, in particular to a test method and device based on a large model. In the embodiment of the invention, the electronic equipment calls the large model to generate a large batch of target questions and corresponding reference answers according to the knowledge data stored in the knowledge base, so that a high-quality question and answer data set is efficiently generated; and calling the to-be-tested system to reply each target question to obtain a target answer corresponding to each target question, scoring the target answers according to the reference answers corresponding to each target question so as to obtain an effect evaluation report of the to-be-tested system, and automatically verifying the accuracy of the to-be-tested system based on the question and answer data set. And the verification quality and efficiency of the to-be-tested system are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a testing method and device based on a large model. Background Art

[0002] In the process of training and testing large models or systems, high-quality training and validation data sets play a key role. In the traditional testing process, questions need to be manually configured, the system to be tested needs to respond to the manually configured questions, generate corresponding answers, and then manually evaluate the answers output by the system to be tested.

[0003] Traditional testing methods take a lot of time and cost, have problems of low efficiency and poor quality, and are difficult to meet the growing data processing needs. Summary of the invention

[0004] The present application provides a testing method and device based on a large model to solve the problem that the traditional testing method relies solely on manual question construction and answer evaluation, resulting in low efficiency, poor quality and difficulty in meeting the growing data processing needs.

[0005] In a first aspect, an embodiment of the present application provides a testing method based on a large model, the method comprising:

[0006] Obtain each target question and corresponding reference answer contained in the saved pre-built question-answering dataset; wherein each target question and corresponding reference answer is generated by the large model based on the knowledge data saved in the knowledge base;

[0007] For each target question, send the target question to the system to be tested, so that the system to be tested generates a target answer corresponding to the target question; and determine the score information corresponding to the target answer according to the reference answer corresponding to the target question;

[0008] Each target question, the target answer corresponding to each target question and the scoring information are input into the report generation model to obtain the effect evaluation report output by the report generation model.

[0009] The above technical solution has the following advantages or beneficial effects: calling the large model to generate a large number of target questions and corresponding reference answers according to the knowledge data stored in the knowledge base, realizing the efficient generation of high-quality question and answer data sets; calling the system to be tested to reply to each target question, obtaining the target answer corresponding to each target question, and scoring the target answer according to the reference answer corresponding to each target question, thereby obtaining an effect evaluation report of the system to be tested, and being able to automatically verify the accuracy of the system to be tested based on the question and answer data set, thereby improving the quality and efficiency of the verification of the system to be tested.

[0010] In some embodiments of the present application, the process of constructing the question-answering dataset includes:

[0011] Receive an input user question and answer demand, wherein the user question and answer demand carries a target identifier for generating a knowledge base of question and answer pairs, a type of each target question and answer pair to be constructed, and a corresponding number of question and answer pairs;

[0012] For each target question-answer pair type, the target prompt template corresponding to the target question-answer pair type is determined according to the pre-saved correspondence between the question-answer pair type and the prompt template; according to the number of question-answer pairs corresponding to the target question-answer pair type, knowledge data is extracted from the knowledge base of the target identifier, and the knowledge data is added to the target prompt template; the added prompt template is input into the big model, and the target question and reference answer output by the big model are obtained.

[0013] In some embodiments of the present application, the target question-answer pair type is at least one of generating a plain text question-answer pair, generating a graph question-answer pair, a summary question-answer pair, and an inference question-answer pair.

[0014] The above technical solution has the following advantages or beneficial effects: different types of question-answer pairs are generated by using a large model and a knowledge base, so that the question-answer data set finally constructed is accurate and comprehensive.

[0015] In some embodiments of the present application, the process of constructing the knowledge base includes:

[0016] Get original documents from a preset data source;

[0017] Converting the original document into a preset format, and cleaning the original document according to a preset cleaning method to obtain a target document;

[0018] Obtaining pictures and corresponding picture names contained in the original document, and storing the pictures and picture names in a picture knowledge base;

[0019] According to a preset document segmentation method, the target document is segmented into document blocks, and the document blocks are stored in a document knowledge base.

[0020] The above technical solution has the following advantages or beneficial effects: constructing a knowledge base based on a large model makes the construction of the knowledge base more accurate and comprehensive.

[0021] In some embodiments of the present application, determining the score information corresponding to the target answer according to the reference answer corresponding to the target question includes:

[0022] Obtaining each selected target analysis algorithm carried in the saved received test instruction; wherein the target analysis algorithm includes at least one of a large model analysis and a similarity algorithm analysis;

[0023] For each target analysis algorithm, obtaining a target scoring standard corresponding to the target analysis algorithm that is saved; using the target scoring standard and the reference answer, determining sub-scoring information corresponding to the target answer;

[0024] The sub-scoring information corresponding to each target analysis algorithm is determined as the scoring information corresponding to the target answer.

[0025] The above technical solution has the following advantages or beneficial effects: multiple analysis algorithms are used to analyze the target object, making the analysis result more accurate.

[0026] In some embodiments of the present application, if the target analysis algorithm is a large model analysis, the use of the target scoring standard and the reference answer to determine the sub-scoring information corresponding to the target answer includes:

[0027] Inputting the big model scoring criteria, the target answer, the target question and the reference answer into the analysis big model, so that the analysis big model scores the target answer from preset dimensions respectively; wherein the preset dimensions include factual accuracy, topic relevance, answer relevance, comprehensiveness and redundancy;

[0028] The output of the analysis model is obtained, and the output is determined as the sub-score information.

[0029] The above technical solution has the following advantages or beneficial effects: it realizes the analysis of the target answer based on the large analysis model, analyzes the target answer from multiple dimensions, and improves the accuracy of the analysis.

[0030] In some embodiments of the present application, the analysis model scores the target answer from preset dimensions respectively, including:

[0031] The analysis model determines the factual accuracy score and the topic relevance score of the target answer based on the reference answer;

[0032] If any one of the fact accuracy score and the topic relevance score is a preset value, determining the total score of the target answer to be the preset value, and outputting the total score, the fact accuracy score and the topic relevance score;

[0033] If any of the factual accuracy score and the topic relevance score is not a preset value, then the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points of the target answer are determined based on the reference answer; the sum of the answer relevance score, comprehensiveness score and redundancy score is determined as the total score of the target answer, and the total score, the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points are output.

[0034] The above technical solution has the following advantages or beneficial effects: it realizes the analysis of the target answer from multiple dimensions based on the analysis of the large model, improves the accuracy of the target answer analysis, and thus improves the accuracy of the test.

[0035] In some embodiments of the present application, if the target analysis algorithm is a similarity algorithm analysis, the use of the target scoring standard and the reference answer to determine the sub-scoring information corresponding to the target answer includes:

[0036] Determine the similarity between the target answer and the reference answer according to a preset similarity algorithm;

[0037] The similarity is determined as the score information of the target answer.

[0038] The above technical solution has the following advantages or beneficial effects: it realizes the analysis of the target answer based on similarity.

[0039] In some embodiments of the present application, before obtaining the stored pre-built data set containing each target question and the corresponding reference answer, the method further includes:

[0040] Receiving an input test instruction carrying the interface address of the system to be tested;

[0041] The sending the target question to the system to be tested comprises:

[0042] The target question is sent to the system to be tested according to the interface address.

[0043] The above technical solution has the following advantages or beneficial effects: the sending of the target question is realized.

[0044] In a second aspect, an embodiment of the present application further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the above-mentioned large model-based testing methods when executing a computer program stored in the memory.

[0045] In an embodiment of the present application, an electronic device obtains each target question and corresponding reference answer contained in a stored pre-built question and answer data set; wherein each target question and corresponding reference answer are generated by a large model according to knowledge data stored in a knowledge base; for each target question, the target question is sent to the system to be tested, so that the system to be tested generates a target answer corresponding to the target question; according to the reference answer corresponding to the target question, the scoring information corresponding to the target answer is determined; each target question, the target answer corresponding to each target question and the scoring information are input into the report generation large model, and an effect evaluation report output by the report generation large model is obtained. In an embodiment of the present application, an electronic device calls a large model to generate a large number of target questions and corresponding reference answers according to the knowledge data stored in the knowledge base, thereby realizing efficient generation of high-quality question and answer data sets; the system to be tested is called to reply to each target question, obtain the target answer corresponding to each target question, and score the target answer according to the reference answer corresponding to each target question, thereby obtaining an effect evaluation report of the system to be tested, and can automatically verify the accuracy of the system to be tested based on the question and answer data set, thereby improving the quality and efficiency of the verification of the system to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A schematic diagram of a test process based on a large model provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a front-end page for determining user question and answer requirements provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a front-end page for uploading an original document provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a front-end page constructed for the knowledge base provided in an embodiment of the present application;

[0051] Figure 5 A flowchart of knowledge base construction and question-answer data set construction provided in the embodiment of the present application;

[0052] Figure 6 A schematic diagram of a front-end page for inputting test instructions provided in an embodiment of the present application;

[0053] Figure 7A test flow chart provided for an embodiment of the present application;

[0054] Figure 8 A schematic diagram of a front-end page for inputting an interface address provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of the framework structure provided for an embodiment of the present application;

[0056] Fig.10 A schematic diagram of the structure of a large-scale model-based testing device provided in an embodiment of the present application;

[0057] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0059] In order to automatically verify the accuracy of the system to be tested based on the question and answer data set and improve the quality and efficiency of the verification of the system to be tested, an embodiment of the present application provides a testing method and device based on a large model.

[0060] In an embodiment of the present application, an electronic device obtains each target question and corresponding reference answer contained in a saved pre-built question and answer data set; wherein each target question and corresponding reference answer are generated by a large model based on knowledge data saved in a knowledge base; for each target question, the target question is sent to a system to be tested, so that the system to be tested generates a target answer corresponding to the target question; based on the reference answer corresponding to the target question, scoring information corresponding to the target answer is determined; each target question, the target answer corresponding to each target question, and the scoring information are input into a report generation large model, and an effect evaluation report output by the report generation large model is obtained.

[0061] Figure 1 A schematic diagram of a retrieval question answering process applied to a large language model provided in an embodiment of the present application, the process includes:

[0062] S101: Obtain each target question and corresponding reference answer contained in a saved pre-built question-answer data set; wherein each target question and corresponding reference answer is generated by the large model based on the knowledge data saved in the knowledge base.

[0063] A large model-based testing method provided in an embodiment of the present application is applied to an electronic device, which may be a PC or a server.

[0064] In the field of artificial intelligence, question-answering datasets are an important resource for training and evaluating question-answering systems. However, existing methods for generating and verifying question-answering datasets have problems such as low efficiency and low accuracy.

[0065] Based on this, in the embodiment of the present application, the large model is called to generate each target question and the corresponding reference answer according to the knowledge data stored in the knowledge base. The electronic device saves each target question and the corresponding reference answer to the question and answer data set, realizing the construction of the data set, and the constructed data set is more in line with user requirements.

[0066] Among them, the knowledge base may contain information such as charts, so the big model used to generate each question text and the corresponding reference answer can be a multimodal big model, or a big language model, etc., which is not limited here.

[0067] In the embodiment of the present application, after the electronic device has constructed the data set, the electronic device can receive a test instruction for the system to be tested sent by the user, wherein the test instruction can carry an identification of the system to be tested, and the electronic device determines the system to be tested according to the identification carried in the test instruction. In the embodiment of the present application, the system to be tested can be a large model, or other equipment, processor, etc. that can provide question-and-answer services, which is not limited here.

[0068] In an embodiment of the present application, after the electronic device receives the test instruction, the electronic device obtains each target question and the corresponding reference answer contained in the saved pre-built question and answer data set. Specifically, if the test instruction carries the number of question and answer pairs required for the test, the electronic device extracts the number of target questions and the corresponding reference answers from the question and answer data set; if the test instruction does not carry the number of question and answer pairs required for the test, the electronic device extracts a preset number of target questions and the corresponding reference answers from the question and answer data set, wherein the preset number is pre-configured and the operator can flexibly adjust it according to needs.

[0069] In addition, in an embodiment of the present application, the question and answer data set also stores the knowledge base and question type corresponding to each target question. The user can also configure the knowledge base identifier and the target question type in the test instruction. After the electronic device receives the test instruction, it can filter each target question and the corresponding reference answer corresponding to the knowledge base identifier and the target question type from the question and answer data set.

[0070] S102: For each target question, send the target question to the system to be tested, so that the system to be tested generates a target answer corresponding to the target question; and determine the score information corresponding to the target answer based on the reference answer corresponding to the target question.

[0071] In an embodiment of the present application, after the electronic device obtains each target question and reference answer from the question-answering data set, the electronic device sends each target question as a test set to the system to be tested. The system to be tested processes each target question and generates a target answer corresponding to each target question. The system to be tested returns each target answer to the electronic device, which analyzes the target answer corresponding to each target question based on the reference answer corresponding to each target question and determines the scoring information corresponding to each target answer.

[0072] Among them, for a target answer and a reference answer corresponding to a target question, the electronic device can perform similarity matching on the target answer and the reference answer to determine the similarity of the two answers. The electronic device also stores a similarity threshold. If the electronic device determines that the similarity is not lower than the similarity threshold, the electronic device determines that the target answer is accurate, and determines that the scoring information corresponding to the target answer is the score corresponding to the accurate answer; if the electronic device determines that the similarity is lower than the similarity threshold, the electronic device determines that the target answer is inaccurate, and determines that the scoring information corresponding to the target answer is the score corresponding to the inaccurate answer. Among them, the scoring information carries information on whether the similarity between the target answer and the reference answer is lower than the similarity threshold.

[0073] In addition, in the embodiment of the present application, the scoring information may carry the score and the reason for deduction, which is not limited here.

[0074] S103: Input each target question, the target answer corresponding to each target question and the scoring information into the report generation model, and obtain the effect evaluation report output by the report generation model.

[0075] In an embodiment of the present application, the electronic device can input each target question, the target answer corresponding to each target question, and the scoring information into the report generation model, so that the report generation model generates and outputs an effect evaluation report of the system to be tested. The electronic device obtains the effect evaluation report output by the report generation model.

[0076] Specifically, the report generation model determines the number of target questions that the model to be tested answers correctly and the number of target questions that it answers incorrectly based on the scoring information of each target answer, and then calculates the accuracy. For example, the report generation model determines that the score is greater than or equal to M points as correct, and the score is less than M points as wrong, and calculates the accuracy based on the number of scores of each number of items.

[0077] The report generation model will also summarize the reasons for deductions below M points, forming a main summary paragraph for deductions in the evaluation report. The generation model generates and outputs an effect evaluation report based on the above calculation standards. The effect evaluation report includes the number of evaluation items, i.e. the number of target questions, the number of correct items, the score of each item, the comprehensive score, the accuracy rate, and the main reasons for deductions.

[0078] Among them, in the embodiment of the present application, the report generation large model can be a large language model, a multimodal large model, or other large models that can generate reports.

[0079] In an embodiment of the present application, the electronic device calls a large model to generate a large number of target questions and corresponding reference answers based on the knowledge data stored in the knowledge base, thereby achieving efficient generation of high-quality question and answer data sets; calls the system to be tested to respond to each target question, obtains the target answer corresponding to each target question, and scores the target answer based on the reference answer corresponding to each target question, thereby obtaining an effectiveness evaluation report of the system to be tested, and can automatically verify the accuracy of the system to be tested based on the question and answer data set, thereby improving the quality and efficiency of the verification of the system to be tested.

[0080] In order to automatically verify the accuracy of the system to be tested based on the question and answer data set and improve the quality and efficiency of the verification of the system to be tested, based on the above embodiments, in the embodiment of the present application, the process of constructing the question and answer data set includes:

[0081] Receive an input user question and answer demand, wherein the user question and answer demand carries a target identifier for generating a knowledge base of question and answer pairs, a type of each target question and answer pair to be constructed, and a corresponding number of question and answer pairs;

[0082] For each target question-answer pair type, the target prompt template corresponding to the target question-answer pair type is determined according to the pre-saved correspondence between the question-answer pair type and the prompt template; according to the number of question-answer pairs corresponding to the target question-answer pair type, knowledge data is extracted from the knowledge base of the target identifier, and the knowledge data is added to the target prompt template; the added prompt template is input into the big model, and the target question and reference answer output by the big model are obtained.

[0083] In the process of fine-tuning and training large models, high-quality data sets play a vital role. A high-quality data set can provide the model with rich and comprehensive information, which helps the model to better understand and learn the characteristics and laws of the data. It can reduce the risk of overfitting of the model and improve the generalization ability of the model, so that the model can perform more stably and reliably in practical applications. Therefore, when conducting large model training and effect evaluation, selecting and preparing high-quality data sets is a crucial step, which will directly affect the performance and effect of the model. High-quality data sets are required for knowledge question and answer training or effect evaluation. If it relies on manual annotation and generalization, it will cost a lot of manpower and material resources, and if it relies on manual verification, it will also incur a lot of workload. Based on this demand, the embodiment of the present application provides a method for constructing a question and answer data set based on a large model.

[0084] Specifically, in an embodiment of the present application, a user can input user question and answer requirements through the front-end page of an electronic device, and the user question and answer requirements may include a target identifier of a knowledge base selected for generating question and answer pairs, the type of each target question and answer pair to be constructed, and the corresponding number of question and answer pairs.

[0085] Among them, in the embodiment of the present application, the user can choose to generate only one type of target question, that is, the number of target question and answer pair types carried in the user's question and answer requirements is 1; the user can also choose to generate multiple types of target questions, that is, the number of target question and answer pair types carried in the user's question and answer requirements is multiple. If the user chooses to generate multiple target question and answer pair types, then when selecting the number of question and answer pairs, the user can select the number of question and answer pairs corresponding to each target question and answer pair type, and then divide the number of question and answer pairs equally into each target question and answer pair type to obtain the number of question and answer pairs corresponding to each target question and answer pair type.

[0086] Figure 2 A schematic diagram of a front-end page for determining user question-and-answer needs provided in an embodiment of the present application, such as the Figure 2 As shown, the user can select question-answer pairs to generate a knowledge base including but not limited to an operation and maintenance knowledge base, a traffic knowledge base, newly uploaded documents, etc.; the user can also select the number of question-answer pairs to generate, such as 200, 500, 1000, and 2000.

[0087] After the electronic device receives the user's question and answer requirements, for each target question and answer pair type, the electronic device determines the target prompt template corresponding to the target question and answer pair type based on the pre-saved correspondence between the question and answer pair type and the prompt template; according to the number of question and answer pairs corresponding to the target question and answer pair type, the electronic device extracts knowledge data from the knowledge base of the target identifier, and adds the knowledge data to the target prompt template; inputs the added prompt template into the big model, and obtains the target question and reference answer output by the big model.

[0088] In the embodiment of the present application, the knowledge data of the knowledge base is a document block or an image, and the knowledge data is the minimum unit for generating a question-answer pair. When the electronic device extracts knowledge data from the knowledge base of the target identifier according to the number of question-answer pairs corresponding to the target question-answer pair type when generating the target question, the amount of knowledge data extracted by the electronic device is not less than the number of question-answer pairs.

[0089] In addition, in an embodiment of the present application, the prompt template includes prompt words for prompting the large model to generate a target question and a reference answer of the target question-answer pair type corresponding to the prompt template, and also includes example knowledge data, example questions and example reference answers generated based on the example knowledge data, wherein the type of the example question is the target question-answer pair type.

[0090] In order to automatically verify the accuracy of the system to be tested based on the question and answer data set and improve the quality and efficiency of the verification of the system to be tested, on the basis of the above embodiments, in an embodiment of the present application, the target question and answer pair type is at least one of generating plain text question and answer pairs, generating chart question and answer pairs, summarizing question and answer pairs and reasoning question and answer pairs.

[0091] In an embodiment of the present application, the target question-answer pair type is at least one of generating a plain text question-answer pair, generating a chart question-answer pair, a summary question-answer pair, and an inference question-answer pair.

[0092] After the electronic device obtains the user's question-and-answer requirements, it obtains the prompt template corresponding to the target question-and-answer pair type carried in the user's question-and-answer requirements, and generates the corresponding type of target questions and reference answers based on the prompt template, not just simply generating simple retrieval question-and-answer pairs. If the user's question-and-answer requirements do not carry the target question-and-answer pair type, the electronic device will generate target questions of the types of questions about tables in documents, questions about pictures, cross-document questions, reasoning questions, summary questions, multi-round questions, etc. by default.

[0093] In an embodiment of the present application, the generation of multiple rounds of questions is a difficult point in the above problems. The prompt template may include prompt words written in a role-playing manner. For example, the prompt words may be "Please simulate roles A and B and have a dialogue. The dialogue format is as follows: Generate multiple rounds of interactive questions and answers based on the content in the document knowledge base, and output them in a question-and-answer format."

[0094] Among them, the prompt template corresponding to the reasoning questions also needs to include high-quality prompts for this type of questions. Among them, reasoning questions are mainly aimed at cross-table questioning. The large model needs to obtain the answer by linking two tables and performing reasoning.

[0095] The following is a description of a specific embodiment:

[0096]

[0097] Table 1

[0098]

[0099] Table 2

[0100] For example, Table 1 is a design speed table, including the design speed corresponding to each highway grade; Table 2 is a lane width table, including the lane width corresponding to each design speed. Among them, for the reasoning question "What is the lane width of a third-level highway?", the large model needs to generate a reference answer by associating Table 1 and Table 2 and performing reasoning.

[0101] In the embodiment of the present application, summary questions also need to be written with high-quality prompt words specifically for this type.

[0102] For example, the target question is: What are the principles for selecting highway technical grades? The large model needs to summarize and respond across two documents to generate a reference answer. The two documents are "JTG B01-2014-Highway Engineering Technical Standards" and "JTG D20—2017-Highway Route Design Specifications".

[0103] Among them, "JTG B01-2014-Highway Engineering Technical Standards" disclosed:

[0104] The selection of highway technical grade should follow the following principles:

[0105] 1 The selection of highway technical grade should be based on road network planning, highway functions, and determined in combination with traffic volume analysis.

[0106] 2Expressways should be used as main trunk roads.

[0107] 3. Secondary trunk roads should be Grade II or above.

[0108] 4. The main distribution roads should be first- and second-class roads.

[0109] 5. Secondary distribution roads should be second- or third-level roads.

[0110] 6. It is advisable to select third- or fourth-class highways for branch roads

[0111] Among them, the "JTGD20-2017-Highway Route Design Specifications" discloses:

[0112] 1 Main trunk roads, as the main channels with the highest structural level in the road network, should be expressways.

[0113] 2. Secondary roads, as a supplement to the main trunk roads, should be of Class II or above.

[0114] 1) When the designed traffic volume reaches 15,000 passenger cars per day, it is advisable to use first-class or above highways.

[0115] 2) When the designed traffic volume reaches 10,000 passenger cars / day and the longitudinal and lateral interference along the route is large, a first-class highway should be selected.

[0116] 3) When the design traffic volume is less than 10,000 passenger cars per day, a secondary highway can be selected; when the truck mixing rate is high, overtaking lanes should be set at intervals to reduce longitudinal interference.

[0117] 3. The main distribution roads connect the trunk roads and branch roads, and it is advisable to use first-class and second-class roads.

[0118] 1) When the designed traffic volume reaches 15,000 passenger cars / day, a first-class highway can be selected.

[0119] 2) When the design traffic volume is between 5,000 and 15,000 cars per day, a second-class highway can be selected; when the design traffic volume reaches 10,000 cars per day and the longitudinal and lateral interference along the line is large, a first-class highway should be selected.

[0120] 3) When the design traffic volume is less than 5,000 cars per day, a secondary highway should be used.

[0121] The 4 secondary distribution roads serve the county and township regional transportation, and it is advisable to use second-class and third-class roads.

[0122] 1) When the designed traffic volume reaches 5,000 passenger cars / day, a secondary highway should be selected.

[0123] 2) When the design traffic volume is less than 5,000 passenger cars / day, it is advisable to use a third-level highway.

[0124] 5. For branch roads, it is advisable to use Class III or Class IV roads. When the designed traffic volume reaches 5,000 passenger cars / day, it is advisable to use Class II roads.

[0125] In an embodiment of the present application, the electronic device completes a high-quality and diverse question-and-answer dataset based on the above user needs and pre-configuration. The question-and-answer dataset contains target questions and answers. The answers at this stage are defined as reference answers. The target questions and reference answers will be used when testing the system to be tested.

[0126] In order to automatically verify the accuracy of the system to be tested based on the question-answer data set and improve the quality and efficiency of the verification of the system to be tested, based on the above embodiments, in the embodiment of the present application, the construction process of the knowledge base includes:

[0127] Get original documents from a preset data source;

[0128] Converting the original document into a preset format, and cleaning the original document according to a preset cleaning method to obtain a target document;

[0129] Obtaining pictures and corresponding picture names contained in the original document, and storing the pictures and picture names in a picture knowledge base;

[0130] According to a preset document segmentation method, the target document is segmented into document blocks, and the document blocks are stored in a document knowledge base.

[0131] In the embodiment of the present application, the electronic device can also construct a knowledge base, which can include an image knowledge base and a document knowledge base. Specific construction process:

[0132] The electronic device obtains the original document from a preset data source, which may be other electronic devices, or a server or device such as a memory or database that can store the original document, or may be uploaded by the user himself.

[0133] Figure 3 A schematic diagram of a front-end page for uploading an original document provided in an embodiment of the present application, such as the Figure 3 As shown, click the "Local Upload" button, the electronic device pops up a selection dialog box, the user selects a file from the local storage space based on the selection dialog box, and clicks the "Confirm Upload" button.

[0134] Since the original document has different formats, such as PDF, Word, TXT, etc., in the application embodiment, the electronic device converts the original document into a preset format, and cleans the original document according to a preset cleaning method to obtain a target document. The preset format can be markdown format or other formats.

[0135] In an embodiment of the present application, the electronic device obtains the pictures and corresponding picture names contained in the original document, and saves the pictures and picture names correspondingly in the picture knowledge base; according to the preset document segmentation method, the target document is segmented into document blocks, and the document blocks are saved in the document knowledge base.

[0136] In an embodiment of the present application, the electronic device formats and cleans batches of original documents according to the above requirements, and after reviewing the processed original documents and images, completes the file segmentation and construction of the document knowledge base.

[0137] In addition, electronic devices also support instant upload and parsing of documents. After users upload the original documents, the formatting process can be completed automatically. Users can divide the documents into blocks according to their needs and complete the storage according to the block settings. The size of the blocks will slightly affect the number of question-answer pairs generated later. Blocking is to divide a large text corpus into smaller, more manageable parts or paragraphs. Each block is an independent information unit, and a separate document block is used as the minimum unit for generating question-answer pairs.

[0138] In the embodiment of the present application, the document data after the above processing is divided into blocks according to the size of the data and stored in the knowledge base. For larger files, the above-mentioned block-based processing is adopted to facilitate the adaptation of the large model to the processing. The image name is stored as an index link in the document knowledge base, and the image is indexed and associated by the name.

[0139] Figure 4 A schematic diagram of a front-end page constructed for the knowledge base provided in the embodiment of the present application, such as the Figure 4 As shown, the user selects the word count range for document segmentation, including 0-200, 200-500 and 500-1000; when the front-end page prompts that the document has been segmented, the front-end page displays whether to generate a question-answer pair, and the user can select "yes" or "no".

[0140] Figure 5 The flowchart of knowledge base construction and question-answer data set construction provided in the embodiment of the present application is as follows: Figure 5 As shown, the process includes: obtaining the original document set, processing the data format of each original document in the original document set, and segmenting the documents to build a document knowledge base; building prompt words, and calling the large model to generate a question-answering dataset.

[0141] In order to automatically verify the accuracy of the system to be tested based on the question and answer data set and improve the quality and efficiency of the verification of the system to be tested, on the basis of the above embodiments, in the embodiment of the present application, the score information corresponding to the target answer is determined according to the reference answer corresponding to the target question, including:

[0142] Obtaining each selected target analysis algorithm carried in the saved received test instruction; wherein the target analysis algorithm includes at least one of a large model analysis and a similarity algorithm analysis;

[0143] For each target analysis algorithm, obtaining a target scoring standard corresponding to the target analysis algorithm that is saved; using the target scoring standard and the reference answer, determining sub-scoring information corresponding to the target answer;

[0144] The sub-scoring information corresponding to each target analysis algorithm is determined as the scoring information corresponding to the target answer.

[0145] In the embodiment of the present application, when the electronic device analyzes the target answer and determines the scoring information of the target answer, the electronic device may analyze according to the target analysis algorithm, wherein the target analysis algorithm includes but is not limited to at least one of large model analysis and similarity algorithm analysis.

[0146] Specifically, when configuring the test instructions, the user can also select a target analysis algorithm based on the front-end page, and can also input scoring requirements based on the front-end page, that is, the test instructions carry each selected target analysis algorithm and scoring requirements.

[0147] Figure 6 A schematic diagram of a front-end page for inputting a test instruction provided in an embodiment of the present application, such as the Figure 6 As shown, the front-end page displays the test result verification methods including large model verification and traditional algorithm verification (similarity algorithm verification); the front-end page also displays a scoring requirement input box, a button to start automatic verification, a button to download verification results, a button to delete verification results, a button to determine whether comprehensive scoring is required, a button to start comprehensive scoring, a button to download comprehensive scoring, and a button to delete comprehensive scoring.

[0148] In the embodiment of the present application, the electronic device stores different scoring standards for different target analysis algorithms. After obtaining each selected target analysis algorithm carried in the received test instruction, the electronic device obtains the target scoring standard corresponding to the target analysis algorithm for each target analysis algorithm; the target scoring standard and the reference answer are used to determine the sub-scoring information corresponding to the target answer.

[0149] The electronic device determines the sub-scoring information corresponding to each target analysis algorithm as the scoring information corresponding to the target answer.

[0150] Figure 7 The test flow chart provided in the embodiment of the present application is as follows: Figure 7 As shown, the process includes: according to the interface of the system / platform to be tested, the question is input into the system / platform to be tested to obtain the generated answer; the generated answer is compared and analyzed with the reference answer to obtain the big model score and the traditional algorithm score; according to the big model score and the traditional algorithm score, a comprehensive score is performed, and the big model is called to generate a comprehensive evaluation report.

[0151] In order to automatically verify the accuracy of the system to be tested based on the question and answer data set and improve the quality and efficiency of the verification of the system to be tested, on the basis of the above embodiments, in the embodiment of the present application, if the target analysis algorithm is a large model analysis, the target scoring standard and the reference answer are used to determine the sub-scoring information corresponding to the target answer, including:

[0152] Inputting the big model scoring criteria, the target answer, the target question and the reference answer into the analysis big model, so that the analysis big model scores the target answer from preset dimensions respectively; wherein the preset dimensions include factual accuracy, topic relevance, answer relevance, comprehensiveness and redundancy;

[0153] The output of the analysis model is obtained, and the output is determined as the sub-score information.

[0154] In the embodiment of the present application, the large model analysis is mainly carried out from the aspects of content similarity, completeness, standardization, etc. Since the reference answer and the target answer are generated or retrieved by the model, there may be differences with the standard answer. Even if the two are highly similar, it does not mean that they are necessarily correct. Therefore, the analysis of the large model also requires separate analysis of the questions and generated answers, and the external knowledge base can be called for comprehensive analysis during the analysis process. For example: What is the process of road hazard inspection? The network answer can be called for reference and comparison to verify the correctness of the generated answer.

[0155] Since the answers to the knowledge quiz are subjective, they must be evaluated from multiple dimensions to obtain a comprehensive score. The core evaluation dimensions include topic relevance and factual accuracy. If the generated answer is irrelevant to the question and has factual errors, it is directly defined as 0 points. The non-core evaluation dimensions are: key point coverage, concise expression, and semantic clarity. Score the above three dimensions separately, and then calculate the comprehensive score.

[0156] Specifically, the analysis model can determine the scoring information based on the following formula:

[0157] IF Score1=0or Score2=0,print 0;

[0158] Else

[0159] Print(Score3+Score4+Score5);

[0160] Among them, Score1 represents the factual accuracy score, Score2 represents the topic relevance score, Score3 represents the answer relevance score, Score4 represents the comprehensiveness score, Score5 represents the redundancy score, and print represents the total score.

[0161] Specifically, in an embodiment of the present application, the electronic device inputs the big model scoring criteria, the target answer, the target question and the reference answer into the analysis big model, so that the analysis big model scores the target answer from preset dimensions respectively; wherein the preset dimensions include factual accuracy, topic relevance, answer relevance, comprehensiveness and redundancy; the electronic device obtains the output of the analysis big model and determines the output as sub-scoring information.

[0162] In a possible implementation, the large model scoring criteria include but are not limited to:

[0163] 1. The scoring range is between 0-N points. The higher the score, the higher the quality of the user's answer.

[0164] 2. Factual accuracy: Factual accuracy measures the consistency of the generated answer with the standard answer. If the user's answer is inconsistent with the standard answer, the score should be 0.

[0165] 3. Topic relevance: Topic relevance measures the consistency of the generated answer with the standard answer. If the user answer is inconsistent with the standard answer topic, the score should be 0.

[0166] 4. Answer relevance: Answer relevance evaluates the relevance of the generated answer to the question. If the deduction range is 1-n1 points, the lower the relevance, the more points should be deducted.

[0167] 5. Comprehensiveness standard: if the user's answer cannot effectively cover the standard answer, then points should be deducted. The deduction range is 1-n2 points. The lower the effective coverage, the more points should be deducted.

[0168] 6. Redundancy standard: if the user's answer is seriously redundant and the redundant content is incorrect, then points should be deducted. The deduction range is 1-n3 points. The greater the proportion of redundancy, the more points should be deducted.

[0169] Based on this, in an embodiment of the present application, before the electronic device inputs the big model scoring criteria, the target answer, the target question and the reference answer into the analysis big model, it can construct a big model prompt based on the big model scoring criteria, and then input the big model prompt, the target answer, the target question and the reference answer into the analysis big model.

[0170] Among them, the large model prompt can be:

[0171] You are a quality assessment master. Your task is to evaluate the user's answers, compare them with the reference answer column, and evaluate the generated answer column according to the correctness, comprehensiveness, and redundancy of the user's answers.

[0172] #Notice

[0173] 1. The scoring range is between 0-N points. The higher the score, the higher the quality of the user's answer.

[0174] 2. Factual accuracy: Factual accuracy measures the consistency of the generated answer with the standard answer. If the user's answer is inconsistent with the standard answer, the score should be 0.

[0175] 3. Topic relevance: Topic relevance measures the consistency of the generated answer with the standard answer. If the user answer is inconsistent with the standard answer topic, the score should be 0.

[0176] 4. Answer relevance: Answer relevance evaluates the relevance of the generated answer to the question. If the deduction range is 1-n1 points, the lower the relevance, the more points should be deducted.

[0177] 5. Comprehensiveness standard: if the user's answer cannot effectively cover the standard answer, then points should be deducted. The deduction range is 1-n2 points. The lower the effective coverage, the more points should be deducted.

[0178] 6. Redundancy standard: if the user's answer is seriously redundant and the redundant content is incorrect, then points should be deducted. The deduction range is 1-n3 points. The greater the proportion of redundancy, the more points should be deducted.

[0179] 7. Please output according to the sample reference format, score according to the above 5 principles, and give a comprehensive score and reason.

[0180] The large model prompt may also include example target answers, example target questions, example reference answers and example scoring information.

[0181] In the embodiment of the present application, the scoring information includes but is not limited to the sub-scoring information of each dimension and the reason for losing points in each dimension. The analysis model can be a large language model, a multimodal large model, or other large models that can be used for analysis.

[0182] In order to automatically verify the accuracy of the system to be tested based on the question-answer data set and improve the quality and efficiency of the verification of the system to be tested, on the basis of the above embodiments, in the embodiment of the present application, the analysis model scores the target answer from the preset dimensions respectively, including:

[0183] The analysis model determines the factual accuracy score and the topic relevance score of the target answer based on the reference answer;

[0184] If any one of the fact accuracy score and the topic relevance score is a preset value, determining the total score of the target answer to be the preset value, and outputting the total score, the fact accuracy score and the topic relevance score;

[0185] If any of the factual accuracy score and the topic relevance score is not a preset value, then the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points of the target answer are determined based on the reference answer; the sum of the answer relevance score, comprehensiveness score and redundancy score is determined as the total score of the target answer, and the total score, the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points are output.

[0186] In an embodiment of the present application, when the analysis model scores the target answer respectively from preset dimensions, the analysis model first determines the factual accuracy score and the topic relevance score of the target answer.

[0187] If any one of the fact accuracy score and the topic relevance score is a preset value, the total score of the target answer is determined to be the preset value, and the total score, the fact accuracy score, and the topic relevance score are output.

[0188] If any of the factual accuracy score and the topic relevance score is not a preset value, then the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points of the target answer are determined based on the reference answer; the sum of the answer relevance score, comprehensiveness score and redundancy score is determined as the total score of the target answer, and the total score, the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points are output.

[0189] In the embodiment of the present application, the preset value is 0. That is, in the embodiment of the present application, if the analysis model determines that any one of the factual accuracy score and the topic relevance score of the target answer is 0, the analysis model determines that the total score of the target answer is 0; if the analysis model determines that both the factual accuracy score and the topic relevance score of the target answer are not 0, the electronic device determines the sum of the answer relevance score, the comprehensiveness score and the redundancy score of the target answer, and determines the sum as the total score of the target answer.

[0190] In the embodiment of the present application, the user can choose to directly export the scoring information or choose manual assisted verification. The electronic device determines the final output scoring information according to the user's choice, and the scoring information includes the number of participating evaluation items, the number of correct items, the score of each item, the comprehensive score, the accuracy rate, and the main reasons for deduction.

[0191] Among them, the scoring evaluation dimensions include three dimensions: key point coverage, conciseness of expression, and semantic clarity.

[0192] In order to automatically verify the accuracy of the system to be tested based on the question and answer data set and improve the quality and efficiency of the verification of the system to be tested, on the basis of the above embodiments, in the embodiment of the present application, if the target analysis algorithm is a similarity algorithm analysis, the target scoring standard and the reference answer are used to determine the sub-scoring information corresponding to the target answer, including:

[0193] Determine the similarity between the target answer and the reference answer according to a preset similarity algorithm;

[0194] The similarity is determined as the score information of the target answer.

[0195] In an embodiment of the present application, when a user selects a target analysis algorithm, the user may also select a similarity algorithm analysis.

[0196] Specifically, if the target analysis algorithm is a similarity algorithm analysis, the electronic device determines the embedded representation vectors of the target answer and the reference answer respectively, and determines the similarity between the target answer and the reference answer according to the preset similarity algorithm and the two embedded representation vectors. The electronic device determines the similarity as the scoring information of the target answer.

[0197] The similarity algorithm may be a cosine similarity algorithm.

[0198] In order to automatically verify the accuracy of the system to be tested based on the question and answer data set and improve the quality and efficiency of the verification of the system to be tested, based on the above embodiments, in the embodiment of the present application, before obtaining the saved pre-built data set containing each target question and the corresponding reference answer, the method further includes:

[0199] Receiving an input test instruction carrying the interface address of the system to be tested;

[0200] The sending the target question to the system to be tested comprises:

[0201] The target question is sent to the system to be tested according to the interface address.

[0202] In the embodiment of the present application, when the user inputs a test instruction, the test instruction may also carry relevant parameters of the interface address of the system to be tested.

[0203] Figure 8 A schematic diagram of a front-end page for inputting an interface address provided in an embodiment of the present application, such as the Figure 8 As shown, the front-end page displays a "Test Interface Address" input box for entering the interface address of the system to be tested; it also displays a "Test Interface Input Parameter Example" and a "Test Interface Output Parameter Example".

[0204] The electronic device sends the target question to the system to be tested according to the interface address carried in the test instruction.

[0205] In order to quickly build a knowledge question answering data set and obtain automated effect evaluation results, saving a lot of manpower. This application embodiment proposes a framework structure that includes four modules: input, generation, analysis, and output. Fig. 9 A schematic diagram of the framework structure provided for an embodiment of the present application.

[0206] exist Fig. 9 In the input module (required input), the main function is to complete the unified formatting of the original document data set, complete the segmentation, and complete the knowledge base construction. The generation module (model generation) mainly calls the large model interface, optimizes and updates the dynamic prompt words according to user needs, calls the large model's understanding and generation capabilities, and generates batches of question-answer pairs based on the segmented data set. Among them, the generated question-answer pairs contain questions and reference answers. According to the system platform entry or interface address and input parameters entered by the user, the relevant code is updated according to the phased time, and the interface is called to generate answers.

[0207] The analysis module (large model analysis and traditional algorithm analysis) mainly compares and analyzes the generated answers with the reference answers in the question-answer pairs generated by the first step. The analysis modules include the large model analysis module and the traditional algorithm analysis module.

[0208] The output module (result output) is mainly used to integrate the large model analysis results and the traditional algorithm analysis results, and output an evaluation report.

[0209] Based on the above embodiments, Fig.10 A schematic diagram of a test device based on a large model provided in an embodiment of the present application, the device comprising:

[0210] Processing module 1001 is used to obtain each target question and corresponding reference answer contained in the stored pre-built question and answer data set; wherein each target question and corresponding reference answer is generated by the large model according to the knowledge data stored in the knowledge base;

[0211] The test module 1002 is used to send each target question to the system to be tested, so that the system to be tested generates a target answer corresponding to the target question; determine the scoring information corresponding to the target answer based on the reference answer corresponding to the target question; input each target question, the target answer corresponding to each target question and the scoring information into the report generation model, and obtain the effect evaluation report output by the report generation model.

[0212] In a possible implementation, the processing module 1001 is also used to receive input user question and answer requirements, wherein the user question and answer requirements carry a target identifier of a knowledge base for generating question and answer pairs, each target question and answer pair type to be constructed, and the corresponding number of question and answer pairs; for each target question and answer pair type, the target prompt template corresponding to the target question and answer pair type is determined according to a pre-saved correspondence between the question and answer pair type and the prompt template; according to the number of question and answer pairs corresponding to the target question and answer pair type, knowledge data is extracted from the knowledge base of the target identifier, and the knowledge data is added to the target prompt template; the added prompt template is input into the big model, and the target question and reference answer output by the big model are obtained.

[0213] In a possible implementation, the target question-answer pair type is at least one of generating a plain text question-answer pair, generating a chart question-answer pair, a summary question-answer pair, and an inference question-answer pair.

[0214] In a possible implementation, the processing module 1001 is also used to obtain an original document from a preset data source; convert the original document into a preset format, and clean the original document according to a preset cleaning method to obtain a target document; obtain the pictures and corresponding picture names contained in the original document, and save the pictures and picture names in a picture knowledge base; according to a preset document segmentation method, segment the target document into document blocks, and save the document blocks in a document knowledge base.

[0215] In one possible implementation, the test module 1002 is specifically used to obtain each selected target analysis algorithm carried in the saved received test instruction; wherein the target analysis algorithm includes at least one of a large model analysis and a similarity algorithm analysis; for each target analysis algorithm, obtain the target scoring standard corresponding to the saved target analysis algorithm; use the target scoring standard and the reference answer to determine the sub-scoring information corresponding to the target answer; and determine the sub-scoring information corresponding to each target analysis algorithm as the scoring information corresponding to the target answer.

[0216] In one possible implementation, the test module 1002 is specifically used to input the big model scoring criteria, the target answer, the target question and the reference answer into the analysis big model if the target analysis algorithm is a big model analysis, so that the analysis big model scores the target answer from preset dimensions respectively; wherein the preset dimensions include factual accuracy, topic relevance, answer relevance, comprehensiveness and redundancy; obtain the output of the analysis big model, and determine the output as the sub-scoring information.

[0217] In one possible implementation, the test module 1002 is specifically used for the analysis model to determine the factual accuracy score and the topic relevance score of the target answer based on the reference answer; if any of the factual accuracy score and the topic relevance score is a preset value, the total score of the target answer is determined to be the preset value, and the total score, the factual accuracy score and the topic relevance score are output; if any of the factual accuracy score and the topic relevance score is not a preset value, the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points of the target answer are determined based on the reference answer; the sum of the answer relevance score, comprehensiveness score and redundancy score is determined as the total score of the target answer, and the total score, the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points are output.

[0218] In a possible implementation, the test module 1002 is specifically used to determine the similarity between the target answer and the reference answer according to a preset similarity algorithm if the target analysis algorithm is a similarity algorithm analysis; and determine the similarity as the scoring information of the target answer.

[0219] In a possible implementation manner, the processing module 1001 is further configured to receive an input test instruction carrying the interface address of the system to be tested; and send the target problem to the system to be tested according to the interface address.

[0220] Based on the above embodiments, the present application also provides an electronic device, Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, it includes: a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104;

[0221] The memory 1103 stores a computer program. When the program is executed by the processor 1101, the processor 1101 executes the steps of the large model-based testing method provided in the above-mentioned embodiments.

[0222] Since the principle of solving the problem by the above electronic device is similar to the testing method based on a large model, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0223] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1102 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0224] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0225] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large model-based testing method provided in the above embodiments.

[0226] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the testing method based on a large model, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0227] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0228] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0229] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0230] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0231] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A testing method based on a large model, characterized in that: The method comprises: Obtain each target question and corresponding reference answer contained in the saved pre-built question-answering dataset; wherein each target question and corresponding reference answer is generated by the large model based on the knowledge data saved in the knowledge base; For each target question, send the target question to the system to be tested, so that the system to be tested generates a target answer corresponding to the target question; and determine the score information corresponding to the target answer according to the reference answer corresponding to the target question; Each target question, the target answer corresponding to each target question and the scoring information are input into the report generation model to obtain the effect evaluation report output by the report generation model.

2. The method according to claim 1, characterized in that The process of constructing the question-answering dataset includes: Receive an input user question and answer demand, wherein the user question and answer demand carries a target identifier for generating a knowledge base of question and answer pairs, a type of each target question and answer pair to be constructed, and a corresponding number of question and answer pairs; For each target question-answer pair type, the target prompt template corresponding to the target question-answer pair type is determined according to the pre-saved correspondence between the question-answer pair type and the prompt template; according to the number of question-answer pairs corresponding to the target question-answer pair type, knowledge data is extracted from the knowledge base of the target identifier, and the knowledge data is added to the target prompt template; the added prompt template is input into the big model, and the target question and reference answer output by the big model are obtained.

3. The method according to claim 2, characterized in that The target question-answer pair type is at least one of generating a plain text question-answer pair, generating a chart question-answer pair, a summary question-answer pair, and an inference question-answer pair.

4. The method according to claim 1, characterized in that: The construction process of the knowledge base includes: Get original documents from a preset data source; Converting the original document into a preset format, and cleaning the original document according to a preset cleaning method to obtain a target document; Obtaining pictures and corresponding picture names contained in the original document, and storing the pictures and picture names in a picture knowledge base; According to a preset document segmentation method, the target document is segmented into document blocks, and the document blocks are stored in a document knowledge base.

5. The method according to claim 1, characterized in that Determining the score information corresponding to the target answer according to the reference answer corresponding to the target question includes: Obtaining each selected target analysis algorithm carried in the saved received test instruction; wherein the target analysis algorithm includes at least one of a large model analysis and a similarity algorithm analysis; For each target analysis algorithm, obtaining a target scoring standard corresponding to the target analysis algorithm that is saved; using the target scoring standard and the reference answer, determining sub-scoring information corresponding to the target answer; The sub-scoring information corresponding to each target analysis algorithm is determined as the scoring information corresponding to the target answer.

6. The method according to claim 5, characterized in that If the target analysis algorithm is a large model analysis, the use of the target scoring standard and the reference answer to determine the sub-scoring information corresponding to the target answer includes: Inputting the big model scoring criteria, the target answer, the target question and the reference answer into the analysis big model, so that the analysis big model scores the target answer from preset dimensions respectively; wherein the preset dimensions include factual accuracy, topic relevance, answer relevance, comprehensiveness and redundancy; The output of the analysis model is obtained, and the output is determined as the sub-score information.

7. The method according to claim 6, characterized in that The analysis model scores the target answer from the preset dimensions respectively, including: The analysis model determines the factual accuracy score and the topic relevance score of the target answer based on the reference answer; If any one of the fact accuracy score and the topic relevance score is a preset value, determining the total score of the target answer to be the preset value, and outputting the total score, the fact accuracy score and the topic relevance score; If any of the factual accuracy score and the topic relevance score is not a preset value, then the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points of the target answer are determined based on the reference answer; the sum of the answer relevance score, comprehensiveness score and redundancy score is determined as the total score of the target answer, and the total score, the answer relevance score, comprehensiveness score, redundancy score and corresponding reasons for losing points are output.

8. The method according to claim 5, characterized in that If the target analysis algorithm is a similarity algorithm analysis, the use of the target scoring standard and the reference answer to determine the sub-scoring information corresponding to the target answer includes: Determine the similarity between the target answer and the reference answer according to a preset similarity algorithm; The similarity is determined as the score information of the target answer.

9. The method according to claim 1, characterized in that: Before obtaining the saved pre-built data set containing each target question and the corresponding reference answer, the method further includes: Receiving an input test instruction carrying the interface address of the system to be tested; The sending the target question to the system to be tested comprises: The target question is sent to the system to be tested according to the interface address.

10. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the large model-based testing method as claimed in any one of claims 1 to 9 when executing a computer program stored in the memory.

Images