A test set construction method of a large model system and an electronic device

Abstract

The application provides a test set construction method of a large model system and an electronic device, and relates to the field of artificial intelligence. A first mapping relationship is constructed in advance. The first mapping relationship includes a corresponding relationship between a function supported by a large model system and an intent recognition type of the large model system, and a corresponding relationship between the intent recognition type and a prompt word. Based on the first mapping relationship, a target intent recognition type and a target prompt word are determined according to a function supported by a to-be-tested large model system. Based on the target prompt word, a use case keyword is obtained through a specified large model, and an initial test case is generated in combination with a predefined use case sentence pattern. The intent category label of the initial test case is the target intent recognition type. Effective use cases are selected from the initial test case to form a test set. The test set accurately adapts to the function of the to-be-tested system, so that the test result can truly reflect the intent recognition capability of the to-be-tested system, a standardized test set construction process is provided, and the construction efficiency and quality are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method for constructing a test set for a large model system and an electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, large model systems (MLS) capable of providing functions such as web retrieval, academic search, knowledge base retrieval, and multimodal question answering have been widely applied in various fields. MLS systems typically possess intent recognition capabilities. Intent recognition refers to determining the user's expressed intent by analyzing user input information, matching the corresponding response path based on the intent, and returning the response result to the user to meet their needs. When intent recognition is inaccurate, the correct response path cannot be matched, and the resulting response will fail to meet the user's requirements. Therefore, the intent recognition capability of large model systems is crucial.

[0003] To test the intent recognition capabilities of a large-scale model system, an intent recognition test suite needs to be constructed, where the test cases represent user input. The system's responses to these test cases are compared with pre-labeled responses, and quantitative metrics such as precision and recall are calculated to measure the system's intent recognition capabilities. However, current intent recognition test suites often rely on manual construction, lacking standardized processes and resulting in low efficiency and quality. Furthermore, current test suites are generally generic, risking incompatibility with the functionality of large-scale model systems, leading to test results that fail to accurately reflect the system's intent recognition capabilities.

[0004] Therefore, there is an urgent need to provide a method for constructing test sets for large model systems to improve the efficiency and quality of test set construction and ensure that the test results can truly reflect the intent recognition capabilities of large model systems. Summary of the Invention

[0005] In view of this, this application provides a method for constructing a test set for a large model system, the method comprising:

[0006] Obtain a pre-constructed first mapping relationship; wherein the first mapping relationship includes the correspondence between the functions supported by the large model system and the intent recognition type of the large model system, and the correspondence between the intent recognition type and the prompt words; the prompt words are used to guide the specified large model to generate initial test cases;

[0007] Based on the first mapping relationship, and according to the functions supported by the large model system under test, the target intent recognition type and target prompt words corresponding to the large model system under test are determined;

[0008] Based on the target prompt words, use case keywords related to the target intent recognition type are obtained through the specified large model, and the initial test cases are generated based on the use case keywords and predefined use case sentence patterns; the intent type label of the initial test cases is the target intent recognition type.

[0009] Valid test cases are selected from the initial test cases, and the valid test cases are combined into a test set to test the intent recognition capability of the large model system under test.

[0010] Optionally, valid test cases are filtered from the initial test cases, including:

[0011] The initial test cases are input into the large model system under test to obtain the predicted intent recognition type determined by the large model system under test for the initial test cases, and the predicted response result for the initial test cases.

[0012] Obtain a pre-constructed second mapping relationship; the second mapping relationship is the correspondence between use case types and verification rules; the use case types include at least intent recognition-oriented and response result-oriented types;

[0013] Based on the second mapping relationship and the target test case type to which the initial test case belongs, the target verification rule corresponding to the initial test case is determined;

[0014] According to the target verification rules, the validity of the initial test cases is verified based on the predicted intent identification type and the predicted response result;

[0015] If the initial test case passes verification, then the initial test case is considered a valid test case.

[0016] Optionally, the target use case type is the intent recognition focus; correspondingly, according to the target verification rules, the validity of the initial test case is verified based on the predicted intent recognition type and the predicted response result, including:

[0017] Determine whether the predicted intent recognition type is consistent with the target intent recognition type;

[0018] If they match, then the initial test case has passed verification.

[0019] If they are inconsistent, then determine whether the predicted response result matches the expected response result;

[0020] If the conditions are met, then the initial test case is determined to have failed verification.

[0021] If it does not meet the requirements, then the initial test case is deemed to have passed verification.

[0022] Optionally, the target use case type is the response result-oriented type; correspondingly, according to the target verification rules, the validity of the initial test case is verified based on the predicted intent identification type and the predicted response result, including:

[0023] Determine whether the predicted intent recognition type is consistent with the target intent recognition type;

[0024] If they match, the validity verification result of the initial test case is determined by comparing the predicted response result with the expected response result.

[0025] If there is a discrepancy, the validity verification result of the initial test cases is determined by comparing the test case keywords in the initial test cases with the keywords stored in the keyword database; wherein, the keyword database is used to store the test case keywords contained in the valid test cases.

[0026] Optionally, the validity verification result of the initial test case is determined by comparing the predicted response result with the expected response result, including:

[0027] Compare whether the predicted response result is consistent with the expected response result;

[0028] If they are consistent, then the validity verification result is determined to be that the initial test case has passed the verification;

[0029] If there is a discrepancy, the analysis results of the initial test cases are obtained;

[0030] If the analysis result is the first result characterizing that the large model system under test supports the response to the initial test case, then the validity verification result is determined to be that the initial test case has passed the verification.

[0031] If the analysis result is a second result indicating that the large model system under test does not support the response to the initial test case, then the validity verification result is determined to be that the initial test case has failed verification.

[0032] Optionally, the validity verification result of the initial test cases is determined by comparing the test case keywords in the initial test cases with the keywords stored in the keyword database, including:

[0033] Determine whether the test case keywords in the initial test cases exist in the keyword database;

[0034] If it exists, then the validity verification result is determined to be that the initial test case has passed the verification;

[0035] If it does not exist, proceed with the following steps:

[0036] Determine whether the predicted response result matches the expected response result;

[0037] If the conditions are met, then the validity verification result is determined to be that the initial test case failed the verification.

[0038] If not, the system will regenerate the predicted response result for the initial test case according to the response link corresponding to the target intent identification type through the large model system under test, and return the step of determining the validity verification result of the initial test case by comparing the predicted response result with the expected response result.

[0039] Optionally, after assembling the valid use cases into a test set, the method further includes:

[0040] Within a specified time period, acquire all historical question information input by the user into the large model system under test;

[0041] Extract question keywords from the historical question information and determine the frequency of occurrence of the question keywords;

[0042] The question keywords with a frequency greater than a preset threshold are used as supplementary keywords;

[0043] If the test case keywords contained in the test set do not contain any keywords that match the supplementary keywords, then supplementary test cases are generated based on the supplementary keywords and the test case sentence structure using the specified large model.

[0044] Add the supplementary test cases to the test set.

[0045] Optionally, the target intent recognition type is multiple, and after assembling the valid use cases into a test set, it further includes:

[0046] Within a specified time period, acquire all historical question information input by the user into the large model system under test;

[0047] For each of the target intent recognition types, perform the following steps:

[0048] Determine a first number of historical question information corresponding to the target intent recognition type; and use the proportion of the first number to the total number of historical question information as a reference proportion;

[0049] Determine a second number of valid use cases corresponding to the target intent recognition type; and use the proportion of the second number to the total number of valid use cases as the actual proportion;

[0050] Based on the reference ratio and the actual ratio, supplement or delete valid use cases corresponding to the target intent recognition type.

[0051] Optionally, after assembling the valid use cases into a test set, the method further includes:

[0052] The test set is input into the evaluation model, and the evaluation results generated by the evaluation model from the specified dimensions are obtained. The specified dimensions include at least one of the following: the coverage of the test set keywords by the effective test sets, the matching degree between the effective test sets and the corresponding target intent recognition type, and the standardization of the expression of the effective test sets.

[0053] If the evaluation result does not meet the preset conditions corresponding to the specified dimension, the test set will be adjusted.

[0054] This application also provides an electronic device, the device comprising:

[0055] Memory, used to store computer programs;

[0056] A processor, used to implement the steps of the test set construction method for any of the above-described large model systems when executing the computer program.

[0057] In summary, this application provides a method for constructing a test set for a large-scale model system. A first mapping relationship is pre-constructed; this first mapping relationship includes the correspondence between the functions supported by the large-scale model system and the intent recognition types of the large-scale model system, as well as the correspondence between intent recognition types and prompt words. Based on the first mapping relationship, the target intent recognition type and target prompt words are determined according to the functions supported by the large-scale model system under test. Based on the target prompt words, by specifying the large-scale model, use case keywords are obtained, and combined with predefined use case sentence structures, initial test cases are generated; the intent category label of the initial test cases is the target intent recognition type. Valid test cases are selected from the initial test cases to form a test set. The test set accurately adapts to the functions of the system under test, enabling the test results to truly reflect the intent recognition capabilities of the system under test. This provides a standardized test set construction process, improving construction efficiency and quality. Attached Figure Description

[0058] Figure 1 A schematic diagram of the first process for constructing a test set for a large model system provided in this application;

[0059] Figure 2 A schematic diagram illustrating the overall principle of a test set construction method for a large model system provided in this application;

[0060] Figure 3 A second flowchart illustrating a method for constructing a test set for a large model system provided in this application;

[0061] Figure 4A schematic diagram of the third process of a test set construction method for a large model system provided in this application;

[0062] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0063] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0064] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0065] Please refer to Figure 1 , Figure 1 This application provides a first flowchart illustrating a method for constructing a test set for a large model system. The method includes:

[0066] S101. Obtain the pre-built first mapping relationship; wherein, the first mapping relationship includes the correspondence between the functions supported by the large model system and the intent recognition type of the large model system, and the correspondence between the intent recognition type and the prompt words; the prompt words are used to guide the specified large model to generate initial test cases.

[0067] After receiving user input, the large model system first analyzes the user's intent using its intent recognition capabilities, and then matches the response path based on the analyzed intent to generate a response result. The intent recognition capabilities of the large model system vary depending on the functions it supports. This application predefines the types corresponding to each intent recognition capability, i.e., intent recognition types, and establishes a correspondence between the functions supported by the large model system and the intent recognition types.

[0068] As an optional embodiment, the intent recognition types include ordinary dialogue, web search, knowledge base search, academic search, document question answering, and image question answering. Correspondingly, if the function of the large model system is to engage in simple communication with the user without invoking external tools, its intent recognition type is ordinary dialogue; if the function involves first performing a web search to obtain real-time information (such as weather, news, and dates), and then calling a large language model to summarize the real-time information, its intent recognition type is web search; if the function involves first searching a professional knowledge base (such as a knowledge base in the field of earth sciences), and then calling a large language model to summarize the search results, its intent recognition type is knowledge base search; if the function involves searching for papers and returning the search results, its intent recognition type is academic search; if the function involves using user-uploaded images (such as geological maps and tables), and then calling a multimodal large model to answer user questions based on the images, its intent recognition type is image question answering. Of course, the above intent recognition types can be expanded according to newly added functions of the large model system, and this application does not impose any special limitations on this. For example, if the large model system supports patent document search and answering related questions, then the intent recognition type for patent search can be added.

[0069] Currently, the test sets used to test the intent recognition capabilities of large-scale model systems are all general-purpose test sets, which risk being incompatible with the functionality and intent recognition capabilities of the large-scale model systems. For example, a general-purpose test set might include test cases in the form of image-based question-and-answer questions, while the large-scale model system under test supports academic retrieval. In this case, the response accuracy and recall of the large-scale model system to the general-purpose test set cannot characterize its ability to recognize academic retrieval intent, resulting in inaccurate test results. Therefore, it is necessary to provide different test cases for different intent recognition types.

[0070] Based on this, this application pre-sets different prompts for different intent recognition types, establishing a correspondence between intent recognition types and prompts. These prompts are used to guide the generation of initial test cases for a specified large model. For each intent recognition type, using its corresponding prompts, test cases specifically designed to test the intent recognition capability corresponding to that type are generated, ensuring that the performance of the large model system on the test set accurately represents its intent recognition capability.

[0071] S102. Based on the first mapping relationship, determine the target intent recognition type and target prompt words corresponding to the large model system under test according to the functions supported by the large model system under test.

[0072] The functions supported by the large model system under test are clearly defined in the model design stage. Based on the above first mapping relationship, the intent recognition type corresponding to the functions supported by the model system under test is taken as the target intent recognition type, and the prompt word corresponding to the target intent recognition type is taken as the target prompt word.

[0073] It should be noted that when the large model system under test supports multiple functions, there are also multiple target intent recognition types. Each target intent recognition type corresponds to its own target prompt word. Subsequently, initial test cases corresponding to each target intent recognition type are generated through each target prompt word to achieve a comprehensive test of the intent recognition capability of the large model system under test.

[0074] S103. Based on the target prompt words, obtain the use case keywords related to the target intent recognition type by specifying the large model, and generate the initial test cases based on the use case keywords and the predefined use case sentence structure; the intent type label of the initial test cases is the target intent recognition type.

[0075] In this application, based on target prompts, initial test cases are generated in batches by specifying a large model and following the method of "use case sentence + use case keywords", thereby improving the efficiency of test set construction.

[0076] First, the target prompts guide the designated large model to obtain use case keywords related to the target intent recognition type. This application does not limit the specific content of the target prompts; they can be set according to actual needs, as long as it ensures that the designated large model obtains the required use case keywords.

[0077] For example, if the target intent recognition type is academic retrieval, the corresponding target prompts can be the following:

[0078] Target prompt 1: "Identify two types of keywords: The first type is keywords specifically for academic search scenarios, which should include expressions such as "paper / literature / research-related texts", "academic carriers", or "research content / conclusions", such as "journal paper" and "references"; the second type is keywords for non-academic search scenarios, which should include expressions such as "daily materials" or "work summaries".

[0079] Target prompt 2: "Distinguish between academic search-specific keywords (such as papers and dissertations) and keywords that are mainly used in daily conversations but occasionally in academic scenarios, and mark the application scenario (e.g., for academic search-specific keywords, mark conference papers; for non-academic search-specific keywords, mark data arrangement)."

[0080] Target suggestion 3: "Classify the retrieved keywords according to whether they belong to the core keywords of academic search, and organize the keywords related to papers, literature, research conclusions and daily materials respectively."

[0081] For example, if the target intent recognition type is image-based question and answer, the corresponding target prompt words could be:

[0082] "Enumerate common images (e.g., landscape images, product images), geological images (e.g., geological section maps, mineral distribution maps), tabular images (e.g., data statistical tables, comparison tables), and keywords in image-related descriptive text to ensure coverage of all image types supported by the large model system."

[0083] For example, if the target intent identification type is patent search, the corresponding target prompt words could be:

[0084] "We sort out the keywords specific to patent search scenarios (e.g., Invention Patent and Patent Specification) and non-patent search keywords (e.g., Paper Abstract), exclude expressions such as Trademark Query that are not supported by the large model system, and present the keyword search results in a table."

[0085] Subsequently, the target prompts guide the specified large model to generate initial test cases based on use case keywords and use case sentence structures. For each use case keyword, the keyword can be sequentially filled into various use case sentence structures; furthermore, by replacing verbs in the use case sentence structures with semantically similar verbs, each use case keyword corresponds to a large number of initial test cases, ensuring that the intent recognition capability test results obtained based on the test set are statistically significant.

[0086] For example, for academic search intent in the field of Earth sciences, use case keywords include papers, journal articles, and conference papers, as well as keywords related to paleomagnetism, geosciences, and climate change. Using a specified large model, the above use case keywords are filled into the use case sentence structure to generate initial test cases. Some initial test cases are as follows: "Please find papers related to paleomagnetism", "Help me search for journal articles on plate tectonics research in the field of geosciences", and "Get conference papers on the impact of climate change" are target prompts.

[0087] Furthermore, target prompts can constrain the number of initial test cases generated by the specified large model for each use case keyword. For example, a threshold for the total number of initial test cases under the intent recognition type can be pre-set, and the number of non-repeating initial test cases to be generated for each use case keyword can be determined based on the total number of use case keywords. The aforementioned total threshold can be set according to actual needs, and this application does not impose any particular limitation on it.

[0088] S104. Select valid test cases from the initial test cases and form a test set to test the intent recognition capability of the large model system under test.

[0089] After obtaining the initial test cases, their validity is verified. A test set is formed from the valid test cases selected from the initial set to further ensure that the performance of the large-scale model system under test accurately reflects its intent recognition capability. The key to validating the initial test cases is determining whether the functionality required to respond to them is supported by the large-scale model system under test. If so, the initial test cases are considered valid; otherwise, they are considered invalid. The process of selecting valid test cases will be described in subsequent embodiments and will not be repeated here.

[0090] The process of testing the intent recognition capability of the large-scale model system under test using a test set is not detailed here. For example, based on the intent type labels of valid use cases and the expected response results according to pre-standards, as well as the predicted intent recognition type and predicted response results of the large-scale model system under test for valid use cases, quantitative indicators such as the accuracy and recall of intent recognition and response results are determined, and the intent recognition capability of the large-scale model system under test is measured based on the quantitative indicators.

[0091] In summary, this application provides a method for constructing a test set for a large-scale model system. A first mapping relationship is pre-constructed; this first mapping relationship includes the correspondence between the functions supported by the large-scale model system and the intent recognition types to be tested, as well as the correspondence between intent recognition types and prompt words. Based on the first mapping relationship, the target intent recognition type and target prompt words are determined according to the functions supported by the large-scale model system under test. Based on the target prompt words, by specifying the large-scale model, use case keywords are obtained, and combined with predefined use case sentence structures, initial test cases are generated; the intent category label of the initial test cases is the target intent recognition type. Valid test cases are selected from the initial test cases to form a test set. The test set accurately adapts to the functions of the system under test, enabling the test results to truly reflect the intent recognition capabilities of the large-scale model system under test. This provides a standardized test set construction process, improving construction efficiency and quality.

[0092] Based on the above embodiments:

[0093] The process of selecting valid use cases is explained in detail below.

[0094] As an optional implementation, valid test cases are selected from the initial test cases, including:

[0095] The initial test cases are input into the large model system under test, and the predicted intent recognition type determined by the large model system under test for the initial test cases, as well as the predicted response result for the initial test cases are obtained.

[0096] Obtain the pre-constructed second mapping relationship; the second mapping relationship is the correspondence between use case types and verification rules; the use case types include at least intent recognition-oriented and response result-oriented types;

[0097] Based on the second mapping relationship and the target test case type to which the initial test case belongs, determine the target verification rule corresponding to the initial test case;

[0098] According to the target verification rules, the effectiveness of the initial test cases is verified based on the predicted intent identification type and the predicted response result;

[0099] If the initial test case passes verification, then the initial test case is considered a valid test case.

[0100] In this embodiment, the large-scale model system under test is used to perform intent recognition on the initial test cases to obtain the predicted intent recognition type. Based on the predicted intent recognition type, the corresponding response link is matched to generate the predicted response result. Please refer to... Figure 2 , Figure 2 This is a schematic diagram illustrating the overall principle of a test set construction method for a large model system provided in this application. The method uses the predicted intent recognition type and predicted response result determined by the large model system under test itself as the basis for validity verification, rather than using the intent type and response result determined by a third-party large model as the initial test cases. This ensures that the selected valid test cases are more closely aligned with the tools and capabilities supported by the large model system under test.

[0101] Furthermore, this application categorizes initial test cases into intent-recognition-focused test cases and response-result-focused test cases. For intent-recognition-focused initial test cases, as long as the large model system correctly identifies the intent of the initial test case and invokes the corresponding response chain, it will generate a clear and expected response result; that is, whether the response result meets expectations mainly depends on whether the large model system can correctly identify the intent of the initial test case. For response-result-focused initial test cases, after the large model system correctly identifies the intent of the initial test case, it still needs to perform comprehensive analysis and retrieval in conjunction with system capabilities before generating a response result; that is, whether the response result meets expectations depends not only on whether the intent recognition is correct, but also on the comprehensive analysis and retrieval capabilities of the large model system.

[0102] In this embodiment, the test case type can be determined based on the target intent recognition type corresponding to the initial test case. For example, the initial test cases corresponding to the image question-and-answer intent and the document question-and-answer intent mentioned above are typical intent recognition-focused test cases. The initial test cases corresponding to the ordinary dialogue intent, web search intent, academic search intent, and knowledge base search intent are typical response result-focused test cases.

[0103] Based on this, this embodiment pre-constructs a second mapping relationship, which is the correspondence between use case types and verification rules. The verification rules for use cases focusing on intent recognition differ from those for use cases focusing on response results, thereby achieving differentiated verification of initial test cases. Based on the target intent recognition type, the target use case type to which the initial test case belongs is determined; based on the second mapping relationship, the verification rules corresponding to the target use case type are used as the target verification rules. According to the target verification rules, the validity of the initial test cases is verified based on the predicted intent recognition type and the predicted response result, and the initial test cases that pass the verification are considered valid use cases.

[0104] In addition, such as Figure 2 As shown, a pre-trained third-party verification model can also be invoked to assist in the verification of valid test cases. As an optional implementation, valid test cases are input into the pre-trained verification model to obtain the verification intent recognition type obtained by the model's intent recognition of the valid test cases, and the verification response result for the valid test cases. If the verification intent recognition type matches the target intent recognition type, and the verification response result meets the expected response result, the valid test case is retained; otherwise, the valid test case is discarded. Of course, further duplication verification can be performed on the test set to deduplicate valid test cases.

[0105] The following describes the validity verification process for intent-based heavy use cases.

[0106] Please refer to Figure 3 ,Figure 3 This is a second flowchart illustrating a method for constructing a test set for a large model system provided in this application. As an optional embodiment, the target test case type is intent recognition-focused; correspondingly, according to the target verification rules, the validity of the initial test cases is verified based on the predicted intent recognition type and the predicted response result, including:

[0107] S301. Determine whether the predicted intent recognition type is consistent with the target intent recognition type; if consistent, trigger step S302; if inconsistent, trigger step S303.

[0108] S302. Confirm that the initial test cases have passed verification;

[0109] S303. Determine whether the predicted response result matches the expected response result; if it matches, trigger step S304; if it does not match, trigger step S302.

[0110] S304. The initial test case failed verification.

[0111] In this embodiment, for initial test cases that emphasize intent recognition, it is first determined whether the predicted intent recognition type is consistent with the target intent recognition type, that is, whether the large-scale model system under test can correctly recognize the intent of the initial test case. If they are consistent, it is considered that the large-scale model system under test supports recognizing the intent of the initial test case and supports responding to the initial test case. Therefore, the initial test case is directly determined to have passed the verification, that is, the initial test case is a valid test case, and it is not necessary to further determine whether the predicted response result meets expectations.

[0112] If there is a discrepancy, further investigation is needed to determine whether the predicted response matches the expected response. The purpose is to analyze whether the initial test case itself is flawed (i.e., the initial test case does not fit the functionality of the large-scale model system under test), or whether the functionality of the large-scale model system itself needs optimization. Specifically, if the predicted response does not match the expected response, the initial test case is considered a valid test case, but the performance of the large-scale model system under test needs optimization. If the predicted response matches the expected response, meaning the large-scale model system under test can still correctly answer the initial test case even with an incorrect intent recognition, it indicates that the initial test case itself is flawed and is an invalid test case.

[0113] This completes the validity verification of the initial test cases focusing on intent recognition, ensuring that the obtained valid test cases are relevant to the functions of the large model system under test, and thus ensuring that the performance of the large model system under test on the test set can accurately reflect the intent recognition capability of the large model system under test.

[0114] The following describes the validity verification process for heavy use cases focusing on response results.

[0115] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the third process of a test set construction method for a large model system provided in this application. As an optional embodiment, the target test case type is response result-oriented; correspondingly, according to the target verification rules, the validity of the initial test cases is verified based on the predicted intent identification type and predicted response result, including:

[0116] S401. Determine whether the predicted intent recognition type is consistent with the target intent recognition type; if consistent, trigger step S402; if inconsistent, trigger step S403.

[0117] S402. By comparing the predicted response results with the expected response results, determine the validity verification results of the initial test cases;

[0118] S403. By comparing the test case keywords in the initial test cases with the keywords stored in the keyword database, the validity verification result of the initial test cases is determined; wherein, the keyword database is used to store the test case keywords contained in valid test cases.

[0119] In this embodiment, it is first determined whether the large-scale model system under test can correctly identify the intent of the initial test case, that is, whether the predicted intent identification type is consistent with the target intent identification type. If they are consistent, it is further determined whether the large-scale model system under test can correctly respond to the initial test case, that is, whether the predicted response result is consistent with the expected response result.

[0120] If the predicted response is consistent with the expected response, it indicates that the large model system under test can correctly identify the intent of the initial test cases and respond correctly to them, demonstrating that the initial test cases fit the functionality of the large model system under test. Therefore, the initial test cases are considered valid test cases.

[0121] If the predicted response differs from the expected response, it is necessary to determine whether the discrepancy stems from a need to optimize the capabilities of the large-scale model system under test, or from defects in the initial test cases themselves. Based on these reasons, the validity verification results of the initial test cases should be determined. Subsequent embodiments will describe this situation in detail, and will not be elaborated upon here.

[0122] In this embodiment, the use case keywords contained in the valid use cases are also stored in the keyword database. It can be understood that if the use case keywords in the initial test case exist in the keyword database, it means that the use case keywords applied by the initial test case are the same as those of the valid use cases. Therefore, the initial test case should also be a valid use case, that is, the initial test case itself does not have any defects.

[0123] Therefore, for cases where the predicted intent recognition type differs from the target intent recognition type, the validity verification result is determined by comparing the keywords in the initial test cases with the keywords stored in the keyword database. Essentially, this involves analyzing whether the intent recognition error stems from an optimization need for the capabilities of the large-scale model system under test, or from defects in the initial test cases themselves, in order to determine the validity verification result of the initial test cases based on these reasons. Subsequent embodiments will describe this situation in detail, and will not be elaborated upon here.

[0124] In summary, the above verification logic completes the validity verification of the initial test cases that focus on the response results, ensuring that the obtained valid test cases are those that fit the functions of the large model system under test, thereby ensuring that the performance of the large model system under test on the test set can accurately reflect the intent recognition capability of the large model system under test.

[0125] As an optional implementation, the validity verification result of the initial test cases is determined by comparing the predicted response results with the expected response results, including:

[0126] Compare whether the predicted response results are consistent with the expected response results;

[0127] If they are consistent, then the validity verification result is determined to be that the initial test case has passed the verification;

[0128] If there is a discrepancy, obtain the analysis results of the initial test cases;

[0129] If the analysis result is the first result characterizing the large model system under test to support the initial test cases, then the validity verification result is determined to be that the initial test cases have passed the verification.

[0130] If the analysis result indicates that the large model system under test does not support the response to the initial test case, then the validity verification result is determined to be that the initial test case failed the verification.

[0131] In this embodiment, if the predicted response result is consistent with the expected response result, it indicates that the large model system under test can accurately identify the intent of the initial test case and accurately respond to the initial test case. This indicates that the initial test case fits the tools and capabilities supported by the large model system under test itself, and therefore the initial test case can be used as a valid test case.

[0132] If the predicted response result is inconsistent with the expected response result, the initial test case will be output to the client to obtain the analysis results of the initial test case from professionals.

[0133] If the analysis result is the first result representing the support of the initial test cases for the large model system under test, it indicates that the reason for the discrepancy between the predicted response result and the expected response result is that the performance of the large model system under test is insufficient and the initial test cases themselves do not have defects. Therefore, the validity verification result is determined to be that the verification is passed, that is, the initial test cases are valid test cases.

[0134] If the analysis result is a second result indicating that the system under test does not support the response of the initial test case, it means that the reason why the predicted response result is inconsistent with the expected response result is that the initial test case does not match the tools and capabilities supported by the system under test. In other words, the initial test case has defects. Therefore, the validity verification result is determined to be that the verification failed, and the initial test case is an invalid test case.

[0135] As an optional implementation, the validity verification result of the initial test cases is determined by comparing the test case keywords in the initial test cases with the keywords stored in the keyword database, including:

[0136] Determine if the test case keywords in the initial test cases exist in the keyword database;

[0137] If it exists, then the validity verification result is determined to be that the initial test case has passed the verification;

[0138] If it does not exist, proceed with the following steps:

[0139] Determine whether the predicted response results match the expected response results;

[0140] If the conditions are met, the validity verification result is determined to be that the initial test case failed the verification.

[0141] If not, the system will use the large model under test to regenerate the predicted response results for the initial test cases according to the response chain corresponding to the target intent identification type, and return the steps of verifying the validity of the initial test cases by comparing the predicted response results with the expected response results.

[0142] As mentioned earlier, if the test case keywords in the initial test case exist in the keyword database, it means that the initial test case generated based on the test case keywords has passed the validity verification. Therefore, the initial test case will not be analyzed again, and it can be directly determined that the initial test case has passed the validity verification.

[0143] If the keywords in the initial test case do not exist in the keyword database, then it is further determined whether the predicted response result matches the expected response result. If it does, it means that the large model system can correctly respond to this initial test case even when the intent recognition is incorrect, by making things up or other means. Therefore, it is determined that the initial test case is not a valid test case for the target intent recognition type, that is, the initial test case itself has a defect, and therefore it is determined that the initial test case has failed the verification.

[0144] As an optional implementation, after assembling the valid use cases into a test set, the following is also included:

[0145] Within a specified time period, acquire all historical question information input by users into the large model system under test;

[0146] Extract question keywords from historical question information to determine the frequency of their occurrence;

[0147] Keywords that appear more frequently than a preset threshold will be used as supplementary keywords.

[0148] If the test case keywords contained in the test set do not contain keywords that match the supplementary keywords, then supplementary test cases are generated based on the supplementary keywords and test case sentence structures by specifying the large model.

[0149] Add supplementary test cases to the test set.

[0150] Specifically, within a specified time period, historical question information input by users into the large-scale model system under test is obtained. Question keywords are extracted from the historical question information using a third-party large-scale model. Keywords whose frequency exceeds a preset threshold are used as supplementary keywords. The type of the third-party large-scale model and the preset threshold can be selected according to actual needs; this embodiment does not impose any particular limitations on them.

[0151] If the test case keywords in the test set do not contain keywords that match the supplementary keywords, meaning the test set does not cover the supplementary keywords, then a supplementary test set is generated based on the supplementary keywords and test case sentence structures by specifying a large model. For example, as mentioned earlier, the supplementary keywords are sequentially filled into each test case sentence structure; and by replacing verbs in the test case sentence structures with semantically similar verbs, each supplementary keyword corresponds to multiple supplementary test cases. Furthermore, the supplementary test cases can be validated using the methods described in the aforementioned embodiments, and test cases that pass the validation are added to the test set.

[0152] In summary, as Figure 2 As shown, in this embodiment, based on real online questions from users, frequently occurring question keywords are statistically analyzed, and test cases in the test set are supplemented based on these question keywords to improve the comprehensiveness of the test set.

[0153] As an optional implementation, the target intent identification type is multiple, and after assembling the valid use cases into a test set, it also includes:

[0154] Within a specified time period, acquire all historical question information input by users into the large model system under test;

[0155] For each target intent recognition type, perform the following steps:

[0156] Determine the first quantity of historical question information corresponding to the target intent identification type; and use the proportion of the first quantity to the total number of historical question information as a reference proportion;

[0157] Determine the second number of valid use cases corresponding to the target intent recognition type; and use the proportion of the second number to the total number of valid use cases as the actual proportion.

[0158] Based on the reference ratio and the actual ratio, supplement or delete valid use cases for the corresponding target intent recognition type.

[0159] In this embodiment, for each target intent recognition type, the reference ratio is the proportion of real questions of the corresponding target intent recognition type among all real questions asked online by users; the actual ratio is the proportion of valid test cases of the corresponding target intent recognition type in the test set among all valid test cases. Based on the reference ratio and the actual ratio, valid test cases of the corresponding target intent recognition type are added or deleted to make the actual ratio of the adjusted test set close to the reference ratio, ensuring the matching of the test set with real application scenarios.

[0160] This embodiment does not specifically limit the specific implementation process for supplementing or deleting valid test cases corresponding to the target intent recognition type. For example, when the reference ratio is greater than the actual ratio, supplementary test cases are generated based on the test case keywords corresponding to the target intent recognition type, and the test cases that pass the validity verification in the supplementary test cases are added to the test set. When the reference ratio is less than the actual ratio, some valid test cases corresponding to the target intent recognition type are deleted, but the number of valid test cases must still be ensured to be no less than a preset threshold to ensure the statistical significance of the intent recognition capability test results determined based on the test set.

[0161] In summary, as Figure 2 As shown, in this embodiment, valid test cases in the test set are supplemented or deleted based on real questions asked by users online, ensuring the matching of the test set with real application scenarios.

[0162] Furthermore, after obtaining the aforementioned valid test cases, a single valid test case is used as a single-round valid test case. Multiple specified valid test cases are combined to obtain multi-round valid test cases, in order to test the intent recognition and response capabilities of the large-scale model system under test in multi-round dialogues. This application does not impose specific restrictions on the rules for combining and generating multi-round valid test cases; a specified number of valid test cases can be randomly selected for combination, or a specified number of semantically similar valid test cases can be selected for combination.

[0163] As an optional implementation, after assembling the valid use cases into a test set, the following is also included:

[0164] The test set is input into the evaluation model, and the evaluation results generated by the evaluation model from the specified dimensions are obtained. The specified dimensions include at least one of the following: the coverage of the test set keywords by the effective test sets, the matching degree between the effective test sets and the corresponding target intent recognition type, and the standardization of the expression of the effective test sets.

[0165] If the evaluation results do not meet the preset conditions corresponding to the specified dimension, the test set will be adjusted.

[0166] Given that the test cases in the test set are valid test cases, not all initial test cases, there is a possibility that the valid test cases in the test set may not cover all test case keywords. To ensure the comprehensiveness of testing the intent recognition capabilities of the large model system under test using the test set, and the statistical significance of the test results, the coverage level of test case keywords by the valid test cases in the test set is set in a specified dimension.

[0167] Specifically, the test set and all use case keywords are input into the evaluation model. The evaluation model determines a coverage score based on the degree to which the effective use cases in the test set cover the use case keywords, and uses the coverage score as the evaluation result. The coverage score is positively correlated with the degree of coverage. If the coverage score is lower than the preset coverage score, the test set is adjusted. This embodiment does not specifically limit the specific method of adjusting the test set. For example, the uncovered keywords output by the evaluation model (i.e., keywords in the use case keywords that are not covered by effective use cases) are obtained; core keywords are selected from the uncovered keywords; supplementary test cases are generated based on the core keywords and use case sentence structures, and added to the test set.

[0168] In this embodiment, the specified dimension also includes the degree of matching between valid use cases and their corresponding target intent recognition types, that is, a secondary verification of valid use cases from the dimension of intent matching accuracy. Specifically, valid use cases are input into the evaluation model to determine the degree of matching between the target intent recognition type and the intent recognition type determined by the evaluation model for valid use cases; a matching score is determined based on the degree of matching, and the matching score is used as the evaluation result, wherein the matching score is positively correlated with the degree of matching.

[0169] If the matching score is lower than the preset matching score, the manual review result is obtained; if the manual review result indicates that the valid use case has defects, the valid use case is regenerated based on the use case keywords contained in the valid use case, and the regenerated valid use case is added to the test set.

[0170] In this embodiment, the specified dimension also includes the degree of standardization of the expression of valid use cases; for example, if a valid use case has problems such as mixing multiple languages, ambiguity, or logical contradiction, it is considered that the valid use case does not meet the preset conditions corresponding to the degree of standardization of expression, and a new valid use case is generated based on the use case keywords contained in the valid use case, and the newly generated valid use case is added to the test set.

[0171] In summary, in this embodiment, the test set is evaluated based on the degree of coverage of test case keywords, the degree of matching between effective test cases and their corresponding target intent recognition types, and the degree of standardization of the expression of effective test cases. The test set is then adjusted in a targeted manner based on the evaluation results to improve its quality.

[0172] Please refer to Figure 5 , Figure 5 This application provides a schematic diagram of the structure of an electronic device, which includes:

[0173] Memory 51 is used to store computer programs;

[0174] Processor 52 is used to implement the steps of the test set construction method for any of the above-mentioned large model systems when executing computer programs.

[0175] The aforementioned electronic devices include, but are not limited to, laptops or desktop computers.

[0176] The processor 52 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 52 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 52 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 52 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 52 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0177] The memory 51 may include one or more computer-readable storage media, which may be non-transitory. The memory 51 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices.

[0178] In some embodiments, the electronic device may further include a display screen 53, an input / output interface 54, a communication interface 55, a power supply 56, and a communication bus 57. Those skilled in the art will understand that... Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0179] For a detailed description of the electronic device provided in this embodiment, please refer to the embodiment of the test set construction method of the large model system described above. This embodiment will not repeat the details here.

[0180] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0181] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0182] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for constructing a test set for a large model system, characterized in that, The method includes: Obtain a pre-constructed first mapping relationship; wherein the first mapping relationship includes the correspondence between the functions supported by the large model system and the intent recognition type of the large model system, and the correspondence between the intent recognition type and the prompt words; the prompt words are used to guide the specified large model to generate initial test cases; Based on the first mapping relationship, and according to the functions supported by the large model system under test, the target intent recognition type and target prompt words corresponding to the large model system under test are determined; Based on the target prompt words, use case keywords related to the target intent recognition type are obtained through the specified large model, and the initial test cases are generated based on the use case keywords and predefined use case sentence patterns; the intent type label of the initial test cases is the target intent recognition type. Valid test cases are selected from the initial test cases, and the valid test cases are combined into a test set to test the intent recognition capability of the large model system under test. Valid test cases are selected from the initial test cases, including: The initial test cases are input into the large model system under test to obtain the predicted intent recognition type determined by the large model system under test for the initial test cases, and the predicted response result for the initial test cases. Obtain a pre-constructed second mapping relationship; the second mapping relationship is the correspondence between use case types and verification rules; the use case types include at least intent recognition-oriented and response result-oriented types; Based on the second mapping relationship and the target test case type to which the initial test case belongs, the target verification rule corresponding to the initial test case is determined; According to the target verification rules, the validity of the initial test cases is verified based on the predicted intent identification type and the predicted response result; If the initial test case passes verification, then the initial test case is considered a valid test case.

2. The method for constructing a test set for a large model system as described in claim 1, characterized in that, The target use case type is the intent recognition focus; correspondingly, according to the target verification rules, the validity of the initial test case is verified based on the predicted intent recognition type and the predicted response result, including: Determine whether the predicted intent recognition type is consistent with the target intent recognition type; If they match, then the initial test case has passed verification. If they are inconsistent, then determine whether the predicted response result matches the expected response result; If the conditions are met, then the initial test case is determined to have failed verification. If it does not meet the requirements, then the initial test case is deemed to have passed verification.

3. The method for constructing a test set for a large model system as described in claim 1, characterized in that, The target test case type is the response result-oriented type; correspondingly, according to the target verification rules, the validity of the initial test case is verified based on the predicted intent identification type and the predicted response result, including: Determine whether the predicted intent recognition type is consistent with the target intent recognition type; If they match, the validity verification result of the initial test case is determined by comparing the predicted response result with the expected response result. If there is a discrepancy, the validity verification result of the initial test cases is determined by comparing the test case keywords in the initial test cases with the keywords stored in the keyword database; wherein, the keyword database is used to store the test case keywords contained in the valid test cases.

4. The method for constructing a test set for a large model system as described in claim 3, characterized in that, The validity verification result of the initial test case is determined by comparing the predicted response result with the expected response result, including: Compare whether the predicted response result is consistent with the expected response result; If they are consistent, then the validity verification result is determined to be that the initial test case has passed the verification; If there is a discrepancy, the analysis results of the initial test cases are obtained; If the analysis result is the first result characterizing that the large model system under test supports the response to the initial test case, then the validity verification result is determined to be that the initial test case has passed the verification. If the analysis result is a second result indicating that the large model system under test does not support the response to the initial test case, then the validity verification result is determined to be that the initial test case has failed verification.

5. The method for constructing a test set for a large model system as described in claim 3, characterized in that, The validity verification result of the initial test cases is determined by comparing the test case keywords in the initial test cases with the keywords stored in the keyword database, including: Determine whether the test case keywords in the initial test cases exist in the keyword database; If it exists, then the validity verification result is determined to be that the initial test case has passed the verification; If it does not exist, proceed with the following steps: Determine whether the predicted response result matches the expected response result; If the conditions are met, then the validity verification result is determined to be that the initial test case failed the verification. If not, the system will regenerate the predicted response result for the initial test case according to the response link corresponding to the target intent identification type through the large model system under test, and return the step of determining the validity verification result of the initial test case by comparing the predicted response result with the expected response result.

6. The method for constructing a test set for a large model system as described in claim 1, characterized in that, After assembling the valid use cases into a test set, the following is also included: Within a specified time period, acquire all historical question information input by the user into the large model system under test; Extract question keywords from the historical question information and determine the frequency of occurrence of the question keywords; The question keywords with a frequency greater than a preset threshold are used as supplementary keywords; If the test case keywords contained in the test set do not contain any keywords that match the supplementary keywords, then supplementary test cases are generated based on the supplementary keywords and the test case sentence structure using the specified large model. Add the supplementary test cases to the test set.

7. The method for constructing a test set for a large model system as described in claim 1, characterized in that, The target intent recognition type is multiple, and after the valid use cases are combined into a test set, it also includes: Within a specified time period, acquire all historical question information input by the user into the large model system under test; For each of the target intent recognition types, perform the following steps: Determine a first number of historical question information corresponding to the target intent recognition type; and use the proportion of the first number to the total number of historical question information as a reference proportion; Determine a second number of valid use cases corresponding to the target intent recognition type; and use the proportion of the second number to the total number of valid use cases as the actual proportion; Based on the reference ratio and the actual ratio, supplement or delete valid use cases corresponding to the target intent recognition type.

8. The method for constructing a test set for a large model system as described in claim 1, characterized in that, After assembling the valid use cases into a test set, the following is also included: The test set is input into the evaluation model, and the evaluation results generated by the evaluation model from the specified dimensions are obtained. The specified dimensions include at least one of the following: the coverage of the test set keywords by the effective test sets, the matching degree between the effective test sets and the corresponding target intent recognition type, and the standardization of the expression of the effective test sets. If the evaluation result does not meet the preset conditions corresponding to the specified dimension, the test set will be adjusted.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the test set construction method for a large model system as described in any one of claims 1 to 8 when executing the computer program.

Images