Automated test result inductive analysis method and system
By using an automated test result inductive analysis method and a large language model to clean and analyze the semantic parsing test results, the problem of low efficiency and inaccurate analysis in existing technologies is solved. This achieves efficient and intelligent error identification and report generation, and is applicable to multiple fields such as video, music, and news.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN CHANGHONG ELECTRIC CO LTD
- Filing Date
- 2026-04-14
- Publication Date
- 2026-07-10
AI Technical Summary
Existing semantic parsing testing methods rely on manual analysis, which is inefficient, cannot handle large-scale test data, lacks systematic error classification, makes it difficult to identify root causes, and lacks intelligent error analysis capabilities.
By using automated test result inductive analysis methods, large language models are used to clean and preprocess the test results, generate standardized error example JSON data structures, classify them using domain mapping and string matching algorithms, and use large language models to summarize patterns and analyze root causes, generating structured analysis reports.
It significantly improves testing efficiency and analysis accuracy, achieves full-process automation, can identify complex error patterns and generate comprehensive and scalable analysis reports, and is suitable for a variety of application scenarios.
Smart Images
Figure CN122364087A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automated testing, specifically to a method and system for summarizing and analyzing automated test results. Background Technology
[0002] With the rapid development of artificial intelligence technology, semantic parsing systems have been widely used in fields such as intelligent dialogue, speech recognition, and natural language processing. The quality of semantic parsing systems directly affects user experience and system performance, thus requiring extensive testing. Existing semantic parsing testing methods mainly suffer from the following technical shortcomings: (1) The analysis of test results mainly relies on manual labor, which is inefficient and cannot handle large-scale test data; (2) The classification and summarization of errors lack a systematic approach, making it easy to overlook important error patterns; (3) It cannot automatically identify the root cause of the error, making it difficult to provide effective guidance for system optimization; (4) Lack of intelligent error analysis capabilities, unable to deeply understand the semantic features of errors. Summary of the Invention
[0003] To improve the efficiency and accuracy of test result analysis, this application provides an automated test result summarization and analysis method and system.
[0004] The technical solution adopted by the present invention to solve the above problems is: Methods for summarizing and analyzing automated test results include: Step 1: Clean and preprocess the test results to generate a standardized JSON data structure for example errors. This includes: extracting error description columns from the semantic test results; describing the mapping relationship of parsed fields in the JSON file according to the pre-configured domain column names; preserving the error row index to establish the correspondence between error data and original data; classifying the error data; and generating a standardized JSON data structure for example errors. Step 2: Analyze test results based on the large language model: Construct a prompt word template for the large language model in the single-class error analysis stage, input the JSON data generated in Step 1 into the large language model, the large language model analyzes the pattern and cause of single-class errors, and outputs the analysis results.
[0005] This invention extracts error descriptions from semantic test results and combines them with domain mapping to transform error data into structured information. Then, it constructs JSON examples based on error description classification and uses a large language model to summarize patterns and analyze root causes of various errors through standardized prompt words. Finally, it outputs structured results containing error indexes and root causes. This process significantly improves the automation and efficiency of error analysis.
[0006] Furthermore, it also includes step 3, aggregating the results of multiple single-class error analysis, constructing a prompt word template for the large language model in the comprehensive analysis stage, inputting the aggregated data into the large language model, and generating the final report.
[0007] This invention aggregates the analysis results of multiple error types and then inputs them into a large language model for comprehensive analysis, ensuring the comprehensiveness of error coverage.
[0008] Furthermore, the domain column name description JSON file contains a mapping of domain-specific field names and meanings.
[0009] This invention, by updating the domain column name description JSON file and prompt word template, can flexibly adapt to new application scenarios and has stronger applicability.
[0010] Furthermore, the areas include video, music, news, sports, or shopping.
[0011] Furthermore, step 1 performs classification based on a string matching algorithm. String matching algorithms can be applied directly to the raw data without the need for complex feature engineering or data transformation, enabling efficient and accurate classification.
[0012] Furthermore, the standardized error example JSON data structure includes an error description, an error index, a number of records, column descriptions, and specific error records.
[0013] Furthermore, the large language model prompt word template includes an output data structure description and error summarization examples.
[0014] An automated test result summarization and analysis system is used to implement automated test result summarization and analysis methods, including: The domain configuration module is used to pre-configure a JSON file describing the domain column names; The data cleaning and preprocessing module extracts error description columns from semantic test results; parses field mapping relationships in JSON files based on pre-configured domain column names; preserves error row indexes to establish a correspondence between error data and original data; classifies error data; and generates standardized error example JSON data structures. The large language model analysis module analyzes test results based on the large language model: it constructs a large language model prompt word template for the single-class error analysis stage, inputs error example JSON data into the large language model, analyzes the pattern and cause of single-class errors, and outputs the analysis results.
[0015] Furthermore, the large language model analysis module is also used to aggregate multiple single-class error analysis results, construct a large language model prompt word template for the comprehensive analysis stage, input the aggregated data into the large language model, and generate the final report.
[0016] The advantages of this invention compared to the prior art are: (1) High degree of automation: It realizes full-process automation from data cleaning to error analysis, which greatly improves testing efficiency; (2) High accuracy of analysis: By utilizing the deep semantic understanding capabilities of large language models, it can accurately identify complex error patterns; (3) High scalability: By configuring different domain column name description files, it can adapt to a variety of application scenarios; (4) Highly practical: Generates structured analysis reports, making it easy for developers to quickly locate and fix problems. Attached Figure Description
[0017] Figure 1 Flowchart for the method of summarizing and analyzing automated test results; Figure 2 Here is a flowchart of the data cleaning and preprocessing process; Figure 3 Flowchart for the single-type error analysis phase; Figure 4 This is a flowchart for the comprehensive analysis phase. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0019] Current test result analysis relies on manual methods, resulting in low processing efficiency. This invention extracts error descriptions from semantic test results and combines them with domain mapping to transform error data into structured information. Then, it constructs JSON examples based on error description classification and uses a large language model to summarize patterns and analyze root causes of various errors through standardized prompt words. Finally, it outputs structured results containing error indexes and root causes. This process significantly improves the automation and efficiency of error analysis, ensures the comprehensiveness and accuracy of error coverage, and can be flexibly adapted to new application scenarios by updating domain configuration files and prompt word templates.
[0020] like Figure 1 As shown, the methods for summarizing and analyzing automated test results include: Step 1: Clean and preprocess the test results to generate a standardized JSON data structure for example errors, including: Step 11: Extract the error description column from the semantic test results; Step 12: Parse the field mapping relationship according to the pre-configured domain column name description JSON file. The domain column name description JSON file contains the specific field names and meaning mappings for each domain, supporting configurations for multiple domains such as video, music, and news; Step 13: Retain the error row index to establish the correspondence between the error data and the original data. The original test data can be traced back through the retained error row index. Step 14: Classify the error data. In this invention, error classification is based on the similarity of error descriptions. A string matching algorithm is used to identify similar errors. Three representative samples are retained for each type of error to ensure coverage of different variations. Step 15: Generate a standardized error example JSON data structure, including error description, error index, number of records, column description, and specific error record.
[0021] Step 2: Analyze test results based on the large language model: Construct a prompt word template for the large language model in the single-class error analysis stage, including an output data structure description and error summary examples; input the error example JSON data generated in Step 1 into the large language model, which analyzes the patterns and root causes of single-class errors and outputs structured analysis results containing error indexes and error descriptions; the large language model uses a pre-trained language model and has deep semantic understanding and text generation capabilities.
[0022] Furthermore, step 3 involves aggregating the results of multiple single-type error analyses to construct a comprehensive analysis stage large language model prompt word template. The aggregated data is then input into the large language model to generate the final report. Engineers can directly locate problems and implement optimizations based on the report, effectively improving the efficiency and accuracy of the repair work.
[0023] Example 1: Taking semantic parsing testing in the video domain as an example, the specific implementation steps are as follows: Step 1: Data cleaning and preprocessing.
[0024] like Figure 2 As shown, the error description column is extracted from the test results Excel file. This Excel file contains the following key columns: 1) TestCase: Test case, such as "Watch the movie adapted from the comic"; 2) Error description: Specific error information, such as "data.semantic.tag / type / award / director result error"; 3) Columns related to expected and actual values.
[0025] Based on the column names in the video domain, describe the JSON configuration file and establish field mapping relationships: { "TestCase": "Test Case", Error Description: The specific error description indicates that the value of xxx is different from the value of Expect_xxx. "Expect_data.semantic.category": "Expected value of semantic category", "data.semantic.category": "The actual return value of the semantic category" } The errors are categorized into types such as "label extraction error" and "category identification error," with three representative samples retained for each category, generating the following JSON structure: { "error_description":"Error in data.semantic.tag / type / award / director result", "error_index": 1, "record_count": 3, "column_descriptions": {...}, "records": [...] } Step 2: Single-type error analysis phase.
[0026] like Figure 3 As shown, using the configured prompt word template, the above JSON data is input into the large language model. The large language model outputs the following after analysis: { "info": { "error_index": [1, 2, 3], "info": "Semantic tag extraction failed: The system cannot recognize modifier tags such as 'comic adaptation', causing the tag field to return null." } } Step 3: Comprehensive Analysis Phase.
[0027] like Figure 4 As shown, after aggregating the analysis results of multiple error types, a prompt word template for the large language model is constructed for the comprehensive analysis stage. This template is then input into the large language model again for comprehensive analysis, ultimately generating a report. { "title": "Error in extracting or parsing singer, actor, director, film title, time, etc." "info": [ { "error_index": [1, 2, 3, 4, 5], "info": "The system has a serious flaw in processing queries containing specific attribute descriptions..." } ] } Example 2: Taking semantic parsing tests in the music field as an example, the specific implementation steps are as follows: Step 1: Data cleaning and preprocessing.
[0028] Read the data from the music semantic parsing test results Excel file, which contains the following key columns: 1) TestCase: Test cases, such as "the first song of the first singer"; 2) Error description: Specific error information, such as "data.semantic.singer / song / style result error"; 3) Columns related to expected and actual values.
[0029] Based on the column names in the music domain, describe the JSON configuration file and establish field mapping relationships: { "TestCase": "Music Test Case", Error Description: The specific error description indicates that the value of xxx is different from the value of Expect_xxx. "Expect_data.semantic.singer": "Expected value of the singer", "data.semantic.singer": "The actual return value of the singer", "Expect_data.semantic.song": "Expected value of the song title", "data.semantic.song": "The actual return value of the song name", "Expect_data.semantic.style": "Expected value of the music style", "data.semantic.style": "The actual return value of the music style" } Errors are categorized into types such as "singer identification error," "song title extraction error," and "music style classification error." Three representative samples are retained for each type, generating the following JSON structure: { "error_description": "Incorrect result from data.semantic.singer", "error_index": 5, "record_count": 3, "column_descriptions": { "TestCase": "Music Test Case", Error Description: "Singer identification error" }, "records": [ {"TestCase": "The first song by the first singer", "Expect_singer": "The first singer", "Actual_singer": "null"}, {"TestCase": "Second singer's second song", "Expect_singer": "Second singer", "Actual_singer": "null"}, {"TestCase": "The third singer's third song", "Expect_singer": "The third singer", "Actual_singer": "null"} ] } Step 2: Single-type error analysis phase.
[0030] Using the configured prompt word template, input the above JSON data into the large language model. For singer recognition errors, the large language model analyzes and outputs: { "info": { "error_index": [5, 6, 7], "info": "Singer identification failed: The system cannot correctly extract singer information from the expression pattern 'singer + song name', causing the 'singer' field to return null." } } Output the following for incorrect music style classification: { "info": { "error_index": [8, 9, 10], "info": "Music style classification error: The system's ability to classify Chinese pop music styles is insufficient, and it cannot accurately distinguish between sub-genres such as folk, rock, and rap." } } Step 3: Comprehensive Analysis Phase.
[0031] After aggregating the analysis results of multiple error types in the music field, a prompt word template for the comprehensive analysis stage of the large language model is constructed. This template is then input into the large language model for comprehensive analysis, ultimately generating a report. { "title": "Error in extracting core elements of music semantic analysis", "info": [ { "error_index": [5, 6, 7, 8, 9, 10, 11, 12], "info": "The semantic parsing system for the music domain has systemic defects when processing Chinese music queries: 1) Low accuracy in singer name recognition, especially for the sentence structure 'singer + of + song'; 2) Song title extraction is easily affected by modifiers, such as adjectives like 'good-sounding' and 'classic', which can affect extraction accuracy; 3) The music style classification is too coarse and lacks the ability to understand the sub-genres of Chinese pop music. It is recommended to optimize the named entity recognition model, enhance the understanding of music domain terminology, and expand the training data for music style classification." } ] } The above embodiments demonstrate the following advantages of the method of the present invention: Strong domain adaptability: Only the configuration JSON file for the music domain needs to be changed to adapt to music semantic parsing tests. No modification to the core algorithm is required, making it more adaptable to different domains. Accurate error pattern recognition: It can accurately identify domain-specific error patterns, such as singer identification, song title extraction, and style classification. Consistent depth of analysis: Whether in the video or music field, it can provide in-depth root cause analysis and improvement suggestions; Extensibility verification: The method of this invention can be easily extended to other semantic parsing application areas such as news, sports, and shopping.
[0032] This invention achieves intelligent analysis of semantic parsing test errors through a hierarchical large language model analysis architecture, significantly improving testing efficiency and analysis quality, and providing strong support for the continuous optimization of the semantic parsing system.
[0033] Correspondingly, this invention also provides an automated test result summarization and analysis system for implementing an automated test result summarization and analysis method, including: The domain configuration module is used to pre-configure a JSON file describing the domain column names; The data cleaning and preprocessing module is used to extract error description columns from semantic test results; parse field mapping relationships in JSON files based on pre-configured domain column names; retain error row indexes to establish a correspondence between error data and original data; classify error data; and generate standardized error example JSON data structures. The large language model analysis module analyzes test results based on the large language model: it constructs a large language model prompt word template for the single-class error analysis stage, inputs error example JSON data into the large language model, the large language model analyzes the patterns and root causes of single-class errors, and outputs structured analysis results containing error indexes and error descriptions.
[0034] Furthermore, the large language model analysis module is also used to aggregate multiple single-class error analysis results, construct a large language model prompt word template for the comprehensive analysis stage, input the aggregated data into the large language model, and generate the final report.
Claims
1. An automated test result summarization and analysis method, characterized in that, include: Step 1: Clean and preprocess the test results to generate a standardized JSON data structure for example errors. This includes: extracting error description columns from the semantic test results; describing the mapping relationship of parsed fields in the JSON file according to the pre-configured domain column names; preserving the error row index to establish the correspondence between error data and original data; classifying the error data; and generating a standardized JSON data structure for example errors. Step 2: Analyze test results based on the large language model: Construct a prompt word template for the large language model in the single-class error analysis stage, input the JSON data generated in Step 1 into the large language model, the large language model analyzes the pattern and cause of single-class errors, and outputs the analysis results.
2. The automated test result summarization and analysis method according to claim 1, characterized in that, It also includes step 3, which aggregates the results of multiple single-type error analysis, constructs a prompt word template for the large language model in the comprehensive analysis stage, inputs the aggregated data into the large language model, and generates the final report.
3. The automated test result summarization and analysis method according to claim 1, characterized in that, The domain column name description JSON file contains a mapping of domain-specific field names and meanings.
4. The automated test result summarization and analysis method according to claim 3, characterized in that, The categories include video, music, news, sports, or shopping.
5. The automated test result summarization and analysis method according to claim 1, characterized in that, Step 1 involves classification based on a string matching algorithm.
6. The automated test result summarization and analysis method according to claim 1, characterized in that, The standardized error example JSON data structure includes an error description, error index, number of records, column descriptions, and specific error records.
7. The automated test result summarization and analysis method according to claim 1, characterized in that, The prompt word template for the large language model includes an explanation of the output data structure and examples of error summarization.
8. An automated test result summarization and analysis system, used to implement the automated test result summarization and analysis method according to any one of claims 1-7, characterized in that, include: The domain configuration module is used to pre-configure a JSON file describing the domain column names; The data cleaning and preprocessing module is used to extract error description columns from semantic test results; parse field mapping relationships in JSON files based on pre-configured domain column names; retain error row indexes to establish a correspondence between error data and original data; classify error data; and generate standardized error example JSON data structures. The large language model analysis module analyzes test results based on the large language model: it constructs a large language model prompt word template for the single-class error analysis stage, inputs error example JSON data into the large language model, analyzes the pattern and cause of single-class errors, and outputs the analysis results.
9. The automated test result summarization and analysis system according to claim 8, characterized in that, The large language model analysis module is also used to aggregate multiple single-class error analysis results, construct large language model prompt word templates for the comprehensive analysis stage, input the aggregated data into the large language model, and generate the final report.