Automatic testing method, device, storage medium and program product for an evaluation tool

Abstract

The embodiment of the invention discloses an automatic testing method and device of an evaluation tool, a storage medium and a program product. The method comprises the steps of obtaining original question data of an evaluation tool, and extracting key data and overall logic data of each question type from the original question data; estimating all answering conditions of each question type based on the key data of each question type; generating a corresponding single test case for each answer condition of each question type; based on the overall logic data, arranging and combining the single test cases of each question type to generate a plurality of overall test cases; and testing the evaluation tool by using each integral test case, and outputting a test report. According to the method, the test case can be generated by obtaining and estimating all the answer conditions of each question type, the evaluation tool can be efficiently and comprehensively tested, the problems that manual testing is low in efficiency and slow in response are solved, and the defects that script testing is incomplete in coverage and high in maintenance cost are overcome.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511290067.8 (22) Application Date 2025.09.10 (71) Applicant Beijing Qingsong Yikang Information Technology Co., Ltd. Address 7th Floor, Block F, Yonghe Building, No. 28 Andingmen East Street, Dongcheng District, Beijing 100007 (72) Inventors Hou Fayun and Li Jing (74) Patent Agency Beijing Youyijia Intellectual Property Agency (General Partnership) 16075 Patent Attorney Sun Jianfeng (51) Int.Cl. G06F 11 / 3668 (2025.01) G06F 40 / 30 (2020.01) (54) Invention Title Automatic Testing Method, Apparatus, Storage Medium and Program Product of Evaluation Tool (57) Abstract This disclosure discloses an automatic testing method, apparatus, storage medium and program product of an evaluation tool. The method includes: acquiring the original question data of the assessment tool; extracting key data and overall logical data for each question type from the original question data; estimating all possible responses for each question type based on the key data; generating corresponding individual test cases for each response for each question type; arranging and combining the individual test cases for each question type based on the overall logical data to generate multiple overall test cases; testing the assessment tool using each overall test case and outputting a test report. This method can generate test cases by acquiring all possible responses for each question type, enabling efficient and comprehensive testing of the assessment tool. It solves the problems of low efficiency and slow response of manual testing, and overcomes the shortcomings of incomplete coverage and high maintenance costs of script testing. Claims (2 pages), Description (16 pages), Drawings (3 pages), CN 121166543 A, 2025.12.19, CN 1 21 16 65 43 A. 1. An automatic testing method for an assessment tool, characterized by comprising: acquiring original question data of the assessment tool; extracting key data and overall logical data for each question type from the original question data; estimating all possible responses for each question type based on the key data for each question type; generating corresponding individual test cases for each response for each question type; arranging and combining the individual test cases for each question type based on the overall logical data to generate multiple overall test cases; testing the assessment tool using each overall test case and outputting a test report.2. The automatic testing method of the assessment tool according to claim 1, characterized in that: when the question type is multiple choice, a jump relationship graph is generated based on the key data of the multiple choice; wherein, the nodes of the jump relationship graph represent multiple choice questions, the attributes of the nodes include question ID, options and mapping values ​​of each option, the edges between nodes are directed edges, the directed edges represent the execution order between multiple choice questions, and the attributes of the directed edges include the jump conditions and path hash values ​​between multiple choice questions; based on the attributes of the nodes and the attributes of the directed edges, a full path traversal is performed on the jump relationship graph; based on all the traversed paths, all answer situations for the multiple choice question type are generated. 3. The automatic testing method for the evaluation tool according to claim 2, characterized in that, the step of performing a full path traversal of the jump relationship graph based on the attributes of nodes and directed edges includes: starting from the starting node, traversing the jump relationship graph; determining whether there is a jump condition from the current node to the next node when traversing each node; when there is a jump condition from the current node to the next node, traversing all possible options of the current node in sequence; when traversing each possible option, determining whether the jump condition from the current node to the next node is satisfied based on the currently selected option; when the jump condition is satisfied, comparing the path hash value from the current node to the next node with the recorded path hash value; when the comparison result is different, moving to the next node to continue traversing, and recording the path hash value from the current node to the next node; after the jump relationship graph traversal is completed, counting all traversed paths. 4. The automatic testing method of the assessment tool according to claim 1, characterized in that: when the question type is a true / false question, all true / false questions are traversed; when each true / false question is traversed, based on the answer of the current true / false question, correct answer content that matches the answer and incorrect answer content that is the opposite of the answer are generated; it is determined whether the content of the current true / false question contains semantically negative words or semantically ambiguous words; when the content of the question contains semantically negative words, answer content containing reverse expression is generated; when the content of the question contains semantically ambiguous words, answer content containing extreme scenarios is generated; when the content of the question contains both semantically negative words and semantically ambiguous words, answer content containing both semantically negative words and semantically ambiguous words is generated; after all true / false questions have been traversed, all answer content of each true / false question is arranged and combined to generate all answer situations of the true / false question type.Claims 1 / 2 Page 2 CN 121166543 A 5. The automatic testing method of the assessment tool according to claim 1, characterized in that: when the question type is a fill-in-the-blank question, all fill-in-the-blank questions are traversed; when each fill-in-the-blank question is traversed, based on the answer of the current fill-in-the-blank question and the format requirements of the answer, the answer content that conforms to the correct format, the answer content that does not conform to the correct format, the correct answer content that conforms to the answer, and the incorrect answer content that does not conform to the answer are generated; after all fill-in-the-blank questions have been traversed, all the answer content of each fill-in-the-blank question is arranged and combined to generate all the answer situations of the fill-in-the-blank question type. 6. The automatic testing method of the assessment tool according to claim 1, characterized in that: when the question type is an open-ended question, all open-ended questions are traversed; when each open-ended question is traversed, answer content of different character lengths is generated based on the answer to the current open-ended question and the character length requirement of the answer; based on the input form requirement of the answer to the current open-ended question and the answer, answer content with diversified input forms is generated, wherein the diversified input forms include at least one of plain text, preset special characters, HTML tags, and multilingual mixing; through semantic parsing of the answer to the current open-ended question, answer content with different semantic relevance to the answer to the current open-ended question is generated; after all open-ended questions have been traversed, all answer content of each open-ended question is arranged and combined to generate all answer situations for the open-ended question type. 7. The automatic testing method for the assessment tool according to claim 1, characterized in that, based on the overall logical data, arranging and combining individual test cases for each question type to generate multiple overall test cases includes: obtaining the order of all question types based on the overall logical data; selecting one individual test case from each question type sequentially according to the order to generate all possible overall test cases; wherein, the number of overall test cases is equal to the product of the number of individual test cases for each question type. 8. A computer device, characterized in that, the computer device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the automatic testing method for the assessment tool according to any one of claims 1-7. 9. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores computer instructions for causing a computer to execute the automatic testing method for the assessment tool according to any one of claims 1-7. 10. A computer program product comprising computer instructions, characterized in that, when executed by a processor, the computer instructions implement the steps of the automatic testing method of the evaluation tool according to any one of claims 1-7.Claims 2 / 2 Page 3 CN 121166543 A Automatic Testing Method, Apparatus, Storage Medium and Program Product of Evaluation Tool Technical Field

[0001] This disclosure relates to the field of automatic testing technology, and in particular to an automatic testing method, apparatus, storage medium and program product of an evaluation tool. Background Art

[0002] In today's digital age, questionnaire platforms, online examination systems, user survey tools and other applications are widely used. These systems need to frequently update questions to meet different business needs and user experience, while also ensuring the stability of system functions. Currently, after question updates or function iterations, existing questionnaire systems usually use manual testing or scripted regression testing to verify system stability. Manual testing requires testers to manually execute various test tasks, while scripted regression testing involves pre-writing test scripts to simulate user operations.

[0003] However, manual testing is inefficient, time-consuming, and cannot quickly respond to high-frequency question updates. For questionnaires containing hundreds of questions, full testing may take several hours to several days. While script testing improves testing efficiency to some extent, it also suffers from incomplete coverage. It requires pre-written test cases, making it difficult to cover complex user interaction paths and leading to the omission of peripheral scenarios. Furthermore, when question rules change, the test script needs to be readjusted, resulting in high maintenance costs.

[0004] In view of this, the present disclosure provides an automatic testing method, apparatus, storage medium, and program product for assessment tools. This method can generate test cases by acquiring all predicted responses for each question type, enabling efficient and comprehensive testing of the assessment tool. It solves the problems of low efficiency and slow response of manual testing, while overcoming the shortcomings of incomplete coverage and high maintenance costs associated with script testing.

[0005] In a first aspect, embodiments of this disclosure provide an automatic testing method for an assessment tool, employing the following technical solution: acquiring the original question data of the assessment tool, and extracting key data and overall logical data for each question type from the original question data; estimating all possible responses for each question type based on the key data for each question type; generating corresponding individual test cases for each response for each question type; arranging and combining the individual test cases for each question type based on the overall logical data to generate multiple overall test cases; testing the assessment tool using each overall test case and outputting a test report.

[0006] Optionally, when the question type is a multiple-choice question, a jump relationship graph is generated based on the key data of the multiple-choice question; wherein, the nodes of the jump relationship graph represent multiple-choice questions, the attributes of the nodes include question ID, options and mapping values ​​of each option, the edges between nodes are directed edges, the directed edges represent the execution order between multiple-choice questions, and the attributes of the directed edges include the jump conditions between multiple-choice questions and the path hash value; based on the attributes of the nodes and the attributes of the directed edges, a full path traversal is performed on the jump relationship graph; based on all the traversed paths, all answer situations for the multiple-choice question type are generated.

[0007] Optionally, the step of performing a full path traversal of the jump relationship graph based on the attributes of nodes and the attributes of directed edges includes: starting from the starting node, traversing the jump relationship graph; determining whether there is a jump condition from the current node to the next node when traversing each node; when there is a jump condition from the current node to the next node, traversing all possible options of the current node in sequence; when traversing each possible option, determining whether the jump condition from the current node to the next node is satisfied based on the currently selected option; when the jump condition is satisfied, comparing the path hash value from the current node to the next node with the recorded path hash value; when the comparison result is different, moving to the next node to continue traversing, and recording the path hash value from the current node to the next node; after the jump relationship graph is traversed, counting all traversed paths.

[0008] Optionally, when the question type is a true / false question, all true / false questions are traversed; each time a true / false question is traversed, based on the answer of the current true / false question, the correct answer content that matches the answer and the incorrect answer content that is the opposite of the answer are generated; it is determined whether the content of the current true / false question contains semantically negative words or semantically ambiguous words; when the content of the question contains semantically negative words, the answer content containing the reverse expression is generated; when the content of the question contains semantically ambiguous words, the answer content containing extreme scenarios is generated; when the content of the question contains both semantically negative words and semantically ambiguous words, the answer content containing both semantically negative words and semantically ambiguous words is generated; after all true / false questions have been traversed, all the answer content of each true / false question is arranged and combined to generate all the answer situations of the true / false question type.

[0009] Optionally, when the question type is a fill-in-the-blank question, all fill-in-the-blank questions are traversed; when each fill-in-the-blank question is traversed, based on the answer of the current fill-in-the-blank question and the format requirements of the answer, the answer content that conforms to the correct format, the answer content that does not conform to the correct format, the correct answer content that conforms to the answer, and the incorrect answer content that does not conform to the answer are generated; after all fill-in-the-blank questions are traversed, all the answer content of each fill-in-the-blank question is arranged and combined to generate all the answer situations of the fill-in-the-blank question type.

[0010] Optionally, when the question type is an open-ended question, all open-ended questions are traversed; when traversing each open-ended question, answer content of different character lengths is generated based on the answer to the current open-ended question and the character length requirement of the answer; based on the answer to the current open-ended question and the input form requirement of the answer, answer content with diversified input forms is generated, wherein diversified input forms include at least one of plain text, preset special characters, HTML tags, and multilingual mixing; through semantic parsing of the answer to the current open-ended question, answer content with different semantic relevance to the answer to the current open-ended question is generated.

[0011] After all open-ended questions have been traversed, all answer content of each open-ended question is arranged and combined to generate all answer situations for the open-ended question type.

[0012] Optionally, the step of arranging and combining individual test cases for each question type based on the overall logical data to generate multiple overall test cases includes: obtaining the order of all question types based on the overall logical data; selecting one individual test case from each question type in sequence according to the order to generate all possible overall test cases; wherein the number of overall test cases is equal to the product of the number of individual test cases for each question type.

[0013] In a second aspect, embodiments of this disclosure also provide an automatic testing system for an assessment tool, employing the following technical solution: an extraction module, used to acquire the original question data of the assessment tool, and extract key data and overall logical data for each question type from the original question data; an estimation module, used to estimate all answer situations for each question type based on the key data for each question type; a generation module, used to generate corresponding individual test cases for each answer situation for each question type; a combination module, used to arrange and combine the individual test cases for each question type based on the overall logical data to generate multiple overall test cases; and a testing module, used to test the assessment tool using each overall test case and output a test report.

[0014] In a third aspect, embodiments of this disclosure also provide a computer device, employing the following technical solution: the computer device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute any of the above-described automatic testing methods for the assessment tool.

[0015] In a fourth aspect, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute an automatic testing method of any of the above-described evaluation tools.

[0016] In a fifth aspect, embodiments of this disclosure also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any of the methods described above.

[0017] The automatic testing method for the evaluation tool provided in embodiments of this disclosure, by acquiring the original question data as the basis of the entire test, can ensure that the test covers all question content of the evaluation tool. By extracting key data and overall logical data for each question type, the internal structure and rules of the evaluation tool can be deeply understood, providing an accurate basis for subsequent testing, ensuring the comprehensiveness and accuracy of the test, and avoiding the omission of important test points. By analyzing the key data for each question type, all possible answer situations can be systematically predicted, which enables the test to cover various normal and abnormal answer scenarios, discover problems that may occur when the evaluation tool handles different answers, and improve the robustness and reliability of the evaluation tool. By generating individual test cases for each answer situation, the function of each question type can be finely tested. Each individual test case is for a specific answer situation, making the test more specific and targeted, and facilitating the location and resolution of problems. Individual test cases have clear inputs and expected outputs, facilitating repeated testing and ensuring the stability of the assessment tool under different environments. Furthermore, when the question types or rules of the assessment tool change, only the corresponding individual test cases need to be modified, improving the maintainability of the test cases. The overall test cases consider the logical relationships and assessment processes between different question types, simulating real-world user responses when using the assessment tool. By arranging and combining individual test cases, various test scenarios can be generated, providing a more comprehensive test of the overall performance and functional completeness of the assessment tool. In actual use, interactions between different question types may produce problems that cannot be detected by individual tests. The overall test cases can uncover these cross-question type issues, such as abnormal interface display during question type switching or data transmission errors, ensuring the overall smoothness and correctness of the assessment tool.

[0018] The above description is merely an overview of the technical solutions of this disclosure. In order to better understand the technical means of this disclosure and to implement it according to the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more obvious and understandable, preferred embodiments are given below, and described in detail with reference to the accompanying drawings. Brief Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a flowchart illustrating the automatic testing method of the assessment tool provided in the embodiments of this disclosure; Figure 2 is a flowchart illustrating the prediction method for all answer situations of multiple-choice questions provided in the embodiments of this disclosure; Figure 3 is a flowchart illustrating the jump relationship graph traversal method provided in the embodiments of this disclosure; Figure 4 is a principle block diagram of the automatic testing system of the assessment tool provided in the embodiments of this disclosure; Figure 5 is a structural schematic diagram of a computer device provided in the embodiments of this disclosure. Detailed Description

[0021] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0022] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. This disclosure can also be implemented or applied through other different specific implementation methods, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this disclosure.

[0023] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities besides one or more of the aspects set forth herein can be used to implement this device and / or practice this method.

[0024] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show components related to this disclosure and are not drawn according to the actual number, shape, and size of the components in implementation. In actual implementation, the type, quantity, and proportion of each component can be arbitrarily changed, and the component layout may also be more complex.

[0025] In addition, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0026] Referring to Figure 1, this disclosure provides an automatic testing method for an assessment tool, comprising the following steps: S1: Obtaining the original question data of the assessment tool, and extracting key data and overall logical data for each question type from the original question data; S2: Based on the key data for each question type, estimating all possible answers for each question type; S3: Generating corresponding individual test cases for each answer for each question type; S4: Based on the overall logical data, arranging and combining the individual test cases for each question type to generate multiple overall test cases; S5: Testing the assessment tool using each overall test case and outputting a test report.

[0027] This disclosure provides an automatic testing method for an assessment tool. Obtaining the original question data is the foundation of the entire test, ensuring that the test covers all question content of the assessment tool. By extracting key data and overall logical data for each question type, a deeper understanding of the internal structure and rules of the assessment tool can be gained, providing an accurate basis for subsequent testing, ensuring the comprehensiveness and accuracy of the test, and avoiding the omission of important test points. Among them, key data reflects the core characteristics of each question type, such as question type, answer options, scoring rules, etc.; overall logic data reflects the relationship between different question types and the evaluation process. Extracting this data helps to conduct targeted testing for different question types and overall logic, improving testing efficiency and effectiveness.

[0028] By analyzing the key data of each question type, all possible answer situations can be systematically predicted. This allows the test to cover various normal and abnormal answer scenarios, discover problems that may occur when the evaluation tool handles different answers, and improve the robustness and reliability of the evaluation tool. Predicting answer situations before testing can help discover some potential logical errors or vulnerabilities in advance. For example, certain special answer combinations may cause calculation errors or system crashes in the evaluation tool. By predicting and testing these situations in advance, problems can be fixed in time before the product goes live, reducing later maintenance costs.

[0029] Generating individual test cases for each answer situation allows for detailed testing of the function of each question type. Each individual test case is for a specific answer situation, making the test more specific and targeted, and facilitating the location and resolution of problems. Individual test cases have clear inputs and expected outputs, facilitating repeated testing and ensuring the stability of the assessment tool under different environments. Furthermore, when the question types or rules of the assessment tool change, only the corresponding individual test cases need to be modified, improving the maintainability of the test cases.

[0030] The overall test cases consider the logical relationships and assessment processes between different question types, simulating the actual answering behavior of users when using the assessment tool. By arranging and combining individual test cases, various different test scenarios can be generated, more comprehensively testing the overall performance and functional integrity of the assessment tool.In practical use, the interaction between different question types may produce some problems that cannot be found in individual tests. The overall test cases can discover these cross-question type problems, such as abnormal interface display when switching question types, data transmission errors, etc., to ensure the overall smoothness and correctness of the assessment tool.

[0031] By using multiple overall test cases to conduct comprehensive testing on the assessment tool, the quality and performance of the assessment tool can be objectively evaluated. The test report records the testing process and results in detail, including the passed test cases, the failed test cases and their reasons, providing developers with clear problem feedback, which facilitates targeted improvements. The test report is an important proof of the quality of the assessment tool, providing a reliable basis for the release and launch of the product. At the same time, the test report can also serve as a reference for subsequent product optimization and upgrades, helping the development team to continuously improve the quality and user experience of the assessment tool.

[0032] In summary, this method generates test cases by obtaining all predicted answers for each question type, which can efficiently and comprehensively test the assessment tool. It solves the problems of low efficiency and slow response of manual testing, and overcomes the shortcomings of incomplete coverage and high maintenance costs of script testing.

[0033] In S1, the original question data containing detailed information of all questions in the assessment tool can be obtained through data interface calls, file reading, etc. These original question data are parsed to extract key data for each question type. The key data for different question types are different. The question types include, but are not limited to, multiple choice, true / false, fill-in-the-blank, and open-ended questions. In terms of identifying question types, a pre-trained BERT classification model is used. The training dataset of this model contains 12 question types and more than 100,000 samples. When using it, you only need to input the question stem text and configure the label. The model can output the question type classification result, such as "multiple choice" and "fill-in-the-blank", with the corresponding confidence level. When the confidence threshold reaches or exceeds 0.95, the classification result is considered reliable. Furthermore, the model supports custom question type extensions to meet the needs of different users. Question content parsing is handled by a multi-format parsing engine. For HTML rich text, this engine can accurately extract tag structure, such as... 、 Meanwhile, it identifies and extracts style attributes, including font size, color, etc., as well as the URLs of multimedia resources, covering images, audio, and other content. For plain text content, the engine will perform sentence segmentation and word segmentation, and will retain format markers such as line breaks and special symbols to ensure that the parsed content is complete and accurate. For answer parsing, the rule engine automatically converts natural language constraints (such as "Please enter an integer from 1 to 100") into executable validation rules: marking required fields (required: true / false), generating format regular expressions (such as date ^\d{4}-\d{2}-\d{2}$), setting the word count range (min_length / max_length), and specifying the data type (integer, string, date).

[0034] In addition, it is also necessary to extract overall logical data, which includes the order of answering each question type, the calculation logic of the overall paper score, etc. In particular, both the score calculation logic for each question type and the overall comprehensive score calculation logic can adopt a calculation logic parsing method. By using a syntax tree parser, variables (such as Q1 score, Q2 score), operators (such as ×, +) and coefficients (such as 0.3, 0.7) can be accurately extracted. Based on the variables, operators and coefficients, expressions that can be directly calculated are generated, thereby accurately calculating the score of each question, such as "total score = Q1 score × 0.3 + Q2 score × 0.7". In addition, the syntax tree parser also supports the processing of nested formulas, such as "Q3 score = max(Q1, Q2)", which further enhances the ability to handle complex calculation logic. Through this method, the score calculation can be ensured to be both accurate and efficient, providing a solid foundation for evaluation and analysis.

[0035] For multiple-choice questions, the option information parsing function can identify the unique identifier of each option (such as "Q1-A") and determine the display order of the options (by integer sorting). Meanwhile, this function can extract the text content of each option and determine the mapping value of the option, such as the value "5" corresponding to option A. These mapping values ​​will affect the calculation of the multiple-choice question score. This function can also obtain the scoring rules of the multiple-choice question. According to the scoring rules, the answer situation and the mapping value of the option, the score of the answer can be determined. For example, suppose a multiple-choice question contains 4 options A, B, C and D. The correct answer is A and D. The mapping value of each correct answer is 1 point. The wrong answer is not scored. If a student chooses A, B and D, then according to the scoring rules, he will score 2 points for this question (A gets 1 point, B gets no points, D gets 1 point). The correct options of the multiple-choice question will be marked in Boolean form to quickly determine the correct answer.

[0036] Extract the jump rule text of the multiple-choice question from the original question data. By parsing the jump rule text, obtain the execution order and jump conditions between multiple-choice questions. Among them, the "keyword matching + semantic dependency analysis" hybrid algorithm can be used to analyze the multiple-choice question.The jump rule text is parsed, specifically including: extracting the condition body of each jump rule from the jump rule text, converting the condition body of each jump rule into standardized entries in the standardized jump rule base; determining the trigger question and target question from each standardized entry, and determining the dependency relationship and jump condition between the trigger question and the target question; integrating all the dependency relationships and jump conditions between the trigger questions and the target questions to obtain the dependency relationships and jump conditions between all multiple-choice questions. For example, for the jump rule "If Q1 selects A and age > 18, then jump to Q3", the algorithm extracts the condition "Q1 = A AND age > 18" and the target question "Q3", then generates a standardized entry in the format {"trigger_id": "Q1", "condition": "A && age > 18", "target_id": "Q3"} and stores it in a standardized jump rule base. This rule base supports AND, OR, and NOT logical combinations and can clearly reflect the jump condition from the trigger question to the target question. According to the entry "{"trigger_id": "Q1", "condition": "A && age > 18", "target_id": "Q3"}" in the rule base, Q1 is the trigger question and Q3 is the target question. When the jump condition "Q1 selects A and age > 18" is met, the jump will be from Q1 to Q3. For multiple-choice questions with show / hide conditions, this method can also correctly parse them. For example, "Q5 is only displayed when Q4 selects B" can be transformed into the Boolean expression "Q5.display=(Q4.value=="B")" in the standardized jump rule base. This determines that the target question Q5 will only be displayed after option B, which triggers the question, is selected. For cases with multiple nested conditions, such as "Q6 display condition: Q4 selects B and Q5>10", this method can also accurately handle them, transforming them into the corresponding Boolean expression "Q6 .display=(Q4 .value=="B)&&(Q5> 10)", thus clarifying the dependency relationship between Q4, Q5, and Q6. Through the above steps, starting with the initial questions and progressively parsing each multiple-choice question, the execution order and jump conditions between multiple-choice questions can be comprehensively and accurately obtained from the original question data, providing a basis for subsequent evaluation of answers. Furthermore, it supports batch parsing of option configuration tables in Excel or JSON format, facilitating users to quickly process large amounts of data.

[0037] Based on the above functions, key data of multiple-choice questions can be obtained. This data includes the question ID, the text content of the question, the unique identifier of the option, the text content of the option, the mapping value of the option, the scoring rules, and the relationships between multiple-choice questions.Dependency relationships and jump conditions, these key data help to comprehensively evaluate the answers to multiple-choice questions and ensure the accuracy of scoring and navigation.

[0038] For true / false questions, fill-in-the-blank questions and open-ended questions, in the NLP semantic analysis layer, TF-IDF and TextRank are integrated to extract keywords from the questions and answers, covering core entities (such as "age" and "occupation"), action words (such as "describe" and "calculate") and constraint words (such as "must" and "at least"). Based on dependency parsing, conditions, negation and limiting relationships are identified, and the position of negation words (such as "not" in "do not choose A") is marked to eliminate ambiguity of use cases. For open-ended questions, the sentiment classification model (TextCNN) predicts the question intent tendency (fact statement, sentiment feedback, opinion solicitation), outputs sentiment labels (neutral, positive, negative) and confidence, thereby generating test input that fits the intent. Through hierarchical clustering, the sentences in the question stem are logically grouped to construct a logical structure tree, which explicitly presents parallel, progressive, causal and other relationships to ensure complete and consistent parsing of complex question stems.

[0039] In S2, referring to the flowchart of the method for predicting all answers to multiple-choice questions shown in Figure 2, the method for predicting all answers to multiple-choice questions includes the following steps: S21: When the question type is multiple-choice, generate a jump relationship graph based on the key data of the multiple-choice questions; S22: Perform a full path traversal of the jump relationship graph based on the attributes of the nodes and the attributes of the directed edges; S23: Generate all answers to the multiple-choice question type based on all the traversed paths.

[0040] Wherein, the nodes of the jump relationship graph represent multiple-choice questions, and the attributes of the nodes include the question ID, the option, and the mapping value of each option. The edges between the nodes are directed edges, which represent the execution order between multiple-choice questions. The attributes of the directed edges include the jump conditions between multiple-choice questions and the path hash value. Wherein, the path hash value is generated by performing a hash operation (such as SHA-256) on the question IDs and jump conditions of the nodes at both ends of the directed edge, and serves as the unique identifier of the directed edge.

[0041] Referring to the flowchart of the jump relationship graph traversal method shown in FIG3, "performing a full path traversal of the jump relationship graph based on the attributes of nodes and the attributes of directed edges" includes the following steps: S221: Starting from the starting node, traverse the jump relationship graph; S222: When traversing each node, determine whether the current node has a jump condition to the next node; S223: When the current node has a jump condition to the next node, traverse all possible options of the current node in sequence; S224: When traversing each possible option, determine whether the jump condition from the current node to the next node is satisfied based on the currently possible option; S225: When the jump condition is satisfied, compare the path hash value from the current node to the next node with the recorded path hash value;S226: When the comparison results are different, move to the next node to continue traversing and record the path hash value from the current node to the next node; S227: After the jump relationship graph is traversed, count all the traversed paths.

[0042] In the above traversal process of expanding the jump relationship graph, when traversing each node, it is necessary to check whether the node has a jump condition pointing to the subsequent node. The jump condition can be a single condition or multiple conditions. If the current node does not have a jump condition to the next node, it means that the current path has ended and the path is recorded as a complete path. If the current node has a jump condition to the next node, it is necessary to traverse all possible options of the current node. For each possible option, it is determined whether it meets the jump condition from the current node to the next node. If the jump condition is not met, the current path cannot be extended further. The path that has been traversed is recorded as a complete path, and then backtrack to the previous node to continue exploring other options of the node. When an option meets the jump condition, obtain the path hash value from the current node to the next node and compare it with the recorded path hash value. If the path hash value is equal to the recorded path hash value, it means that the path has entered a loop. The traversal of the current branch is terminated, and the path backtracks to the previous node to continue exploring other options of the node. If it is not equal to the recorded path hash value, the path jumps to the next node and continues the above-mentioned checks, option traversal, condition judgment and hash value comparison operations. During the entire traversal, the backtracking mechanism is used to ensure that all possible options of each node can be traversed. When jumping from the current node to the next node along an option, after completing the exploration of all paths starting from the next node, the program will return to the current node and continue to try other unexplored options of the current node. Finally, all complete paths recorded during the traversal are counted. These paths are all non-repeating valid paths in the jump relationship graph.

[0043] Based on each traversed path, at least one answer to the multiple-choice question is generated. These answer situations include four categories: normal answer situation, boundary answer situation, abnormal answer situation, and related answer situation. In multiple-choice questions, "normal answering" refers to the answering situation where the selection is made according to the normal logic and rules set in the question. This means choosing the correct or incorrect option, typically selecting from the given valid options that aligns with the question's intent. This is used to verify the accuracy of the assessment tool's answer judgment and page navigation functions under normal circumstances. For example, in a single-choice question, choosing the correct option from options A, B, C, and D; in a multiple-choice question, reasonably selecting multiple valid options. "Boundary answering" involves answering methods in multiple-choice questions that fall into boundary states. For multiple-choice questions, selecting all or none are typical boundary situations. Additionally, shuffling the order of the options also falls under boundary situations. These combinations are mainly used to verify the accuracy of the assessment tool.The assessment tool must demonstrate logical consistency under boundary conditions to correctly handle these special selection methods. Abnormal response scenarios refer to inputting non-option values ​​that do not meet the question requirements or making selections outside the range of options, such as entering the number "5" in a question with only three options (A, B, C) or entering the text "invalid." The purpose is to test the tool's error handling capabilities and see if it can accurately identify and prompt such abnormal inputs. Related response scenarios involve combining options from preceding questions to form a logically related series of choices, verifying the correctness of the logical chain between questions. In a question-and-answer process consisting of multiple multiple-choice questions, the selection of earlier questions affects the display and processing of later questions. Related response scenarios simulate this coherent response process, generating complete path test cases. By comprehensively utilizing these four response types, all response scenarios for multiple-choice questions can be generated based on the traversal path, comprehensively testing the functionality and logic of the assessment tool. Instruction manual, page 8 / 16, 11 CN 121166543 A

[0044] The key data of the judgment questions include the question ID, the content of the question, the answer, and the scoring rules. All judgment questions are traversed. For each judgment question, based on its key data, all possible answers are generated. First, for the current judgment question being traversed, based on the answer to the current judgment question, correct answers matching the answer and incorrect answers opposite to the answer are generated. That is, the answer content contains a basic combination of affirmative words (such as "√", "correct" and "true") and negative words (such as "×", "false" and "incorrect"). This is the most basic possible answer for all judgment questions, used to make a preliminary binary judgment on the question. It can directly test the evaluation system's ability to identify the basic correctness of the question statement. For example, for the question "The moon is a satellite of the earth," the basic answer is {correct, incorrect}, which can quickly test the system's basic judgment ability on this statement. The system determines whether the current true / false question contains semantically negative or semantically ambiguous words. When the question contains semantically negative words, it generates an answer containing a reverse statement. The reverse statement is a judgment based on a reverse understanding of the original question's semantics. This is used to test whether the assessment system can correctly understand negative semantics and avoid making incorrect judgments due to interference from negative words. For example, for the question "Coral is not a living organism," in addition to the basic {True, False}, the reverse statement answer could be set as {False (i.e., coral is considered a living organism)}, thus verifying the system's ability to handle negative semantics. When the question contains semantically ambiguous words, it generates an answer including extreme scenarios. Extreme scenarios are the most specific and boundary-point cases within the scope of the ambiguous statement, testing the respondent's judgment ability in complex and varied situations, and testing the assessment system's logical fault tolerance, verifying whether it can consider various possibilities under ambiguous statements. For example, for the question "Metals are usually conductive..."In extreme scenarios, the answer could be {false (some special alloys are non-conductive at specific low temperatures)}, prompting the system to make accurate judgments in complex situations. When the question contains both semantically negative and semantically ambiguous words, an answer containing both is generated. This combination is more complex, requiring a comprehensive analysis of the various possibilities arising from negative semantics and ambiguous expressions. This type of answer is used to comprehensively assess the system's ability to make comprehensive judgments in complex semantic situations. For example, in the question "Generally speaking, deep-sea creatures are not warm-blooded animals," a comprehensive assessment is needed. Complex responses can be {True (most deep-sea creatures are indeed not warm-blooded), False (there may be unknown special warm-blooded deep-sea creatures)}, thus assessing the system's ability to handle complex semantics and multiple possibilities. After all true / false questions have been traversed, all responses for each true / false question are permuted and combined to generate all possible responses for each question type. For example, when there are 2 true / false questions, each true / false question has 4 possible responses, so the total number of possible responses is 4 × 4 = 4² = 16. Similarly, when there are n true / false questions, the total number of possible responses is 4n.

[0045] The key data for fill-in-the-blank questions includes the question ID, question content, answer, format requirements for the answer, keywords in the question and answer, and scoring rules. All fill-in-the-blank questions are iterated over. For each fill-in-the-blank question, based on its key data, all possible answers are generated. For the current fill-in-the-blank question, based on the answer and format requirements, correct answers, incorrect answers, correct answers, and incorrect answers are generated. When generating the other three types of answers (excluding correct answers), the question text, the correct answer, and key words in the question and answer must be comprehensively considered. Logical deduction and language organization are performed based on these elements to generate appropriate and suitable answers that fit the question context. After all fill-in-the-blank questions have been iterated over, all possible answers for each fill-in-the-blank question are arranged and combined to generate all possible answer scenarios for the fill-in-the-blank question type.

[0046] Key data for open-ended questions includes the question ID, question content, answer, character length requirement for the answer, input format requirement for the answer, semantic parsing result of the answer, and scoring rules. All open-ended questions are iterated over. For each open-ended question, based on its key data, all possible answers are generated. For the currently iterated open-ended question, based on the answer and character length requirement, answers of different lengths are generated. These answers include zero-character length (empty input) and the minimum allowed length of the answer (e.g., page 9 / 16, CN 121166543 A in the specification).The system generates responses with varying input formats based on the current open-ended question's answer and input requirements. These formats include plain text, preset special characters (e.g., #, ￥, %), HTML tags, and at least one of the following: [List of input formats would be inserted here]. Semantic analysis of the current open-ended question's answer is used to generate responses with different semantic relevance. These include responses that closely match the question and answer and receive high scores (strong semantic relevance); responses that partially match the question and answer and receive medium scores (medium semantic relevance); responses that only slightly match the question and receive low scores (low semantic relevance); and responses that resemble gibberish and receive no points (no semantic relevance). After all open-ended questions have been traversed, all responses for each open-ended question are arranged and combined to generate all possible responses for each question type.

[0047] The above method analyzes key data for each question type, enabling prediction of all possible responses for each question type. This helps to understand and simulate the performance of test takers on different question types, providing valuable reference for the evaluation of the assessment tool and facilitating the generation of targeted test cases.

[0048] In S3, corresponding individual test cases are generated for all responses to multiple-choice questions. For each response to a multiple-choice question, a different number of individual test cases can be generated based on a preset weight list. Since the responses to multiple-choice questions can be divided into four categories, the individual test cases for multiple-choice questions also include four types of sub-test cases: normal test cases, boundary test cases, abnormal test cases, and related test cases. These test cases include input values, operation steps, expected scores, expected jump results, and verification points. Among them, normal test cases refer to test cases designed based on normal response situations, used to verify the answer judgment and basic jump function of the assessment tool under normal selection conditions. For example, if the correct option B is selected, the expected score is 10 points, and a jump to page Q3 is triggered. Boundary test cases are test cases generated based on boundary response scenarios. They are used to check the assessment tool's logical processing capabilities under boundary conditions. For example, in a multiple-choice question, selecting all options verifies whether the assessment tool supports storing all selections; or, by shuffling the order of options, it checks whether the assessment tool can handle the selection correctly. Exception test cases are test cases designed based on abnormal response scenarios. They are used to test the assessment tool's error handling mechanism. For example, inputting "5" (with only A / B / C options) is expected to prompt "Please select a valid option"; inputting the text "Invalid" checks whether the assessment tool can provide the corresponding error message. Related test cases are test cases created based on related response scenarios to verify cross-question logical chains.The correctness of the choices, such as Q1 choosing A to display Q3, Q3 choosing B to display Q5, is ensured through complete path test cases to ensure the logical coherence of the assessment tool under associated selection. Through the detailed definition of different answer situations and the generation of corresponding test cases, all possible answer situations of multiple choice questions can be fully covered, and the assessment tool can be fully tested.

[0049] For all answer situations of the judgment question type, corresponding individual test cases are generated respectively, and for each answer situation of the judgment question type, different numbers of individual test cases can be generated based on the preset weight list. Since the answer situations of judgment questions can be divided into 4 categories, the individual test cases of the judgment question type also contain 4 types of sub-test cases, namely basic test cases, semantic negation test cases, fuzzy judgment test cases and comprehensive test cases. These test cases include answer content, expected score and semantic verification points. Basic test cases are used to test the system's ability to recognize the basic correctness of the question statement; semantic negation test cases are used to examine the system's ability to understand the negation semantics of questions containing semantic negation words; fuzzy judgment test cases are used to test the system's logical fault tolerance when facing questions containing semantic fuzzy words; comprehensive test cases are used to evaluate the system's comprehensive judgment ability to handle questions containing both semantic negation words and semantic fuzzy words.

[0050] For all answer situations of fill-in-the-blank questions, corresponding individual test cases are generated respectively, and for each answer situation of fill-in-the-blank questions, different numbers of individual test cases can be generated based on a preset weight list. Since the answer situations of fill-in-the-blank questions can be clearly divided into 4 categories, the individual test cases of fill-in-the-blank questions also contain 4 types of sub-test cases, namely format compliance test cases, format abnormality test cases, content correctness association test cases, and content error instruction manual 10 / 16 pages 13 CN 121166543 A association test cases. These test cases include input content, expected format verification results, expected scores, and verification points. Among them, format compliance test cases are mainly used to verify whether the answers entered by the test takers strictly conform to the format requirements of the question. These test cases focus on the format specifications of the answers, ensuring that they not only meet the basic format rules, but also cover edge cases. For example, when the answer requires a date, the input must conform to the standard format "2024-01-01" and cannot exceed the maximum date "2024-12-31". Based on this requirement, standard format and edge format answers conforming to regular expression rules will be generated, such as "2024-06-15" (standard format), "2024-01-01" (minimum edge format value), "2024-12-31" (maximum edge format value), etc., to test the system's ability to process correct format input. Format exception test cases are used to check the system's response mechanism when faced with incorrect format input, ensuring that the system can accurately identify and provide the corresponding error.The note indicates that these test cases cover various possible formatting errors. For example, for date input, using a separator format like "2024 / 01 / 01" is incorrect; inputting the text "abc" in a numerical question results in a type error; inputting "12345" in a phone number question results in insufficient length. When these incorrectly formatted answers appear, the system should anticipate and display a "format error" message. These test cases ensure the system's robustness in format validation. Content-related test cases verify whether the candidate's input correctly matches the question's requirements, emphasizing semantic accuracy and logical rationality. These test cases primarily assess the candidate's understanding of the question content and their ability to apply knowledge. For example, in a game-related question like "The Kingdom's Royal City," the correct answer should be "Radiant City," demonstrating semantic matching; for the question "1+1=__," inputting "2" is the correct answer. By setting these test cases, the system can ensure accurate judgment of correct answers. Content error association test cases are used to test the system's ability to identify content errors or irrelevant answers. These test cases include semantically irrelevant or logically incorrect answers, aiming to simulate possible incorrect answers from test takers. For example, in the game-related question "The Kingdom's Royal City", inputting "apple" is semantically irrelevant; for the question stem "1+1=__", inputting "3" is a logically incorrect answer, and inputting "abc" is completely irrelevant to the question. Through these test cases, it can be ensured that the system can accurately judge and process incorrect content input.

[0051] For all answer situations of open-ended questions, corresponding unit test cases are generated respectively, and for each answer situation of open-ended questions, different numbers of unit test cases can be generated based on a preset weight list. Since the answer situations of open-ended questions can be clearly divided into 3 categories, the unit test cases of open-ended questions also include 3 types of sub-test cases, namely length boundary test cases, content type test cases, and semantically related test cases. These test cases include input text, expected storage results, expected display effects, and security verification points. The length boundary test cases are used to verify the system's ability to process responses of varying lengths. They assess the system's stability and accuracy when response lengths are at critical limits. For example, inputting 1000 characters of text verifies whether the system truncates or stores the entire text, thus determining if the system can correctly handle longer responses and avoid data loss or storage anomalies due to length issues. The content type test cases are used to test the system's compatibility and security with different input content types, ensuring the system can identify and process various legal content types while effectively filtering content with potential security risks. For example, inputting <imgsrc=xonerror=alert(1)>, verify whether XSS scripts are filtered to prevent malicious scripts from posing a security threat to the system and to ensure the security of the system and user data. Semantic related test cases are used to evaluate the system's ability to identify and judge the semantic relevance of the answer content, examine whether the system can accurately distinguish the answer content with different semantic relevance, and give a reasonable evaluation based on its relevance to the question stem. For example, if the question stem is "feedback experience", input "good experience" (strong relevance) and "good weather today" (irrelevant) to verify the accuracy and reliability of the system in semantic understanding and relevance judgment.

[0052] It can be seen that generating individual test cases for each answer situation of each question type and adjusting the number of individual test cases generated for different answer situations according to the preset weight list can comprehensively and meticulously test the system's ability to handle different question types, accurately discover potential problems, and ensure that the system runs stably and accurately in various actual answer situations. Instruction manual, pages 11 / 16, 14 CN 121166543 A

[0053] Further, the K-means algorithm is used to cluster the historical defect data of each question type. The historical defect data covers key information such as defect type, application scenario and frequency. High-risk scenarios are identified through clustering operations. For example, in the health question questionnaire scenario, the "fill-in-the-blank question format error cluster" accounts for 30%. For the identified high-risk scenarios, the generation weight of the corresponding type of test cases is dynamically increased, such as increasing the original weight of 20% to 40%, so that the number of test cases generated for this type increases, thereby carrying out testing and risk prevention work in a more targeted manner.

[0054] In S4, based on the overall logical data, the order of all question types is obtained; according to the order, a single test case is selected from each question type in turn to combine them to generate all possible overall test cases; wherein, the number of overall test cases is equal to the product of the number of single test cases of each question type. When generating overall test cases, attention should be paid to incorporating the calculation logic of the overall comprehensive score into the scope of consideration, while also taking into account other important related factors, so that the generated test cases have a high degree of professionalism and the ability to simulate real evaluation.

[0055] In S5, the probability of finding defects in the overall test cases is predicted based on the random forest model. This random forest model will comprehensively consider several key features, including the types of sub-test cases included in the overall test cases, the historical defect rate, and the scenario complexity. Among them, the type of sub-test cases can reflect the characteristics of the software functional modules or business processes targeted by the test cases; the historical defect rate reflects the frequency of defects found in past tests by different types of sub-test cases, which to some extent suggests its potential to find new defects; the scenario complexity describes the software usage scenarios simulated by the overall test cases.Complexity and complexity often lead to hidden defects. By analyzing and learning these characteristics using a random forest model, a probability value for finding defects is output for each overall test case. Overall test cases are sorted in descending order of probability and executed in that order. Test cases ranked higher are considered high-value cases; prioritizing high-value cases maximizes defect discovery efficiency even with limited testing resources (such as time and manpower). These cases are more likely to uncover potential software defects, and concentrating testing resources on them allows for the discovery of more serious problems early in the testing process, providing strong support for software quality assurance. Furthermore, this data-driven test case execution order arrangement reduces the blindness of test execution, improves the scientific nature and predictability of the testing process, helps optimize the testing workflow, reduces testing costs, and enhances the overall efficiency of software testing.

[0056] Both individual test cases and overall test cases can be generated using AI technology, transforming the overall test cases to be executed into executable operations to simulate user interaction. During the testing process, detailed operation logs, response times, element states, and other test process data are recorded for easy review and analysis later. To ensure that the test can cover different platforms, targeted measures were taken in terms of execution engine adaptation. The Web client uses Selenium WebDriver, which supports mainstream browsers such as Chrome, Edge, and Firefox; the mobile client uses Appium, which is compatible with both Android and iOS systems. Furthermore, by encapsulating operation methods such as click() and input() through a unified interface, cross-platform compatible execution of test cases was achieved. In terms of element positioning, a hybrid positioning strategy of "dynamic XPath + text matching + attribute filtering" was adopted. For dynamic ID elements, positioning was achieved through the parent node hierarchy. For example, for an element such as id=rand_123, positioning was achieved through a method such as / / div[@class='options'] / child::*[2]. At the same time, the PageObject pattern was used to manage page elements. If an element changes, the system can automatically update to ensure the stability and reliability of the test cases. During testing, core operation simulation is crucial for verifying the assessment tool's functionality. Selection operations simulate mouse or touchscreen clicks, recording changes in element styles before and after selection to verify option selection. Input operations support character-by-character input (simulating a real typing speed of 30 characters per minute) and paste input, with real-time monitoring of input box feedback. Page redirection verification monitors page URL changes and AJAX request responses, utilizing Selenium's `wait_for_url()` method to ensure the target question is correctly displayed after redirection. Furthermore, to evaluate the assessment tool's performance, [further details are needed].Performance data is collected, recording operation timestamps, system response time (≤200ms as the normal standard), page loading completion time, element interaction success rate, and other data. A performance index time-series diagram is generated to visually present the performance of the evaluation tool under different operating scenarios, providing a basis for tool optimization.

[0057] After completing the automatic testing of the evaluation tool, the test results need to be analyzed and corresponding reports generated to comprehensively evaluate the tool's quality and provide a basis for subsequent improvements. This mainly involves two key parts: the test result analysis module and the report generation module. The test result analysis module is responsible for comparing the execution results with the expected results, using AI technology to intelligently identify defects and classify and grade them, ultimately outputting a test report containing defect analysis. Regarding the result comparison mechanism, it includes functional comparison, rendering comparison, and content comparison. Functional comparison is used to verify score calculation (such as whether the score of multiple-choice questions conforms to the rules), jump logic (whether the jump to the target question is in accordance with the rules), and format verification (whether the incorrect input is displayed correctly). Rendering comparison uses the OpenCV image recognition library to compare the page screenshot with the benchmark image, and uses structural similarity (SSIM≥0.95 is normal) to identify UI defects such as image misalignment and text overlap. Content comparison is for open-ended question input, and uses the cosine similarity algorithm to compare the stored content with the input content (similarity≥0.98 is complete storage) to verify the integrity of text processing. In the AI ​​defect identification stage, defect features (such as "jump error" and "format validation failure") are classified based on the support vector machine (SVM) model, and the types such as functional defects, UI defects, and performance defects are output. The priority (P0-P3) is calculated from three dimensions: scope of impact (all users / part of users), probability of reproduction (100% / 50%), and cost of repair (low / medium / high), where P0 is the highest priority (such as core process blockage). An example of a defect is shown below: json { "defect_id": "D123", "question_id": "Q1", "type": "functional defect", "description": "After selecting A in Q1, the user was not redirected to Q3 as per the rules, and Q2 was still displayed", "severity": "P1", "reproduce_steps": ["select Q1-A", "observe the next question display"], "screenshot_url": "defect / D123.png", "expected_result": "redirected to Q3", "actual_result": "displayed Q2"} The report generation module integrates test execution data and defect analysis results to generate a multi-dimensional visual report, supporting JSON format.The report can be output in formats such as PDF, Excel, and HTML, and covers core metrics, defect details, and improvement suggestions. The core content includes a test overview, showing the total number of test cases, the number of pass / fail / unexecuted cases and percentages, and a pie chart showing the execution distribution by question type / module; the defect analysis section presents the defect type distribution (e.g., functional defects account for 60%), priority distribution (e.g., P1 accounts for 40%), and lists the details of the top 3 most frequent defects (including screenshots and reproduction steps); performance metrics are displayed as trend charts of average response time (e.g., 150ms), 95% response time (e.g., 300ms), and error rate (e.g., 0.5%); improvement suggestions recommend optimization points based on defect clustering results (e.g., "strengthen multi-condition jump logic testing"). Furthermore, this module supports customized output; users can customize report templates (configure display fields and chart types), and can also push reports on a schedule (via email / WeChat Work robot), while providing defect tracking links (linked to Jira / Trello) for convenient subsequent defect handling and tracking.

[0058] In summary, this method greatly improves the testing efficiency of the evaluation tool, avoids the time-consuming problem of manual testing, and can quickly respond to the high-frequency question updates of the evaluation tool. Even when facing an evaluation tool with a large number of questions, it does not require several hours or days to conduct full testing. At the same time, it achieves more comprehensive test coverage, overcomes the shortcomings of script testing in covering complex user interaction paths, and can cover edge scenarios such as multi-condition logical jumps and dynamically displayed questions, reducing the omission of edge scenarios. This method also reduces maintenance costs. Since test cases can be automatically generated and adjusted according to the original question data, when the question rules change, there is no need to readjust a large number of test scripts as in traditional script testing, effectively reducing maintenance costs. This method also has intelligent analysis capabilities, which can automatically identify problems such as question configuration errors and abnormal interface rendering, changing the previous situation of relying on manual visual inspection, improving the accuracy and reliability of testing, and ensuring that the evaluation tool can maintain functional stability even after frequent question updates.

[0059] Referring to FIG4, this disclosure provides an automatic testing system for an assessment tool, comprising: an extraction module 101, used to acquire the original question data of the assessment tool, and extract key data and overall logical data for each question type from the original question data; an estimation module 102, used to estimate all answer situations for each question type based on the key data for each question type; a generation module 103, used to generate corresponding individual test cases for each answer situation for each question type; a combination module 104, used to arrange and combine the individual test cases for each question type based on the overall logical data to generate multiple overall test cases; and a testing module 105, used to test the assessment tool using each overall test case and output a test report.

[0060] The various variations and specific examples of the automatic testing methods for the evaluation tools provided above are also applicable to the automatic testing system for the evaluation tools provided in this disclosure. Through the foregoing detailed description of the automatic testing methods for the evaluation tools, those skilled in the art can clearly understand the implementation method of the automatic testing system for the evaluation tools. For the sake of brevity, it will not be described in detail here.

[0061] A computer device according to an embodiment of this disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0062] The processor may be a central processing unit (CPU) or other forms of processing unit with data processing capability and / or instruction execution capability, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute the computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the automatic testing method of the evaluation tool of the foregoing embodiments of this disclosure.

[0063] Those skilled in the art should understand that, in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0064] FIG5 is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. It shows a schematic diagram of the structure of a computer device suitable for implementing the embodiments of this disclosure. The computer device shown in FIG5 is merely an example and should not impose any limitations on the function and scope of use of the embodiments of this disclosure.

[0065] As shown in FIG5, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). Various programs and data required for the operation of the computer device are also stored in the RAM. The processor, ROM, and RAM are interconnected through a bus. Input / output (I / O) interfaces are also connected to the bus.

[0066] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard disks; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to describe the specifications.Page 14 / 16 17 CN 121166543 A Data exchange. Although FIG5 shows a computer device with various means, it should be understood that it is not required to implement or have all the means shown. More or fewer means may be implemented or have instead.

[0067] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the automated testing method of the evaluation tool of embodiments of the present disclosure are performed.

[0068] Detailed description of this embodiment can be found in the corresponding descriptions of the foregoing embodiments, and will not be repeated here.

[0069] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions thereon. When the non-transitory computer-readable instruction is executed by the processor, all or part of the steps of the automatic testing method of the evaluation tool of the various embodiments of the present disclosure described above are performed.

[0070] The computer-readable storage media described above include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard disk), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0071] For detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0072] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are merely examples and not limitations, and should not be considered as essential features of the various embodiments of the present disclosure. In addition, the specific details of the above disclosure are only for illustrative purposes and for ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details for implementation.

[0073] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that connections, arrangements, or configurations must be made in the manner shown in the block diagrams. As those skilled in the art...As will be recognized by the practitioner, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context explicitly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0074] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0075] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalents of this disclosure.

[0076] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope. Specification 15 / 16 pages 18 CN 121166543 A

[0077] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0078] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereto. Specification 16 / 16 pages 19 CN 121166543 A Figure 1 Figure 2 Specification Drawings 1 / 3 pages 20 CN 121166543 A Figure 3 Specification Drawings 2 / 3 pages 21 CN121166543 A Figure 4 Figure 5说明书附图3 / 3页22 CN 121166543 A Abstract Abdominal ultrasound examination method, system and device Abstract The embodiments of the present disclosure disclose an automatic testing method, device, storage medium and program product for an evaluation tool. The method includes: obtaining the original question data of an evaluation tool, extracting the key data and overall logic data of each question type from the original question data; estimating all the answer situations of each question type based on the key data thereof; generating corresponding individual test cases for each answer situation of each question type; generating multiple overall test cases by permuting and combining the individual test cases of each question type based on the overall logic data; and testing the evaluation tool by using each overall test case and outputting a test report. This method can generate test cases by obtaining and estimating all the answer situations of each question type,which can efficiently and comprehensively test the evaluation tool. It not only solves the problems of low efficiency and slow response of manual testing, but also overcomes the defects of incomplete coverage and high maintenance cost of script testing.

Claims

1. An automated test method for an assessment tool, characterized by, The method comprises: obtaining original question data of the evaluation tool, extracting key data and overall logical data of each type of question from the original question data; estimating all answer conditions of each type of question based on the key data of each type of question; generating a corresponding single test case for each answer condition of each type of question; based on the overall logical data, arranging and combining the single test cases of each type of question to generate a plurality of overall test cases; using each overall test case to test the evaluation tool respectively, and outputting a test report.

2. The automatic test method of the evaluation tool according to claim 1, wherein when the type of question is a selection question, a jump relationship graph is generated based on the key data of the selection question; wherein the nodes of the jump relationship graph represent the selection questions, the attributes of the nodes include question ID, options and mapping values of each option, the edges between the nodes are directed edges, the directed edges represent the execution order between the selection questions, and the attributes of the directed edges include jump conditions and path hash values between the selection questions; based on the attributes of the nodes and the attributes of the directed edges, performing full-path traversal on the jump relationship graph; based on all the paths traversed, generating all the answer conditions of the selection question type. The full-path traversal on the jump relationship graph based on the attributes of the nodes and the attributes of the directed edges comprises:

3. The method of claim 2, wherein, starting from a starting node, traversing the jump relationship graph; when traversing each node, determining whether there is a jump condition from the current node to the next node; when there is a jump condition from the current node to the next node, traversing all the possible selected options of the current node in turn; when traversing each possible selected option, determining whether the jump condition from the current node to the next node is met based on the current possible selected option; when the jump condition is met, comparing the path hash value from the current node to the next node with the recorded path hash value; when the comparison result is different, transferring to the next node for continuous traversal, and recording the path hash value from the current node to the next node; after the traversal of the jump relationship graph is completed, counting all the traversed paths.

4. The automatic test method of the evaluation tool according to claim 1, wherein when the type of question is a judgment question, all the judgment questions are traversed; when traversing each judgment question, generating correct answer content conforming to the answer and incorrect answer content opposite to the answer based on the answer of the current judgment question; determining whether the content of the current judgment question contains semantic negative words or semantic ambiguous words; when the content of the question contains semantic negative words, generating answer content containing reverse expressions; when the content of the question contains semantic ambiguous words, generating answer content containing extreme scenarios; when the content of the question contains both semantic negative words and semantic ambiguous words, generating answer content containing both semantic negative words and semantic ambiguous words; after the traversal of all the judgment questions is completed, arranging and combining all the answer content of each judgment question to generate all the answer conditions of the judgment question type.

5. The automatic test method of the evaluation tool according to claim 1, wherein when the type of question is a fill-in-the-blank question, all the fill-in-the-blank questions are traversed; ​ ​ ​ When each fill-in-the-blank question is traversed, based on the answer of the current fill-in-the-blank question and the format requirement of the answer, the correct format answer content, the incorrect format answer content, the correct answer content and the incorrect answer content are generated; After all the fill-in-the-blank questions are traversed, all the answer contents of each fill-in-the-blank question are arranged and combined to generate all the answer conditions of the fill-in-the-blank question type.

6. The automatic test method of the evaluation tool according to claim 1, characterized in that, when the question type is an open question, all the open questions are traversed; When each open question is traversed, based on the answer of the current open question and the character length requirement of the answer, answer contents of different character lengths are generated; Based on the answer of the current open question and the input form requirement of the answer, answer contents of diversified input forms are generated, wherein the diversified input forms include at least one of pure text, preset special characters, HTML tags and multi-language mixing; By semantic analysis of the answer of the current open question, answer contents with different semantic correlation degrees are generated; After all the open questions are traversed, all the answer contents of each open question are arranged and combined to generate all the answer conditions of the open question type.

7. The method of claim 1, wherein, The generating of the plurality of overall test cases by arranging and combining the single test cases of each question type based on the overall logical data comprises: Based on the overall logical data, the order between all the question types is obtained; According to the order, one single test case is selected from each question type in turn to combine to generate all possible overall test cases; The number of overall test cases is equal to the product of the number of single test cases of each question type.

8. A computer apparatus, comprising: The computer device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the automatic test method of the evaluation tool according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the automatic test method of the evaluation tool according to any one of claims 1-7.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions, when executed by a processor, implement the steps of the automatic test method of the evaluation tool according to any one of claims 1-7.

Images