An execution result checking method, device and storage medium

Abstract

The application discloses an execution result checking method and device and a storage medium, relates to the technical field of software testing, and comprises the following steps: in response to a checking request, generating a to-be-checked case set based on a preset selection strategy according to a project identifier and a checking rule, wherein the to-be-checked case set comprises each test case and an expected result of each test case; obtaining an execution screenshot corresponding to each test case, extracting text information in the execution screenshot as an actual execution result; respectively performing cleaning processing on the actual execution result and the expected result to obtain a cleaned actual execution result and a cleaned expected result; comparing the cleaned actual execution result and the cleaned expected result based on a preset scoring rule, generating an evaluation score, and determining a judgment result of each test case according to the evaluation score. The application automatically extracts the text information in the execution screenshot and compares the text information with the expected result, thereby improving the checking efficiency and guaranteeing the objectivity and consistency of the judgment standard.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and in particular to a method, apparatus and storage medium for verifying execution results. Background Technology

[0002] In the field of software testing, verifying the execution results of test cases is a crucial step in ensuring software quality. Test cases typically contain pre-defined expected results. After testers execute the test cases and obtain screenshots, they need to compare the actual results in the screenshots with the expected results to determine whether the test case has passed.

[0003] Currently, the verification of test case execution results mainly relies on manual operation. The test manager subjectively decides whether to check the execution results and the proportion of checks based on factors such as project staffing, product characteristics, and the number of test cases. During manual verification, testers need to review each screenshot of the test case execution and manually compare it with the pre-set expected results of the case. When the number of test cases is large, manual comparison is inefficient; at the same time, different testers may have different judgment standards for the same case, leading to subjective factors affecting the verification results, resulting in misjudgments and insufficient accuracy.

[0004] Therefore, how to improve the efficiency of verification while ensuring the objectivity and consistency of judgment criteria in the verification of test case execution results is a technical problem that urgently needs to be solved.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a method, apparatus and storage medium for verifying execution results, aiming to solve the technical problem of how to improve verification efficiency while ensuring the objectivity and consistency of judgment criteria in the verification of test case execution results.

[0007] To achieve the above objectives, this application proposes a method for verifying execution results, the method comprising: In response to a received verification request, a set of test cases to be verified is generated based on a preset selection strategy according to the project identifier and verification rules carried in the verification request. The set of test cases to be verified includes each test case and the expected result of each test case. Obtain the execution screenshots corresponding to each test case in the set of test cases to be verified, extract the text information from the execution screenshots, and use the text information as the actual execution result of each test case; The actual execution results and expected results of each test case are cleaned to obtain the cleaned actual execution results and cleaned expected results of each test case. Based on preset scoring rules, the actual execution results after cleaning and the expected results after cleaning are compared to generate an evaluation score, and the judgment result of each test case is determined based on the evaluation score.

[0008] In one embodiment, the step of extracting text information from the execution screenshot and using the text information as the actual execution result of each test case includes: Input the execution screenshot into the preset artificial intelligence model; The AI ​​model is used to analyze visual elements and extract text information from the execution screenshot to obtain the text information in the execution screenshot. The text information is used as the actual execution result of each test case.

[0009] In one embodiment, the step of using the text information as the actual execution result of each test case includes: The text information is classified according to checkpoint type, which includes at least one of data checkpoint, user interface checkpoint, and function checkpoint; The text information is organized into structured text according to the classified checkpoint types; The structured text is used as the actual execution result for each test case.

[0010] In one embodiment, the step of cleaning the actual execution results and expected results of each test case to obtain the cleaned actual execution results and cleaned expected results of each test case includes: For any of the test cases, the actual execution results and the expected results of the test cases are standardized in format to remove redundant symbols and spaces. The actual execution results and expected results after format unification are semantically standardized according to the checkpoint type to map the text of the actual execution results and expected results after format unification into a unified semantic representation, thus obtaining the actual execution results and expected results after test case cleaning.

[0011] In one embodiment, the step of comparing the actual execution result after cleaning with the expected result after cleaning based on a preset scoring rule to generate an evaluation score, and determining the judgment result of each test case based on the evaluation score, includes: Obtain the preset weights corresponding to each of the aforementioned checkpoint types; For each checkpoint type, calculate the semantic difference rate between the actual execution result after cleaning and the expected result after cleaning for each test case; Based on the semantic difference rate and the scoring criteria corresponding to the checkpoint type, determine the checkpoint score for each checkpoint type in each test case; An evaluation score is generated based on the checkpoint scores of each checkpoint type for each test case and the preset weights, and the judgment result of each test case is determined based on the evaluation scores.

[0012] In one embodiment, the step of determining the checkpoint score for each checkpoint type of each test case based on the semantic difference rate and the scoring criteria corresponding to the checkpoint type includes: For each checkpoint type, determine whether the semantic difference rate is less than or equal to a preset difference rate threshold corresponding to the checkpoint type; When the semantic difference rate is less than or equal to the preset difference rate threshold, the checkpoint score of the checkpoint type is determined to be the preset full score. When the semantic difference rate is greater than the preset difference rate threshold, the deduction amount is determined by the product of the semantic difference rate and the preset decreasing coefficient corresponding to the checkpoint type, and the checkpoint score of the checkpoint type is obtained by subtracting the deduction amount from the preset full score. Among them, the preset difference rate thresholds corresponding to data checkpoints, user interface checkpoints, and function checkpoints increase sequentially, while the preset decreasing coefficients corresponding to data checkpoints, user interface checkpoints, and function checkpoints decrease sequentially.

[0013] In one embodiment, the verification rules include minimum dimension verification rules, and the step of generating a set of cases to be verified based on a preset selection strategy includes: When the verification rule carried in the verification request is the minimum dimension verification rule, candidate test cases are determined according to the project identifier; Test cases that meet the preset important functional conditions are selected from the candidate test cases to generate the test case set to be checked; The preset key functional conditions include: the test case belongs to at least one of the core scenarios, core functions, or core business processes.

[0014] In one embodiment, after the step of determining the judgment result of each test case based on the evaluation score, the method further includes: A verification report is generated based on the judgment results of each test case and the evaluation scores; The verification report is pushed to the party that initiated the verification request.

[0015] Furthermore, to achieve the above objectives, this application also proposes an execution result verification device, which includes: The case generation module is used to respond to the received verification request and generate a set of cases to be verified based on the project identifier and verification rules carried in the verification request and a preset selection strategy. The set of cases to be verified includes each test case and the expected result of each test case. The case execution module is used to obtain the execution screenshots corresponding to each test case in the case set to be verified, extract the text information in the execution screenshots, and use the text information as the actual execution result of each test case; The data cleaning module is used to clean the actual execution results and the expected results of each test case, respectively, to obtain the cleaned actual execution results and the cleaned expected results of each test case. The result determination module is used to compare the actual execution result after cleaning with the expected result after cleaning based on preset scoring rules, generate an evaluation score, and determine the determination result of each test case based on the evaluation score.

[0016] In addition, to achieve the above objectives, this application also proposes an execution result verification device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the execution result verification method as described above.

[0017] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the execution result verification method as described above.

[0018] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the execution result verification method described above.

[0019] This application provides a method for verifying execution results. First, in response to a verification request, a set of test cases to be verified is generated based on the project identifier and verification rules. Then, the execution screenshots corresponding to each test case are obtained, and the text information is extracted from them as the actual execution results. The actual execution results and expected results are cleaned separately. Finally, the cleaned results are compared based on preset scoring rules to generate an evaluation score and determine the judgment result of each test case.

[0020] Therefore, this application automatically extracts text information from the execution screenshots as the actual execution result and compares the cleaned actual execution result with the cleaned expected result, replacing the manual operation of checking each screenshot and performing manual comparison, thus improving the verification efficiency. At the same time, by cleaning the actual execution result and the expected result separately, the impact of format differences and semantic differences in the original data on subsequent comparisons is eliminated. Based on the preset scoring rules, a unified quantitative comparison is performed, ensuring that the verification of all test cases follows the same judgment standard. This eliminates the problem of inconsistent judgment standards caused by the subjective differences of testers in manual verification, thereby improving the verification efficiency while ensuring the objectivity and consistency of the verification quality. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating the method for verifying the execution results of this application is provided in Embodiment 1. Figure 2 This is a schematic diagram showing the process details of Embodiment 1 of the method for verifying the execution results of this application; Figure 3 This is a partial flowchart illustrating the second embodiment of the method for verifying the execution results of this application. Figure 4 This is a schematic diagram of the module structure of the execution result verification device in an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the execution result verification method in this embodiment of the application.

[0024] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0026] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0027] In the field of software testing, test cases typically contain pre-defined expected results. After testers execute the test cases, they obtain screenshots of the execution. The actual results in these screenshots need to be compared with the expected results to determine whether the test case passed. Currently, the verification of test case execution results mainly relies on manual operation. The test manager subjectively determines the inspection ratio, and testers compare each screenshot with the expected results. This process suffers from low verification efficiency, subjective judgment criteria leading to misjudgments, and insufficient accuracy.

[0028] Based on this, the embodiments of this application provide a method for verifying execution results, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for verifying the execution results of this application.

[0029] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a test management platform server, a control host of a test execution system, or a cloud test service platform. The following description uses a test management platform server as an example to illustrate this embodiment and the subsequent embodiments.

[0030] Example 1 In this embodiment, the execution result verification method includes steps S10 to S40: Step S10: In response to the received verification request, generate a set of test cases to be verified based on the project identifier and verification rules carried in the verification request and a preset selection strategy. The set of test cases to be verified includes each test case and the expected result of each test case.

[0031] It should be noted that the verification request is triggered by the tester through the user interface of the test management platform to initiate the verification of execution results. The tester selects the item to be verified, specifies the verification rules, and submits the request, which is then received by the system. The verification request carries a project identifier and verification rules. The project identifier is used to determine the scope of the test items targeted by this verification; for example, the project identifier can be the project name, project number, or a unique identifier for the project in the system. The verification rules are used to determine the scope and method of verification.

[0032] It should be noted that verification rules can include full verification rules and minimum-dimensional verification rules. Full verification rules mean verifying all test cases within the project; minimum-dimensional verification rules mean verifying only the test case set of essential functions within the project. Testers can flexibly choose the verification rules based on the project's requirements.

[0033] It should be noted that the preset selection strategy refers to the pre-configured strategy logic used to dynamically generate a set of test cases to be verified based on the verification rules. When the verification rule is a full verification rule, the selection strategy includes all test cases under the project in the set of test cases to be verified; when the verification rule is a minimum-dimensional verification rule, the selection strategy filters test cases according to preset important functional conditions. There are various ways to implement the selection strategy. For example, it can be based on the tags of test cases, or it can be based on the position or priority of test cases in the business process. This embodiment does not specifically limit the specific implementation method of the selection strategy.

[0034] It should be noted that the test case set to be verified is a data set generated after the strategy is executed, containing each test case that needs to be verified and its corresponding expected results. The expected results are included in the definition phase of each test case; these expected results are inherent attributes of the test case, describing the correct result that should be presented after the case is executed. Therefore, when generating the test case set to be verified, each test case and its corresponding expected result are determined together, without needing to be obtained separately. For example, the test case set to be verified can be a data table structure, where each record contains information such as the test case number, test case name, test step description, expected result text, and the functional module to which the case belongs.

[0035] Step S20: Obtain the execution screenshots corresponding to each test case in the test case set to be verified, and extract the text information from the execution screenshots, using the text information as the actual execution result of each test case.

[0036] It should be noted that execution screenshots refer to screen images that reflect the software interface state after testers execute test cases, captured automatically by the system or manually by the testers. Execution screenshots are typically stored in the execution records of the test management platform in image formats such as PNG (Portable Network Graphics) and JPEG (Joint Photographic Experts Group). In one feasible implementation, the system can query the test case execution record database, obtain the corresponding execution screenshot storage path based on the test case number, and then read the execution screenshot file.

[0037] It should be noted that extracting text information from the execution screenshot refers to using text recognition technology to identify and extract the text content contained within the image data of the execution screenshot. This text information describes the actual content displayed on the software interface after the test case is executed, and serves as the basis for subsequent comparison with the expected result; therefore, it is used as the actual execution result. Since the execution screenshot is in image format, while the expected result is usually in text format, extracting the text information from the execution screenshot as the actual execution result in text format allows subsequent cleaning and comparison operations to be performed on a unified text data basis.

[0038] It should be noted that, in one feasible implementation, obtaining the execution screenshots corresponding to each test case can be achieved using an asynchronous multithreaded approach. Specifically, after generating the test case set to be verified, the system creates one or more asynchronous tasks using an asynchronous multithreaded approach. These asynchronous tasks are responsible for querying and retrieving the execution screenshots corresponding to each test case from the execution record database and storing the retrieved screenshots in a preset storage location. This asynchronous task executes in the background, allowing testers to immediately execute other test tasks after submitting a verification request without waiting for the screenshots to be retrieved. This decouples the verification process from the test execution process, improving the efficiency of testers.

[0039] Step S30: Clean the actual execution results and expected results of each test case to obtain the cleaned actual execution results and cleaned expected results of each test case.

[0040] It's important to note that text cleaning refers to the process of formatting and semantic standardization of text data. The actual execution result comes from the text recognition in the execution screenshot, which may contain noise characters and format fragments caused by recognition errors; the expected result comes from the test case definition, which may contain various special characters, redundant formats, and other non-semantic noise that could affect the accuracy of subsequent comparisons. Although the data sources and noise types differ, both require text cleaning to improve the accuracy of subsequent comparisons. "Performing separate cleaning processes" means that the actual execution result and the expected result are cleaned independently. The cleaning process framework is the same for both, but the focus of processing can be emphasized according to the characteristics of their respective data.

[0041] It should be noted that the actual and expected results after cleaning have been standardized in terms of format and semantic expression, eliminating the risk of misjudgment caused by differences in format or expression, so that subsequent scoring comparisons can be carried out on a unified and clean data basis.

[0042] Step S40: Based on the preset scoring rules, compare the actual execution results after cleaning with the expected results after cleaning, generate an evaluation score, and determine the judgment result of each test case based on the evaluation score.

[0043] It should be noted that the preset scoring rules refer to predefined rules used to quantify the consistency between the actual execution results after cleaning and the expected results after cleaning. These scoring rules may include the method of dividing checkpoints, the weight configuration of each checkpoint, the scoring criteria for each checkpoint, and the mapping relationship from evaluation scores to judgment results. Through these preset scoring rules, the verification of all test cases follows a unified judgment standard, eliminating the inconsistency in judgment standards caused by subjective differences among testers during manual verification.

[0044] It should be noted that the evaluation score is a quantitative representation of the degree to which the actual execution result matches the expected result. For example, the evaluation score can be on a percentage scale, with a higher score indicating a greater consistency between the actual and expected results. The judgment result is a qualitative conclusion regarding the execution of the test case, determined based on the evaluation score. It indicates whether the test case has passed, transforming the test case verification conclusion, which previously relied on subjective human judgment, into an automatic, objective, and quantifiable output from the system.

[0045] It should be noted that in practical applications, the judgment results can be further used to drive subsequent test management processes. For example, test cases judged as passed can be automatically archived, while test cases judged as failed can automatically create defect tickets and notify relevant developers. The judgment results can also serve as the basis for generating verification reports, which are automatically pushed to the party that initiated the verification request, enabling the initiator to obtain the verification results in a timely manner and achieving a complete closed loop in the verification process.

[0046] In this embodiment, through steps S10 to S40, the application first responds to the verification request by generating a set of test cases to be verified, containing each test case and its expected results, based on the project identifier and verification rules. Then, it obtains the execution screenshots corresponding to each test case and extracts text information from them as the actual execution results. After cleaning the actual execution results and expected results respectively, they are compared based on preset scoring rules to generate an evaluation score and determine the judgment result for each test case. By automatically extracting text information from the execution screenshots and comparing them using unified scoring rules, the manual operation of reviewing each screenshot and performing manual comparison is replaced, improving verification efficiency while ensuring the objectivity and consistency of verification quality.

[0047] Reference Figure 2 , Figure 2 This is a schematic diagram of the overall process of one embodiment of the method for verifying the execution results of this application.

[0048] like Figure 2As shown, the overall flow of the execution result verification method in this embodiment is a top-down vertical flow. The process begins with "Start," first entering the "Select Sampling Test Task" step, which corresponds to the process where the tester selects the project, specifies the verification rules, and submits the verification request in the system interface.

[0049] After selecting the sampling test task, the process enters the diamond-shaped decision node "Verification Rules". This node performs branching decisions based on the verification rules selected by the tester, and includes two branch paths: The first branch is the "All Sampling Inspection" branch, corresponding to the "Full Verification" rule in the verification rules. When the tester selects "All Sampling Inspection," the process proceeds downwards along the main line, directly entering the "Save Case Information" step. Under this branch, the case set to be verified includes all test cases under the project.

[0050] The second branch is the associated case sampling branch, corresponding to the minimum dimension verification rule in the verification rules. When testers select associated case sampling, the process enters the "AI Identification of Important Functional Cases" step from the horizontal branch on the right side of the diamond. In this step, the system, based on a preset selection strategy, filters out test cases that meet the preset important functional conditions from the candidate test cases—that is, important functional test cases belonging to core scenarios, core functions, or core business processes. After the AI ​​identification of important functional cases step is completed, it merges into the main line "Save Case Information" node via a horizontal return arrow, and the set of selected important functional test cases becomes the set of cases to be verified. The two branches converge at the "Save Case Information" node, which saves the generated set of cases to be verified and its corresponding expected results.

[0051] After saving the case information, the process enters the "sampling and checking each case in a loop" step. This step means that the system performs subsequent verification processing on each test case in the case set to be verified, including: obtaining the execution screenshots corresponding to each test case; performing large-scale OCR recognition on the execution screenshots and extracting summary information (i.e., extracting the text information in the execution screenshots as the actual execution result); after extracting information from the execution screenshots, data cleaning can be performed on the extracted information; intelligent cleaning of the expected results can be performed (i.e., cleaning processing is performed on the actual execution result and the expected result separately); and the actual result and the expected result can be compared using a large-scale model (i.e., comparison is performed based on preset scoring rules and evaluation scores and judgment results are generated). After verification is completed, the "report push" step is entered, and the verification report is pushed to the initiator. Finally, the process reaches "end".

[0052] It should be noted that, Figure 2The steps of "performing screenshot large-scale OCR recognition, extracting and summarizing information" and "intelligent cleaning of expected results" are arranged sequentially in the flowchart. In actual execution, the cleaning of actual results and the cleaning of expected results can be performed independently. While the cleaning process framework is the same, the focus differs depending on the characteristics of their respective data. After cleaning, they proceed to the comparison step. Furthermore, Figure 2 The "large model comparison of actual results and expected results" step covers the complete process of comparison based on preset scoring rules, generation of evaluation scores, and determination of judgment results based on evaluation scores. The specific processing details involved, such as checkpoint classification, weight configuration, difference rate calculation, difference scoring, and threshold determination, have been described in detail in the text description of the above embodiments and will not be repeated here.

[0053] Example 2 Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description, and will not be repeated hereafter.

[0054] In one feasible implementation, step S20 involves extracting text information from the execution screenshot and using the text information as the actual execution result for each test case. This can be achieved through the following steps: Step S201: Input the screenshot into the preset artificial intelligence model.

[0055] It should be noted that the preset artificial intelligence model refers to a pre-trained AI (Artificial Intelligence) model with image understanding and text recognition capabilities. Unlike traditional OCR (Optical Character Recognition) engines, which can only recognize characters, the artificial intelligence model can perform deeper visual element analysis on the screenshot. It can not only recognize the text content in the screenshot, but also understand the layout of the text in the interface and the type of interface element it belongs to (such as button text, input box content, pop-up prompts, table data, etc.), thereby extracting more semantically meaningful text information.

[0056] It should be noted that the training method for the artificial intelligence model can be flexibly selected according to actual needs. For example, a large-scale multimodal model (such as a vision-language model) can be used, with screenshots as input and manually labeled text information as tags, for fine-tuning training, enabling the model to learn the ability to accurately extract structured text information from software interface screenshots. Alternatively, a general-purpose large model can be used to directly perform zero-shot or few-shot inference, utilizing the model's general image understanding capabilities to complete text extraction. This embodiment does not specifically limit the specific type and training method of the artificial intelligence model.

[0057] Step S202: Visual element analysis and text information extraction are performed on the execution screenshot using an artificial intelligence model to obtain the text information in the execution screenshot.

[0058] It should be noted that visual element analysis refers to the process by which the artificial intelligence model analyzes and identifies various visual elements in the screenshot, including but not limited to: identifying text and non-text areas in the screenshot, distinguishing different interface controls (such as buttons, text boxes, drop-down lists, data tables, etc.), and extracting text information from each control. Text information extraction, based on visual element analysis, organizes the identified text content according to its logical relationship within the interface to form readable text information. For example, the artificial intelligence model can output recognition results that include the type of interface element and the text content, such as recognizing "Account Balance: 1000.00 yuan" as coming from a cell in a data table, "Transaction Successful" as coming from a pop-up notification text, and "Confirm" as coming from a button label, providing richer semantic context for subsequent structured processing and comparison.

[0059] Step S203: Use the text information as the actual execution result for each test case.

[0060] In this embodiment, by introducing an artificial intelligence model to analyze visual elements and extract text information from the execution screenshot, the text information extracted from the execution screenshot not only includes the text content itself, but also the semantic context information of the text in the interface. This provides higher quality input data for subsequent structured processing and accurate comparison, thereby improving the accuracy of the verification results.

[0061] In one feasible implementation, based on the text information extracted through the artificial intelligence model, the step S20, which uses the text information as the actual execution result of each test case, may further include: Step S204: Classify the text information according to the checkpoint type, which includes at least one of data checkpoints, user interface checkpoints, and function checkpoints.

[0062] It's important to note that checkpoint types refer to the categories of content that needs to be verified in test cases, categorized according to their business attributes. Data checkpoints refer to checks related to business data, such as account balances, transaction amounts, and order status codes. These checkpoints are directly related to core business logic, and data errors may lead to business interruptions. User interface checkpoints refer to checks related to interface display, such as button text, page titles, and prompts. These checkpoints affect user experience but are not core business logic. Functional checkpoints refer to checks related to operational flow verification, such as button click responses and page navigation logic. These checkpoints verify whether the function executes as designed.

[0063] It's important to note that the deeper significance of categorizing text information by checkpoint type lies in the fact that different checkpoint types have varying degrees of importance in the business, requiring differentiated scoring standards for different types during subsequent comparisons. By categorizing the text information immediately after extraction, the data's structure is aligned with the scoring rules from the outset, providing a data-level foundation for subsequent differentiated scoring.

[0064] Step S205: Organize the text information into structured text according to the classified checkpoint types.

[0065] It should be noted that structured text refers to text information organized according to a predefined data structure. Organizing categorized text information into structured text means organizing text content belonging to the same checkpoint type according to the format corresponding to that type. For example, the structured text corresponding to data checkpoints can adopt a key-value pair format, such as "Account Balance: 1000.00; Transaction Amount: 500.00; Order Status: Paid"; the structured text corresponding to user interface checkpoints can adopt a format such as "Page Title: Transaction Successful; Button Text: Confirm"; and the structured text corresponding to function checkpoints can adopt a format such as "Operation Steps: Click the Confirm Button → Page Jumps to Transaction Record Page". Through structured organization, the check content of different checkpoint types is explicitly distinguished, which facilitates subsequent processing according to checkpoint type during cleaning and scoring comparison.

[0066] Step S206: Use the structured text as the actual execution result for each test case.

[0067] In this embodiment, by classifying and organizing text information into structured text according to checkpoint type, the actual execution result has a clear hierarchical structure. The inspection content of different checkpoint types is explicitly distinguished, providing a structural basis for subsequent cleaning and differential scoring according to checkpoint type. This allows subsequent comparison operations to be precise to each checkpoint, rather than making a general text similarity judgment on the entire case.

[0068] In one feasible implementation, step S30 involves cleaning the actual execution results and expected results of each test case to obtain the cleaned actual execution results and cleaned expected results of each test case. This can be achieved through the following steps: Step S301: For any test case, the actual execution result and the expected result of the test case are formatted to remove redundant symbols and spaces.

[0069] It should be noted that format standardization refers to preprocessing operations such as denoising and format normalization of text, aiming to eliminate the interference of non-semantic noise in the text on subsequent comparisons. For the actual execution result, format standardization mainly targets format fragments that may be introduced during OCR recognition, including redundant spaces, line break fragments, and special characters generated by recognition errors. For the expected result, format standardization mainly targets special characters and redundant formats that may be included in the case definition, such as redundant punctuation marks, inconsistent number formats (e.g., "1,000" vs. "1000"), and template tags introduced during case writing. Format standardization ensures consistency in the format of the two types of data, creating conditions for subsequent semantic standardization. For example, after format standardization, both "Account Balance: 1,000.00 yuan" in the actual execution result and "Account Balance: 1000.00 yuan" in the expected result are standardized into a unified format.

[0070] Step S302: After the format unification process, the actual execution result and the expected result are semantically standardized according to the checkpoint type, so as to map the text of the actual execution result and the expected result after the format unification process into a unified semantic representation, and obtain the actual execution result and the expected result after the test case cleaning.

[0071] It's important to note that semantic standardization refers to the process of mapping text to a unified semantic representation based on checkpoint type, building upon format standardization. The aim is to eliminate misjudgments caused by different expression methods. Different testers may use different expressions for the same check content when writing expected results; similarly, the text expression in OCR recognition output may vary depending on the recognition context. Semantic standardization maps text with different expressions but the same meaning to a unified semantic representation, allowing the comparison operation to be performed at the semantic level, rather than a simple string matching.

[0072] It should be noted that the semantic standardization methods for different checkpoint types can have different emphases. For example, for data checkpoints, "Account balance: 1000.00 yuan" and "Balance: 1000.00" can both be mapped to the unified semantic representation "balance=1000.00"; for user interface checkpoints, "Login successful" and "Login successful!" can both be mapped to the unified semantic representation "login_result=success"; for function checkpoints, "Click the confirmation button, the page jumps to the transaction record page" and "Confirm → Transaction record page" can both be mapped to the unified semantic representation "action=confirm, target=transaction_record". By mapping different expressions to a unified semantic representation, the comparison operation can determine the consistency of content at the semantic level, rather than relying solely on a complete match at the character level.

[0073] In this embodiment, non-semantic noise in the two types of data is removed by format unification processing, and the possibility of misjudgment caused by differences in expression is eliminated by semantic standardization processing. This enables the subsequent scoring comparison to be carried out at a clean and unified semantic level, avoiding misjudgment caused by differences in format and expression, thereby improving the accuracy of the comparison.

[0074] In one feasible implementation, refer to Figure 3 , Figure 3 This is a partial flowchart of the second embodiment of the method for verifying the execution results of this application.

[0075] Based on classifying and organizing the text information into structured text according to checkpoint type, step S40 compares the actual execution results and the expected results after cleaning according to preset scoring rules, generates an evaluation score, and determines the judgment result of each test case based on the evaluation score. This can be achieved through the following steps: Step S401: Obtain the preset weights corresponding to each checkpoint type.

[0076] It should be noted that preset weights refer to pre-configured coefficients reflecting the importance of each checkpoint type in the overall assessment. Weight configuration is dynamically defined by testers based on the project's business characteristics. Different checkpoint types can be assigned different weights, typically decimals between 0 and 1, with the sum of the weights for all checkpoint types being 1. For example, in a financial project, data checkpoints, involving core transaction data, can be configured with a weight of 0.6; user interface checkpoints, affecting user experience but not core business logic, can be configured with a weight of 0.3; and functional checkpoints, verifying operational processes, have a relatively low error rate and can be configured with a weight of 0.1. Through weight configuration, deviations in key business risks receive higher attention in the overall assessment, prioritizing the exposure of core issues. It should be noted that the weights corresponding to data checkpoints, user interface checkpoints, and functional checkpoints decrease sequentially, reflecting the decreasing importance of the three checkpoint types in the business.

[0077] Step S402: For each checkpoint type, calculate the semantic difference rate between the actual execution result after cleaning and the expected result after cleaning for each test case.

[0078] It should be noted that the semantic difference rate is a quantitative indicator of the degree of semantic difference between the actual execution result after cleaning and the expected result after cleaning. Based on the semantic standardization process mapping the two types of results to a unified semantic representation, the semantic difference rate can be determined by calculating the matching degree between the two sets of semantic representations. For example, text similarity algorithms can be used to calculate the difference rate, such as calculating the proportion of differences at the string level based on edit distance, or calculating the degree of semantic difference based on semantic vector distance. The semantic difference rate typically ranges from 0% to 100%, where a difference rate of 0% indicates that the two are completely identical, and a higher difference rate indicates a greater difference.

[0079] Step S403: Determine the checkpoint score for each checkpoint type in each test case based on the semantic difference rate and the scoring criteria corresponding to the checkpoint type.

[0080] It should be noted that the scoring criteria refer to the quantitative rules for determining checkpoint scores based on semantic discrepancy rates. Each checkpoint type can correspond to different scoring criteria and different tolerances for discrepancy rates. Since data checkpoints are directly related to core business logic, their scoring criteria can be stricter—points are deducted even for small semantic discrepancy rates; the scoring criteria for user interface checkpoints and functional checkpoints can be relatively lenient—tolerating larger semantic discrepancy rates. This differentiated scoring criterion setting reflects the different levels of importance of different checkpoint types in the business.

[0081] Step S404: Generate an evaluation score based on the checkpoint scores and preset weights for each checkpoint type of each test case, and determine the judgment result for each test case based on the evaluation score.

[0082] It should be noted that the evaluation score can be calculated by weighting the checkpoint scores for each checkpoint type with their corresponding preset weights. The formula for the weighted sum is:

[0083] Where F represents the assessment score, This represents the checkpoint score for the i-th checkpoint type. This represents the preset weight corresponding to the i-th checkpoint type.

[0084] The weighted summation method ensures that the scores of high-weighted checkpoints have a greater impact on the final evaluation score, while the scores of low-weighted checkpoints have a smaller impact. For example, if the data checkpoint score is 95 with a weight of 0.6, the user interface checkpoint score is 85 with a weight of 0.3, and the functionality checkpoint score is 90 with a weight of 0.1, then the evaluation score = 95 × 0.6 + 85 × 0.3 + 90 × 0.1 = 91.5. The range of values ​​for the evaluation score is consistent with the range of values ​​for the checkpoint scores. For example, if the checkpoint scores are on a percentage scale, the evaluation score is also on a percentage scale.

[0085] It should be noted that the determination of the judgment result for each test case based on the evaluation score can be performed by interval mapping based on a preset threshold, mapping continuous evaluation scores to discrete judgment results. In one feasible implementation, a first threshold and a second threshold can be pre-configured, wherein the first threshold is greater than the second threshold. When the evaluation score is greater than or equal to the first threshold, the judgment result is determined to be pass, indicating that the actual execution result is highly consistent with the expected result, the test case execution is qualified, and no manual review is required; when the evaluation score is less than the first threshold but greater than or equal to the second threshold, the judgment result is determined to be partially pass, indicating that there is a certain deviation but the core checkpoints are basically correct, and manual intervention is required to determine whether repair is needed; when the evaluation score is less than the second threshold, the judgment result is determined to be fail, indicating that the deviation is significant, the core checkpoints may be incorrect, the test case execution is unqualified, and it is necessary to investigate, repair, and retest.

[0086] It's important to note that the three-tier judgment system retains a grayscale range during automatic judgment. The purpose of this intermediate level is that when there are deviations in UI or functional checkpoints but normal data checkpoints, the total score may fall within the middle range. In this case, it shouldn't be directly judged as failing (core data is correct, business operations won't be interrupted), nor should it be directly judged as passing (UI or functional issues do exist). Instead, it should be marked for manual review, achieving effective collaboration between automated and manual verification. For example, the first threshold can be configured to 90, the second threshold can be configured to 80, and these threshold values ​​can be flexibly configured according to project characteristics and business risk preferences to adapt to the verification needs of different business scenarios.

[0087] In this implementation, preset weights reflect the business importance of different checkpoint types, semantic difference rate quantifies the deviation between actual execution results and expected results, weighted summation integrates the scores of each checkpoint level into an overall evaluation score, and three levels of judgment results are determined based on preset thresholds. This achieves a systematic quantitative process from multi-dimensional checkpoints to a unified evaluation conclusion, so that the verification results can reflect both overall consistency and the weight influence of core checkpoints. At the same time, grayscale ranges are retained in the automatic judgment for manual review, balancing verification efficiency and judgment accuracy.

[0088] In one feasible implementation, step S403, determining the checkpoint score for each checkpoint type of each test case based on the semantic difference rate and the scoring criteria corresponding to the checkpoint type, can be achieved through the following steps: Step S4031: For each checkpoint type, determine whether the semantic difference rate is less than or equal to the preset difference rate threshold corresponding to the checkpoint type.

[0089] It should be noted that the preset difference rate threshold refers to the upper limit of the semantic difference rate tolerance for each checkpoint type. When the semantic difference rate is less than or equal to this threshold, the actual execution result is considered to be basically consistent with the expected result at the semantic level, and no points need to be deducted; when the semantic difference rate is greater than this threshold, points will be deducted based on the excess.

[0090] Step S4032: When the semantic difference rate is less than or equal to the preset difference rate threshold, the checkpoint score of the checkpoint type is determined to be the preset full score.

[0091] It should be noted that the preset maximum score refers to the score for this checkpoint type when the actual execution result is substantially consistent with the expected result at the semantic level. For example, the preset maximum score can be 100, indicating that the checkpoint type scores the maximum.

[0092] Step S4033: When the semantic difference rate is greater than the preset difference rate threshold, the deduction amount is determined by multiplying the semantic difference rate with the preset reduction coefficient corresponding to the checkpoint type, and the checkpoint score of the checkpoint type is obtained by subtracting the deduction amount from the preset full score.

[0093] It should be noted that the preset reduction coefficient is a multiplier used to convert the excess semantic difference rate into a deduction factor when the semantic difference rate exceeds a preset difference rate threshold. The deduction amount and checkpoint score can be determined in the following ways:

[0094] Where P represents the deduction amount, λ represents the preset reduction coefficient, D represents the semantic difference rate, and T represents the preset difference rate threshold.

[0095]

[0096] Where S represents the checkpoint score and S0 represents the preset maximum score.

[0097] It should be noted that the preset difference rate thresholds for data checkpoints, user interface checkpoints, and functional checkpoints increase sequentially. This reflects the different tolerances for differences among the checkpoints: data checkpoints are the most sensitive to differences, as even minor data deviations may indicate core business errors, hence their lowest preset difference rate threshold; functional checkpoints have the highest tolerance for differences because there may be multiple reasonable expressions for the textual descriptions of functional operations, hence their highest preset difference rate threshold; user interface checkpoints fall in between. For example, the preset difference rate threshold for data checkpoints can be configured to 5%, for user interface checkpoints to 10%, and for functional checkpoints to 15%.

[0098] It should be noted that the preset decreasing coefficients for data checkpoints, user interface checkpoints, and functional checkpoints decrease sequentially. The larger the preset decreasing coefficient, the more points are deducted for the same excess difference rate, and the more sensitive the scoring is to differences. The data checkpoint has the largest decreasing coefficient, indicating the fastest deduction once the difference rate threshold is exceeded; the functional checkpoint has the smallest decreasing coefficient, indicating a relatively mild deduction even if the difference rate threshold is exceeded. For example, the preset decreasing coefficient for data checkpoints can be configured to 10, the preset decreasing coefficient for user interface checkpoints can be configured to 5, and the preset decreasing coefficient for functional checkpoints can be configured to 3.

[0099] It should be noted that differentiated scoring for different checkpoint types is achieved through a combination of the difference rate threshold and the reduction coefficient. Taking a preset maximum score of 100 as an example, if the semantic difference rate of a data checkpoint is 8% (greater than the 5% threshold) and the reduction coefficient is 10, then the deduction = 10 × (8% - 5%) = 30, and the checkpoint score = 100 - 30 = 70. If the semantic difference rate of a user interface checkpoint is 8% (less than the 10% threshold), then the checkpoint score is 100. If the semantic difference rate of a functional checkpoint is 20% (greater than the 15% threshold) and the reduction coefficient is 3, then the deduction = 3 × (20% - 15%) = 15, and the checkpoint score = 100 - 15 = 85. The above values ​​are for illustrative purposes only; in actual applications, they can be flexibly configured according to business needs.

[0100] In this implementation, by setting differentiated preset difference rate thresholds and preset reduction coefficients for different checkpoint types, the scoring criteria are matched with the importance of the business—data checkpoints are scored the most strictly, functional checkpoints are scored the most leniently, and user interface checkpoints are scored in the middle, so that the scoring results can accurately reflect the actual business risks of each checkpoint type.

[0101] In one feasible implementation, the verification rules in step S10 include minimum dimension verification rules. The step of generating a set of cases to be verified based on a preset selection strategy can be achieved through the following steps: Step S101: When the verification rule carried in the verification request is the minimum dimension verification rule, determine the candidate test cases based on the project identifier.

[0102] It should be noted that candidate test cases refer to all test cases belonging to the project as determined by the project identifier. After determining the scope of candidate test cases, the system needs to further filter them to select test cases that meet the important functional conditions.

[0103] Step S102: Select test cases that meet the preset important functional conditions from the candidate test cases and generate a set of test cases to be checked.

[0104] It should be noted that the preset important function conditions are used to define which test cases fall into the category of "important functions". Specifically, the preset important function conditions can include: the test case belongs to at least one of the core scenarios, core functions, or core business processes. Core scenarios refer to user usage scenarios that play a key role in the operation of the business, such as transfer scenarios and payment scenarios in a financial system; core functions refer to the basic functional modules that the system must operate correctly, such as user authentication and order processing; core business processes refer to the end-to-end business chain that runs through multiple functional modules, such as the complete transaction chain from order placement to payment.

[0105] It should be noted that there are multiple ways to determine whether a test case meets the preset important functional conditions. For example, it can be based on test case tags—core test cases are pre-labeled with tags such as "core," "important," and "critical path" in the test case management system, and selection strategies are used to filter by matching tags; it can also be based on test case priority—high-priority test cases are considered important functional test cases; or it can be based on the test case's position in the business process—test cases located in the main business process are considered important functional test cases. This embodiment does not specifically limit the method for determining whether a test case meets the preset important functional conditions.

[0106] In this implementation, by combining minimum-dimensional verification rules with preset key functional conditions, testers can trigger automated verification of key functional test cases with a single click. This is independent of project size and the testers' subjective experience—regardless of the number of test cases in a project, the system dynamically generates a set of test cases to be verified according to uniform preset key functional conditions, ensuring the standardization of verification rules. Testers prioritize covering the most important test cases within the limited verification time, ensuring that critical business risks are exposed first while avoiding the time overhead of full verification, thus achieving a balance between verification efficiency and verification quality.

[0107] In one feasible implementation, after determining the judgment result of each test case based on the evaluation score in step S40, the following steps may also be included: Step S50: Generate a verification report based on the judgment results and evaluation scores of each test case.

[0108] It should be noted that the verification report is a document or data file summarizing the results of this verification. Its content includes the judgment results, evaluation scores, and related detailed information for each test case (such as the actual execution result text, expected result text, semantic difference rate, and scores for each checkpoint). The verification report can be in structured formats such as JSON (JavaScript Object Notation, a lightweight data exchange format) and XML (Extensible Markup Language), or in visual formats such as HTML (HyperText Markup Language) pages and PDF (Portable Document Format) documents. The generation method of the verification report can be flexibly selected according to actual needs. It can be based on a preset report template, where the verification result data is populated into the template to generate the report; or the system can automatically organize the verification result data and output a standard format report file. For example, the verification report can include a summary statistics section and a detailed list section: the summary statistics section displays the number and proportion of cases that passed, partially passed, and failed; the detailed list section lists the judgment results, evaluation scores, scores for each checkpoint, and the corresponding actual execution results and expected result text for each case.

[0109] Step S60: Push the verification report to the party that initiated the verification request.

[0110] It should be noted that the initiator of the verification request refers to the tester or test role who submits the verification request. Pushing the verification report to the initiator allows them to receive the verification results promptly. The push method can be flexibly selected according to actual needs. For example, it can be pushed via system messages, emails, instant messaging tools, or internal notifications from the test management platform; alternatively, the report can be stored in a preset location and the report link sent to the initiator. By generating and pushing verification reports in real time, the initiator does not need to manually check the verification progress. The system automatically notifies upon completion of the verification, achieving immediate feedback on the verification results. This allows testers to continue performing other testing tasks immediately after submitting the verification request without waiting, further improving testing efficiency.

[0111] In this implementation, by automatically generating a verification report and pushing it to the initiator, testers can continue to perform other test tasks after submitting a verification request. Once the verification is completed, the system automatically notifies the results, thus achieving a complete closed loop in the verification process.

[0112] It should be noted that the technical features in the above embodiments can be arbitrarily combined according to actual needs. As long as the combined technical solutions do not contradict each other, they all fall within the protection scope of this application. For example, the implementation method of extracting text information using an artificial intelligence model can be combined with the implementation method of semantic standardization processing according to checkpoint type; the implementation method of differential scoring according to checkpoint type can be combined with the implementation method of three-level threshold judgment; the implementation method of minimum dimension verification rule can also be combined with the implementation method of asynchronous multi-threaded screenshot acquisition. This application does not specifically limit the combination method between the various implementation methods.

[0113] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method for verifying the execution results of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0114] This application also provides an execution result verification device, please refer to... Figure 4 The execution result verification device includes: The case generation module 10 is used to respond to the received verification request and generate a set of cases to be verified based on the project identifier and verification rules carried in the verification request and a preset selection strategy. The set of cases to be verified includes each test case and the expected result of each test case. The case execution module 20 is used to obtain the execution screenshots corresponding to each test case in the case set to be verified, extract the text information in the execution screenshots, and use the text information as the actual execution result of each test case; The data cleaning module 30 is used to clean the actual execution results and expected results of each test case, respectively, to obtain the cleaned actual execution results and cleaned expected results of each test case. The result determination module 40 is used to compare the actual execution result after cleaning with the expected result after cleaning based on the preset scoring rules, generate an evaluation score, and determine the determination result of each test case based on the evaluation score.

[0115] The execution result verification device provided in this application, employing the execution result verification method in the above embodiments, can solve the technical problem of how to improve verification efficiency while ensuring the objectivity and consistency of judgment criteria in test case execution result verification. Compared with the prior art, the beneficial effects of the execution result verification device provided in this application are the same as those of the execution result verification method provided in the above embodiments, and other technical features in the execution result verification device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0116] This application provides an execution result verification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the execution result verification method in the above embodiment 1.

[0117] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the execution result verification device in the embodiments of this application. The execution result verification device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The execution result verification device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0118] like Figure 5 As shown, the execution result verification device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the execution result verification device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the execution result verification device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows execution result verification devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0119] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0120] The execution result verification device provided in this application, employing the execution result verification method in the above embodiments, can solve the technical problem of how to improve verification efficiency while ensuring the objectivity and consistency of judgment criteria in test case execution result verification. Compared with the prior art, the beneficial effects of the execution result verification device provided in this application are the same as those of the execution result verification method provided in the above embodiments, and other technical features in this execution result verification device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0121] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0122] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0123] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the execution result verification method in the above embodiments.

[0124] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0125] The aforementioned computer-readable storage medium may be included in the execution result verification device; or it may exist independently and not be assembled into the execution result verification device.

[0126] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the execution result verification device, cause the execution result verification device to: In response to the received verification request, a set of test cases to be verified is generated based on the project identifier and verification rules carried in the verification request and a preset selection strategy. The set of test cases to be verified includes each test case and the expected result of each test case. Obtain the execution screenshots corresponding to each test case in the test case set to be verified, and extract the text information from the execution screenshots, and use the text information as the actual execution result of each test case; The actual execution results and expected results of each test case are cleaned separately to obtain the cleaned actual execution results and cleaned expected results of each test case. Based on the preset scoring rules, the actual execution results after cleaning are compared with the expected results after cleaning to generate an evaluation score, and the judgment result of each test case is determined according to the evaluation score.

[0127] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0130] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described execution result verification method. This solves the technical problem of improving verification efficiency while ensuring the objectivity and consistency of the judgment criteria in test case execution result verification. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the execution result verification method provided in the above embodiments, and will not be repeated here.

[0131] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the execution result verification method described above.

[0132] The computer program product provided in this application solves the technical problem of improving verification efficiency while ensuring the objectivity and consistency of judgment criteria in test case execution result verification. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the execution result verification method provided in the above embodiments, and will not be repeated here.

[0133] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for verifying execution results, characterized in that, The method includes: In response to a received verification request, a set of test cases to be verified is generated based on a preset selection strategy according to the project identifier and verification rules carried in the verification request. The set of test cases to be verified includes each test case and the expected result of each test case. Obtain the execution screenshots corresponding to each test case in the set of test cases to be verified, extract the text information from the execution screenshots, and use the text information as the actual execution result of each test case; The actual execution results and expected results of each test case are cleaned to obtain the cleaned actual execution results and cleaned expected results of each test case. Based on preset scoring rules, the actual execution results after cleaning and the expected results after cleaning are compared to generate an evaluation score, and the judgment result of each test case is determined based on the evaluation score.

2. The method according to claim 1, characterized in that, The step of extracting text information from the execution screenshot and using the text information as the actual execution result of each test case includes: Input the execution screenshot into the preset artificial intelligence model; The AI ​​model is used to analyze visual elements and extract text information from the execution screenshot to obtain the text information in the execution screenshot. The text information is used as the actual execution result of each test case.

3. The method according to claim 1, characterized in that, The step of using the text information as the actual execution result of each test case includes: The text information is classified according to checkpoint type, which includes at least one of data checkpoint, user interface checkpoint, and function checkpoint; The text information is organized into structured text according to the classified checkpoint types; The structured text is used as the actual execution result for each test case.

4. The method according to claim 3, characterized in that, The step of cleaning the actual execution results and expected results of each test case to obtain the cleaned actual execution results and cleaned expected results of each test case includes: For any of the test cases, the actual execution results and the expected results of the test cases are standardized in format to remove redundant symbols and spaces. The actual execution results and expected results after format unification are semantically standardized according to the checkpoint type to map the text of the actual execution results and expected results after format unification into a unified semantic representation, thus obtaining the actual execution results and expected results after test case cleaning.

5. The method according to claim 3, characterized in that, The steps of comparing the actual execution results after cleaning with the expected results after cleaning based on preset scoring rules, generating an evaluation score, and determining the judgment result of each test case based on the evaluation score include: Obtain the preset weights corresponding to each of the aforementioned checkpoint types; For each checkpoint type, calculate the semantic difference rate between the actual execution result after cleaning and the expected result after cleaning for each test case; Based on the semantic difference rate and the scoring criteria corresponding to the checkpoint type, determine the checkpoint score for each checkpoint type in each test case; An evaluation score is generated based on the checkpoint scores of each checkpoint type for each test case and the preset weights, and the judgment result of each test case is determined based on the evaluation scores.

6. The method according to claim 5, characterized in that, The step of determining the checkpoint score for each checkpoint type of each test case based on the semantic difference rate and the scoring criteria corresponding to the checkpoint type includes: For each checkpoint type, determine whether the semantic difference rate is less than or equal to a preset difference rate threshold corresponding to the checkpoint type; When the semantic difference rate is less than or equal to the preset difference rate threshold, the checkpoint score of the checkpoint type is determined to be the preset full score. When the semantic difference rate is greater than the preset difference rate threshold, the deduction amount is determined by the product of the semantic difference rate and the preset decreasing coefficient corresponding to the checkpoint type, and the checkpoint score of the checkpoint type is obtained by subtracting the deduction amount from the preset full score. Among them, the preset difference rate thresholds corresponding to data checkpoints, user interface checkpoints, and function checkpoints increase sequentially, while the preset decreasing coefficients corresponding to data checkpoints, user interface checkpoints, and function checkpoints decrease sequentially.

7. The method according to claim 1, characterized in that, The verification rules include minimum dimension verification rules, and the step of generating a set of cases to be verified based on a preset selection strategy includes: When the verification rule carried in the verification request is the minimum dimension verification rule, candidate test cases are determined according to the project identifier; Test cases that meet the preset important functional conditions are selected from the candidate test cases to generate the test case set to be checked; The preset key functional conditions include: the test case belongs to at least one of the core scenarios, core functions, or core business processes.

8. The method according to claim 1, characterized in that, Following the step of determining the judgment result of each test case based on the evaluation score, the method further includes: A verification report is generated based on the judgment results of each test case and the evaluation scores; The verification report is pushed to the party that initiated the verification request.

9. An execution result verification device, characterized in that, The execution result verification device includes: The case generation module is used to respond to the received verification request and generate a set of cases to be verified based on the project identifier and verification rules carried in the verification request and a preset selection strategy. The set of cases to be verified includes each test case and the expected result of each test case. The case execution module is used to obtain the execution screenshots corresponding to each test case in the case set to be verified, extract the text information in the execution screenshots, and use the text information as the actual execution result of each test case; The data cleaning module is used to clean the actual execution results and the expected results of each test case, respectively, to obtain the cleaned actual execution results and the cleaned expected results of each test case. The result determination module is used to compare the actual execution result after cleaning with the expected result after cleaning based on preset scoring rules, generate an evaluation score, and determine the determination result of each test case based on the evaluation score.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the execution result verification method as described in any one of claims 1 to 8.

Images