Methods and related equipment for detecting test case execution results

Abstract

This application discloses a method and related equipment for detecting the execution results of test cases, to solve the problem in the prior art that it is impossible to accurately determine whether a test case has been truly completed and whether there are any execution omissions based on execution records. The method includes: obtaining the test steps of the test case to be detected and the execution time of the test case to be detected recorded in the test case management platform; determining a log query time window based on the execution time, and obtaining multi-source log records of the target business system within the log query time window; determining a query identifier for querying the operation object in the multi-source log records based on the operation object in the test steps; determining the matching feature information between the test steps and the multi-source log records based on the query results of the query identifier in the multi-source log records, the matching feature information being used to characterize the execution status of the test steps; and predicting the execution completion probability of the test case to be detected using a trained probability prediction model based on the matching feature information.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and in particular to a method and related equipment for detecting the execution results of test cases. Background Technology

[0002] In the field of software testing, the execution record of software test cases is the key basis for determining whether test cases have been executed completely and for tracing whether any test cases have been missed.

[0003] However, in existing test management systems, the execution records of software test cases rely heavily on manual input and marking by test engineers. Therefore, test engineers may, due to subjective negligence or objective reasons, not actually execute test cases, but still mark the execution status of the test cases as completed, thus generating false execution records that do not match the actual situation. Ultimately, this leads to the problem that it is impossible to accurately determine whether the test cases have been truly completed based on the execution records. Summary of the Invention

[0004] This application provides a method for detecting the execution results of test cases, which solves the problem in the prior art that the execution records of test cases are highly dependent on manual marking, which may lead to inconsistencies between the execution records and the actual execution situation, and makes it impossible to accurately determine whether the test cases have been actually completed or whether there are any execution omissions based on the execution records.

[0005] This application also provides a device for detecting the execution results of test cases, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] The embodiments of this application adopt the following technical solutions: In a first aspect, embodiments of this application provide a method for detecting the execution result of test cases, including: Obtain the test steps of the test cases to be tested and the execution time of the test cases to be tested recorded in the test case management platform; Determine the log query time window based on the execution time, and obtain the multi-source log records of the target business system within the log query time window; Determine the query identifier used to query the operation object in the multi-source log records based on the operation object in the test steps; Based on the query results of the query identifier in the multi-source log records, the matching feature information between the test steps and the multi-source log records is determined. The matching feature information is used to characterize the execution status of the test steps. Based on the matching feature information, the probability of execution completion of the test cases to be detected is predicted by the trained probability prediction model.

[0007] Secondly, embodiments of this application provide a device for detecting test case execution results, including a test case information acquisition module, a log recording acquisition module, a query identifier determination module, a feature information determination module, and a probability prediction module, wherein: The test case information acquisition module is used to acquire the test steps of the test cases to be tested and the execution time of the test cases to be tested recorded in the test case management platform; The log acquisition module is used to determine the log query time window based on the execution time and acquire multi-source log records of the target business system within the log query time window; The query identifier determination module is used to determine the query identifier for querying the operation object in multi-source log records based on the operation object in the test steps; The feature information determination module is used to determine the matching feature information between the test steps and the multi-source log records based on the query results of the query identifier in the multi-source log records. The matching feature information is used to characterize the execution status of the test steps. The probability prediction module is used to predict the probability of execution completion of the test cases to be detected based on the matching feature information and the probability prediction model trained.

[0008] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for detecting the execution results of test cases as described above.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for detecting the execution results of test cases as described above.

[0010] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method for detecting the execution results of test cases as described above.

[0011] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: The method provided in this application can determine a log query time window based on the test steps of the test case to be tested and their execution time recorded in the test case management platform. It then obtains multi-source log records from the target business system within this time window, generates query identifiers based on the operation objects in the test steps, and retrieves these identifiers from the multi-source log records. This yields matching feature information characterizing the actual execution status of each test step. Finally, a trained probability prediction model is used to predict the probability of the test case to be tested completing. This allows for the correlation and verification of the execution status marked on the platform with objective log evidence from the target business system, quantifying the actual execution degree of the test steps in probabilistic form. This reduces the risk of false execution records caused by manual entry and subjective marking, thus solving the problems in existing technologies where manual execution records cannot accurately determine whether test cases have actually been completed and where it is difficult to trace execution omissions. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram illustrating the implementation process of a test case execution result detection method according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation flow of a method for determining a query identifier for querying an operation object in multi-source log records, according to an embodiment of this application. Figure 3 This is a schematic diagram of the specific structure of a test case execution result detection device according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0015] To address the problem in existing technologies where test case execution records heavily rely on manual labeling, potentially leading to discrepancies between execution records and actual execution, and making it impossible to accurately determine whether test cases have been truly completed or if any execution omissions exist based on the execution records, this application provides a method for detecting test case execution results.

[0016] The execution subject of this method can be various types of computing devices, or it can be an application or app installed on the computing device. The computing device can be a user terminal such as a mobile phone, tablet computer, or smart wearable device, or it can be a server.

[0017] For ease of description, this application uses a server as the execution subject of the method in its embodiments to illustrate the method. Those skilled in the art will understand that this embodiment uses a server as an example to describe the method, which is merely an illustrative example and does not limit the scope of protection of the corresponding claims.

[0018] Specifically, the implementation flow of the method provided in this application embodiment is as follows: Figure 1 As shown, it includes the following steps 102 to 110.

[0019] Step 102: Obtain the test steps of the test cases to be tested and the execution time of the test cases to be tested recorded in the test case management platform.

[0020] In this embodiment, the test case content of the test case to be tested can be obtained from the test case management platform by calling the interface of the test case management platform. The test case content includes at least the test steps and the execution time recorded in the test case management platform.

[0021] The test steps describe the specific sequence of operations that the test cases to be tested need to execute during the testing process. Test steps typically include three basic elements: operation type, operation object, and operation data.

[0022] The operation type describes the type of basic operation action that the test case to be tested needs to perform. For example, common operation types may include, but are not limited to: Interface operations: click, double-click, long press, swipe, scroll, drag; Input operations: Input, Clear, Paste, Select; Navigation operations: Open, Close, Back, Refresh, Jump; Verification operations: check, verify, assert, compare; Waiting operation: wait, pause.

[0023] The operation object identifies the target element to which the specific operation performed by the test case during the testing process acts. For example, operation types may include, but are not limited to: Interface elements: buttons, links, input boxes, dropdown lists, checkboxes, radio buttons; Pages / Views: Login page, homepage, details page, pop-ups, menus; Data elements: username, password, order number, product ID; Functional modules: search function, sorting function, payment module.

[0024] Correspondingly, operation data refers to the specific data values ​​of the operations performed by the test case during the testing process. For example, operation data may include, but is not limited to: Input data: text, numbers, dates, files; Expected data: expected results, expected values, expected status; Configuration data: environment parameters, user credentials, test configuration.

[0025] In some embodiments, to distinguish the test steps and execution times corresponding to different test cases to be detected, in addition to obtaining the test steps and execution times, the test case number or test case name of the test case to be detected can also be obtained. The test case number or test case name is used to uniquely identify the test case to be detected, so as to establish a relationship between the obtained test steps, execution times and corresponding test cases. This is used to accurately distinguish different test cases in subsequent processes such as log matching and feature calculation, ensuring the accuracy and traceability of the detection process.

[0026] For example, suppose a test case to be tested is numbered TC-2025-001, and its test case content is as follows: Test case name: User login function verification; Execution time: 2025-03-15 11:23:34; Test steps: ① Open the login page ② Enter the username "test_user" ③ Enter the password "xxx" ④ Enter the verification code “AGW5” ⑤ Click the login button You can then obtain structured data such as the test case name, test steps, and execution time by calling the interface of the test case management platform, which can be used as input for subsequent processing.

[0027] It should be noted that the test steps and execution time for obtaining the test cases to be tested listed above are merely an exemplary description of the embodiments of this application and do not impose any limitations on the embodiments of this application.

[0028] Step 104: Determine the log query time window based on the execution time, and obtain the multi-source log records of the target business system within the log query time window.

[0029] Among them, multi-source log recording includes at least: front-end access log recording, back-end application log recording, and back-end database operation log recording.

[0030] To avoid affecting query efficiency and result accuracy due to improper log query time window settings—for example, setting the log query time window too large can lead to an excessively large log query range, resulting in low log query efficiency and the introduction of irrelevant log noise; or setting the log query time window too small can lead to an excessively small log query range, resulting in the omission of relevant execution records—in this embodiment, when determining the log query time window, the execution time of the test case to be tested can be used as the midpoint, and then extended forward and backward by a preset time offset as the log query time window.

[0031] For example, suppose the execution time of the test case to be tested is T, and the preset time offset is... Then the log query time window It can be defined as: .

[0032] Taking the execution time listed in step 102, 2025-03-15 11:23:34, as an example, assuming a preset time offset... A time window of 60 minutes can be used to determine the log query time window. [2025-03-15 10:23:34, 2025-03-15 12:23:34].

[0033] In some embodiments, a preset time offset can be determined based on the log volume or matching conditions recorded by multiple sources. Adjustments need to be made. For example, during the log query time window. When the number of multi-source log records retrieved exceeds the first threshold N_max, the number can be reduced. To narrow down the query scope; when querying logs within the time window If the number of multi-source log records retrieved is lower than the second threshold N_min or the query identifier is not matched, the threshold can be increased. This expands the query scope until the preset minimum / maximum log query time window limit is reached.

[0034] The first threshold N_max is used to limit the upper limit of the number of multi-source log records obtained within the log query time window, so as to avoid the log data volume being too large, causing the time overhead or resource consumption of log retrieval, parsing and subsequent feature calculation to exceed the system's capacity; N_min is used to limit the lower limit of the number of multi-source log records obtained within the log query time window, so as to ensure that the log data volume is sufficient to support the hit of query identifiers and the stable calculation of matching feature information.

[0035] Furthermore, N_max can be determined based on system performance evaluation / stress test results or historical log volume statistics, while N_min can be determined based on the minimum sample requirement for subsequent feature calculations and historical hit rate statistics.

[0036] For example, suppose Given a time window of 60 minutes, N_max = 10000 records, and N_min = 50 records, if a total of 35000 multi-source log records are returned within the log query time window W, then the time window can be narrowed down. For example, Reduced to 30 minutes.

[0037] Alternatively, if within the log query time window If the returned multi-source log records are only 12 or the query identifier is hit 0 times, then the increase can be made. For example, Increased to 90 minutes.

[0038] It should be noted that the above-listed content regarding determining the log query time window is merely an exemplary illustration in the embodiments of this application and does not impose any limitations on the embodiments of this application.

[0039] Step 106: Determine the query identifier for querying the operation object in the multi-source log records based on the operation object in the test steps.

[0040] The query identifier includes a front-end route list and a back-end application programming interface (API) list for the operation object. The front-end route list includes the best-matching route for the operation object, and the back-end API list includes the best-matching API for the operation object.

[0041] In this embodiment of the application, the purpose of determining the query identifier for querying the operation object in the multi-source log records based on the operation object in the test step is to convert the operation object described in natural language in the test step, such as login page, login button, etc., into a structured identifier that can be directly queried in the system log, such as front-end route, back-end API, thereby avoiding problems such as missed detection / false detection caused by the fact that natural language object names usually do not appear directly in multi-source log records.

[0042] In some embodiments, the query identifier can be determined as follows: First, the test steps of the test cases to be tested are parsed to obtain the operation objects in the test steps; for example, entity recognition is performed on the test steps to identify the operation objects from the test steps. Then, the operation objects are matched against the front-end route tag table and the back-end API tag table respectively: the front-end route tag table stores front-end route addresses and their description text, and the back-end API tag table stores back-end APIs and their description text, thereby providing a candidate set for the semantic mapping of operation objects—routes / APIs. During matching, the text of the operation object and the description text in the aforementioned tag tables can be vectorized and their similarity calculated. For example, Sentence-BERT can be used to generate vectors and cosine similarity can be applied. The front-end route with the highest similarity is selected as the best matching route, and the back-end API with the highest similarity is selected as the best matching API, forming a front-end route list and a back-end API list respectively. Finally, the front-end route list and the back-end API list are jointly determined as the query identifier, used for retrieval and hit statistics of the corresponding operation objects in multi-source log records.

[0043] For example, Tables 1 and 2 below are examples of a front-end routing tag table and a back-end API tag table, respectively, according to embodiments of this application.

[0044] Table 1 Front-end route tagging table

[0045] Table 2 Backend API Tag Table

[0046] It should be noted that the front-end routing tag table and back-end API tag table listed above are merely exemplary illustrations of the embodiments of this application and do not impose any limitations on the embodiments of this application.

[0047] Step 108: Based on the query results of the query identifier in the multi-source log records, determine the matching feature information between the test steps and the multi-source log records.

[0048] In this embodiment of the application, after obtaining the structured query identifiers corresponding to the test steps, namely the front-end route list, the back-end API list, and the data value list, these identifiers can be used as retrieval fields to search in multi-source log records, and a set of numerical features that can quantify the execution status of the test steps can be extracted from the retrieval results.

[0049] Matching feature information can be used to characterize the execution status of test steps. For example, matching feature information can include hit strength features (such as the number of hits, the number of log sources covered by the hits, etc.), time consistency features (such as the degree of deviation between the hit log time and the execution time, cross-log source time consistency, etc.), and anomaly indication features (such as whether error codes / abnormal keywords appear, etc.), thereby providing quantifiable input for subsequent execution probability prediction or execution conclusion determination.

[0050] Specifically, based on the type of query identifier, precise queries can be performed in the corresponding log storage. For example, for each route address in the frontend route list, such as / login, it can be matched against the frontend access logs within a set time window, recording all matching log entries and their detailed content, such as timestamps and session IDs. The same operation is performed on the backend API list and data value list in the backend application logs and database operation logs, respectively.

[0051] In one alternative implementation, the matching feature information between the test steps and the multi-source log records can be determined according to the following steps: (1) Based on the query results of the query identifier in the multi-source log records, count the number of times the query identifier hits the multi-source log records to obtain the number of matches.

[0052] The match count measures the number of times a query identifier in a test step of a test case appears in multi-source log records. This match count is equal to the sum of the number of times all query identifiers in the test case are found in multi-source log records within the log query time window.

[0053] Specifically, based on the query identifier, the total number of successfully matched entries in the front-end access log records, back-end application log records, and back-end database operation log records can be accumulated to obtain the number of matching entries in the front-end access log records, the number of matching entries in the back-end application log records, and the number of matching entries in the back-end database operation log records.

[0054] For example, you can first query the front-end access log records within the log query time window to find the route addresses in the front-end route list, and then count the number of times the route addresses in the front-end route list match multi-source log records to obtain the number of matches. For example, assuming that the front-end route list for test case 001's normal login is [" / login", " / index"], you can get the query results shown in Table 3 below: Table 3

[0055] Next, you can query the backend APIs in the backend application log records within the log query time window, and then count the number of times the backend APIs in the backend API list match multi-source log records to get the number of matches.

[0056] In one optional implementation, to ensure the accuracy of the test case execution results, in addition to determining the matching feature information between the test steps and the multi-source log records based on the query results of the query identifier in the multi-source log records, the matching feature information between the test steps and the multi-source log records can be further determined based on the query results of the operation data (or data values) in the multi-source log records. Specifically, the data values ​​in the backend data value list can be queried from the backend database operation log records within the log query time window, and the number of times the data values ​​in the backend data value list match the multi-source log records can be counted to obtain the number of matches.

[0057] (2) Obtain the log record time of the log record that the query identifier matches in the multi-source log records.

[0058] In this embodiment of the application, for each hit, the precise timestamp of the hit log can be parsed and extracted. As shown in Table 3 above, assuming that the timestamp of the hit log record when querying the multi-source log record for the routing address / login for the first time is 2025-3-11 13:40:32, then this timestamp can be recorded as the log record time.

[0059] (3) Calculate the time difference between log recording time and execution time.

[0060] Time difference information includes at least the minimum time difference, the standard deviation of the minimum time difference, and the range of the minimum time difference.

[0061] The minimum time difference represents the minimum time interval between the execution time of the test case to be tested and the logging time of the log record that the query identifier matches in the multi-source log records. It is used to verify whether the operation responds to the execution of the test case within a reasonable time. In this embodiment, the minimum time difference can be the minimum value of the difference between the logging time and the execution time of all matched log records.

[0062] The standard deviation of the minimum time difference measures the volatility of the minimum time difference between the front-end, back-end, and database systems. The smaller the value, the more synchronized the response times of the three systems are.

[0063] The range of minimum time differences reflects the largest discrepancy between the minimum time differences of the front-end, back-end, and database systems. A larger range indicates that at least one of them has a significant delay, which may indicate an anomaly.

[0064] (4) The matching quantity and time difference information are determined as the matching feature information between the test steps and the multi-source log records.

[0065] Step 110: Based on the matching feature information, predict the execution completion probability of the test cases to be detected using the trained probability prediction model.

[0066] In some embodiments of this application, after obtaining the matching feature information, the matching feature information can be used as an input feature vector and input into the trained probability prediction model to output the execution completion probability of the test case to be detected.

[0067] Optionally, considering that the number of features relied upon for the test cases to perform realistic predictions is moderate and fixed, and that the model is required to have good interpretability and high computational efficiency, the XGBoost (eXtreme Gradient Boosting) model can be used as the probabilistic prediction model in this embodiment. This model is an efficient gradient boosting decision tree algorithm that can effectively handle structured features and output the probability that a sample belongs to a certain category.

[0068] Furthermore, the probabilistic prediction model can be trained as follows: Sample data is extracted from existing test cases and test libraries, for example, 3000 samples are extracted, and each sample test case is marked as a real execution to form a label, such as is_real_executed, thereby constructing a training dataset; the input features are preprocessed, for example, missing features caused by no log matching are filled, such as filling the missing minimum time difference value of each log source with 3600 seconds, and the numerical features are standardized and then divided into training and test sets; then the XGBoost model is trained based on a binary classification objective (such as binary:logistic) and validated.

[0069] In the inference and prediction phase, the matching feature information can be processed in the same way as in the training phase, then input into the trained model. The probability output interface of the model can be used to obtain the probability value of the actual execution / completion of the use case, which is used as the execution completion probability.

[0070] In some embodiments, after obtaining the execution completion probability, a hierarchical judgment rule can be further set based on the execution completion probability, for example: When the execution probability is ≥75%, it is judged as high credibility and the use case is determined to have been actually executed.

[0071] When the execution probability is between 60% and 75%, it is judged as having low credibility and its actual execution is considered questionable, requiring manual review.

[0072] When the execution probability is less than 60%, it is judged as unreliable, and the test case is considered not to have been executed and needs to be checked and reviewed.

[0073] The method provided in this application can determine a log query time window based on the test steps of the test case to be tested and their execution time recorded in the test case management platform. It then obtains multi-source log records from the target business system within this time window, generates query identifiers based on the operation objects in the test steps, and searches the multi-source log records to obtain matching feature information characterizing the actual execution status of each test step. Finally, it uses a trained probability prediction model to predict the probability of the test case to be tested completing. This allows for the correlation and verification of the execution status marked on the platform with objective log evidence from the target business system, quantifying the actual execution degree of the test steps in probabilistic form. This reduces the risk of false execution records caused by manual entry and subjective marking, thus solving the problems in existing technologies where manual execution records cannot accurately determine whether test cases have actually been completed and where it is difficult to trace execution omissions.

[0074] like Figure 2 As shown, in an optional embodiment, step 106 can also be implemented according to the following steps 202 to 206, namely, determining the query identifier used to query the operation object in the multi-source log records.

[0075] Step 202: Analyze the test steps to obtain the operation objects in the test steps.

[0076] Since test steps typically contain mixed information such as operation type, operation object, and operation data, directly using the complete text corresponding to the test step to retrieve multi-source log records may introduce a lot of noise. Therefore, this embodiment can parse the test steps to obtain the operation object.

[0077] To achieve automated and high-precision structured parsing of test steps described in natural language, this embodiment employs a deep learning-based sequence labeling model, i.e., a test case parsing model, to parse the test steps and obtain the operation objects within them. Specifically, the test steps can be input into the trained test case parsing model to perform entity recognition and obtain the operation objects within the test steps.

[0078] Among them, the test case parsing model is a model specifically trained for the characteristics of text in the software testing field. It can be trained based on the test steps of the test case sample set and the labeled data of the test steps of the test case sample set. The labeled data includes at least the labeled information of the operation objects in the test steps of the test case sample set. It can accurately identify and extract key semantic entities from the test step description, including operation type, operation object and operation data.

[0079] Optionally, the use case parsing model can adopt a joint architecture of Bidirectional Encoder Representations from Transformers and Conditional Random Field (i.e., BERT-CRF joint architecture). In this architecture, BERT (Bidirectional Encoder Representations from Transformers) acts as the encoder, responsible for understanding the deep contextual semantics of the test step text; CRF (Conditional Random Field) acts as the decoding layer, responsible for sequence constraints on the label predictions of each character or subword in the BERT output, ensuring that the output entity label sequence conforms to grammar and business logic. This model can be obtained by performing domain-adaptive training and fine-tuning on a general Chinese pre-trained model (such as bert-base-chinese) using a large amount of domain-specific test case data. This use case parsing model can also be called the BERT-CRF-TESTCASE model.

[0080] In some embodiments, the use case parsing model can be trained as follows: (1) First, construct a test case sample set and its labeled data for training. Specifically, existing test case data can be exported from the test case management platform, for example, about 30,000 test cases can be exported, and the test steps in the test cases can be used as sample input text. Sequence labeling is performed on each test step to form a labeled corpus file, for example, denoted as test_corpus.txt. When performing sequence labeling on each test step, at least the operation object should be labeled; in some embodiments, the operation type and operation data can also be labeled at the same time to improve the model's overall ability to parse the semantics of the steps. For example, the entity set can be set as entities=[operation type, operation object, operation data], and the word / character sequence in the step can be labeled using the BIO labeling method.

[0081] (2) After preparing the sample set and labeled data, the test case parsing model can be implemented using a sequence labeling model and trained based on the test steps and labeled data of the sample set. For example, domain-adaptive training and fine-tuning can be performed based on the BERT-CRF structure: BERT is used to perform context encoding on the word / character sequence of the test step, and CRF is used to perform global decoding constraints on the output label sequence to improve the consistency between entity boundaries and labels; the label set can include 7 types of labels such as B-OpT / I-OpT, B-Obj / I-Obj, B-Data / I-Data, and O. During training, fine-tuning can be performed based on the parameters of the pre-trained Chinese model, and training configurations such as learning rate, batch size, and training epochs can be set, such as learning rate 3e-5, batch size=16, training for 10 epochs and evaluating by epoch, so as to obtain the trained test case parsing model, which can be denoted as BERT-CRF-TESTCASE.

[0082] (3) In the model inference stage, the test steps to be parsed are input into the test case parsing model trained above. The test steps are segmented / segmented and entity labeled, and the label sequence corresponding to each word / character is output. Based on this, the operation object entity is extracted as the parsing result. Specifically, the operation object can be determined as the continuous fragment of the predicted label B-Obj / I-Obj or the label of the corresponding object element / operation object. In some embodiments, the model can also simultaneously identify the operation type, such as click, input, submit, jump, etc.; and data values, such as username=test_user, keyword=mobile phone, etc., so as to obtain a structured step parsing result and provide input for the subsequent mapping of the operation object to searchable route / API and other query identifiers.

[0083] Step 204: Based on the operation object, determine the front-end route list and back-end API list by comparing them with the front-end route tag table and back-end API tag table of the operation object, respectively.

[0084] In an alternative implementation, the front-end route list and the back-end API list can be determined according to the following steps (1) to (3): (1) Determine the best matching route for the operation object based on the description text of the front-end route address in the front-end route tag table and the operation object.

[0085] (2) Determine the best matching API for the operation object based on the description text of the backend API in the backend API tag table.

[0086] (3) Determine the front-end route list and the back-end API list based on the best matching route and the best matching API, respectively.

[0087] In this embodiment of the application, after parsing the operation object from the test steps, in order to map the operation object into a query identifier that can be directly retrieved in different log sources, a front-end route tag table and a back-end API tag table can be pre-built. The front-end route tag table is used to store the front-end route address and its description text. For example, the route address exported from the front-end code repository or database of the business system is written into the tag table, and the menu name / page name is used as the default tag. The tag can be manually modified to adapt to the business name. The back-end API tag table is used to store the back-end API and its description text. For example, the API and its description text are crawled from the Swagger document, and the description text is used as the default tag of the API. The tag can be manually revised.

[0088] After constructing the front-end route tagging table and the back-end API tagging table, semantic matching can be performed on the operation object and the description text in the front-end route tagging table / back-end API tagging table to determine the best matching route and the best matching API for the operation object. Specifically, the operation object text and the description text of each route in the front-end route tagging table can be input into the Sentence-BERT model for vectorization encoding, and the similarity between the operation object and each route description text can be calculated based on cosine similarity; the front-end route with the highest similarity is selected as the best matching route (best_match_route) for the operation object. Similarly, the similarity between the operation object text and the description text of each API in the back-end API tagging table is calculated, and the back-end API with the highest similarity is selected as the best matching API (best_match_api).

[0089] Finally, a front-end route list and a back-end API list for log retrieval are generated based on the best-match route and the best-match API, respectively. For example, for multiple operation object elements identified in the same test case, their corresponding `best_match_route` is calculated and obtained one by one, and these are summarized to form the front-end route list for that test case; similarly, the `best_match_api` corresponding to each operation object element is summarized to form the back-end API list. Ultimately, the front-end route list and the back-end API list are jointly determined as query identifiers to enable retrieval and hit statistics in multi-source log records by route address or API path, respectively.

[0090] It should be noted that the reason for determining the best matching route and the best matching API in this application embodiment is mainly for the controllability and accuracy of log retrieval. Specifically, in actual business, a relatively complex test step may involve multiple page jumps or trigger multiple interface calls simultaneously. For example, after a user clicks a button, the first API is called to complete the verification, and then the user is redirected to another page and further calls the second API. That is, a test step may correspond to multiple routes or multiple APIs during execution. However, if a set of candidate routes / APIs is matched for each operation element and all of them are searched in the logs, the search scope will be greatly expanded, thereby introducing a large amount of irrelevant or accidental log hit noise within the time window. Therefore, this application adopts a best matching strategy based on semantic similarity. By determining the best matching route and best matching API with the highest similarity for each operation object, the candidate set is converged. In this way, while ensuring semantic closeness, the search identifiers are converged into a set of routes / APIs with a controllable size and the most relevant information. This allows for effective location of the log segment most relevant to the operation object when performing retrieval and hit statistics based on the best_match route / API within the log query time window, while reducing the risk of misjudgment caused by irrelevant hits, thereby improving the reliability and stability of the execution judgment results.

[0091] Step 206: Determine the front-end route list and the back-end API list as query identifiers.

[0092] The front-end route tag table stores the front-end route address and its description text for the operation object; the back-end API tag table stores the back-end API and its description text for the operation object.

[0093] Based on the implementation methods described in steps 202 to 206 above, this embodiment of the application can extract operation objects from test steps described in natural language through a use case parsing model. Combining the route / API address-description text mapping relationship recorded in the front-end route tag table and the back-end API tag table, it determines the best matching route and best matching API for the operation object based on semantic similarity, thereby forming a front-end route list and a back-end API list as query identifiers. This allows test steps to be converted into structured identifiers that can be directly retrieved from multi-source logs, thus solving the location difficulties caused by the inconsistency between natural language steps and the searchable fields in logs. Simultaneously, by employing the best matching strategy to converge the candidate route / API set to a semantically closest and controllable set of identifiers, it avoids introducing a large amount of irrelevant or accidental hit noise within the time window, reducing the risk of misjudging non-executed actions as executed actions, and improving the reliability and stability of execution determination. Furthermore, using the front-end route list and the back-end API list together as query identifiers can adapt to the recording dimensions of different log sources, improving cross-log source coverage and robustness. The tag table can also adapt to business iterations by maintaining description text or entries, thereby improving the maintainability and scalability of the solution.

[0094] To address the problem in existing technologies where test case execution records heavily rely on manual labeling, potentially leading to discrepancies between the execution records and actual execution, and making it impossible to accurately determine whether test cases have been fully executed or if any execution omissions exist based solely on the execution records, this application provides a test case execution result detection device. A schematic diagram of the device's specific structure is shown below. Figure 3 As shown, the system includes a use case information acquisition module 302, a log recording acquisition module 304, a query identifier determination module 306, a feature information determination module 308, and a probability prediction module 310. The functions of each module are as follows: The test case information acquisition module 302 is used to acquire the test steps of the test cases to be tested and the execution time of the test cases to be tested recorded in the test case management platform; The log recording acquisition module 304 is used to determine the log query time window based on the execution time and acquire multi-source log records of the target business system within the log query time window; The query identifier determination module 306 is used to determine the query identifier for querying the operation object in the multi-source log records based on the operation object in the test steps; The feature information determination module 308 is used to determine the matching feature information between the test steps and the multi-source log records based on the query results of the query identifier in the multi-source log records. The matching feature information is used to characterize the execution status of the test steps. The probability prediction module 310 is used to predict the execution completion probability of the test cases to be detected by using a trained probability prediction model based on matching feature information.

[0095] Optionally, the query identifiers include a front-end route list and a back-end application programming interface (API) list for the operation object; the front-end route list includes the best-matching route for the operation object, and the back-end API list includes the best-matching API for the operation object.

[0096] Optionally, the query identifier determination module 306 includes: The parsing unit is used to parse the test steps to obtain the operation objects in the test steps; The list determination unit is used to determine the front-end route list and the back-end API list based on the operation object and the front-end route tag table and the back-end API tag table of the operation object, respectively. The identifier determination unit is used to determine the front-end route list and the back-end API list as query identifiers; The front-end route tag table stores the front-end route address and its description text for the operation object; the back-end API tag table stores the back-end API and its description text for the operation object.

[0097] Optional, parsing unit, used for: The test steps are input into the trained test case parsing model to perform entity recognition on the test steps and obtain the operation objects in the test steps; The test case parsing model is trained based on the test steps of the test case sample set and the labeled data of the test steps of the test case sample set; the labeled data includes at least the labeled information of the operation objects in the test steps of the test case sample set.

[0098] Optional, list to determine the cells, used for: Based on the description text of the operation object and the front-end route address in the front-end route tagging table, determine the best matching route for the operation object; and Based on the description text of the backend API in the backend API tag table, determine the best matching API for the operation object. The front-end route list and back-end API list are determined based on the best matching route and the best matching API, respectively.

[0099] Optionally, the feature information determination module 308 is used for: Based on the query results of the query identifier in the multi-source log records, count the number of times the query identifier hits the multi-source log records to obtain the number of matches; Retrieve the log record time of the log record that matches the query identifier in the multi-source log records; Calculate the time difference between log recording time and execution time; The number of matches and the time difference information are determined as the matching feature information between the test steps and the multi-source log records.

[0100] The apparatus provided in this application can determine a log query time window based on the test steps of the test case to be tested and their execution time recorded in the test case management platform. It then obtains multi-source log records from the target business system within this time window, generates a query identifier based on the operation object in the test steps, and searches the multi-source log records to obtain matching feature information characterizing the actual execution status of each test step. Finally, it uses a trained probability prediction model to predict the probability of the test case to be tested completing. This allows for the correlation and verification of the execution status marked on the platform with objective log evidence from the target business system, quantifying the actual execution degree of the test steps in probabilistic form. This reduces the risk of false execution records caused by manual entry and subjective marking, thus solving the problems in the prior art where it is difficult to accurately determine whether a test case has actually been completed based on manual execution records and where it is difficult to trace execution omissions.

[0101] Figure 4 To illustrate the hardware structure of an electronic device according to various embodiments of this application, the electronic device may include a processor 401 and a memory 402 storing computer program instructions. Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this application.

[0102] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to an electronic device. In a particular embodiment, memory 402 may be a non-volatile solid-state memory.

[0103] In one embodiment, memory 402 may be read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0104] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the test case execution result detection methods in the above embodiments.

[0105] In one example, the electronic device may also include a communication interface 403 and a bus 410. For example, Figure 4 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 410 and complete communication with each other.

[0106] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0107] Bus 410 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0108] Furthermore, in conjunction with the test case execution result detection method in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the test case execution result detection methods in the above embodiments.

[0109] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0110] The above description is merely a specific implementation example of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0111] Secondly, those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0117] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0118] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0119] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting the execution result of test cases, characterized in that, include: Obtain the test steps of the test cases to be tested and the execution time of the test cases to be tested recorded in the test case management platform; The log query time window is determined based on the execution time, and multi-source log records of the target business system within the log query time window are obtained. Determine the query identifier for querying the operation object in the multi-source log records based on the operation object in the test steps; Based on the query results of the query identifier in the multi-source log records, the matching feature information between the test step and the multi-source log records is determined, and the matching feature information is used to characterize the execution status of the test step; The probability of execution completion of the test case to be detected is predicted by a probability prediction model trained based on the matching feature information.

2. The method as described in claim 1, characterized in that, The query identifier includes the front-end route list and the back-end application programming interface (API) list of the operation object; The front-end route list includes the best matching route for the operation object, and the back-end API list includes the best matching API for the operation object.

3. The method as described in claim 2, characterized in that, Determine the query identifier for querying the operation object in the multi-source log records based on the operation object in the test step, including: The test steps are analyzed to obtain the operation objects in the test steps; Based on the operation object, the front-end route list and the back-end API list are determined by comparing them with the front-end route tag table and the back-end API tag table of the operation object, respectively; The front-end route list and the back-end API list are determined as the query identifier; The front-end routing tag table is used to store the front-end routing address of the operation object and the description text of the front-end routing address; the back-end API tag table is used to store the back-end API of the operation object and the description text of the back-end API.

4. The method as described in claim 3, characterized in that, The test steps are parsed to obtain the operation objects in the test steps, including: The test steps are input into the trained test case parsing model to perform entity recognition on the test steps and obtain the operation objects in the test steps; The test case parsing model is trained based on the test steps of the test case sample set and the labeled data of the test steps of the test case sample set; the labeled data includes at least the labeled information of the operation objects in the test steps of the test case sample set.

5. The method as described in claim 3, characterized in that, Based on the operation object, the front-end route list and the back-end API list are determined by comparing them with the front-end route tag table and the back-end API tag table of the operation object, respectively, including: Based on the description text of the operation object and the front-end route address in the front-end route tagging table, determine the best matching route for the operation object; and Based on the operation object and the description text of the backend API in the backend API tag table, determine the best matching API for the operation object; The front-end route list and the back-end API list are determined based on the best matching route and the best matching API, respectively.

6. The method according to any one of claims 1-5, characterized in that, Based on the query results of the query identifier in the multi-source log records, determine the matching feature information between the test step and the multi-source log records, including: Based on the query results of the query identifier in the multi-source log records, the number of times the query identifier hits the multi-source log records is counted to obtain the number of matches; Obtain the log record time of the log record whose query identifier is matched in the multi-source log records; Calculate the time difference between the log recording time and the execution time; The number of matches and the time difference information are determined as the matching feature information between the test step and the multi-source log records.

7. A device for detecting the execution result of test cases, characterized in that, It includes a use case information acquisition module, a log recording acquisition module, a query identifier determination module, a feature information determination module, and a probability prediction module, among which: The test case information acquisition module is used to acquire the test steps of the test cases to be tested and the execution time of the test cases to be tested recorded in the test case management platform; The log recording acquisition module is used to determine the log query time window based on the execution time, and to acquire multi-source log records of the target business system within the log query time window; The query identifier determination module is used to determine the query identifier for querying the operation object in the multi-source log records based on the operation object in the test step. The feature information determination module is used to determine the matching feature information between the test step and the multi-source log record based on the query result of the query identifier in the multi-source log record. The matching feature information is used to characterize the execution status of the test step. The probability prediction module is used to predict the execution completion probability of the test case to be detected by using a trained probability prediction model based on the matching feature information.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for detecting the results of test case execution as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for detecting the execution result of test cases as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method for detecting the execution results of test cases as described in any one of claims 1 to 6.

Images