A test oracle generation method and apparatus

Abstract

The embodiment of the application provides a test prediction generation method and device, and relates to the technical field of software testing. The method comprises the following steps: generating at least one test step described in natural language according to a user interface knowledge base of an application program and a user interface event generated in a test application program process; respectively searching for user interfaces and control operations for executing each test step in a mind map data knowledge base to obtain a plurality of most similar nodes; obtaining a mind map path corresponding to each most similar node; the mind map path corresponding to any most similar node is a mind map path from a root node of a test mind map to the most similar node; generating a search result according to the mind map path corresponding to each most similar node; generating a model prompt word according to the search result and the at least one test step; inputting the model prompt word into a test prediction generation model, and obtaining a test prediction of the application program output by the test prediction generation model. The embodiment of the application is used for generating a test prediction.

Description

[0001] This application claims priority to patent application No. 202510501174.4, filed on April 21, 2025, the contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of software testing technology, and in particular to a test oracle generation method and apparatus. Background Technology

[0003] A test oracle, also known as a test oracle, is a mechanism or basis for determining whether the output of software is correct based on expected logic, rules, or business requirements.

[0004] Testing graphical user interfaces (GUIs) is notoriously difficult due to the challenges of accurately predicting GUI behavior, the difficulty in aligning predicted behavior with GUI elements, and the resulting large volume of test oracle data. This has made test oracle generation a persistent challenge in automated testing. Current test oracle generation methods can be categorized into exploration-based and information-based methods. Exploration-based methods can generate test oracle data related to common sense, but they struggle to distinguish between application features and bugs, often requiring extensive normal behavior to correct defects. Information-based methods offer more precise test oracle generation, better aligning with testing requirements. However, current methods demand overly precise additional information, severely limiting automated test oracle generation. Summary of the Invention

[0005] In view of this, embodiments of this application provide a test oracle generation method and apparatus for automatically generating test oracles, thereby improving the testing efficiency of applications.

[0006] To achieve the above objectives, the technical solutions provided in this application are as follows:

[0007] In a first aspect, embodiments of this application provide a test oracle generation method, including:

[0008] Generate at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application;

[0009] The user interface and control operations for each test step are searched in the mind map data knowledge base to obtain multiple most similar nodes. The mind map data knowledge base stores data of each node in the test mind map of the application.

[0010] Obtain the mind map path corresponding to each most similar node; the mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to that most similar node;

[0011] Retrieval results are generated based on the mind map paths corresponding to each most similar node;

[0012] Generate model suggestion words based on the search results and at least one test step;

[0013] Input the model prompt words into the test oracle generation model, and obtain the test oracle of the application output by the test oracle generation model.

[0014] As an optional implementation of this application, the step of generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application includes:

[0015] Based on the user interface events generated during the testing of the application, obtain screenshots of each control to be operated.

[0016] Based on the screenshots of each control to be operated and the aforementioned user interface knowledge base, determine the control name of each control to be operated.

[0017] At least one test step is generated based on the control name of each control to be operated.

[0018] As an optional implementation of this application, the step of obtaining control annotation information corresponding to each control to be operated from the user interface knowledge base based on the control screenshots of each control to be operated includes:

[0019] Input the screenshot of the first control and the control names of each control in the user interface knowledge base into the first network model, and obtain the similarity between the screenshot of the first control and each control name output by the first network model. The first control is any control to be operated, and the first network model is the model obtained by training the CLIP model.

[0020] The first control is named according to the control name that has the greatest similarity to the screenshot of the first control.

[0021] As an optional implementation of this application, the step of generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application includes:

[0022] The interface structure of each user interface to be operated is obtained based on the user interface events generated during the testing of the application.

[0023] Based on the interface structure of each user interface to be operated, obtain the user interface annotation information corresponding to each user interface to be operated from the user interface knowledge base. The user interface annotation information includes: interface structure and user interface name.

[0024] The interface name of each user interface to be operated is determined based on the interface label information corresponding to each user interface to be operated.

[0025] At least one test step is generated based on the interface name of each user interface to be operated.

[0026] As an optional implementation of this application, the step of obtaining the user interface annotation information corresponding to each user interface to be operated from the user interface knowledge base according to the interface structure of each user interface to be operated includes:

[0027] The structure vectors of the interface structure of each user interface to be operated are obtained based on the second network model, which is the model obtained by training the InfoGraph model.

[0028] Based on the second network model, obtain the structural vector of the page structure of each interface annotation information;

[0029] Obtain the similarity between each user interface to be operated and each interface annotation information. The similarity between the user interface to be operated and the interface annotation information is the similarity between the structure vector of the page structure of the user interface to be operated and the structure vector of the page structure of the control annotation information.

[0030] The interface annotation information with the highest similarity to each interface to be operated is determined as the interface annotation information corresponding to each interface to be operated.

[0031] As an optional implementation of this application, before retrieving the most similar nodes for each test operation and each user interface in the mind map data knowledge base, the method further includes:

[0032] Obtain the metadata of each node in the test mind map. The metadata of any node includes at least one of the following: the test mind map to which the node belongs, the position of the node in the test mind map, and the number of nodes in the node tree with the node as the root node.

[0033] The mind map data knowledge base is constructed based on the metadata of each node in the test mind map.

[0034] As an optional implementation of this application, generating the search results based on the mind map paths corresponding to each most similar node includes:

[0035] Obtain the retrieval score of the mind map path corresponding to each most similar node; the retrieval score of any mind map path is the average similarity between each node in the retrieval path and the corresponding test operation or user interface.

[0036] Select the preset number of mind map paths with the highest scores from the mind map paths corresponding to each most similar node;

[0037] Obtain the node tree corresponding to the preset number of mind map paths respectively; the node tree corresponding to any mind map path is a node tree with the last node of the mind map path as the root node;

[0038] The preset number of mind map paths are completed based on the node trees corresponding to the preset number of mind map paths, so as to obtain a preset number of completion search results;

[0039] The search results are obtained by combining the preset number of completion search results.

[0040] As an optional implementation of this application, the step of completing the preset number of mind map paths based on the node trees corresponding to the preset number of mind map paths to obtain a preset number of completion retrieval results includes:

[0041] Determine whether the number of nodes in the node tree corresponding to the preset number of mind map paths is greater than a preset threshold.

[0042] If the number of nodes in the node tree corresponding to the first mind map path in the preset number of mind map paths is less than or equal to the preset threshold, then the first mind map path is supplemented based on the node tree corresponding to the first mind map path to obtain the completion search result corresponding to the first mind map path.

[0043] If the number of nodes in the node tree corresponding to the second mind map path in the preset number of mind map paths is greater than the preset threshold, then the number of nodes in the child node tree of each node in the node tree corresponding to the second mind map path is obtained, and the preset threshold number of nodes with the largest number of nodes in the child node tree of the node tree corresponding to the second mind map path is selected to supplement the second mind map path, so as to obtain the completion search result corresponding to the second mind map path.

[0044] As an optional implementation of this application, before generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application, the method further includes:

[0045] Obtain user interface screenshots and user interface structure files of the application generated during the testing process;

[0046] Extract controls from the user interface screenshot to obtain a first set of controls;

[0047] Extract controls from the user interface structure file to obtain a second set of controls;

[0048] Merge the first control set and the second control set to obtain a third control set;

[0049] The controls in the third control set are merged based on their positional relationships and / or size to obtain a fourth control set.

[0050] Annotate each control in the fourth control set in the user interface screenshot to obtain interface annotation information;

[0051] The user interface events are generated based on the interface annotation information.

[0052] As an optional implementation of this application, before merging the first control set and the second control set to obtain the third control set, the method further includes:

[0053] Perform a consistency check on the user interface screenshots and the user interface structure files.

[0054] As an optional implementation of this application, before inputting the model prompt words into the test oracle generation model, the method further includes:

[0055] Add the interface annotation information to the model prompts.

[0056] Secondly, this application provides a test oracle generation apparatus, comprising:

[0057] Alignment unit, used to generate at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application;

[0058] The retrieval unit is used to search the user interface and control operations of each test step in the mind map data knowledge base to obtain multiple most similar nodes. The mind map data knowledge base stores data of each node in the test mind map of the application.

[0059] The processing unit is used to obtain the mind map path corresponding to each most similar node; the mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to the most similar node, and to generate search results based on the mind map path corresponding to each most similar node.

[0060] A generation unit is configured to generate model hint words based on the search results and the at least one test step, input the model hint words into a test oracle generation model, and obtain the test oracle of the application output by the test oracle generation model.

[0061] As an optional implementation of this application, the alignment unit is specifically used to obtain control screenshots of each control to be operated based on user interface events generated during the testing of the application; obtain control annotation information corresponding to each control to be operated from the user interface knowledge base based on the control screenshots of each control to be operated, the control annotation information including: control name; determine the control name of each control to be operated based on the control annotation information corresponding to each control to be operated; and generate the at least one test step based on the control name of each control to be operated.

[0062] As an optional implementation of this application, the alignment unit is specifically used to input the control screenshot of the first control and the control names of each control annotation information in the user interface knowledge base into the first network model, and obtain the similarity between the control screenshot of the first control output by the first network model and each control name, and name the first control according to the control name with the highest similarity to the control screenshot of the first control;

[0063] Wherein, the first control is any control to be operated, and the first network model is a model obtained by training a contrastive language image pre-trained CLIP model.

[0064] As an optional implementation of this application, the alignment unit is specifically used to obtain the interface structure of each user interface to be operated based on the user interface events generated during the testing of the application; obtain the user interface annotation information corresponding to each user interface to be operated from the user interface knowledge base based on the interface structure of each user interface to be operated, wherein the user interface annotation information includes: interface structure and user interface name; determine the interface name of each user interface to be operated based on the interface annotation information corresponding to each user interface to be operated; and generate the at least one test step based on the interface name of each user interface to be operated.

[0065] As an optional implementation of this application, the alignment unit is specifically used to obtain the structure vector of the interface structure of each user interface to be operated based on the second network model, wherein the second network model is a model obtained by training the InfoGraph model; obtain the structure vector of the page structure of each interface annotation information based on the second network model; obtain the similarity between each user interface to be operated and each interface annotation information, wherein the similarity between the user interface to be operated and the interface annotation information is the similarity between the structure vector of the page structure of the user interface to be operated and the structure vector of the page structure of the control annotation information; and determine the interface annotation information with the highest similarity to each user interface to be operated as the interface annotation information corresponding to each user interface to be operated.

[0066] As an optional implementation of this application, the retrieval unit is further configured to obtain the metadata of each node in the test mind map before retrieving the most similar nodes of each test operation and each user interface in the mind map data knowledge base. The metadata of any node includes at least one of the following: the test mind map to which the node belongs, the position of the node in the test mind map, and the number of nodes in the node tree with the node as the root node; and to construct the mind map data knowledge base based on the metadata of each node in the test mind map.

[0067] As an optional implementation of this application, the processing unit is specifically used to obtain the retrieval score of the mind map path corresponding to each most similar node; the retrieval score of any mind map path is the average similarity between each node in the retrieval path and the corresponding test operation or user interface; select a preset number of mind map paths with the highest retrieval scores among the mind map paths corresponding to each most similar node; obtain the node tree corresponding to the preset number of mind map paths respectively; the node tree corresponding to any mind map path is a node tree with the last node of the mind map path as the root node; complete the preset number of mind map paths based on the node trees corresponding to the preset number of mind map paths respectively to obtain a preset number of completed retrieval results; combine the preset number of completed retrieval results to obtain the retrieval result.

[0068] As an optional implementation of this application, the processing unit is specifically used to determine whether the number of nodes in the node tree corresponding to the preset number of mind map paths is greater than a preset threshold; if the number of nodes in the node tree corresponding to the first mind map path in the preset number of mind map paths is less than or equal to the preset threshold, then the first mind map path is supplemented based on the node tree corresponding to the first mind map path to obtain the completion retrieval result corresponding to the first mind map path; if the number of nodes in the node tree corresponding to the second mind map path in the preset number of mind map paths is greater than the preset threshold, then the number of nodes in the child node tree of each node in the node tree corresponding to the second mind map path is obtained, and the preset threshold number of nodes with the largest number of nodes in the child node tree of the node tree corresponding to the second mind map path are selected to supplement the second mind map path to obtain the completion retrieval result corresponding to the second mind map path.

[0069] As an optional implementation of this application, the alignment unit is further configured to: acquire a user interface screenshot generated during the testing of the application and a user interface structure file of the application before generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application; extract controls from the user interface screenshot to obtain a first control set; extract controls from the user interface structure file to obtain a second control set; merge the first control set and the second control set to obtain a third control set; merge the controls in the third control set based on the positional relationship and / or size between the controls to obtain a fourth control set; annotate each control in the fourth control set in the user interface screenshot to obtain interface annotation information; and generate the user interface event based on the interface annotation information.

[0070] As an optional implementation of this application, the alignment unit is further configured to perform a consistency check on the user interface screenshot and the user interface structure file before merging the first control set and the second control set to obtain the third control set.

[0071] As an optional implementation of this application, the generation unit is further configured to add the interface annotation information to the model prompts before inputting the model prompts into the test oracle to generate the model.

[0072] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program and the processor is used to cause the electronic device to implement the test oracle generation method described in any of the above embodiments when executing the computer program.

[0073] Fourthly, embodiments of this application provide a computer-readable storage medium that, when executed by a computing device, causes the computing device to implement the test oracle generation method described in any of the above embodiments.

[0074] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, enables the computer to implement the test oracle generation method described in any of the above embodiments.

[0075] The test oracle generation method provided in this application first generates at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application. Then, it searches the mind map data knowledge base for the user interface and control operations of each test step to obtain multiple most similar nodes. Next, it obtains the mind map paths corresponding to each most similar node and generates search results based on these paths. Finally, it generates model hints based on the search results and the at least one test step, and inputs these model hints into a test oracle generation model to obtain the test oracle of the application output by the model. Because the test oracle generation method provided in this application can automatically generate test steps described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application, and then search the mind map data knowledge base for the most similar nodes of the user interface and control operations of the test steps, obtain their mind map paths to generate search results, and add the search results to the model hints, this application can automatically and accurately generate additional information for the model hints, thereby automatically generating test oracles and improving the testing efficiency of the application. Attached Figure Description

[0076] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings referenced in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0077] Figure 1 This is a schematic diagram of the business process of the test oracle generation method provided in the embodiments of this application;

[0078] Figure 2 This is one of the flowcharts of the test oracle generation method provided in the embodiments of this application;

[0079] Figure 3 A schematic diagram of the node tree of the test mind map provided in the embodiments of this application;

[0080] Figure 4 This is the second flowchart of the test oracle generation method provided in the embodiments of this application;

[0081] Figure 5 The third flowchart of the test oracle generation method provided in this application embodiment;

[0082] Figure 6 The fourth flowchart of the test oracle generation method provided in this application embodiment;

[0083] Figure 7 The fifth step of the test oracle generation method provided in this application embodiment;

[0084] Figure 8 This is a schematic diagram of the test oracle generation device provided in the embodiments of this application;

[0085] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0086] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0087] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0088] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0089] Reference Figure 1 As shown, Figure 1 This is a schematic diagram of the business process for the test oracle generation method provided in the application embodiment. For example... Figure 1 As shown in the illustration, the business process diagram of the test oracle generation method provided in this application embodiment includes:

[0090] 1. Align the application's user interface knowledge base 11 with the user interface events 12 generated during the testing of the application to generate at least one test step 13 described in natural language.

[0091] 2. Retrieve the user interface and control operations of at least one test step 13 in the mind map data knowledge base 14 to obtain multiple most similar nodes, and obtain retrieval results 14 based on the multiple most similar nodes.

[0092] 3. Construct model prompt words 15 based on the search results 14 and at least one test step 13.

[0093] 4. Input the model prompt word 15 into the test oracle generation model 16, and obtain the test oracle of the application output by the test oracle generation model 16.

[0094] Based on the above, this application provides a method for generating test oracles, referring to... Figure 2 As shown, the test oracle generation method includes the following steps S21 to S26:

[0095] S21. Generate at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application.

[0096] During the testing of the application, multiple user interface events may be generated. Each user interface event may include: the operation type (e.g., click, input, long press, swipe, etc.), the operation object (a control in the user interface), and a screenshot of the user interface that generated the event. Furthermore, user interface events are generally described using attribute values ​​and Android class descriptions.

[0097] The application's user interface knowledge base can include: user interface annotation information corresponding to each user interface of the application and control annotation information corresponding to the controls in each user interface of the application. The user interface annotation information can include the interface structure and name of the user interface, and the control annotation information can include screenshots of the controls and their names.

[0098] Since user interface events are generally described by attribute values ​​and Android classes, and the test brain Figure 1 Generally, these are freely written documents with no fixed format. Therefore, by generating test steps described in natural language based on the application's user interface knowledge base and the user interface events generated during the testing of the application, the test steps and test mind maps can be semantically and logically matched and associated to achieve more accurate knowledge base retrieval and subsequent processing.

[0099] S22. Search the user interface and control operations of each test step in the mind map data knowledge base to obtain the most similar nodes.

[0100] The mind map data knowledge base stores data on each node in the test mind map of the application.

[0101] A test mind map is a test document written and stored in a tree structure, describing the goals, process, and expected performance of the testing work. The mind map information implicitly contains precise application design information, testing information, and testing requirements; therefore, using test mind maps can enhance the accuracy of generated predictions, making them more closely aligned with testing needs.

[0102] For example, when at least one test step, described in natural language, includes: 1. Performing operation 1 on control a on user interface A, 2. Performing operation 2 on control b on user interface A, and 3. Performing operation 1 on control c on user interface B, then the following objects will be used as retrieval input to retrieve the most similar node in the mind map data knowledge base:

[0103] 1. User Interface A;

[0104] 2. Perform operation 1 on control a;

[0105] 3. Perform operation 2 on control b;

[0106] 4. User Interface B;

[0107] 5. Perform operation 1 on control c.

[0108] In some embodiments, each test operation and each user interface can be used as retrieval input, and the similarity between the retrieval input and each node stored in the mind map data knowledge base can be calculated. The node with the highest similarity to the retrieval input is then selected as the most similar node to the retrieval input. The similarity calculation between the retrieval input and nodes can employ similarity algorithms such as cosine similarity of vector controls, string-based edit distance algorithms, and deep learning pre-trained model similarity calculation methods.

[0109] S23. Obtain the mind map path corresponding to each most similar node.

[0110] The mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to that most similar node.

[0111] Since the mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to the most similar node, obtaining the mind map path corresponding to each most similar node is: for each most similar node, obtain the mind map path from the root node of the test mind map to the most similar node.

[0112] For example, refer to Figure 3 As shown, node A is the root node of the test mind map. If node E is the most similar node to a certain test operation or user interface, since the path from node A to node E is: node A → node B → node C → node E, the mind map path corresponding to node E is: node A → node B → node C → node E.

[0113] S24. Generate search results based on the mind map paths corresponding to each most similar node.

[0114] In some embodiments, generating search results based on the mind map paths corresponding to each most similar node may include: directly combining the mind map paths corresponding to the most similar nodes to generate the search results.

[0115] In other embodiments, generating the search results based on the mind map paths corresponding to each most similar node includes: filtering and / or supplementing the mind map paths corresponding to each most similar node, and generating the search results based on the results of the filtering and / or supplementing.

[0116] S25. Generate model prompts based on the search results and the test steps.

[0117] In some embodiments, generating model suggestion words based on the search results and the test steps includes adding the search results and the test steps as additional information to the model suggestion words.

[0118] S26. Input the model prompt words into the test oracle generation model, and obtain the test oracle of the application output by the test oracle generation model.

[0119] For example, the test oracle generation model in this application embodiment can be a Large Language Model (LLM). That is, the model prompt words are input into the Large Language Model, and the test oracle of the test application is obtained from the output of the Large Language Model.

[0120] The test oracle generation method provided in this application first generates at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application. Then, it searches the mind map data knowledge base for the user interface and control operations of each test step to obtain multiple most similar nodes. Next, it obtains the mind map paths corresponding to each most similar node and generates search results based on these paths. Finally, it generates model hints based on the search results and the at least one test step, and inputs these model hints into a test oracle generation model to obtain the test oracle of the application output by the model. Because the test oracle generation method provided in this application can automatically generate test steps described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application, and then search the mind map data knowledge base for the most similar nodes of the user interface and control operations of the test steps, obtain their mind map paths to generate search results, and add the search results to the model hints, this application can automatically and accurately generate additional information for the model hints, thereby automatically generating test oracles and improving the testing efficiency of the application.

[0121] Reference Figure 4 As shown, in some embodiments, the implementation of generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application may include the following steps:

[0122] S401. Obtain screenshots of each control to be operated based on the user interface events generated during the testing of the application.

[0123] That is, to determine the screenshots of each control that needs to be operated in the test steps.

[0124] S402. Based on the screenshots of each control to be operated and the user interface knowledge base, determine the control name of each control to be operated.

[0125] In some embodiments, determining the control name of each control to be operated based on the control screenshots and the user interface knowledge base may include the following steps a and b:

[0126] Step a: Input the screenshot of the first control and the control names of each control in the user interface knowledge base into the first network model, and obtain the similarity between the screenshot of the first control and each control name output by the first network model.

[0127] Wherein, the first control is any control to be operated, and the first network model is a model obtained by training a Contrastive Language-Image Pretraining (CLIP) model.

[0128] CLIP model is a contrastive language image pre-training model that can map images (control screenshots) and text (control names) to embedding vectors in the same modality. Therefore, in this embodiment, the similarity between control screenshots and control names can be obtained based on the first network model.

[0129] Step b: Name the first control based on the control name that has the greatest similarity to the screenshot of the first control.

[0130] S403. Generate at least one test step based on the control name of each control to be operated.

[0131] Reference Figure 5 As shown, in some embodiments, the implementation of generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application may include the following steps:

[0132] S501. Obtain the interface structure of each user interface to be operated based on the user interface events generated during the testing of the application.

[0133] That is, determine the interface structure of the user interface that needs to be operated in the test steps.

[0134] S502. Based on the interface structure of each user interface to be operated, obtain the user interface annotation information corresponding to each user interface to be operated from the user interface knowledge base.

[0135] The interface annotation information includes: interface structure and user interface name.

[0136] In some embodiments, based on the interface structure of each user interface to be operated, the user interface annotation information corresponding to each user interface to be operated is obtained from the user interface knowledge base, including the following steps 1 to 4:

[0137] Step 1: Obtain the structure vectors of each user interface to be operated based on the second network model.

[0138] The second network model is the model obtained by training the InfoGraph model.

[0139] The InfoGraph model is a model for information propagation and representation learning of graph-structured data, which can convert the interface structure of a user interface into a corresponding structure vector.

[0140] Step 2: Obtain the structure vector of the page structure based on the annotation information of each interface based on the second network model.

[0141] Step 3: Obtain the similarity between each user interface to be operated and the annotation information of each interface.

[0142] The similarity between the user interface to be operated and the interface annotation information is the similarity between the structure vector of the page structure of the user interface to be operated and the structure vector of the page structure of the control annotation information.

[0143] Step 4: Determine the interface annotation information that has the highest similarity to each interface to be operated as the interface annotation information corresponding to each interface to be operated.

[0144] S503. Determine the interface name of each user interface to be operated based on the interface name in the interface label information corresponding to each user interface to be operated.

[0145] In some embodiments, determining the interface name of each user interface to be operated based on the interface annotation information corresponding to each user interface to be operated includes: naming the user interface to be operated as the interface name in the corresponding interface annotation information.

[0146] S504. Generate at least one test step based on the interface name of each user interface to be operated.

[0147] Reference Figure 6 As shown, in some embodiments, obtaining search results described in natural language may include the following steps:

[0148] S601. Search the user interface and control operations of each test step in the mind map data knowledge base to obtain multiple most similar nodes.

[0149] The mind map data knowledge base stores data on each node in the test mind map of the application.

[0150] S602. Obtain the mind map path corresponding to each most similar node.

[0151] The mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to that most similar node.

[0152] The implementation methods of steps S601 and S602 can refer to steps S22 and S23 above. To avoid redundancy, they will not be repeated here.

[0153] S603. Obtain the retrieval score of the mind map path corresponding to each most similar node.

[0154] The retrieval score for any mind map path is the average similarity between each node in that retrieval path and its corresponding test operation or user interface.

[0155] For example, the mind map path corresponding to a most similar node includes four nodes A, B, C, and D, and the similarity between A, B, C, and D and the corresponding test operation or user interface is 0.6, 0.75, 0.85, and 0.9, respectively. Then the retrieval score of the mind map path is (0.6+0.75+0.85+0.9) / 4=0.775.

[0156] S604. Select the preset number of mind map paths with the highest scores from the mind map paths corresponding to each most similar node.

[0157] In some embodiments, the mind map paths corresponding to each most similar node can be sorted in descending order according to the search score, and a preset number of mind map paths with the highest search score can be selected from the sorted results to obtain a preset number of mind map paths with the highest search score among the selected mind map paths corresponding to the most similar nodes.

[0158] S605. Obtain the node tree corresponding to the preset number of mind map paths respectively.

[0159] In this context, the node tree corresponding to any mind map path is a node tree with the last node of that mind map path as its root node.

[0160] That is, the node tree corresponding to the mind map path includes all child nodes of the last node of the mind map path.

[0161] S606. Complete the preset number of mind map paths based on the node tree corresponding to the preset number of mind map paths to obtain a preset number of completion retrieval results.

[0162] In some embodiments, the preset number of mind map paths are completed based on the node trees corresponding to the preset number of mind map paths to obtain a preset number of completion retrieval results, including the following steps ① to ③:

[0163] Step ①: Determine whether the number of nodes in the node tree corresponding to the preset number of mind map paths is greater than a preset threshold.

[0164] In step ① above, if the number of nodes in the node tree corresponding to the first mind map path in the preset number of mind map paths is less than or equal to the preset threshold, then step ② is executed as follows:

[0165] Step 2: Supplement the first mind map path based on the node tree corresponding to the first mind map path to obtain the completion search results corresponding to the first mind map path.

[0166] In some embodiments, supplementing the first mind map path based on the node tree corresponding to the first mind map path includes: supplementing the node tree corresponding to the first mind map path to the end of the first mind map path.

[0167] In step ① above, if the number of nodes in the node tree corresponding to the first mind map path in the preset number of mind map paths is greater than the preset threshold, then step ③ is executed as follows:

[0168] Step 3: Obtain the number of nodes in the child node tree of each node in the node tree corresponding to the second mind map path, and select the preset threshold number of nodes with the largest number of nodes in the child node tree of the node tree corresponding to the second mind map path to supplement the second mind map path, so as to obtain the completion search result corresponding to the second mind map path.

[0169] Since the node tree corresponding to the mind map path is a node tree with the last node of the mind map path as the root node, the node tree corresponding to the mind map path can describe the application behavior after the mind map path, covering the expected performance of the application in different aspects and under different conditions. Therefore, by completing the preset number of mind map paths based on the node trees corresponding to the preset number of mind map paths respectively, a preset number of completion search results can be obtained to provide more complete information.

[0170] S607. Combine the preset number of completion search results to obtain the search results.

[0171] Since the above embodiments require retrieval from a mind map data knowledge base, the test prediction generation method provided in this application further includes constructing a mind map data knowledge base before performing the retrieval. In some embodiments, constructing the mind map data knowledge base includes: obtaining metadata of each node in the test mind map, and constructing the mind map data knowledge base based on the metadata of each node in the test mind map.

[0172] That is, the mind map data knowledge base in this application embodiment will store each node in each test mind map separately, and the element data of each node may include: the test mind map to which the node belongs, the position of the node in the test mind map, and the number of nodes in the node tree with the node as the root node.

[0173] Since the above embodiments require generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application, the test oracle generation method provided in this application further includes acquiring user interface events generated during the testing of the application before generating the at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application. (Refer to...) Figure 7 As shown, in some embodiments, the interface annotation information of the application can be obtained through the following steps:

[0174] S701. Obtain the user interface screenshots and user interface structure files of the application generated during the testing process.

[0175] S702. Extract controls from the user interface screenshot to obtain a first set of controls.

[0176] In some embodiments, controls can be extracted from user interface screenshots based on computer vision (CV) to obtain a first set of controls.

[0177] For example, the first set of controls may include all recognizable controls in the user interface screenshot, or it may include only the important interactive controls in the user interface screenshot.

[0178] S703. Extract controls from the user interface structure file to obtain a second set of controls.

[0179] S704. Merge the first control set and the second control set to obtain the third control set.

[0180] That is, duplicate controls are deleted from the control set (first control set) obtained by extracting controls from the user interface screenshot and the control set (second control set) obtained by extracting controls from the user interface structure file.

[0181] In some embodiments, merging the first set of controls and the second set of controls includes: if the first control and the second control are duplicated, determining the attribute information contained in the first control and the second control, and retaining the control with more attribute information.

[0182] S705. Merge the controls in the third control set based on the positional relationship and / or size between the controls to obtain a fourth control set.

[0183] In some embodiments, controls in the third control set are merged based on their positional relationships and / or size to obtain a fourth control set, including:

[0184] Calculate the overlap rate of all controls in the third control set, and merge controls with an overlap rate exceeding a certain threshold into a control group.

[0185] For example, if the view of control B is overlaid on the view of control A, then control A and control B can be merged into a single control group.

[0186] In some embodiments, controls in the third control set are merged based on their positional relationships and / or size to obtain a fourth control set, including:

[0187] Group controls that are similar in size, located in the same horizontal area, and are less than a threshold distance apart into a single control group.

[0188] For example, all controls at the bottom of the user interface can be merged into one control group.

[0189] S706. Mark each control in the fourth control set in the user interface screenshot to obtain interface annotation information.

[0190] S707. Generate the user interface event based on the interface annotation information.

[0191] In some embodiments, before merging the first control set and the second control set to obtain the third control set, the test oracle generation method provided in this application further includes: performing consistency detection on the user interface screenshot and the user interface structure file.

[0192] In some embodiments, the implementation of consistency detection between the user interface screenshot and the user interface structure file may include: determining a first ratio in which controls in the user interface structure file are identified in the user interface screenshot, and a second ratio in which controls identified in the user interface screenshot are successfully matched in the user interface structure file; if both the first ratio and the second ratio are less than a preset ratio, then the user interface screenshot and the user interface structure file are determined to be mismatched; otherwise, the user interface screenshot and the user interface structure file are determined to be matched.

[0193] In some embodiments, the test oracle generation provided in this application further includes: adding the interface annotation information to the model prompts before inputting the model prompts into the test oracle generation model.

[0194] That is, the model prompts include: test steps described in natural language, search results, and application interface annotation information.

[0195] Based on the same inventive concept, as an implementation of the above method, this application embodiment also provides a test oracle generation device. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the test oracle generation device in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0196] This application provides a test oracle generation apparatus. Figure 8This is a schematic diagram of the test oracle generation device, as shown below. Figure 8 As shown, the test oracle generation device 800 includes:

[0197] Alignment unit 81 is used to generate at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application.

[0198] The retrieval unit 82 is used to search the user interface and control operations of each test step in the mind map data knowledge base to obtain multiple most similar nodes. The mind map data knowledge base stores data of each node in the test mind map of the application.

[0199] Processing unit 83 is used to obtain the mind map path corresponding to each most similar node; the mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to the most similar node, and to generate search results based on the mind map path corresponding to each most similar node.

[0200] The generation unit 84 is configured to generate model prompt words based on the search results and the at least one test step, input the model prompt words into the test oracle generation model, and obtain the test oracle of the application output by the test oracle generation model.

[0201] As an optional implementation of this application, the alignment unit 81 is specifically used to obtain control screenshots of each control to be operated based on user interface events generated during the testing of the application; obtain control annotation information corresponding to each control to be operated from the user interface knowledge base based on the control screenshots of each control to be operated, the control annotation information including: control name; determine the control name of each control to be operated based on the control annotation information corresponding to each control to be operated; and generate the at least one test step based on the control name of each control to be operated.

[0202] As an optional implementation of this application, the alignment unit 81 is specifically used to input the control screenshot of the first control and the control name of each control annotation information in the user interface knowledge base into the first network model, and obtain the similarity between the control screenshot of the first control output by the first network model and each control name, and name the first control according to the control name with the highest similarity to the control screenshot of the first control.

[0203] Wherein, the first control is any control to be operated, and the first network model is a model obtained by training a contrastive language image pre-trained CLIP model.

[0204] As an optional implementation of this application, the alignment unit 81 is specifically used to obtain the interface structure of each user interface to be operated based on the user interface events generated during the testing of the application; obtain the user interface annotation information corresponding to each user interface to be operated from the user interface knowledge base based on the interface structure of each user interface to be operated, wherein the user interface annotation information includes: interface structure and user interface name; determine the interface name of each user interface to be operated based on the interface annotation information corresponding to each user interface to be operated; and generate the at least one test step based on the interface name of each user interface to be operated.

[0205] As an optional implementation of this application, the alignment unit 81 is specifically used to obtain the structure vector of the interface structure of each user interface to be operated based on the second network model, wherein the second network model is a model obtained by training the InfoGraph model; obtain the structure vector of the page structure of each interface annotation information based on the second network model; obtain the similarity between each user interface to be operated and each interface annotation information, wherein the similarity between the user interface to be operated and the interface annotation information is the similarity between the structure vector of the page structure of the user interface to be operated and the structure vector of the page structure of the control annotation information; and determine the interface annotation information with the highest similarity to each user interface to be operated as the interface annotation information corresponding to each user interface to be operated.

[0206] As an optional implementation of this application, the retrieval unit 82 is further configured to obtain the metadata of each node in the test mind map before retrieving the most similar nodes of each test operation and each user interface in the mind map data knowledge base. The metadata of any node includes at least one of the following: the test mind map to which the node belongs, the position of the node in the test mind map, and the number of nodes in the node tree with the node as the root node; and to construct the mind map data knowledge base based on the metadata of each node in the test mind map.

[0207] As an optional implementation of this application, the processing unit 83 is specifically used to obtain the retrieval score of the mind map path corresponding to each most similar node; the retrieval score of any mind map path is the average similarity between each node in the retrieval path and the corresponding test operation or user interface; select a preset number of mind map paths with the highest retrieval score among the mind map paths corresponding to each most similar node; obtain the node tree corresponding to the preset number of mind map paths respectively; the node tree corresponding to any mind map path is a node tree with the last node of the mind map path as the root node; complete the preset number of mind map paths based on the node trees corresponding to the preset number of mind map paths respectively to obtain a preset number of completed retrieval results; combine the preset number of completed retrieval results to obtain the retrieval result.

[0208] As an optional implementation of this application, the processing unit 83 is specifically used to determine whether the number of nodes in the node tree corresponding to the preset number of mind map paths is greater than a preset threshold; if the number of nodes in the node tree corresponding to the first mind map path in the preset number of mind map paths is less than or equal to the preset threshold, then the first mind map path is supplemented based on the node tree corresponding to the first mind map path to obtain the completion search result corresponding to the first mind map path; if the number of nodes in the node tree corresponding to the second mind map path in the preset number of mind map paths is greater than the preset threshold, then the number of nodes in the child node tree of each node in the node tree corresponding to the second mind map path is obtained, and the preset threshold number of nodes with the largest number of nodes in the child node tree of the node tree corresponding to the second mind map path are selected to supplement the second mind map path to obtain the completion search result corresponding to the second mind map path.

[0209] As an optional implementation of this application, the alignment unit 81 is further configured to: acquire a user interface screenshot generated during the testing of the application and a user interface structure file of the application before generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application; extract controls from the user interface screenshot to obtain a first control set; extract controls from the user interface structure file to obtain a second control set; merge the first control set and the second control set to obtain a third control set; merge the controls in the third control set based on the positional relationship and / or size between the controls to obtain a fourth control set; annotate each control in the fourth control set in the user interface screenshot to obtain interface annotation information; and generate the user interface event based on the interface annotation information.

[0210] As an optional implementation of this application, the alignment unit 81 is further configured to perform a consistency check on the user interface screenshot and the user interface structure file before merging the first control set and the second control set to obtain the third control set.

[0211] As an optional implementation of this application, the generation unit 84 is further configured to add the interface annotation information to the model prompt words before inputting the model prompt words into the test oracle to generate the model.

[0212] The test oracle generation apparatus provided in this application embodiment can execute the test oracle generation method provided in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0213] Based on the same inventive concept, embodiments of this application also provide an electronic device. Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 9 As shown, the electronic device provided in this embodiment includes a memory 901 and a processor 902. The memory 901 is used to store a computer program, and the processor 902 is used to execute the test oracle generation method provided in the above embodiment when executing the computer program.

[0214] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the computing device to implement the test oracle generation method provided in the above embodiments.

[0215] Based on the same inventive concept, this application also provides a computer program product that, when run on a computer, enables the computing device to implement the test oracle generation method provided in the above embodiments.

[0216] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0217] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0218] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0219] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating test oracles, characterized in that, include: Generate at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application; The user interface and control operations for each test step are searched in the mind map data knowledge base to obtain multiple most similar nodes. The mind map data knowledge base stores data of each node in the test mind map of the application. Obtain the mind map path corresponding to each most similar node; the mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to that most similar node; Retrieval results are generated based on the mind map paths corresponding to each most similar node; Generate model suggestion words based on the search results and at least one test step; Input the model prompt words into the test oracle generation model, and obtain the test oracle of the application output by the test oracle generation model.

2. The method according to claim 1, characterized in that, The step of generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application includes: Based on the user interface events generated during the testing of the application, obtain screenshots of each control to be operated. Based on the screenshots of each control to be operated and the user interface knowledge base, determine the control name of each control to be operated; At least one test step is generated based on the control name of each control to be operated.

3. The method according to claim 2, characterized in that, The step of obtaining control annotation information corresponding to each control to be operated from the user interface knowledge base based on the control screenshots of each control to be operated includes: Input the screenshot of the first control and the control names of each control in the user interface knowledge base into the first network model, and obtain the similarity between the screenshot of the first control and each control name output by the first network model. The first control is any control to be operated, and the first network model is a model obtained by training a contrastive language image pre-trained CLIP model. The first control is named according to the control name that has the greatest similarity to the screenshot of the first control.

4. The method according to claim 1, characterized in that, The step of generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application includes: The interface structure of each user interface to be operated is obtained based on the user interface events generated during the testing of the application. Based on the interface structure of each user interface to be operated, obtain the user interface annotation information corresponding to each user interface to be operated from the user interface knowledge base. The user interface annotation information includes: interface structure and user interface name. The interface name of each user interface to be operated is determined based on the interface name in the interface label information corresponding to each user interface to be operated. At least one test step is generated based on the interface name of each user interface to be operated.

5. The method according to claim 4, characterized in that, The step of obtaining user interface annotation information corresponding to each user interface to be operated from the user interface knowledge base according to the interface structure of each user interface to be operated includes: The structure vectors of the interface structure of each user interface to be operated are obtained based on the second network model, which is the model obtained by training the InfoGraph model. Based on the second network model, obtain the structural vector of the page structure of each interface annotation information; Obtain the similarity between each user interface to be operated and each interface annotation information. The similarity between the user interface to be operated and the interface annotation information is the similarity between the structure vector of the page structure of the user interface to be operated and the structure vector of the page structure of the control annotation information. The interface annotation information with the highest similarity to each interface to be operated is determined as the interface annotation information corresponding to each interface to be operated.

6. The method according to claim 1, characterized in that, Before retrieving the most similar nodes for each test operation and each user interface in the mind map data knowledge base, the method further includes: Obtain the metadata of each node in the test mind map. The metadata of any node includes at least one of the following: the test mind map to which the node belongs, the position of the node in the test mind map, and the number of nodes in the node tree with the node as the root node. The mind map data knowledge base is constructed based on the metadata of each node in the test mind map.

7. The method according to claim 1, characterized in that, The process of generating the search results based on the mind map paths corresponding to each most similar node includes: Obtain the retrieval score of the mind map path corresponding to each most similar node; the retrieval score of any mind map path is the average similarity between each node in the retrieval path and the corresponding test operation or user interface. Select the preset number of mind map paths with the highest scores from the mind map paths corresponding to each most similar node; Obtain the node tree corresponding to the preset number of mind map paths respectively; the node tree corresponding to any mind map path is a node tree with the last node of the mind map path as the root node; The preset number of mind map paths are completed based on the node trees corresponding to the preset number of mind map paths, so as to obtain a preset number of completion search results; The search results are obtained by combining the preset number of completion search results.

8. The method according to claim 7, characterized in that, The step of completing the preset number of mind map paths based on the node trees corresponding to the preset number of mind map paths to obtain a preset number of completion retrieval results includes: Determine whether the number of nodes in the node tree corresponding to the preset number of mind map paths is greater than a preset threshold. If the number of nodes in the node tree corresponding to the first mind map path in the preset number of mind map paths is less than or equal to the preset threshold, then the first mind map path is supplemented based on the node tree corresponding to the first mind map path to obtain the completion search result corresponding to the first mind map path. If the number of nodes in the node tree corresponding to the second mind map path in the preset number of mind map paths is greater than the preset threshold, then the number of nodes in the child node tree of each node in the node tree corresponding to the second mind map path is obtained, and the preset threshold number of nodes with the largest number of nodes in the child node tree of the node tree corresponding to the second mind map path is selected to supplement the second mind map path, so as to obtain the completion search result corresponding to the second mind map path.

9. The method according to claim 1, characterized in that, Before generating at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application, the method further includes: Obtain user interface screenshots and user interface structure files of the application generated during the testing process; Extract controls from the user interface screenshot to obtain a first set of controls; Extract controls from the user interface structure file to obtain a second set of controls; Merge the first control set and the second control set to obtain a third control set; The controls in the third control set are merged based on their positional relationships and / or size to obtain a fourth control set. Annotate each control in the fourth control set in the user interface screenshot to obtain interface annotation information; The user interface events are generated based on the interface annotation information.

10. The method according to claim 9, characterized in that, Before merging the first control set and the second control set to obtain the third control set, the method further includes: Perform a consistency check on the user interface screenshots and the user interface structure files.

11. The method according to claim 1, characterized in that, Before inputting the model prompt words into the test oracle generation model, the method further includes: Add the interface annotation information to the model prompts.

12. A test oracle generation device, characterized in that, include: Alignment unit, used to generate at least one test step described in natural language based on the application's user interface knowledge base and user interface events generated during the testing of the application; The retrieval unit is used to search the user interface and control operations of each test step in the mind map data knowledge base to obtain multiple most similar nodes. The mind map data knowledge base stores data of each node in the test mind map of the application. The processing unit is used to obtain the mind map path corresponding to each most similar node; the mind map path corresponding to any most similar node is the mind map path from the root node of the test mind map to the most similar node, and to generate search results based on the mind map path corresponding to each most similar node. A generation unit is configured to generate model hint words based on the search results and the at least one test step, input the model hint words into a test oracle generation model, and obtain the test oracle of the application output by the test oracle generation model.

13. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program and the processor being used to cause the electronic device to implement the test oracle generation method according to any one of claims 1-11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the test oracle generation method according to any one of claims 1-11.

15. A computer program product, characterized in that, When the computer program product is run on a computer, the computer enables the computer to implement the test oracle generation method according to any one of claims 1-11.

Images