Test case generation method and device, problem scenario automatic reproduction method and device

Abstract

The present invention discloses a test case generation method and device, the method comprising: obtaining a user interface screenshot of the software to be tested; using a pre-trained interface element recognition model to recognize the user interface screenshot, and obtaining each control in the user interface screenshot and related information of the control; generating an interface test case set according to each control in the user interface screenshot and related information. The present invention also discloses a method and device for automatically reproducing a problem scenario. By using the present invention, the efficiency of test case generation can be improved, and a comprehensive test of the user interface can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a test case generation method and device, and also to a problem scenario automatic reproduction method and device. Background Art

[0002] A Graphical User Interface (GUI) is a computer operation user interface that is displayed in graphical form. It is an interface display format for communication between people and computers, allowing users to use input devices such as a mouse to manipulate icons or menu options on the screen to select commands, call files, start programs, or perform other daily tasks.

[0003] Existing tests for GUIs usually require manual design of test cases based on the interface control type and layout, and define the tested object, test method, expected results, etc., and then the test terminal executes the test case to complete the GUI test. Even if automated testing is used, that is, the machine simulates the user's operation on the GUI to test, it is necessary to rely on manually designed test cases in advance and specify the operation elements for execution, or to obtain elements and execute test cases by analyzing the program source code. Since GUI software contains a large number of windows, and each window contains many elements, these windows and different combinations of elements therein constitute different functions of the software. The diversity of windows and elements in GUI software and the complexity of the combination make it often take a lot of time to manually write test cases, which is inefficient. Summary of the invention

[0004] On one hand, an embodiment of the present invention provides a test case generation method and device to improve the test case generation efficiency.

[0005] Another aspect of the embodiments of the present invention provides a method and device for automatically reproducing a problem scenario, which can automatically reproduce the problem scenario.

[0006] To this end, the present invention provides the following technical solutions:

[0007] A test case generation method, the method comprising:

[0008] Get a screenshot of the user interface of the software under test;

[0009] Using a pre-trained interface element recognition model to recognize the user interface screenshot, and obtaining each control in the user interface screenshot and related information of the control; the related information of the control includes the coordinates and type of the control;

[0010] Generate an interface test case set based on each control and related information in the user interface screenshot.

[0011] Optionally, the method further includes the step of training the interface element recognition model, including:

[0012] Determine the topological structure of the interface element recognition model;

[0013] Collect a large number of software interface screenshots as training interface screenshots;

[0014] Annotate the controls in the screenshot of the training interface, where the annotation information includes the coordinates and type of the controls;

[0015] The interface screenshots and their annotation information are used to train an interface element recognition model.

[0016] Optionally, the annotation information also includes: an operation set corresponding to the control.

[0017] Optionally, the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to a combination of controls;

[0018] Generate a subset of independent test cases corresponding to each control, including:

[0019] Obtaining an operation set corresponding to the type of each control in the user interface screenshot, and generating an independent test case corresponding to the control according to the operation set corresponding to the type of each control and the coordinates of the control;

[0020] Adding the independent test case to the independent test case collection corresponding to each control;

[0021] A subset of combined test cases that generate corresponding control combinations include:

[0022] Performing full permutations and combinations on all controls in the user interface screenshot to obtain various combinations;

[0023] Determine the order of operations on different controls in each combination;

[0024] Calling a subset of independent test cases corresponding to each control in sequence according to the operation sequence to generate a combination test case corresponding to the operation sequence under the combination;

[0025] The combined test case is added to the combined test case collection of the corresponding control combination.

[0026] Optionally, generating an independent test case corresponding to each control according to an operation set corresponding to the type of each control and the coordinates of the control includes:

[0027] Obtain an executable script template corresponding to each operation in an operation set corresponding to the type to which the control belongs, fill the coordinates of the control into the executable script template, and generate an independent test case corresponding to the control.

[0028] Optionally, the method further comprises:

[0029] Using each test case in the interface test case set to test the user interface to obtain a test result;

[0030] Obtaining a screenshot of the test result;

[0031] Establish and save the correspondence between each test case and the screenshot of the test results.

[0032] A method for automatically reproducing a problem scenario, the method comprising:

[0033] Obtain a screenshot of the problem scenario provided by the user as the problem screenshot;

[0034] Using a pre-trained interface element recognition model to recognize the problematic screenshot, obtaining each control in the problematic screenshot and related information of the control; the related information of the control includes the coordinates and type of the control;

[0035] Generate a problem test case set according to each control in the problem screenshot and related information of the control;

[0036] The problem scenario is reproduced using the problem test case set.

[0037] Optionally, the reproducing the problem scenario by using the problem test case set includes:

[0038] Using each test case in the problem test case set to test the problem scenario, and obtaining a screenshot of the problem scenario test result;

[0039] Select a problem reproduction scenario screenshot from the screenshots of the problem scenario test results;

[0040] Get the test case corresponding to the screenshot of the problem reproduction scenario.

[0041] Optionally, selecting a problem reproduction scenario screenshot from the screenshots of the problem scenario test result includes:

[0042] Calculate the similarity between the screenshots of the test results of each problem scenario and the screenshots of the problem;

[0043] The screenshots of the problem scenario test results with a similarity greater than a set threshold are used as the problem reproduction scenario screenshots; or

[0044] A set number of screenshots of the problem scenario test results are selected from large to small according to similarity as the problem reproduction scenario screenshots.

[0045] A test case generating device, the device comprising:

[0046] A screenshot acquisition module is used to obtain a screenshot of the user interface of the software under test;

[0047] An identification module, used to identify the user interface screenshot using a pre-trained interface element identification model, and obtain each control in the user interface screenshot and related information of the control; the related information of the control includes the coordinates and type of the control;

[0048] The interface test case generation module is used to generate an interface test case set according to each control in the user interface screenshot and its related information.

[0049] Optionally, the device further includes a model training module for training an interface element recognition model; the model training module includes:

[0050] A structure determination unit, used to determine the topological structure of the interface element recognition model;

[0051] An information collection unit, used to collect a large number of software interface screenshots as training interface screenshots;

[0052] A labeling unit, used to label the controls in the screenshot of the training interface, wherein the labeling information includes the coordinates and type of the controls;

[0053] The training unit is used to train an interface element recognition model using the interface screenshots and their annotation information.

[0054] Optionally, the annotation information also includes: an operation set corresponding to the control.

[0055] Optionally, the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to a combination of controls;

[0056] The interface test case generation module includes: a first subset generation module for generating a subset of independent test cases corresponding to each control and a second subset generation module for generating a subset of combined test cases corresponding to a combination of controls;

[0057] The first subset generation module comprises:

[0058] An independent test case generating unit, used to obtain an operation set corresponding to the type of each control in the user interface screenshot, and generate an independent test case corresponding to the control according to the operation set corresponding to the type of each control and the coordinates of the control;

[0059] An independent test case adding unit, which adds the independent test case to the independent test case collection corresponding to each control;

[0060] The second subset generation module comprises:

[0061] A combination unit, used for performing full permutation and combination of all controls in the user interface screenshot to obtain various combinations;

[0062] An operation sequence determination unit, used to determine the operation sequence of different controls in each combination;

[0063] A calling unit, used to call a subset of independent test cases corresponding to each control in sequence according to the operation sequence, and generate a combined test case corresponding to the operation sequence under the combination;

[0064] The combination test case adding unit is used to add the combination test case to the combination test case collection of the corresponding control combination.

[0065] Optionally, the independent test case generation unit is specifically used to obtain an executable script template corresponding to each operation in an operation set corresponding to the type of the control, fill the coordinates of the control into the executable script template, and generate an independent test case corresponding to the control.

[0066] Optionally, the device further comprises: a test case execution module and a corresponding relationship generation module;

[0067] The test case execution module is used to test the user interface using each test case in the interface test case set to obtain a test result;

[0068] The screenshot acquisition module is further used to obtain a screenshot of the test result;

[0069] The corresponding relationship generating module is used to establish and save the corresponding relationship between each test case and the screenshot of the test result.

[0070] A device for automatically reproducing a problem scenario, the device comprising:

[0071] A problem scenario acquisition module is used to acquire a screenshot of a problem scenario provided by a user as a problem screenshot;

[0072] An identification module, used to identify the problematic screenshot using a pre-trained interface element identification model, and obtain each control in the problematic screenshot and related information of the control;

[0073] A problem test case generation module, used to generate a problem test case set according to each control in the problem screenshot and related information of the control;

[0074] A scenario reproduction module is used to reproduce the problem scenario using the problem test case set.

[0075] Optionally, the scene reproduction module includes:

[0076] A testing unit, used to test the problem scenario using each test case in the problem test case set, and obtain a screenshot of the problem scenario test result;

[0077] A scenario screening unit, used to select a problem reproduction scenario screenshot from the screenshots of the problem scenario test results;

[0078] The test case acquisition unit is used to acquire the test case corresponding to the problem reproduction scenario screenshot.

[0079] Optionally, the scene screening unit includes:

[0080] A similarity calculation subunit, used to calculate the similarity between the screenshots of the test results of each problem scenario and the screenshots of the problem;

[0081] The screening subunit is used to use the screenshots of the problem scenario test results whose similarity is greater than a set threshold as the problem reproduction scenario screenshots; or to select a set number of screenshots of the problem scenario test results from large to small similarity as the problem reproduction scenario screenshots.

[0082] An electronic device, comprising: one or more processors and a memory;

[0083] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the aforementioned method.

[0084] A readable storage medium stores instructions thereon, wherein the instructions are executed to implement the above-mentioned method.

[0085] The test case generation method and device provided in the embodiments of the present invention are based on image detection and recognition technology, and use a pre-trained interface element recognition model to automatically identify controls and related information of the controls in a user interface screenshot, and then use the controls and related information to generate an interface test case set, thereby greatly improving the test case generation efficiency.

[0086] Furthermore, the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to various combinations of controls, so that the generated interface test case set includes both test cases corresponding to reasonable operations and test cases corresponding to abnormal operations, breaking the inertial thinking when manually designing test cases. These test cases can be used to better test the robustness of the software system.

[0087] Furthermore, the user interface can be tested using each test case in the interface test case set to obtain test results, and a screenshot of the test results can be obtained to establish and save a correspondence between each test case and the screenshot of the test result. Providing the correspondence to the tester can help the tester quickly determine whether each test result is correct.

[0088] The method and device for automatically reproducing problem scenarios provided by the embodiments of the present invention automatically recognize the screenshots of the problem scenarios provided by the user using the pre-trained interface element recognition model, and generate a problem test case set according to the recognition result, so that the tester can reproduce the problem scenario according to the test cases in the problem test case set to achieve problem backtracking. Compared with manual multiple clueless problem searches, it can quickly and effectively discover problems and then solve them. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0090] Figure 1 is a training flow chart of the interface element recognition model in an embodiment of the present invention;

[0091] Figure 2 It is a flow chart of a test case generation method according to an embodiment of the present invention;

[0092] Figure 3 This is a flow chart of a method for automatically reproducing a problem scenario according to an embodiment of the present invention;

[0093] Figure 4 is a flow chart for reproducing a problem scenario in an embodiment of the present invention;

[0094] Figure 5 It is a structural block diagram of a test case generating device according to an embodiment of the present invention;

[0095] Figure 6 It is another structural block diagram of the test case generating device according to an embodiment of the present invention;

[0096] Figure 7 This is a structural block diagram of a device for automatically reproducing a problem scenario according to an embodiment of the present invention;

[0097] Figure 8 is a structural block diagram of a scene reproduction module in an embodiment of the present invention;

[0098] Fig. 9is a block diagram of a device for a test case generation method according to an exemplary embodiment;

[0099] Fig.10 It is a schematic diagram of the structure of the server in the embodiment of the present invention. DETAILED DESCRIPTION

[0100] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.

[0101] In view of the problem of low efficiency in generating graphical user interfaces in the prior art, an embodiment of the present invention provides a test case generating method and device, which utilizes a pre-trained interface element recognition model to identify a screenshot of the user interface of the software to be tested, and obtains each control in the screenshot of the user interface and related information of the control; an interface test case set is generated according to each control in the screenshot of the user interface and related information, and each test case in the interface test case set can be used to comprehensively and effectively test the user interface, thereby improving the robustness of the software system.

[0102] In addition, an embodiment of the present invention also provides a method and device for automatically reproducing a problem scenario.

[0103] The interface element recognition model can adopt a neural network structure, such as a CNN (Convolutional Neural Network) structure. The specific training process is as follows: Figure 1 As shown, the following steps are included:

[0104] Step 101, determining the topological structure of the interface element recognition model.

[0105] Step 102, collect a large number of software interface screenshots as training interface screenshots.

[0106] Step 103, annotating the controls in the screenshot of the training interface, wherein the annotation information includes the coordinates and types of the controls.

[0107] For example, mark the search box, buttons, text links and other controls in the training interface screenshot, and mark the coordinates and type of each control.

[0108] Furthermore, the annotation information may also include: an operation set corresponding to the control.

[0109] Controls are encapsulations of data and methods. The content and style of each control can be set by the software developer, but the corresponding operation set for each type of control is fixed. For example, the operation set corresponding to Button is click; the operation set corresponding to Link is click; and the operation set corresponding to Text is input and delete. Therefore, after the type of each control is determined, the operation set of the control can be obtained.

[0110] Step 104: train an interface element recognition model using the interface screenshot and its annotation information.

[0111] The specific training method is similar to the existing neural network training method and will not be described in detail here.

[0112] By using the above-mentioned pre-trained interface element recognition model to recognize the user interface screenshot of the software to be tested, each control in the user interface screenshot and related information of the control can be obtained, and then the corresponding test case can be automatically generated in combination with the operation set corresponding to each control type.

[0113] like Figure 2 FIG. 1 is a flow chart of a test case generation method according to an embodiment of the present invention, comprising the following steps:

[0114] Step 201, obtaining a screenshot of the user interface of the software to be tested.

[0115] The user interface screenshot can be obtained through an existing screenshot tool or a photo taking tool, and the user interface screenshot is saved in a picture format, such as BMP format, TIF format, GIF format, JPEG format, etc., which is not limited in this embodiment of the present invention.

[0116] Step 202: Use a pre-trained interface element recognition model to recognize the user interface screenshot to obtain each control in the user interface screenshot and related information of the control.

[0117] When the user interface screenshot is identified using the interface element recognition model, the user interface screenshot can be input into the interface element recognition model, and each control in the user interface screenshot and related information of the control can be obtained according to the output of the interface element recognition model.

[0118] In an embodiment of the present invention, the controls identified by using the interface element recognition model may include any controls used in a graphical user interface, such as a text control, a link control, a slider control, a paging control, and the like.

[0119] The relevant information of the control includes, but is not limited to, information such as the coordinates and type of the control.

[0120] Step 203: Generate an interface test case set based on each control in the user interface screenshot and its related information.

[0121] In an embodiment of the present invention, the interface test case set may include a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to a combination of controls.

[0122] Among them, a subset of independent test cases corresponding to each control can be generated in the following way:

[0123] Obtain an operation set corresponding to the type of each control in the user interface screenshot, for example, the operation corresponding to button is click, the operation corresponding to text box is input, delete, etc., and generate independent test cases corresponding to the controls according to the operation set corresponding to the type of each control and the coordinates of the controls; then, add the independent test cases to the independent test case set corresponding to each control.

[0124] For different types of controls, corresponding executable script templates can be pre-set according to their corresponding operations. Accordingly, after obtaining the controls and their coordinate information in the user interface screenshot, the executable script template corresponding to each operation in the operation set corresponding to the type of the control is obtained, and the coordinates of the control are filled into the executable script template, and an independent test case corresponding to the control can be generated.

[0125] Among them, the subset of combined test cases corresponding to the control combinations may include test cases for normal operations of each combination according to the software design. Furthermore, in order to achieve comprehensive testing of the user interface and improve the robustness of the software system, all controls in the user interface screenshot can be fully arranged and combined, and corresponding test cases can be generated for all operations under the operation sequence in each combination. Specifically, the process of generating a subset of combined test cases corresponding to the control combinations is as follows:

[0126] (1) performing full permutation and combination of all controls in the user interface screenshot to obtain various combinations;

[0127] (2) Determine the operation order of different controls in each combination;

[0128] (3) calling a subset of independent test cases corresponding to the controls in sequence according to the operation sequence to generate a combination test case corresponding to the operation sequence under the combination;

[0129] (4) Add the combination test case to the combination test case collection of the corresponding control combination.

[0130] For example, a combination includes a control buttonA and an input box B. The operation sequence in this combination is: buttonA-->input box B. The independent test case subsets corresponding to buttonA and input box B are called respectively. The test cases corresponding to the above operation sequence in this combination include:

[0131] click A-->input B;

[0132] click A-->delete B;

[0133] click A-->input B-->delete B;

[0134] click A-->delete B-->input B.

[0135] The test case generation method provided by the embodiment of the present invention is based on image detection and recognition technology, and uses a pre-trained interface element recognition model to automatically identify controls and related information of the controls in the user interface screenshot, and then uses the controls and related information to generate an interface test case set, thereby greatly improving the test case generation efficiency.

[0136] Furthermore, the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to various combinations of controls. It can be seen from the above examples that not only all controls in the user interface screenshot are fully arranged and combined, but also for each combination, the subset of independent test cases corresponding to each control is called according to the operation sequence of each control to generate a combined test case corresponding to the operation sequence. In other words, the test cases under each combination cover all operation conditions under the operation sequence, so that the generated interface test case set contains both test cases corresponding to reasonable operations and test cases corresponding to abnormal operations, breaking the inertial thinking when manually designing test cases, making the test cases for the user interface richer and more comprehensive, so that these test cases can be used to better test the robustness of the software system.

[0137] Furthermore, in another embodiment of the test case generation method of the present invention, the user interface can also be tested using each test case in these interface test case sets (including each test case in the independent test case set corresponding to each control and each test case in the combined test case set corresponding to the control combination) to obtain test results, and obtain a screenshot of the test results, establish and save a correspondence between each test case and the screenshot of the test results, and provide the correspondence to the tester, which can help the tester quickly determine whether each test result is correct.

[0138] Correspondingly, an embodiment of the present invention further provides a method for automatically reproducing a problem scenario, which can reproduce the problem scenario provided by a user.

[0139] like Figure 3 FIG. 1 is a flow chart of a method for automatically reproducing a problem scenario according to an embodiment of the present invention, comprising the following steps:

[0140] Step 301: Obtain a screenshot of a problem scenario provided by a user as a problem screenshot.

[0141] The problem scenario refers to an abnormal situation that occurs in the user graphical interface when the software is running, that is, it is not the interface desired during design.

[0142] Step 302: Use a pre-trained interface element recognition model to recognize the problematic screenshot, and obtain each control in the problematic screenshot and related information of the control.

[0143] Step 303: Generate a problem test case set according to each control in the problem screenshot and related information of the control.

[0144] The above method of identifying the problem screenshots and generating a problem test case set based on the identification results is the same as the previous Figure 2 The process of generating an interface test case set for the user interface is similar and will not be described in detail here.

[0145] Step 304: Reproduce the problem scenario using the problem test case set.

[0146] When using the problem test case set to reproduce the problem scenario, it is necessary to use each test case in the problem test case set to test the problem scenario, find out the possible problem reproduction scenario from the test results, and obtain the test case corresponding to the problem reproduction scenario, so that the problem scenario can be reproduced using these test cases.

[0147] like Figure 4 FIG. 1 is a flowchart of reproducing a problem scenario in an embodiment of the present invention, comprising the following steps:

[0148] Step 401 , using each test case in the problem test case set generated according to the problem scenario to test the problem scenario, and obtaining a screenshot of the problem scenario test result.

[0149] For each test case in the problem test case set, a problem scenario test result can be obtained.

[0150] Step 402: Select a problem reproduction scenario screenshot from the screenshots of the problem scenario test results.

[0151] Specifically, the similarity between the screenshots of each problem scenario test result and the problem screenshot can be calculated; the screenshots of the problem scenario test results whose similarity is greater than a set threshold are used as the problem reproduction scenario screenshots; or a set number of screenshots of the problem scenario test results are selected from large to small in terms of similarity as the problem reproduction scenario screenshots.

[0152] In actual applications, one or more problem reproduction scenario screenshots may be selected in the above manner, which is not limited in this embodiment of the present invention.

[0153] Step 403: Obtain the test case corresponding to the problem reproduction scenario screenshot.

[0154] Since each problem scenario test result corresponds to a test case, the test case corresponding to the problem reproduction scenario screenshot can be directly obtained according to the correspondence, and the problem scenario can be automatically reproduced by using the test case. Furthermore, the correspondence between the problem reproduction scenario screenshot and the corresponding test case can be saved, and the correspondence can be provided to the tester so that the tester can find the problem as soon as possible and then solve the problem. It should be noted that in actual applications, one or more problem reproduction scenario screenshots that meet the above conditions can be selected in the above step 402. If multiple problem reproduction scenario screenshots are selected, it is necessary to obtain the corresponding test case for each problem reproduction scenario screenshot.

[0155] The method for automatically reproducing problem scenarios provided by an embodiment of the present invention uses a pre-trained interface element recognition model to identify screenshots of problem scenarios provided by users, automatically identifies controls in the screenshots of the problem scenarios and related information of the controls, generates a problem test case set for the problem scenarios, and uses the problem test case set to automatically reproduce the problem scenarios. Compared with manual and clueless problem searches, problems can be discovered quickly and effectively, and then solved.

[0156] Accordingly, an embodiment of the present invention also provides a test case generating device, such as Figure 5 Shown is a structural block diagram of the device.

[0157] In this embodiment, the test case generation device includes the following modules:

[0158] A screenshot acquisition module 51 is used to acquire a screenshot of the user interface of the software to be tested;

[0159] The recognition module 52 is used to recognize the user interface screenshot using a pre-trained interface element recognition model to obtain each control in the user interface screenshot and related information of the control; the related information includes but is not limited to information such as coordinates and types of the control;

[0160] The interface test case generation module 53 is used to generate an interface test case set according to each control in the user interface screenshot and its related information.

[0161] The screenshot acquisition module 51 can acquire the user interface screenshot of the software to be tested by using an existing screenshot tool or a photo taking tool, and save the user interface screenshot in a picture format.

[0162] The interface element recognition model can adopt a neural network structure and be constructed by a corresponding model training module. The model training module can be used as a part of the test case generation device of the present invention or can be independent of the test case generation device. This embodiment of the present invention is not limited to this.

[0163] A specific structure of the model training module may include the following units:

[0164] A structure determination unit, used to determine the topological structure of the interface element recognition model;

[0165] An information collection unit, used to collect a large number of software interface screenshots as training interface screenshots;

[0166] A labeling unit, used to label the controls in the screenshot of the training interface, wherein the labeling information includes the coordinates and type of the controls;

[0167] The training unit is used to train the interface element recognition model using the interface screenshots and their annotation information. The specific training method is similar to the existing neural network training method and will not be described in detail here.

[0168] It should be noted that, in the interface element recognition model, the labeling unit can also obtain the operation set corresponding to the control according to the type of the control. That is to say, the operation set corresponding to the control can also be used as a label to participate in the training of the interface element recognition model, which is not limited to this embodiment of the present invention.

[0169] In the example of the present invention, the interface test case set may include a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to the control combination. The subset of combined test cases corresponding to the control combination may include various possible operations under each combination of all permutations and combinations of all controls in the user interface screenshot, that is, it includes both test cases corresponding to reasonable operations under the combination and test cases corresponding to abnormal operations under the combination.

[0170] Accordingly, if Figure 5As shown, the interface test case generation module 53 may specifically include: a first subset generation module 531 for generating a subset of independent test cases corresponding to each control and a second subset generation module 532 for generating a subset of combined test cases corresponding to a combination of controls.

[0171] The first subset generation module 531 may include the following units:

[0172] An independent test case generation unit is used to obtain an operation set corresponding to the type of each control in the user interface screenshot, and generate an independent test case corresponding to the control according to the operation set corresponding to the type of each control and the coordinates of the control; specifically, obtain an executable script template corresponding to each operation in the operation set corresponding to the type of the control, fill the coordinates of the control into the executable script template, and generate an independent test case corresponding to the control;

[0173] An independent test case adding unit, which adds the independent test case to the independent test case collection corresponding to each control;

[0174] The second subset generation module 532 may include the following units:

[0175] A combination unit, used for performing full permutation and combination of all controls in the user interface screenshot to obtain various combinations;

[0176] An operation sequence determination unit, used to determine the operation sequence of different controls in each combination;

[0177] A calling unit, used to call a subset of independent test cases corresponding to each control in sequence according to the operation sequence, and generate a combined test case corresponding to the operation sequence under the combination;

[0178] The combination test case adding unit is used to add the combination test case to the combination test case collection of the corresponding control combination.

[0179] It should be noted that, for the above-mentioned test case generation device embodiments, since the functional implementation of each module and unit is similar to that in the corresponding method, the description of the test case generation device embodiments is relatively simple, and the relevant parts can be found in the corresponding part description of the method embodiment.

[0180] The test case generation device provided by the embodiment of the present invention is based on image detection and recognition technology, and uses a pre-trained interface element recognition model to automatically identify controls and related information of the controls in user interface screenshots, and then uses the controls and related information to generate an interface test case set, thereby greatly improving the test case generation efficiency.

[0181] Furthermore, the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to various combinations of controls. As can be seen from the above examples, not only are all controls in the user interface screenshot fully arranged and combined, but for each combination, the subset of independent test cases corresponding to the controls is called according to the operation sequence of each control to generate a combined test case corresponding to the operation sequence. In other words, the test cases under each combination cover all operation conditions under the operation sequence, so that the generated interface test case set includes both test cases corresponding to reasonable operations and test cases corresponding to abnormal operations, breaking the inertial thinking when manually designing test cases, making the test cases for the user interface richer and more comprehensive, so that these test cases can be used to better test the robustness of the software system.

[0182] like Figure 6 FIG. 2 is another structural block diagram of the test case generating device according to an embodiment of the present invention.

[0183] and Figure 5 The difference between the illustrated embodiment is that, in this embodiment, the test case generating device further includes: a test case executing module 54 and a corresponding relationship generating module 55 .

[0184] In this embodiment, the test case execution module 54 is used to test the user interface using each test case in the interface test case set generated by the test case generation module 53 to obtain the test results; the screenshot acquisition module 51 obtains the screenshot of the test results; the correspondence generation module 55 is used to establish and save the correspondence between each test case and the screenshot of the test results.

[0185] The test case generation device provided in this embodiment can not only automatically identify the controls and related information of the controls in the user interface screenshot according to the pre-trained interface element recognition model, and then generate an interface test case set using the controls and related information, but also use each test case in these interface test case sets to test the user interface, obtain the test results, and obtain the screenshots of the test results, and establish and save the corresponding relationship between each test case and the screenshot of the test results. Providing the corresponding relationship to the tester can help the tester quickly determine whether each test result is correct.

[0186] Correspondingly, an embodiment of the present invention further provides a problem scenario automatic reproduction device, which can reproduce the problem scenario provided by the user.

[0187] like Figure 7 As shown, it is a structural block diagram of the automatic reproduction device of the problem scenario.

[0188] In this embodiment, the problem scenario automatic reproduction device includes the following modules:

[0189] The problem scene acquisition module 71 is used to acquire a screenshot of the problem scene provided by the user as a problem screenshot;

[0190] The recognition module 72 is used to recognize the problematic screenshot using a pre-trained interface element recognition model to obtain each control in the problematic screenshot and related information of the control;

[0191] The problem test case generation module 73 is also used to generate a problem test case set according to each control in the problem screenshot and related information of the control;

[0192] The scenario reproduction module 74 is used to reproduce the problem scenario using the problem test case set.

[0193] It should be noted that the interface test case set can be generated by the above-mentioned test case generation device, that is, based on image detection and recognition technology, or it can be generated based on other devices or in other ways, which is not limited by the embodiments of the present invention.

[0194] Accordingly, the identification module 72 and the problem test case generation module 73 are respectively Figure 5 The identification module 52 and the interface test case generation module 53 in the test case generation device embodiment shown have the same principle. If an integrated system including the above-mentioned test case generation device and the above-mentioned problem scenario automatic reproduction device is adopted, the above-mentioned corresponding modules in these two devices can be independent of each other or integrated into one, and this embodiment of the present invention is not limited to this.

[0195] It should be noted that in actual applications, there may be multiple problem scenarios. In the embodiment of the present invention, it is necessary to generate a problem test case set corresponding to each problem scenario. In other words, each problem scenario corresponds to a problem test case set, and the scenario reproduction module 74 reproduces the problem scenario using the problem test case set corresponding to the problem scenario.

[0196] When the scenario reproduction module 74 uses the problem test case set to reproduce the problem scenario, it is necessary to use each test case in the problem test case set to test the problem scenario, find out the possible problem reproduction scenario from the test results, and obtain the test case corresponding to the problem reproduction scenario, so that the problem scenario can be reproduced using these test cases.

[0197] Accordingly, a specific structure of the scene reproduction module 74 is as follows: Figure 8 As shown, the following units may be included:

[0198] The testing unit 741 is used to test the corresponding problem scenario using each test case in the problem test case set, and obtain a screenshot of the problem scenario test result;

[0199] A scenario screening unit 742, configured to select a problem reproduction scenario screenshot from the screenshots of the problem scenario test results;

[0200] The test case acquisition unit 743 is used to acquire the test case corresponding to the problem reproduction scenario screenshot.

[0201] The scene screening unit 742 can select the problem reproduction scene screenshot by comparing the screenshot of the problem scene test result with the image of the problem scene. Accordingly, a specific structure of the scene screening unit 742 may include the following sub-units:

[0202] A similarity calculation subunit, used to calculate the similarity between the screenshots of the test results of each problem scenario and the screenshots of the problem;

[0203] The screening subunit is used to use the screenshots of the problem scenario test results whose similarity is greater than a set threshold as the problem reproduction scenario screenshots; or to select a set number of screenshots of the problem scenario test results from large to small similarity as the problem reproduction scenario screenshots.

[0204] It should be noted that, since there may be multiple problem scenarios, the scenario reproduction module 68 needs to reproduce the problem scenario for each problem scenario using a test case in the problem test case set corresponding to the problem scenario.

[0205] For the above-mentioned embodiments of the automatic reproduction device for problem scenarios, since the functional implementation of each module and unit is similar to that in the corresponding method, the description of the embodiments of the automatic reproduction device for problem scenarios is relatively simple, and the relevant parts can be found in the corresponding part description of the method embodiment.

[0206] The problem scenario automatic reproduction device provided by the embodiment of the present invention uses a pre-trained interface element recognition model to identify a screenshot of the problem scenario provided by a user, automatically identifies the controls in the problem scenario screenshot and related information of the controls, generates a problem test case set for the problem scenario, and uses the problem test case set to automatically reproduce the problem scenario. Compared with manual and clueless problem searching multiple times, problems can be discovered quickly and effectively, and then solved.

[0207] Fig. 98 is a block diagram of an apparatus 800 for a test case generation method according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0208] Reference Fig. 9 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0209] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0210] The memory 804 is configured to store various categories of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any category of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0211] The power component 806 provides power to the various components of the device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0212] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0213] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0214] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0215] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0216] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0217] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0218] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to complete the above-mentioned key mis-touch error correction method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0219] The present invention also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can execute all or part of the steps in the above-mentioned method embodiment of the present invention.

[0220] Fig.101 is a schematic diagram of the structure of the server in an embodiment of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0221] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0222] Obviously, the embodiments described above are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0223] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0224] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0225] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0226] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A test case generation method, characterized in that: The method comprises: Get a screenshot of the user interface of the software under test; Using a pre-trained interface element recognition model to recognize the user interface screenshot, and obtaining each control in the user interface screenshot and related information of the control; the related information of the control includes the coordinates and type of the control; Generate an interface test case set based on each control and related information in the user interface screenshot; The interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to the control combination; generating a subset of combined test cases corresponding to the control combination includes: Performing full permutations and combinations on all controls in the user interface screenshot to obtain various combinations; Determine the order of operations on different controls in each combination; The independent test case subset corresponding to each control is called in sequence according to the operation sequence to generate a combination test case corresponding to the operation sequence under the combination; the test case under each combination covers all operation conditions under the operation sequence, so that the generated interface test case set includes both test cases corresponding to reasonable operations and test cases corresponding to abnormal operations; Adding the combined test case to the combined test case collection of the corresponding control combination; The following are the subsets of independent test cases generated for each control: Obtain an operation set corresponding to the type of each control in the user interface screenshot, and obtain an executable script template corresponding to each operation in the operation set corresponding to the type of the control, fill the coordinates of the control into the executable script template, and generate an independent test case corresponding to the control; The independent test case is added to the independent test case collection corresponding to each control.

2. The method according to claim 1, characterized in that The method further includes the step of training the interface element recognition model, comprising: Determine the topological structure of the interface element recognition model; Collect a large number of software interface screenshots as training interface screenshots; Annotate the controls in the screenshot of the training interface, where the annotation information includes the coordinates and type of the controls; The interface screenshots and their annotation information are used to train an interface element recognition model.

3. The method according to claim 2, characterized in that The annotation information also includes: an operation set corresponding to the control.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Using each test case in the interface test case set to test the user interface to obtain a test result; Obtaining a screenshot of the test result; Establish and save the correspondence between each test case and the screenshot of the test results.

5. A method for automatically reproducing a problem scenario, characterized in that: The method comprises: Obtain a screenshot of the problem scenario provided by the user as the problem screenshot; Using a pre-trained interface element recognition model to recognize the problematic screenshot, and obtaining each control in the problematic screenshot and related information of the control; the related information of the control includes the coordinates and type of the control; Generate a problem test case set according to each control in the problem screenshot and the related information of the control; the problem test case set is an interface test case set under the problem scenario; the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to the control combination; generating a subset of combined test cases corresponding to the control combination includes: performing full permutation and combination of all controls in the user interface screenshot to obtain each combination; determining the operation sequence of different controls in each combination; calling the subset of independent test cases corresponding to each control in sequence according to the operation sequence to generate a combined test case corresponding to the operation sequence under the combination; the test case under each combination covers all operation conditions under the operation sequence, so that the generated interface test case set contains both test cases corresponding to reasonable operations and test cases corresponding to abnormal operations; adding the combined test case to the combined test case set corresponding to the control combination; Reproduce the problem scenario using the problem test case set; The following are the subsets of independent test cases generated for each control: Obtain an operation set corresponding to the type of each control in the user interface screenshot, and obtain an executable script template corresponding to each operation in the operation set corresponding to the type of the control, fill the coordinates of the control into the executable script template, and generate an independent test case corresponding to the control; The independent test case is added to the independent test case collection corresponding to each control.

6. The method according to claim 5, characterized in that The reproducing the problem scenario by using the problem test case set includes: Using each test case in the problem test case set to test the problem scenario, and obtaining a screenshot of the problem scenario test result; Select a problem reproduction scenario screenshot from the screenshots of the problem scenario test results; Get the test case corresponding to the screenshot of the problem reproduction scenario.

7. The method according to claim 6, characterized in that The step of selecting a problem reproduction scenario screenshot from the screenshots of the problem scenario test results includes: Calculate the similarity between the screenshots of the test results of each problem scenario and the screenshots of the problem; The screenshots of the problem scenario test results with a similarity greater than a set threshold are used as the problem reproduction scenario screenshots; or A set number of screenshots of the problem scenario test results are selected from large to small according to similarity as the problem reproduction scenario screenshots.

8. A test case generating device, characterized in that: The device comprises: A screenshot acquisition module is used to obtain a screenshot of the user interface of the software under test; An identification module, used to identify the user interface screenshot using a pre-trained interface element identification model, and obtain each control in the user interface screenshot and related information of the control; the related information of the control includes the coordinates and type of the control; An interface test case generation module is used to generate an interface test case set according to each control and related information in the user interface screenshot; the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to the control combination; The interface test case generation module includes: a first subset generation module for generating a subset of independent test cases corresponding to each control and a second subset generation module for generating a subset of combined test cases corresponding to a combination of controls; The first subset generation module comprises: An independent test case generating unit, used to obtain an operation set corresponding to the type of each control in the user interface screenshot, and generate an independent test case corresponding to the control according to the operation set corresponding to the type of each control and the coordinates of the control; An independent test case adding unit, which adds the independent test case to the independent test case collection corresponding to each control; The second subset generation module comprises: A combination unit, used for performing full permutation and combination of all controls in the user interface screenshot to obtain various combinations; An operation sequence determination unit, used to determine the operation sequence of different controls in each combination; A calling unit, used to call the independent test case subset corresponding to each control in sequence according to the operation sequence, and generate a combination test case corresponding to the operation sequence under the combination; the test case under each combination covers all operation conditions under the operation sequence, so that the generated interface test case set includes both test cases corresponding to reasonable operations and test cases corresponding to abnormal operations; A combination test case adding unit, used to add the combination test case to the combination test case collection of the corresponding control combination; The independent test case generation unit is specifically used to obtain the executable script template corresponding to each operation in the operation set corresponding to the type of the control, fill the coordinates of the control into the executable script template, and generate an independent test case corresponding to the control.

9. The device according to claim 8, characterized in that The device also includes a model training module for training an interface element recognition model; the model training module includes: A structure determination unit, used to determine the topological structure of the interface element recognition model; An information collection unit, used to collect a large number of software interface screenshots as training interface screenshots; A labeling unit, used to label the controls in the screenshot of the training interface, wherein the labeling information includes the coordinates and type of the controls; The training unit is used to train an interface element recognition model using the interface screenshots and their annotation information.

10. The device according to claim 9, characterized in that The annotation information also includes: an operation set corresponding to the control.

11. The device according to any one of claims 8 to 10, characterized in that The device also includes: a test case execution module and a corresponding relationship generation module; The test case execution module is used to test the user interface using each test case in the interface test case set to obtain a test result; The screenshot acquisition module is further used to obtain a screenshot of the test result; The corresponding relationship generating module is used to establish and save the corresponding relationship between each test case and the screenshot of the test result.

12. A device for automatically reproducing a problem scenario, characterized in that: The device comprises: A problem scenario acquisition module is used to acquire a screenshot of a problem scenario provided by a user as a problem screenshot; An identification module, used to identify the problematic screenshot using a pre-trained interface element identification model, and obtain each control in the problematic screenshot and related information of the control; A problem test case generation module is used to generate a problem test case set according to each control in the problem screenshot and the related information of the control; the problem test case set is an interface test case set under the problem scenario; the interface test case set includes a subset of independent test cases corresponding to each control and a subset of combined test cases corresponding to the control combination; generating a subset of combined test cases corresponding to the control combination includes: performing full permutation and combination of all controls in the user interface screenshot to obtain each combination; determining the operation sequence of different controls in each combination; calling the subset of independent test cases corresponding to each control in sequence according to the operation sequence to generate a combined test case corresponding to the operation sequence under the combination; the test case under each combination covers all operation conditions under the operation sequence, so that the generated interface test case set contains both test cases corresponding to reasonable operations and test cases corresponding to abnormal operations; adding the combined test case to the combined test case set corresponding to the control combination; The generating of the subset of independent test cases corresponding to each control includes: obtaining an operation set corresponding to the type of each control in the user interface screenshot, and obtaining an executable script template corresponding to each operation in the operation set corresponding to the type of the control, filling the coordinates of the control into the executable script template, and generating an independent test case corresponding to the control; and adding the independent test case to the subset of independent test cases corresponding to each control; A scenario reproduction module is used to reproduce the problem scenario using the problem test case set.

13. The device according to claim 12, characterized in that The scene reproduction module includes: A testing unit, used to test the problem scenario using each test case in the problem test case set, and obtain a screenshot of the problem scenario test result; A scenario screening unit, used to select a problem reproduction scenario screenshot from the screenshots of the problem scenario test results; The test case acquisition unit is used to acquire the test case corresponding to the problem reproduction scenario screenshot.

14. The device according to claim 13, characterized in that The scene screening unit comprises: A similarity calculation subunit, used to calculate the similarity between the screenshots of the test results of each problem scenario and the screenshots of the problem; The screening subunit is used to use the screenshots of the problem scenario test results whose similarity is greater than a set threshold as the problem reproduction scenario screenshots; or to select a set number of screenshots of the problem scenario test results from large to small similarity as the problem reproduction scenario screenshots.

15. An electronic device, characterized in that: include: One or more processors, memory; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method according to any one of claims 1 to 4.

16. A readable storage medium having instructions stored thereon, wherein the instructions are executed to implement the method according to any one of claims 1 to 4.

Images