Control Testing Method, Device, Storage Medium and Electronic Device

Abstract

The present disclosure relates to a control testing method, apparatus, storage medium, and electronic device, which are used to reduce the modification of test scripts during the automated testing process, thereby improving the efficiency of automated testing. The method includes: obtaining a keyword corresponding to the control to be tested, where the keyword is used to identify the content represented by the control to be tested; identifying the controls in the page to be tested through a pre-trained recognition model; when the control to be tested cannot be identified in the page to be tested, searching for the corresponding control image in the control image set based on the keyword, where the control image set includes a plurality of control images annotated with keywords; when the corresponding control image is found in the control image set based on the keyword, performing image matching between the found control image and the page screenshot of the page to be tested, and positioning the control to be tested in the page to be tested based on the result of the image matching for testing.

Description

Technical Field

[0001] The present disclosure relates to the field of testing technologies, and in particular, to a control testing method, apparatus, storage medium, and electronic device. Background Art

[0002] With the increasing maturity of AI (Artificial Intelligence) technology, AI is increasingly widely used in automated testing. For example, through AI image recognition technologies such as object detection, the control content of the page to be tested can be recognized, and then control testing can be performed according to the recognition results.

[0003] In practical applications, there are often situations where the AI image recognition model fails to recognize the control to be tested. In such cases, the related technology usually manually modifies the test script to achieve automated testing of the control through template matching. However, this method requires a lot of manpower and time, thus affecting the efficiency of automated testing of the control. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a control testing method, apparatus, storage medium, and electronic device to reduce the modification of the test script during the testing process, thereby improving the efficiency of automated testing of the control.

[0005] To achieve the above purpose, in a first aspect, the present disclosure provides a control testing method, and the method includes:

[0006] Obtain a keyword corresponding to the control to be tested, where the keyword is used to identify the content represented by the control to be tested;

[0007] Identify the controls in the page to be tested through a pre-trained recognition model;

[0008] When the control to be tested cannot be recognized in the page to be tested, search for the corresponding control picture in the control picture set based on the keyword, where the control picture set includes multiple control pictures labeled with keywords;

[0009] When the corresponding control picture is found in the control picture set based on the keyword, perform image matching on the found control picture and the page screenshot of the page to be tested, and locate the control to be tested in the page to be tested based on the result of the image matching for testing.

[0010] Optionally, the recognition model includes an image recognition model, and the method further includes:

[0011] When the target control in the page to be tested cannot be recognized by the image recognition model, capture the control picture corresponding to the target control, and label the target keyword for the control picture to obtain the target control picture;

[0012] Add the target control picture to the control picture set, and generate a control annotation file based on the control picture set after adding the target control picture;

[0013] In response to the generation of the control annotation file, call the control annotation file to train the image recognition model to obtain a target recognition model;

[0014] The recognition of the control in the page to be tested by the pre-trained recognition model includes:

[0015] Perform image recognition on the control in the page to be tested through the target recognition model.

[0016] Optionally, the calling the control annotation file to train the image recognition model to obtain a target recognition model includes:

[0017] Call the control annotation file to train the image recognition model in the first process to obtain a target recognition model;

[0018] The recognition of the control in the page to be tested through the target recognition model includes:

[0019] Replace the image recognition model in the second process with the target recognition model, where the image recognition model in the second process is the same as the image recognition model in the first process before training;

[0020] Perform image recognition on the page to be tested through the target recognition model in the second process to obtain a recognition result.

[0021] Optionally, the calling the control annotation file to train the image recognition model to obtain a target recognition model includes:

[0022] If the preset training time is reached, call the current control annotation file generated between the last model training and the current model training, and use the current control annotation file as the target annotation file, or perform data fusion on the current control annotation file and the historical control annotation file obtained during the last model training to obtain the target annotation file;

[0023] Call the target control annotation file to train the image recognition model to obtain a target recognition model.

[0024] Optionally, locating the control to be tested in the page to be tested based on the result of image matching for testing includes:

[0025] If a control with a similarity exceeding a preset similarity to the control in the control picture is matched in the page screenshot, perform control testing based on the position of the control in the page screenshot.

[0026] Optionally, the method further includes:

[0027] When the control to be tested cannot be recognized in the page to be tested, if the corresponding control picture cannot be found in the control picture set based on the keyword, output a first prompt message indicating that the control to be tested cannot be located; or

[0028] If the corresponding control is not successfully matched in the page screenshot based on the found control picture, output a second prompt message indicating that the control to be tested cannot be located.

[0029] In a second aspect, the present disclosure provides a control testing device, the device includes:

[0030] An acquisition module, configured to acquire a keyword corresponding to the control to be tested, where the keyword is used to identify the content represented by the control to be tested;

[0031] An identification module, configured to identify the controls in the page to be tested through a pre-trained identification model;

[0032] A first testing module, configured to, when the control to be tested cannot be recognized in the page to be tested, search for the corresponding control picture in the control picture set based on the keyword, where the control picture set includes multiple control pictures marked with keywords;

[0033] A second testing module, configured to, when the corresponding control picture is found in the control picture set based on the keyword, perform image matching on the found control picture and the page screenshot of the page to be tested, and locate the control to be tested in the page to be tested based on the result of the image matching for testing.

[0034] Optionally, the identification model includes an image recognition model, and the device further includes:

[0035] A first annotation module, configured to, when the target control in the page to be tested cannot be recognized by the image recognition model, capture the control picture corresponding to the target control, and annotate the target keyword on the control picture to obtain the target control picture;

[0036] A second annotation module, configured to add the target control image to the control image set and generate a control annotation file based on the control image set after adding the target control image;

[0037] A training module, configured to, in response to the generation of the control annotation file, call the control annotation file to train an image recognition model in a first process to obtain a target recognition model;

[0038] The recognition module is configured to:

[0039] Perform image recognition on the controls in the page to be tested through the target recognition model.

[0040] In a third aspect, the present disclosure provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method according to any one of the first aspects are implemented.

[0041] In a fourth aspect, the present disclosure provides an electronic device, including:

[0042] A memory, on which a computer program is stored;

[0043] A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of the first aspects.

[0044] Through the above technical solution, the control images in the control image set are annotated with keywords. When the target control to be tested cannot be recognized in the page to be tested by the recognition model, the corresponding control image can be searched in the control image set based on the keyword, and then image matching is performed on the page screenshot of the page to be tested based on the found control image, and further the target control to be tested is located in the page to be tested for testing. Thus, when the target control to be tested cannot be recognized by the recognition model, there is no need to manually modify the test script for template matching, which can reduce the manpower and time consumed in the test process, thereby improving the automation test efficiency of the control.

[0045] Other features and advantages of the present disclosure will be described in detail in the subsequent specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings are used to provide a further understanding of the present disclosure, and constitute a part of the specification, and are used to explain the present disclosure together with the following specific implementation manners, but do not constitute a limitation to the present disclosure. In the drawings:

[0047] Figure 1 is a flowchart of a control test method shown according to an exemplary embodiment of the present disclosure;

[0048] Figure 2It is a schematic diagram of a control picture in a control test method shown according to an exemplary embodiment of the present disclosure;

[0049] Figure 3 It is a schematic diagram of a control picture set in a control test method shown according to an exemplary embodiment of the present disclosure;

[0050] Figure 4 It is a flowchart of a control test method shown according to another exemplary embodiment of the present disclosure;

[0051] Figure 5 It is a block diagram of a control test device shown according to an exemplary embodiment of the present disclosure;

[0052] Figure 6 It is a block diagram of an electronic device shown according to an exemplary embodiment of the present disclosure. Detailed Embodiments

[0053] The following will describe the detailed embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the detailed embodiments described herein are only for explaining and illustrating the present disclosure, and are not used to limit the present disclosure.

[0054] The inventors have found through research that in the related art, test scripts are usually written to perform control positioning on the page to be tested through AI image recognition. If the AI image recognition model fails to recognize the control to be tested, it is necessary to manually modify the control positioning method in the test script to the template matching method. Among them, the template matching method refers to matching according to a pre-set control template picture in the page to be tested. Then, by retraining the AI image recognition model, after the AI image recognition model is retrained, that is, after the AI image recognition model is updated, the control positioning method in the test script is manually modified to the AI image recognition method.

[0055] It can be seen that in the related art, when the AI image recognition model fails to recognize the control to be tested, it is necessary to manually modify the test script twice, which is cumbersome and consumes a lot of manpower and time, thus affecting the automation test efficiency of the control.

[0056] Therefore, the present disclosure provides a control test method, device, storage medium and electronic device to reduce the modification of the test script during the test process and improve the automation test efficiency of the control.

[0057] Figure 1 It is a flowchart of a control test method shown according to an exemplary embodiment of the present disclosure. Referring to Figure 1 , the control test method includes:

[0058] Step 101: Obtain the keyword corresponding to the control to be tested, where the keyword is used to identify the content represented by the control to be tested.

[0059] Step 102: Identify the controls in the page to be tested through a pre-trained recognition model.

[0060] Step 103: When the control to be tested cannot be identified in the page to be tested, search for the corresponding control picture in the control picture set based on the keyword. The control picture set includes multiple control pictures marked with keywords.

[0061] Step 104: When the corresponding control picture is found in the control picture set based on the keyword, perform image matching on the found control picture and the page screenshot of the page to be tested, and locate the control to be tested in the page to be tested based on the result of the image matching for testing.

[0062] Exemplarily, the keyword corresponding to the control to be tested can be used to identify the content represented by the control to be tested. For example, Figure 2 For the two controls shown, the content they represent is "mine", so the keyword corresponding to these two controls can be "mine". In each control test, the keyword of the control to be tested can be given, so that when the control to be tested cannot be identified by the recognition model, the corresponding control picture can be searched in the control picture set based on this keyword for automatic template matching, without the need to manually modify the test script for template matching, thereby improving the efficiency of automatic testing of controls.

[0063] Exemplarily, the recognition model can include an OCR (Optical Character Recognition) recognition model, and can also include various types of AI image recognition models, which are not limited in the embodiments of the present disclosure. By identifying the controls in the page to be tested through the recognition model, the recognition result can include the keyword used to represent the content of the identified control. Therefore, it can be determined whether the control to be tested is identified in the page to be tested based on whether the keyword corresponding to the control to be tested is included in the recognition result.

[0064] Exemplarily, if the keyword corresponding to the control to be tested is included in the recognition result, it indicates that the control to be tested is recognized in the page to be tested by the recognition model, so that the control test can be performed on the recognized control to be tested. On the contrary, if the keyword corresponding to the control to be tested is not included in the recognition result, it indicates that the recognition model cannot recognize the control to be tested in the page to be tested. Further, the corresponding control image can be searched in the control image set based on the keyword corresponding to the control to be tested for control positioning. Among them, if the corresponding control image is found in the control image set based on the keyword, the found control image can be matched with the page screenshot of the page to be tested, so as to locate the control to be tested in the page to be tested based on the result of the image matching.

[0065] Exemplarily, the page screenshot of the page to be tested can be obtained by taking a screenshot of the page to be tested before performing image recognition on the page to be tested. Matching the control image with the page screenshot of the page to be tested can be, for example: first, divide the page screenshot into multiple sub-images according to the size of the control image, and then match the control image with each sub-image respectively. Of course, other possible image matching methods can also be used to match the control image with the page screenshot of the page to be tested, and the embodiments of the present disclosure do not limit this.

[0066] In a possible way, positioning the control to be tested in the page to be tested for testing based on the result of the image matching can be: if a control with a similarity exceeding the preset similarity to the control in the control image is matched in the page screenshot, the control test is performed based on the position of the control in the page screenshot.

[0067] Exemplarily, the preset similarity can be set according to the actual situation, and the embodiments of the present disclosure do not limit this.

[0068] Continuing with the above example, after matching the control image with each sub-image respectively, if the similarity between the control in a certain sub-image and the control in the control image exceeds the preset similarity, the control test can be performed based on the position of the control in the sub-image in the page screenshot. Thus, when the recognition model cannot recognize the control to be tested, there is no need to manually modify the test script for template matching, which can reduce the manpower and time consumed in the test process, thereby improving the control test efficiency.

[0069] In a possible way, when the control to be tested cannot be recognized in the page to be tested, if the corresponding control image cannot be found in the control image set based on the keyword, a first prompt message for indicating that the control to be tested cannot be located can be output; or, if the corresponding control cannot be successfully matched in the page screenshot based on the found control image, a second prompt message for indicating that the control to be tested cannot be located can be output.

[0070] It should be understood that if the keyword in the recognition result does not include the keyword corresponding to the control to be tested, it means that the control to be tested cannot be recognized by the recognition model. In this case, if the corresponding control picture cannot be found in the control picture set based on the keyword, the first prompt message indicating that the control to be tested cannot be located can be output to prompt the user that the control to be tested corresponding to the keyword cannot be located on the page to be tested. Alternatively, if the control to be tested cannot be located on the page to be tested based on the target control picture, the second prompt message indicating that the control to be tested cannot be located can be output to prompt the user that the control to be tested corresponding to the keyword does not exist on the page to be tested.

[0071] In a possible way, when the target control on the page to be tested cannot be recognized by the image recognition model, the control picture corresponding to the target control can also be intercepted, the target keyword can be marked on the control picture to obtain the target control picture, then the target control picture can be added to the control picture set, and the control annotation file can be generated based on the control picture set after adding the target control picture. Finally, in response to the generation of the control annotation file, the control annotation file can be called to train the image recognition model to obtain the target recognition model. Correspondingly, step 102 can be: performing image recognition on the controls on the page to be tested through the target recognition model.

[0072] It should be understood that in the related art, when encountering a target control that cannot be recognized by the image recognition model, usually the picture corresponding to the target control is intercepted and sent to the backend system. After waiting for the backend system to update the image recognition model based on the picture corresponding to the target control, the image recognition model can be used again for control positioning only by modifying the test script. It can be seen that in this way in the related art, a long model training time is required, thus affecting the efficiency of control testing.

[0073] In the embodiment of the present disclosure, when encountering a target control that cannot be recognized by the image recognition model on the page to be tested, the control picture corresponding to the target control can be intercepted, the target keyword can be marked on the control picture to obtain the target control picture, then the control annotation file can be automatically generated based on the target control picture, and in response to the generation of the control annotation file, the control annotation file can be called to train the image recognition model. That is to say, in the embodiment of the present disclosure, after the control annotation file is generated, the control annotation file can be applied to the training of the image recognition model in real time, so as to perform control testing according to the trained image recognition model (i.e., the target recognition model). Thus, an integrated process of annotation, training, and application can be realized, the waiting time for model training can be reduced, and the control testing efficiency can be further improved.

[0074] Exemplarily, the intercepted control picture can be the control icon corresponding to the target control. For example, Figure 2The two control icons shown in the figure. After capturing the corresponding control picture, keywords can be labeled for the control picture based on the content represented by the control picture. For example, if the control picture is as Figure 2 shown, then the keyword "mine" can be labeled for it to obtain the target control picture.

[0075] Exemplarily, adding the target control picture to the control picture set can be: adding the control picture and information such as the resolution and pixel density of the device to which the control picture belongs to the control picture set. Thus, each control picture in the control picture set can correspond to picture basic information. For example, Figure 3 shown, in the control picture set corresponding to the keyword "mine", there are multiple control pictures, and each control picture corresponds to a file name file_name, a pixel ratio value density, the resolution device_resolution of the device to which it belongs, and the pixel density shot_resolution.

[0076] It should be understood that control pictures with the same keyword can be stored in the same control picture set, so one control picture set can correspond to one keyword. In this case, after obtaining the keyword of the control to be tested, the target control picture set with the same keyword can be determined first based on the keyword of the control to be tested, and then image matching can be performed on each control picture in the target control picture set in the page screenshot. Or, control pictures with different keywords can be stored in the same control picture set, and at least one control picture corresponds to one keyword in the control picture set, then the target control picture corresponding to the keyword of the control to be tested can be found in the control picture set, and then the corresponding image matching can be performed.

[0077] Exemplarily, the control annotation file is data generated based on the control picture set and can be directly used to train the image recognition model. Compared with the data in the control picture set, the data in the control annotation file can include information such as the position information of the control picture in the corresponding sample page in addition to the picture basic information described above, which is convenient for the training of the image recognition model.

[0078] After generating the control annotation file, the control annotation file can be called through a script to train the image recognition model, that is, the labeled results can be applied to model training in real time through the script. Thus, the waiting time for model training can be reduced, and the control test efficiency can be improved.

[0079] In a possible way, the control annotation file can be called to train the image recognition model in the first process to obtain the target recognition model. Correspondingly, the image recognition model in the second process can be replaced with the target recognition model, where the image recognition model in the second process is the same as the image recognition model in the first process before training. Then, the target recognition model in the second process is used to perform image recognition on the page to be tested.

[0080] That is to say, the annotation result can be applied to model training in real time through two processes to realize the online training of the image recognition model. Among them, the image recognition model in the first process is used to execute the model training process, and the image recognition model in the second process is used to execute the model application process, that is, to perform real-time recognition on the page to be tested. After the image recognition model in the first process is trained, the currently running image recognition model in the second process can be replaced through a script.

[0081] Thus, model training and model application can be carried out simultaneously, and the model application process can be uninterrupted during the model training process, thereby reducing the waiting time for model training and improving the control test efficiency.

[0082] In a possible way, another way to call the control annotation file to train the image recognition model to obtain the target recognition model is: if the preset training time is reached, the current control annotation file generated between the last model training and the current model training is called, and the current control annotation file is used as the target annotation file, or the current control annotation file and the historical control annotation file obtained during the last model training are fused to obtain the target annotation file. Finally, the target control annotation file is called to train the image recognition model to obtain the target recognition model.

[0083] Among them, the preset training time can be set according to the actual situation, and the embodiments of the present disclosure do not limit this. For example, it can be set to perform model training periodically to reduce the running loss while ensuring the training effect. In this case, the training time can be the start time of each cycle, so as to trigger model training regularly.

[0084] When the control annotation file is generated and the preset training time is reached, the current control annotation file generated between the last model training and the current model training can be called through a script. It should be understood that during each model training process, the called control annotation file can be saved. Therefore, during a certain model training, only the control annotation file generated between the last model training and the current model training can be called to reduce unnecessary data transmission.

[0085] In addition, it should be understood that, in this embodiment, although the triggering of model training needs to meet the two conditions of generating a control annotation file and reaching a preset training time at the same time, the control annotation file can be transmitted to the model training process in real time after it is generated, and the model training can be performed immediately after reaching the preset training time, so that the control test can be performed according to the trained image recognition model. In other words, the embodiment of the present disclosure can also realize the integrated process of annotation, training and application, and reduce the waiting time of model training.

[0086] After obtaining the current control annotation file, the image recognition model can be supplemented with training based on the current control annotation file. Since the amount of data in the current control annotation file is not large, the training efficiency of this method is relatively high. Alternatively, after obtaining the current control annotation file, data fusion can be performed based on the current control annotation file and the historical control annotation file obtained during the last model training to obtain the target annotation file for complete template training. Since this method fuses the current control annotation file and the historical control annotation file for training, the training efficiency is lower than that of the previous method, but due to the comprehensiveness of the training data, the accuracy of the training results will be improved compared to the previous method. In actual application, the corresponding training method can be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0087] After training the image recognition model, there is no need to modify the test script. The trained target recognition model can be used to directly perform image recognition on the test page, thereby improving the control testing efficiency.

[0088] The control test method provided by the present disclosure is described below by another exemplary embodiment. Figure 4 , the control testing method includes the following steps:

[0089] Step 401, capture a sample image of the control to be annotated.

[0090] Among them, the control to be annotated can be a target control that cannot be identified in the page to be tested by the recognition model, and of course it can also be a control determined by other means, such as a user-specified control, etc., which is not limited in this embodiment of the present disclosure.

[0091] Step 402: Obtain sample keywords corresponding to the sample images.

[0092] The sample keywords may be determined according to the content represented by the control to be annotated.

[0093] Step 403: Add the sample image and information such as the resolution and pixel density of the device to which it belongs to the control image set corresponding to the sample keyword.

[0094] Step 404, generating a control annotation file.

[0095] Step 405: If the preset training time is reached, call the control annotation file through a script.

[0096] Among them, the called control annotation file can be the current control annotation file generated between the last model training and the current model training, or it can also be all generated control annotation files. This disclosure embodiment does not limit this.

[0097] Step 406: Train the image recognition model in the first process based on the called control annotation file to obtain the target recognition model.

[0098] Step 407: Replace the currently running image recognition model in the second process with the target recognition model.

[0099] Step 408: Obtain the target keyword. This target keyword is the keyword of the control to be tested.

[0100] Step 409: Obtain a page screenshot of the page to be tested.

[0101] Step 410: Perform image recognition on the page screenshot.

[0102] Step 411: Perform OCR character recognition on the page screenshot.

[0103] Step 412: Obtain the OCR recognition result and the image recognition result to obtain the target recognition result.

[0104] Step 413: Determine whether the target recognition result includes the target keyword. If it includes, go to Step 414; otherwise, go to Step 415.

[0105] Step 414: Determine whether the target keyword exists in the control picture set. If it exists, go to Step 416; otherwise, go to Step 417.

[0106] Step 415: Locate the control to be tested corresponding to the target keyword for testing.

[0107] Step 416: Match the target control picture corresponding to the target keyword in the control picture set with the page screenshot, and go to Step 418.

[0108] Step 417: Prompt that the control to be tested cannot be located.

[0109] Step 418: Determine whether the match is successful. If the match is successful, go to Step 415; otherwise, go to Step 417.

[0110] The specific implementation manners of the above steps have been described in detail by way of examples above and will not be elaborated here. In addition, it should be understood that for the above method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present disclosure is not limited by the order of actions described above. Secondly, those skilled in the art should also know that the embodiments described above are preferred embodiments, and the steps involved are not necessarily essential to the present disclosure.

[0111] In the above manner, when the image recognition fails to recognize, the corresponding target control picture can be found in the control picture set based on the given target keyword, and then the control test can be performed based on the target control picture. Thus, when the image recognition fails to recognize, there is no need to manually modify the test script for template matching, which can reduce the manpower and time consumed in the test process, thereby improving the control test efficiency. On the other hand, after the control annotation file is generated, the control annotation file can be applied to the training of the image recognition model in real time, and then the control test can be performed according to the trained image recognition model (i.e., the target recognition model). Thus, an integrated process of annotation, training, and application can be realized through script control, reducing the waiting time for model training and further improving the control test efficiency.

[0112] Based on the same concept, the present disclosure also provides a control test device, which can become part or all of an electronic device in a manner of software, hardware, or a combination of both. Referring to Figure 5 , the control test device 500 may include:

[0113] An acquisition module 501, configured to acquire a keyword corresponding to a control to be tested, where the keyword is used to identify the content represented by the control to be tested;

[0114] An identification module 502, configured to identify the control in the page to be tested through a pre-trained identification model;

[0115] A first test module 503, configured to, when the control to be tested cannot be recognized in the page to be tested, search for a corresponding control picture in the control picture set based on the keyword, where the control picture set includes a plurality of control pictures annotated with keywords;

[0116] A second test module 504, configured to, when a corresponding control picture is found in the control picture set based on the keyword, perform image matching on the found control picture and a page screenshot of the page to be tested, and locate the control to be tested in the page to be tested based on the result of the image matching for testing.

[0117] Optionally, the identification model includes an image recognition model, and the device 500 further includes:

[0118] The first annotation module is used to intercept the control picture corresponding to the target control and annotate the target keyword for the control picture to obtain the target control picture when the target control in the page to be tested cannot be recognized by the image recognition model;

[0119] The second annotation module is used to add the target control picture to the control picture set and generate a control annotation file based on the control picture set after adding the target control picture;

[0120] The training module is used to, in response to the generation of the control annotation file, call the control annotation file to train the image recognition model in the first process to obtain a target recognition model;

[0121] The recognition module 502 is used for:

[0122] Performing image recognition on the controls in the page to be tested through the target recognition model.

[0123] Optionally, the training module is used for:

[0124] Calling the control annotation file to train the image recognition model in the first process to obtain a target recognition model;

[0125] The recognition module 502 is used for:

[0126] Replacing the image recognition model in the second process with the target recognition model, where the image recognition model in the second process is the same as the image recognition model in the first process before training;

[0127] Performing image recognition on the page to be tested through the target recognition model in the second process.

[0128] Optionally, the training module is used for:

[0129] When reaching the preset training time, calling the current control annotation file generated between the last model training and the current model training, and using the current control annotation file as the target annotation file, or performing data fusion on the current control annotation file and the historical control annotation file obtained during the last model training to obtain the target annotation file;

[0130] Calling the target control annotation file to train the image recognition model to obtain a target recognition model.

[0131] Optionally, the second testing module 504 is used for:

[0132] If a control with a similarity to the control in the control picture exceeding a preset similarity is matched in the page screenshot, control testing is performed based on the position of the control in the page screenshot.

[0133] Optionally, the apparatus 500 further includes:

[0134] A prompt module, configured to, when the keyword in the recognition result does not include the target keyword and when the corresponding target control picture cannot be found in the control picture set based on the target keyword, output a first prompt message for indicating that the to-be-tested control cannot be located; or, when the corresponding control cannot be located in the to-be-tested page based on the target control picture, output a second prompt message for indicating that the to-be-tested control cannot be located.

[0135] Regarding the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0136] Based on the same inventive concept, the present disclosure also provides an electronic device, including:

[0137] A memory, having a computer program stored thereon;

[0138] A processor, configured to execute the computer program in the memory to implement the steps of any of the above control testing methods.

[0139] In a possible manner, the block diagram of the electronic device may be as Figure 6 shown. Referring to Figure 6 , the electronic device 600 may include: a processor 601, a memory 602. The electronic device 600 may further include one or more of a multimedia component 603, an input / output (I / O) interface 604, and a communication component 605.

[0140] Among them, the processor 601 is used to control the overall operation of the electronic device 600 to complete all or part of the steps in the above control test method. The memory 602 is used to store various types of data to support the operation of the electronic device 600. These data may include, for example, instructions for any application or method operating on the electronic device 600, as well as application-related data, such as contact data, received and sent messages, pictures, audio, video, and so on. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 603 may include a screen and an audio component. Among them, the screen can be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, and the microphone is used to receive external audio signals. The received audio signals can be further stored in the memory 602 or sent through the communication component 605. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 604 provides an interface between the processor 601 and other interface modules. The above other interface modules can be a keyboard, a mouse, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 605 may include: a Wi-Fi module, a Bluetooth module, an NFC module, and so on.

[0141] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and is used to execute the above control test method.

[0142] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above control test method are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 602 including program instructions, and the above program instructions may be executed by the processor 601 of the electronic device 600 to complete the above control test method.

[0143] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above control test method when executed by the programmable device.

[0144] The preferred embodiments of the present disclosure have been described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0145] In addition, it should be noted that, in the above specific embodiments, the various specific technical features described can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present disclosure does not separately describe various possible combination methods.

[0146] Furthermore, any combination can be made between various different embodiments of the present disclosure as long as it does not violate the idea of the present disclosure, and it should also be regarded as the content disclosed by the present disclosure.

Claims

1. A control testing method, characterized in that, the method includes: Obtain the keyword corresponding to the control to be tested, where the keyword is used to identify the content represented by the control to be tested; Identify the controls in the page to be tested through a pre-trained recognition model; When the control to be tested cannot be identified in the page to be tested, search for the corresponding control picture in the control picture set based on the keyword, where the control picture set includes multiple control pictures marked with keywords; When the corresponding control picture is found in the control picture set based on the keyword, perform image matching on the found control picture and the page screenshot of the page to be tested, and locate the control to be tested in the page to be tested based on the result of the image matching for testing; The recognition model includes an image recognition model, and the method further includes: When the target control in the page to be tested cannot be recognized by the image recognition model, intercept the control picture corresponding to the target control, and mark the target keyword on the control picture to obtain the target control picture; Add the target control picture to the control picture set, and generate a control annotation file based on the control picture set after adding the target control picture, where the control annotation file is used to indicate the picture information of each control picture in the control picture set and / or the position information of each control picture in the corresponding sample page; In response to the generation of the control annotation file, call the control annotation file to train the image recognition model to obtain a target recognition model; The step of identifying the controls in the page to be tested through the pre-trained recognition model includes: Perform image recognition on the controls in the page to be tested through the target recognition model.

2. The method according to claim 1, characterized in that, the step of calling the control annotation file to train the image recognition model to obtain a target recognition model includes: Call the control annotation file to train the image recognition model in the first process to obtain a target recognition model; The step of identifying the controls in the page to be tested through the target recognition model includes: Replace the image recognition model in the second process with the target recognition model, where the image recognition model in the second process is the same as the image recognition model in the first process before training; Perform image recognition on the page to be tested through the target recognition model in the second process.

3. The method according to claim 1, characterized in that, the step of calling the control annotation file to train the image recognition model to obtain a target recognition model includes: If a preset training time is reached, call the current control annotation file generated between the last model training and the current model training, and use the current control annotation file as the target annotation file, or perform data fusion on the current control annotation file and the historical control annotation file obtained during the last model training to obtain a target annotation file; Call the target control annotation file to train the image recognition model to obtain a target recognition model.

4. The method according to any one of claims 1-3, characterized in that, locating the control to be tested in the page to be tested based on the result of image matching for testing includes: If a control with a similarity exceeding a preset similarity to the control in the control picture is matched in the page screenshot, control testing is performed based on the position of the control in the page screenshot.

5. The method according to any one of claims 1-3, characterized in that, the method further includes: When the control to be tested cannot be recognized in the page to be tested, if the corresponding control picture cannot be found in the control picture set based on the keyword, a first prompt message for indicating that the control to be tested cannot be located is output; or If the corresponding control is not successfully matched in the page screenshot based on the found control picture, a second prompt message for indicating that the control to be tested cannot be located is output.

6. A control testing device, characterized in that, the device includes: An acquisition module for acquiring a keyword corresponding to the control to be tested, where the keyword is used to identify the content represented by the control to be tested; An identification module for identifying the controls in the page to be tested through a pre-trained identification model; A first testing module for, when the control to be tested cannot be recognized in the page to be tested, searching for the corresponding control picture in the control picture set based on the keyword, where the control picture set includes multiple control pictures marked with keywords; A second testing module for, when the corresponding control picture is found in the control picture set based on the keyword, performing image matching between the found control picture and the page screenshot of the page to be tested, and locating the control to be tested in the page to be tested based on the result of the image matching for testing; The identification model includes an image recognition model, and the device further includes: A first annotation module for, when the target control in the page to be tested cannot be recognized by the image recognition model, intercepting the control picture corresponding to the target control and annotating the control picture with the target keyword to obtain the target control picture; A second annotation module for adding the target control picture to the control picture set and generating a control annotation file based on the control picture set after adding the target control picture, where the control annotation file is used to indicate the picture information of each control picture in the control picture set and / or the position information of each control picture in the corresponding sample page; A training module for, in response to the generation of the control annotation file, calling the control annotation file to train the image recognition model in the first process to obtain the target recognition model; The identification module is used for: Performing image recognition on the controls in the page to be tested through the target recognition model.

7. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the method according to any one of claims 1-5 are implemented.

8. An electronic device, characterized in that, includes: A memory, on which a computer program is stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images