A method for generating an automated program based on selenium and image recognition

Abstract

This invention discloses a method for generating automated programs based on Selenium and image recognition, relating to the field of automated program management technology. According to the expected operation steps, materials are archived according to rules; an extraction program is executed; the archived materials are read; an image recognition service is called to extract the coordinates of elements; a JSON configuration file for triggering the program is generated based on config.json; different JSON configuration files are generated for different materials archived according to requirements; the trigger program is executed; the JSON configuration file is specified; the trigger program performs automated operations based on preset operations and element coordinates; and different inspection processes are implemented by specifying different JSON configuration files according to requirements.

Description

Technical Field

[0001] This invention discloses a method for generating automated programs based on Selenium and image recognition, which relates to the field of automated program management technology. Background Technology

[0002] The main tasks of automating development using Selenium consist of two parts. The first part is writing the logic code for the inspection steps using programming languages ​​such as Java and Python. The second part is locating elements using methods such as XPath and CSS selectors, which is also the most labor-intensive part in typical scenarios.

[0003] To lower the barrier to entry for automated program development, users need to learn process configuration rules and replace writing inspection step logic code by dragging and dropping elements to build flowcharts and configuring JSON. However, element localization still requires users to learn positioning methods such as XPath and CSS selectors. Image recognition for element localization can indeed solve this problem to some extent, but relying solely on element localization has issues such as low recognition rate and element occlusion. Summary of the Invention

[0004] This invention addresses the problems of existing technologies by providing a method for generating automated programs based on Selenium and image recognition. By archiving operations and related materials of the operation objects, image recognition technology is used to extract the coordinates of the operation objects. Selenium then performs automated operations based on the coordinates, and finally completes the corresponding automated program operations according to different preset steps.

[0005] The specific solution proposed in this invention is as follows: This invention provides a method for generating automated programs based on Selenium and image recognition, which archives materials according to rules and anticipated operational steps. The extraction program is executed to read archived materials, call image recognition services to extract element coordinates, and generate a JSON configuration file for triggering the program based on config.json. Different JSON configuration files are generated according to the requirements of archived materials. The trigger program is executed by specifying a JSON configuration file. The trigger program performs automated operations based on preset operations and element coordinates. Different inspection processes can be implemented by specifying different JSON configuration files according to requirements.

[0006] Furthermore, the method for generating automated programs based on Selenium and image recognition includes preset rules for numbering the order of steps, specifying the content stored in the step directory, and indicating the content identified in the JSON of the operation steps.

[0007] Furthermore, in the method for generating automated programs based on Selenium and image recognition, the steps are numbered according to rules, including: starting with a number, separating the number and the step name with a hyphen, and customizing the step name.

[0008] Furthermore, in the method for generating automated programs based on Selenium and image recognition, the content stored in the step directory is specified according to the rules, including: specifying that each step directory stores a reference image, an operation object icon, and an operation step json file. There are no requirements for the naming of the reference image and the operation object icon, but there is only one operation step json file, and it must be named config.json.

[0009] Furthermore, in the method for generating automated programs based on Selenium and image recognition, the operation steps are specified in the JSON file according to the rules, including: identifying the operation type, recognition type, reference image, operation, timeout, input, horizontal offset, and vertical offset in config.json.

[0010] This invention also provides an apparatus for generating automated programs based on Selenium and image recognition, including an archiving module, an extraction module, and a trigger generation module. The archiving module archives materials according to the expected operational steps and rules. The extraction module executes the extraction program, reads archived materials, calls the image recognition service to extract the coordinates of elements, and generates a JSON configuration file for triggering the program based on config.json. Different JSON configuration files are generated according to the requirements of archiving different materials. The trigger generation module executes the trigger program, which specifies a JSON configuration file. The trigger program performs automated operations based on preset operations and element coordinates. Different inspection processes can be implemented by specifying different JSON configuration files according to requirements.

[0011] Furthermore, the archiving module in the device based on Selenium and image recognition to generate automated programs has preset rules for numbering the order of steps, specifying the content stored in the step directory, and indicating the content identified in the operation step JSON.

[0012] Furthermore, in the device for generating automated programs based on Selenium and image recognition, the archiving module numbers the steps in sequence according to rules, including: starting with a number, separating the number and the step name with a hyphen, and customizing the step name.

[0013] Furthermore, in the device for generating automated programs based on Selenium and image recognition, the archiving module stores content in the specified steps according to the rules, including: specifying that each step directory stores a reference image, an operation object icon, and an operation step JSON. The naming of the reference image and the operation object icon is not required, but there is only one operation step JSON, and the name must be config.json.

[0014] Furthermore, in the device for generating automated programs based on Selenium and image recognition, the archiving module specifies the content identified in the operation steps JSON according to the rules, including: identifying the operation type, recognition type, reference image, operation, timeout, input, horizontal offset, and vertical offset in config.json.

[0015] The advantages of this invention are: Based on the archived materials according to the steps, the extraction program reads the archived materials, extracts the operation object JSON, and combines image recognition to collect element coordinates. The code then uses these coordinates to locate the elements more efficiently and accurately, triggering the program to read the operation object JSON and complete the automated operation. Ultimately, this allows for the completion of corresponding automated operations based on different preset steps and materials. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the extraction program's execution flow.

[0017] Figure 2 This is a diagram illustrating the process of triggering the program.

[0018] Figure 3 This is a schematic diagram of the element location process. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention. Example

[0020] This invention provides a method for generating automated programs based on Selenium and image recognition: Step 1: File the materials according to the rules and following the expected operating procedures.

[0021] The preset rules are used to number the steps, specify the content stored in the step directory, and indicate the content identified in the operation step JSON.

[0022] Number the steps in the following order: start with a number, separate the number and the step name with a hyphen, and use a custom name for the step.

[0023] The rules stipulate that the following content should be stored in the step directory: each step directory should contain a baseline diagram, an operation object icon, and an operation step JSON file. There are no naming requirements for the baseline diagram and the operation object icon. There should be only one operation step JSON file, and it must be named config.json.

[0024] The operation steps are specified in the JSON file according to the rules, including: operation type, identification type, baseline graph, operation, timeout, input, horizontal offset, and vertical offset.

[0025] Step 2: Execute the extraction program, read the archived materials, and call the image recognition service to extract the coordinates of the elements. The image recognition service can provide a baseline image and the target screenshot or text to be recognized. The specific coordinates of the target screenshot or text are returned through image recognition and OCR technology. Based on the coordinates, the relevant element set is determined by JavaScript, and then interfering elements are eliminated by iframes and user operation types, and finally the target element is determined.

[0026] The trigger program's configuration file JSON is generated based on config.json. Different configuration file JSONs are generated according to the different materials to be archived as needed. Step 3: Execute the trigger program and specify the configuration file JSON. The trigger program will perform automated operations based on preset operations and element coordinates. Different inspection processes can be implemented by specifying different configuration files JSON according to requirements. Example

[0027] This invention also provides an apparatus for generating automated programs based on Selenium and image recognition, including an archiving module, an extraction module, and a trigger generation module. The archiving module archives materials according to the expected operational steps and rules. The extraction module executes the extraction program, reads archived materials, calls the image recognition service to extract the coordinates of elements, and generates a JSON configuration file for triggering the program based on config.json. Different JSON configuration files are generated according to the requirements of archiving different materials. The trigger generation module executes the trigger program, which specifies a JSON configuration file. The trigger program performs automated operations based on preset operations and element coordinates. Different inspection processes can be implemented by specifying different JSON configuration files according to requirements.

[0028] The information interaction and readable program execution processes of the processor in the above-mentioned device are based on the same concept as those in the method embodiments of the present invention, and the specific details can be found in the descriptions in the method embodiments of the present invention, and will not be repeated here.

[0029] Similarly, the device of this invention archives materials according to steps, extracts the archived materials through a program, extracts the JSON object of the operation, and combines image recognition to collect element coordinates. Then, the element is located using code based on these coordinates, enabling more efficient and accurate element location. This triggers the program to read the JSON object of the operation and completes the automated operation. Ultimately, it achieves the ability to complete corresponding automated operations based on different preset steps and materials.

[0030] It should be noted that not all steps and modules in the above processes and device structures are mandatory; some steps or modules may be omitted as needed. The execution order of the steps is not fixed and can be adjusted as required. The device structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0031] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for generating automated programs based on Selenium and image recognition, characterized by: File the materials according to the expected operational procedures and in accordance with the rules. The extraction program is executed to read archived materials, call image recognition services to extract element coordinates, and generate a JSON configuration file for triggering the program based on config.json. Different JSON configuration files are generated according to the requirements of archived materials. The trigger program is executed by specifying a JSON configuration file. The trigger program performs automated operations based on preset operations and element coordinates. Different inspection processes can be implemented by specifying different JSON configuration files according to requirements.

2. The method for generating automated programs based on Selenium and image recognition according to claim 1, characterized by preset rules for numbering the order of steps, specifying the content stored in the step directory, and indicating the content identified in the operation step JSON.

3. The method for generating automated programs based on Selenium and image recognition according to claim 2, characterized in that: Number the steps in the following order: start with a number, separate the number and the step name with a hyphen, and use a custom name for the step.

4. The method for generating automated programs based on Selenium and image recognition according to claim 2, characterized in that: The rules stipulate that the following content should be stored in the step directory: each step directory should contain a baseline diagram, an operation object icon, and an operation step JSON file. There are no naming requirements for the baseline diagram and the operation object icon. There should be only one operation step JSON file, and it must be named config.json.

5. The method for generating automated programs based on Selenium and image recognition according to claim 2, characterized in that: The operation steps are specified in the JSON file according to the rules, including: operation type, identification type, baseline graph, operation, timeout, input, horizontal offset, and vertical offset.

6. An apparatus for generating automated programs based on Selenium and image recognition, characterized in that: It includes an archiving module, an extraction module, and a trigger generation module. The archiving module archives materials according to the expected operational steps and rules. The extraction module executes the extraction program, reads archived materials, calls the image recognition service to extract the coordinates of elements, and generates a JSON configuration file for triggering the program based on config.json. Different JSON configuration files are generated according to the requirements of archiving different materials. The trigger generation module executes the trigger program, which specifies a JSON configuration file. The trigger program performs automated operations based on preset operations and element coordinates. Different inspection processes can be implemented by specifying different JSON configuration files according to requirements.

7. The apparatus for generating an automated program based on Selenium and image recognition according to claim 6, characterized in that: The archiving module has preset rules for numbering the order of steps, specifying the content stored in the step directory, and indicating the content identified in the JSON of the operation steps.

8. The apparatus for generating an automated program based on Selenium and image recognition according to claim 7, characterized in that: The archiving module numbers the steps according to the following rules: in the order of steps, the steps begin with a number, and the number and step name are separated by a hyphen (-). Step names can be customized.

9. The apparatus for generating an automated program based on Selenium and image recognition according to claim 7, characterized in that: The archiving module stores content in the specified steps according to the rules, including: each step directory contains a baseline diagram, an operation object icon, and an operation step JSON file. There are no naming requirements for the baseline diagram and the operation object icon, but there must be only one operation step JSON file, and it must be named config.json.

10. The apparatus for generating an automated program based on Selenium and image recognition according to claim 7, characterized in that: The archiving module specifies the content identified in the operation steps JSON according to the rules, including: operation type, identification type, baseline graph, operation, timeout, input, horizontal offset, and vertical offset in config.json.

Images