Selenium and picture matching combined positioning method in automatic test

Abstract

The invention relates to the field of automatic testing, discloses a selenium and picture matching and combining positioning method in automatic testing, and solves the problem that positioning is difficult when no DOM (Document Object Model) node exists in a web page. According to the method, firstly, action functions are compiled, action keywords are defined, the action functions are packaged into a library, and different action sequences are compiled through a JSON file; when web element positioning is carried out, the DOM associated nodes of the elements are searched firstly, if the DOM associated nodes are searched, positioning is carried out through the DOM associated nodes, and a script sequence is executed after positioning succeeds; and if the DOM association node of the element is not found, carrying out picture identification positioning, and executing the script sequence after the positioning is successful.

Description

Technical Field

[0001] The present invention relates to the field of automated testing, and in particular to a method for combining selenium (a tool for testing Web application programs) and image matching for positioning in automated testing. Background Art

[0002] When writing automation scripts in Selenium, Web element positioning needs to rely on basic Web elements. If you encounter content drawn by the JavaScript API, since these contents are not part of the HTML DOM structure, Selenium cannot locate them like ordinary HTML elements. <canvas>This means that Selenium cannot use conventional positioning strategies (such as ID, Class, XPath, etc.) to locate <canvas>Specific graphics or text within. Summary of the invention

[0003] The technical problem to be solved by the present invention is to propose a method for combining selenium and image matching in automated testing to solve the problem of difficulty in positioning when there is no DOM node in the web page.

[0004] The technical solution adopted by the present invention to solve the above technical problems is:

[0005] The method of combining selenium and image matching for positioning in automated testing includes:

[0006] Write action functions, define action keywords, encapsulate action functions into libraries, and write them into different action sequences through JSON files;

[0007] When locating a web element, first search for the DOM associated node of the element. If found, locate it through the DOM associated node. After successful locating, execute the script sequence.

[0008] If the DOM associated node of the element is not found, image recognition and positioning are performed, and the script sequence is executed after successful positioning.

[0009] Furthermore, the action keywords include: selenium click, selenium find element and selenium input information and the like.

[0010] Furthermore, the action functions include: image recognition and positioning, text recognition and positioning, image recognition and click, and text recognition and click.

[0011] Furthermore, the image recognition and positioning includes: based on the defined action function, passing in the xpath (XML path parsing language) of the web page, thereby obtaining the stream data of screenshot_as_base64 (base64 encoded data of the web page screenshot), converting the stream data into an image through base64.b64decode (decoding the base64 encoded data), and then performing template matching through the target template image to determine whether the matching result reaches a preset confidence level. If so, returning the matching result and the positioning is successful, otherwise the positioning fails.

[0012] Furthermore, the target template image is a target element image obtained by taking a screenshot and saving the target element.

[0013] The beneficial effects of the present invention are:

[0014] The present invention provides a method combining Selenium element positioning with image matching positioning, which are used together in the same action sequence. When there is no DOM node in a web page, the position information of the target element is obtained by image matching positioning, thereby solving the problem of difficult positioning when there is no DOM node. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of a method for combining selenium and image matching for positioning in automated testing in an embodiment of the present invention;

[0016] Figure 2 The figure is a flowchart of image matching and positioning in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention aims to enrich the selenium positioning method and enhance the function of selenium by providing additional libraries and tools. Selenium is used for web automation, and image recognition positioning is more used for image recognition and text recognition. When developing automation scripts, the two are effectively combined to enhance the fault tolerance of automation scripts. In the scheme of the present invention, when locating a web element, the DOM associated node of the element is first searched. If it is found, it is located through the DOM associated node, and the script sequence is executed after the positioning is successful; if the DOM associated node of the element is not found, the image recognition positioning is performed, and the script sequence is executed after the positioning is successful. In addition, after the image positioning is completed, the operation of selenium can be continued without interfering with each other. At the same time, in a continuous test script action sequence, the selenium and image positioning return information can be switched at any time and written into the same json operation sequence, so as to realize the effective switching of DOM node positioning and image positioning.

[0018] Example

[0019] See the flow of the method for combining selenium and image matching and positioning in the automated test provided in this embodiment. Figure 1 , which includes the following implementation process:

[0020] Define the action function:

[0021] Specifically, by writing action functions, action keywords are defined, such as selenium click, selenium find element, selenium input information, etc. Combined with image recognition and positioning, text recognition and positioning, image recognition and click, and text recognition and click methods, this series of action functions are encapsulated into a library, and written into different action sequences through JSON files. Each action sequence contains a series of action keywords, that is, the operations performed by the script on the web interface, so as to achieve effective switching between DOM node positioning and image positioning.

[0022] Find the DOM associated node:

[0023] Specifically, when locating a web element, the DOM associated node of the element is first searched. If the associated node can be found, the element is located through the DOM associated node. After the location is successful, the script sequence is executed.

[0024] However, if the element to be located is not part of the HTML DOM structure and has no DOM associated node, Selenium cannot use conventional positioning strategies (such as ID, Class, XPath) to locate it. <canvas>To identify specific graphics or text in the image, you need to switch to image recognition and positioning mode.

[0025] Image recognition and positioning:

[0026] Specifically, the process of image recognition and positioning can be found in Figure 2 , which includes: based on the defined action function, passing in the xpath (XML path parsing language) of the webpage, thereby obtaining the stream data of screenshot_as_base64 (base64 encoded data of the webpage screenshot), converting the stream data into an image through base64.b64decode (decoding the base64 encoded data), and then performing template matching through the target template image, judging whether the matching result reaches the preset confidence level, if so, returning the matching result, and the positioning is successful, otherwise the positioning fails, and an error processing can be performed. The target template image is a target element image that is obtained and saved in advance through a screenshot.

[0027] Based on the above, this embodiment can realize the effective switching between DOM node positioning and image positioning, thereby solving the problem of difficulty in positioning when there is no DOM node in the web page.

[0028] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.< / canvas> < / canvas> < / canvas>

Claims

1. A method for combining selenium and image matching for positioning in automated testing, characterized in that: include: Write action functions, define action keywords, encapsulate action functions into libraries, and write them into different action sequences through JSON files; When locating a web element, first search for the DOM associated node of the element. If found, locate it through the DOM associated node. After successful locating, execute the script sequence. If the DOM associated node of the element is not found, image recognition and positioning are performed, and the script sequence is executed after successful positioning.

2. The method for combining selenium and image matching and positioning in automated testing as claimed in claim 1, characterized in that: The action keywords include: selenium click, selenium search element and selenium input information.

3. The method for combining selenium and image matching and positioning in automated testing as claimed in claim 1, characterized in that: The action functions include: image recognition and positioning, text recognition and positioning, image recognition and clicking, and text recognition and clicking.

4. The method for combining selenium and image matching and positioning in automated testing according to any one of claims 1 to 3, characterized in that: The image recognition and positioning includes: based on the defined action function, passing the xpath of the web page to obtain the stream data of screenshot_as_base64, converting the stream data into an image through base64.b64decode, and then performing template matching through the target template image to determine whether the matching result reaches a preset confidence level. If so, returning the matching result and the positioning is successful, otherwise the positioning fails.

5. The method for combining selenium and image matching and positioning in automated testing as claimed in claim 4, characterized in that: The target template image is a target element image obtained by taking a screenshot and saving the target element.

Images