A grid-based image matching positioning method, system, device and medium

Abstract

The present application relates to the technical field of image processing, in particular to a grid-based image matching positioning method, system, device and medium; the method divides the input image into grids, performs template matching of the image according to the grid position, and reduces the input image matched to the smallest range, greatly reducing the number of matching comparisons, improving the running efficiency of the RPA process and the accuracy of image positioning; by setting a priority quadrant, the efficiency of positioning matching is improved, the input image is reduced to the smallest range, the number of matching is reduced under different resolutions, and the running efficiency and positioning accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a grid-based image matching and localization method, system, device, and medium. Background Technology

[0002] In RPA process development, when locating and manipulating UI elements, the process essentially involves calling interfaces provided by the software containing those elements. Examples include desktop applications developed using MFC, WTL, WinForm, and WPF frameworks. However, many software programs within desktop systems do not provide UI element interfaces, such as RDP (Windows Remote Desktop), VMware, Citrix, and other virtual environments, or UI elements in DirectUI-based software and games like Flash. In these cases, image-based element location is necessary, combined with mouse and keyboard actions to automate operations. Image location techniques typically employ template matching based on OpenCV.

[0003] Template matching based on OpenCV refers to finding the most similar part of image B within the current image A. Image A is generally called the input image, and image B is called the template image. The operation method of template matching is to slide the template image B across image A, traversing all pixels to complete the matching.

[0004] When image A is directly used for template matching with image B under different resolutions, the matching similarity is low and a match cannot be made normally. During the development of the RPA process, the execution efficiency and matching accuracy of image localization are not high. In addition, the RPA process is incompatible with different desktop scaling and resolutions when running in development and production environments. Furthermore, when there are multiple template images in the input image, there are problems with inaccurate matching. Summary of the Invention

[0005] This invention addresses the problems of low similarity, inaccurate matching, and incompatibility when directly using image B for template matching of image A under different resolutions. It proposes a grid-based image matching and localization method, system, device, and medium. This method first divides the input image into a grid, then performs matching and localization according to the grid positions. By setting priority quadrants, the efficiency of localization and matching is improved. The input image is reduced to a minimum range, and the number of matching operations is reduced under different resolutions, thus improving operational efficiency and localization accuracy.

[0006] The specific implementation details of this invention are as follows:

[0007] A grid-based image matching and localization method specifically includes the following steps:

[0008] Step 1: Generate a template image based on the current screen resolution;

[0009] Step 2: Preprocess the template image according to the set image scaling ratio;

[0010] Step 3: Match the input image with the preprocessed template image according to the set image scaling ratio to obtain a matching image, and set the matching priority quadrant according to the matching image;

[0011] Step 4: Obtain the grid coordinates in the selector, and capture the corresponding position in the input image according to the grid coordinates. Match the captured image with the matching image and output the matching result.

[0012] To better implement the present invention, the specific operation of step 1 is as follows: according to the resolution of the current screen, select the target screen to divide into grids according to the set ratio, generate a template image, and record the grid position of the first point in the upper left corner of the template image and the current screen resolution.

[0013] To better realize the present invention, step 2 further includes the following steps:

[0014] Step 21: Obtain a screenshot of the current screen and the current screen resolution;

[0015] Step 22: Read the selector information, save the template image to the selector to obtain the matching image, and obtain the selector resolution;

[0016] Step 23: Determine the relationship between the current screen resolution and the selector resolution. If the current screen resolution is greater than the selector resolution, enlarge the matching image according to the set ratio. If the current screen resolution is less than the selector resolution, shrink the screenshot according to the set ratio and save it.

[0017] To better implement the present invention, step 3 further includes the following steps: dividing the image to be input into a grid according to a set image scaling ratio, and matching the image to be input with the preprocessed template image according to a set priority quadrant;

[0018] The priority quadrants include the first priority quadrant, the second priority quadrant, and the third priority quadrant;

[0019] The first quadrant is used to specify the grid where the grid position is located;

[0020] The second quadrant is used to match the four grids above, below, left, and right of a specified grid position after the first priority quadrant fails to match;

[0021] The third quadrant is used to specify a nine-square grid centered on a specified grid for matching after the second priority quadrant fails to match.

[0022] To better realize the present invention, step 4 further includes the following steps:

[0023] Step 41: Divide the screenshot image into a grid;

[0024] Step 42: Obtain the grid coordinates in the selector;

[0025] Step 43: Based on the grid coordinates, obtain the screenshot image at the corresponding position in the screenshot image;

[0026] Step 44: Match the screenshot image with the matching image according to the priority quadrant, and output the matching result.

[0027] Based on the above-mentioned grid-based image matching and localization method, in order to better realize the present invention, a grid-based image matching and localization system is further proposed, including an initialization unit, a partitioning unit, and a matching unit;

[0028] The initialization unit is used to generate a template image according to the current screen resolution and to preprocess the template image according to a set image scaling ratio.

[0029] The segmentation unit is used to match the input image with the preprocessed template image according to the set image scaling, and set the matching priority quadrant;

[0030] The matching unit is used to obtain the grid coordinates in the selector, and according to the grid coordinates, to capture a screenshot image of the corresponding position in the input image, match the screenshot image with the template image, and output the matching result.

[0031] Based on the above-mentioned grid-based image matching and localization method, in order to better realize the present invention, an electronic device is further proposed, including a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, the above-mentioned grid-based image matching and localization method is implemented.

[0032] Based on the grid-based image matching and localization method proposed above, in order to better realize the present invention, a computer-readable storage medium is further proposed, wherein computer instructions are stored on the computer-readable storage medium; when the computer instructions are executed on the above-mentioned electronic device, the above-mentioned grid-based image matching and localization method is realized.

[0033] The present invention has the following beneficial effects:

[0034] (1) The present invention divides the input image into grids and performs template matching of the image according to the grid position, reducing the input image to the smallest range for image matching, which greatly reduces the number of matching comparisons and improves the running efficiency of the RPA process and the accuracy of image positioning.

[0035] (2) When performing image positioning and matching, the present invention improves the running efficiency by setting different levels of quadrants.

[0036] (3) This invention improves the stability of the RPA process by processing images at different resolutions. At the same time, the element search and location capabilities of the RPA system are greatly improved by using a grid-based image localization method. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating the principle of finding a 2x2 image in a 10x10 resolution image based on pixel matching.

[0038] Figure 2 This is a schematic diagram of an image A with a resolution of 200x100 provided in an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of a matching image B generated at a resolution of 200x100, provided as an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of an image A with a resolution of 200x100 provided in an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of the image preprocessing process provided in an embodiment of the present invention.

[0042] Figure 6 This is a schematic diagram of the first priority quadrant provided for an embodiment of the present invention.

[0043] Figure 7 This is a schematic diagram of the second priority quadrant provided for an embodiment of the present invention.

[0044] Figure 8 A schematic diagram of the third priority quadrant provided for an embodiment of the present invention.

[0045] Figure 9 This is a schematic diagram of the positioning and matching process provided in an embodiment of the present invention. Detailed Implementation

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0048] The terms and explanations used in the embodiments of this invention are as follows:

[0049] RPA: Robotic Process Automation is used to replace or assist in completing repetitive, fixed-process tasks in computer equipment.

[0050] Selector: Use the mouse on the Windows desktop to select relevant information about the target you want to focus on.

[0051] Template matching: A technique for finding the part of an image that best matches (is similar to) another template image.

[0052] Input image: This is the image we want to find that matches the template; it is usually a large image.

[0053] Template image: The "object image patch" we search for in the input image, usually a known small image.

[0054] Example 1:

[0055] This embodiment proposes a grid-based image matching and localization method, which specifically includes the following steps:

[0056] Step 1: Generate a template image based on the current screen resolution.

[0057] Further, the specific operation of step 1 is as follows: according to the current screen resolution, select the target screen to divide into grids according to the set ratio, generate a template image, and record the grid position of the first point in the upper left corner of the template image and the current screen resolution.

[0058] Step 2: Preprocess the template image according to the set image scaling ratio.

[0059] Furthermore, step 2 specifically includes the following steps:

[0060] Step 21: Obtain a screenshot of the current screen and the current screen resolution;

[0061] Step 22: Read the selector information, save the template image to the selector to obtain the matching image, and obtain the selector resolution;

[0062] Step 23: Determine the relationship between the current screen resolution and the selector resolution. If the current screen resolution is greater than the selector resolution, reduce the size of the matching image according to the set ratio. If the current screen resolution is less than the selector resolution, reduce the size of the screenshot according to the set ratio and save it.

[0063] Step 3: Match the input image with the preprocessed template image according to the set image scaling ratio to obtain a matching image, and set the matching priority quadrant according to the matching image.

[0064] Further, step 3 specifically includes the following steps: dividing the image to be input into a grid according to a set image scaling ratio, and matching the image to be input with the preprocessed template image according to a set priority quadrant;

[0065] The priority quadrants include the first priority quadrant, the second priority quadrant, and the third priority quadrant;

[0066] The first quadrant is used to specify the grid where the grid position is located;

[0067] The second quadrant is used to match the four grids above, below, left, and right of a specified grid position after the first priority quadrant fails to match;

[0068] The third quadrant is used to specify a nine-square grid centered on a specified grid for matching after the second priority quadrant fails to match.

[0069] Step 4: Obtain the grid coordinates in the selector, and capture the corresponding position in the input image according to the grid coordinates. Match the captured image with the matching image and output the matching result.

[0070] Furthermore, step 4 specifically includes the following steps:

[0071] Step 41: Divide the screenshot image into a grid;

[0072] Step 42: Obtain the grid coordinates in the selector;

[0073] Step 43: Based on the grid coordinates, obtain the screenshot image at the corresponding position in the screenshot image;

[0074] Step 44: Match the screenshot image with the matching image according to the priority quadrant, and output the matching result.

[0075] Working principle: This embodiment uses grid-based localization to narrow down the input image for image matching to the smallest possible range, significantly reducing the number of matching comparisons and improving the efficiency and accuracy of RPA process localization. Simultaneously, processing images at different resolutions enhances the stability of the RPA process. Furthermore, the grid-based image localization method greatly improves the element search and localization capabilities of the RPA system.

[0076] Example 2:

[0077] This embodiment is based on the above embodiment 1, such as... Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 As shown, a specific embodiment will be described in detail.

[0078] This embodiment mainly addresses the issues of execution efficiency and matching accuracy during image localization in RPA process development, as well as compatibility issues caused by different desktop scaling and resolutions when the RPA process runs in development and production environments. It also solves the problem of inaccurate matching when multiple template images exist in the input image during image localization.

[0079] Based on existing technology, this embodiment first divides the image into grids and records the grid positions of the positioning elements when using image positioning. During matching, the input image is also divided into grids according to the same rules. Precise positioning is achieved through grid positions, thereby narrowing the range of the input image, improving execution efficiency, and avoiding the problem of matching multiple results.

[0080] When generating a matching template image B, the resolution of the generated image is recorded and saved to the image data resource. The next time template matching is performed, the current resolution and the resolution of the template image are compared. Based on this, the image's aspect ratio is scaled to form a new matching template image B, thereby improving the accuracy of template matching at different resolutions.

[0081] The resolution of the current screen when the image is selected is saved to the current image B data resource. When the next matching is performed, if image A is generated in high resolution, the resolution difference is used to calculate the image reduction ratio. The reduction ratio is used to re-acquire the input image of image B in high resolution, and then the template matching method is used to perform image matching.

[0082] like Figure 1 The diagram illustrates the principle of finding a 2x2 image within a 10x10 resolution image using pixel matching. OpenCV-based template matching involves finding the most similar part of image B within the current image A. Image A is generally referred to as the input image, and image B as the template image. The template matching operation involves sliding the template image B across image A, traversing all pixels to complete the match.

[0083] If the size of the input image (original image) is W * H and the size of the template is w * h, then the number of comparisons is (W-w+1)*(H-h+1).

[0084] like Figure 1 As shown, the input image size is 10*10, and the template image size is 2*2. Therefore, the number of comparisons is (10-2+1)×(10-2+1), or 9×9. In other words, the template image needs to be compared a total of 9×9=81 times within the input image.

[0085] The input image is usually the size of the current screen, such as 1366*768, while the positioning image is usually smaller, such as a tiny button which is about 80 * 28. The number of comparisons would then be (1366-80+1)×(768-28+1), or 953667.

[0086] To improve efficiency, the common approach is to reduce the size of the input image, specify the recognition range of the input image, and then take a screenshot. However, this range can also cause problems when there are scaling issues or different resolutions.

[0087] This embodiment uses grid positioning to reduce the input image to the smallest possible range, greatly reducing the number of comparisons.

[0088] In practical RPA scenarios, traditional image template matching can lead to situations where the main image A is generated at different resolutions, but the matching template image B remains unchanged. For example... Figure 2 The image shown is a schematic diagram of the formation of image A at a resolution of 200x100; as shown Figure 3 The image shown is a schematic diagram of the matching image B generated at a resolution of 200x100; as shown Figure 4 The image shown is a schematic diagram of the formation of image A with a resolution of 400x200.

[0089] When image A is directly matched with image B as a template at a resolution of 400x200, the similarity of the match is low and the match cannot be found.

[0090] This embodiment uses images A generated at different resolutions. Before matching, the images are reasonably scaled to improve the similarity of template matching. The specific implementation steps are as follows:

[0091] Step 1: Select the target image.

[0092] Based on the current screen resolution and following best practice proportions, the screen containing the target is divided into a grid. During image localization, a template image is generated, and the grid position of the top-left corner of the template image, along with the current screen resolution, is recorded. The specific process is as follows:

[0093] {

[0094] / / Template image resource "name":"ff6a5634-d47e-4c66-a2f8-83635d70be30.png",

[0095] / / Grid position where the template image is located

[0096] "grid":{

[0097] "x":4,

[0098] "y":2

[0099] },

[0100] / / Current resolution information

[0101] "screen":{

[0102] "height":1366,

[0103] "width":768

[0104] }

[0105] }

[0106] Step 2: Preprocessing.

[0107] Before performing image matching, the matching template image is optimized, and after calculating and dividing the image into a grid, the matching range is narrowed down according to the grid. The template matching of the image is then performed. The processing steps are as follows:

[0108] Step 21: Capture the current screen image and save it to PICA, then obtain the current screen display resolution W1:H1.

[0109] Step 22: Read selector information, read the template matching image and save it to image PICB, and obtain the resolution information W2:H2 from the selector.

[0110] Step 23: When the resolution of the screenshot is greater than the resolution in the selector, the matching image needs to be scaled down proportionally.

[0111] Image length = Image length * (W2 / W1);

[0112] Image width = Image width * (H2 / H1);

[0113] Step 24: When the screenshot resolution is smaller than the resolution in the selector, the screenshot needs to be scaled down proportionally.

[0114] Image length = Image length * (W1 / W2);

[0115] Image width = Image width * (H1 / H2);

[0116] Step 25: Regenerate and save the scaled image according to the resolution to the screenshot image PICA and the matching image PICB.

[0117] Step 3: Set the matching priority quadrant.

[0118] When performing image matching, the input image is divided into grids according to the same proportion. Images within the specified grid positions and the surrounding nine-square grid are matched according to priority.

[0119] Three levels of quadrants:

[0120] First priority quadrant: The grid where the specified grid location is located;

[0121] Second priority quadrant: If a match fails, then match the four grids above, below, left, and right of the specified grid position.

[0122] Third priority quadrant: If the match still fails, expand the matching area to a 3x3 grid centered on the specified grid position.

[0123] If no match is found in any of the three quadrants, a matching failure message will be displayed.

[0124] Step 4: Matching steps.

[0125] Step 41: Divide the screenshot image PICA into a grid.

[0126] Step 42: Obtain the grid coordinates in the selector.

[0127] Step 43: Based on the coordinates, obtain the image PICC at the corresponding position in the screenshot image PICA.

[0128] Step 44: Perform template matching between PICC and PICB.

[0129] Step 45: If the matching fails, expand the range of the input image according to the matching strategy and try matching again.

[0130] This embodiment divides the input image into a grid and performs image template matching according to the grid position; when performing image positioning matching, different levels of quadrants are used to improve efficiency; in desktop image template matching, it is not affected by screen resolution, thus improving the matching success rate.

[0131] The other parts of this embodiment are the same as those in Embodiment 1 above, so they will not be described again.

[0132] Example 3:

[0133] Based on any one of Embodiments 1-2 above, this embodiment proposes a grid-based image matching and localization system, including an initialization unit, a partitioning unit, and a matching unit;

[0134] The initialization unit is used to generate a template image according to the current screen resolution and to preprocess the template image according to a set image scaling ratio.

[0135] The segmentation unit is used to match the input image with the preprocessed template image according to the set image scaling, and set the matching priority quadrant;

[0136] The matching unit is used to obtain the grid coordinates in the selector, and according to the grid coordinates, to capture a screenshot image of the corresponding position in the input image, match the screenshot image with the template image, and output the matching result.

[0137] This embodiment also proposes an electronic device, including a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, it implements the above-described grid-based image matching and localization method.

[0138] This embodiment also proposes a computer-readable storage medium storing computer instructions; when the computer instructions are executed on the aforementioned electronic device, the aforementioned grid-based image matching and localization method is implemented.

[0139] The other parts of this embodiment are the same as any one of the above embodiments 1-2, so they will not be described again.

[0140] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A grid-based image matching and localization method, characterized in that, Specifically, the following steps are included: Step 1: Generate a template image based on the current screen resolution; Step 2: Preprocess the template image according to the set image scaling ratio; Step 2 specifically includes the following steps: Step 21: Take a screenshot of the current screen and save it to PICA with the current screen resolution W1:H1; Step 22: Read the selector information, save the template image to the selector to obtain the matching image PICB, and obtain the selector resolution W2:H2; Step 23: Determine the relationship between the current screen resolution and the selector resolution. If the current screen resolution is greater than the selector resolution, enlarge the matching image according to the set ratio. If the current screen resolution is less than the selector resolution, shrink the screenshot according to the set ratio and save it. Regenerate and save the image after scaling according to the resolution to the screenshot image PICA and the matching image PICB. Step 3: Set the matching priority quadrant based on the matched image; The specific operation of setting the matching priority quadrant in step 3 is as follows: when performing image matching, the input image is divided into grids according to the same proportion, and the images in the specified grid positions and the surrounding nine grids are matched according to priority. Step 4: Obtain the grid coordinates in the selector, and capture the corresponding position in the input image according to the grid coordinates. Match the screenshot image PICA with the matching image PICB, and output the matching and positioning result. Step 4 specifically includes the following steps: Step 41: Divide the screenshot image PICA into a grid; Step 42: Obtain the grid coordinates in the selector; Step 43: Based on the grid coordinates, obtain the screenshot image PICC at the corresponding position in the screenshot image PICA; Step 44: Match the screenshot image PICC with the matching image PICB according to the priority quadrant set in Step 3, and output the matching and positioning results.

2. The grid-based image matching and localization method according to claim 1, characterized in that, The specific operation of step 1 is as follows: according to the current screen resolution, select the target screen to divide into grids according to the set ratio, generate a template image, and record the grid position of the first point in the upper left corner of the template image and the current screen resolution.

3. The grid-based image matching and localization method according to claim 1, characterized in that, Step 3 specifically includes the following steps: dividing the image to be input into a grid according to a set image scaling ratio, and matching the image to be input with the preprocessed template image according to a set priority quadrant; The priority quadrants include the first priority quadrant, the second priority quadrant, and the third priority quadrant; The first priority quadrant is used to specify the grid where the grid location is located; The second priority quadrant is used to match the four grids above, below, left, and right of a specified grid position after the first priority quadrant fails to match; The third priority quadrant is used to specify a nine-square grid centered on a specified grid for matching after the second priority quadrant fails to match.

4. A grid-based image matching and localization system, used to execute the grid-based image matching and localization method as described in claim 1; characterized in that, Includes initialization unit, partitioning unit, and matching unit; The initialization unit is used to generate a template image according to the current screen resolution and to preprocess the template image according to a set image scaling ratio. The partitioning unit is used to set the matching priority quadrant; The matching unit is used to obtain the grid coordinates in the selector, and according to the grid coordinates, to capture a screenshot image of the corresponding position in the input image, match the screenshot image with the template image, and output the matching and positioning result.

5. An electronic device, characterized in that, It includes a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, it implements the grid-based image matching and localization method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions; when the computer instructions are executed on the electronic device as described in claim 5, they implement the grid-based image matching and localization method as described in any one of claims 1-3.

Images