Match rate calculation device, match rate calculation method, and match rate calculation program

Abstract

In the present invention, a background detection unit (15b) detects a background region in a template image for image matching. A calculation unit (15d) excludes the detected background region and calculates the match rate between the template image and a retrieved image.

Description

Technical Field

[0001] The present invention relates to a matching rate calculation device, a matching rate calculation method, and a matching rate calculation program.

Background Art

[0002] In the automation of operations on a PC or the analysis of operations on a PC, such as RPA (Robotic Process Automation), image matching is used to search for regions that match a template image in a captured image (searched image) on the PC screen (see Non-Patent Document 1). In image matching, in addition to the process of finding a part that exactly matches the template image in the searched image, a process of finding a similar part in the searched image whose matching rate with the template image is equal to or higher than a predetermined threshold is used.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with conventional technologies, setting a threshold for the match rate in image matching can be difficult. For example, the match rate in image matching changes if the size of the margin area changes depending on how the template image is cropped. Also, controls (operable elements) on a PC screen may have visual effects such as changing color when the mouse cursor is over the control (mouseover), and if the non-searched image changes due to the visual effect, the match rate will also change. However, the size of the margin area of ​​the template image and the changes in the non-searched image due to visual effects are difficult for humans to notice, so the resulting fluctuations in the match rate create a discrepancy with human intuition, making it difficult for humans to set a threshold for determining similarity.

[0005] The present invention has been made in view of the above, and aims to make it possible for humans to easily set a threshold for the match rate by aligning the calculation of the match rate in image matching with human intuition. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, the match rate calculation device according to the present invention is characterized by comprising: a background detection unit that detects the background region of a template image for image matching; and a calculation unit that excludes the detected background region and calculates the match rate between the template image and the image to be searched. [Effects of the Invention]

[0007] According to the present invention, it becomes possible for humans to easily set the threshold for the match rate in image matching. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a diagram illustrating the overview of the match rate calculation device. [Figure 2] Figure 2 is a diagram illustrating the overview of the match rate calculation device. [Figure 3]Figure 3 is a diagram illustrating the overview of the match rate calculation device. [Figure 4] Figure 4 is a diagram illustrating the overview of the match rate calculation device. [Figure 5] Figure 5 is a schematic diagram illustrating the general configuration of the match rate calculation device of this embodiment. [Figure 6] Figure 6 is a diagram illustrating the matching rate calculation process. [Figure 7] Figure 7 is a diagram illustrating the processing of the background detection unit. [Figure 8] Figure 8 is a diagram illustrating the processing of the edge detection unit. [Figure 9] Figure 9 is a diagram illustrating the processing of the calculation unit. [Figure 10] Figure 10 is a diagram illustrating the processing of the calculation unit. [Figure 11] Figure 11 is a diagram illustrating the processing of the calculation unit. [Figure 12] Figure 12 is a diagram illustrating the processing of the calculation unit. [Figure 13] Figure 13 is a flowchart showing the matching rate calculation process. [Figure 14] Figure 14 is a diagram illustrating the effect of the matching rate calculation process. [Figure 15] Figure 15 shows an example of a computer running a match rate calculation program. [Modes for carrying out the invention]

[0009] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0010] [Overview of the Match Rate Calculation Device] Figures 1 to 4 are diagrams for explaining the outline of the matching rate calculation device. Image matching is a process of comparing a search image and a template image to find a position in the search image that has a high degree of similarity with the template image. As illustrated in the concept in FIG. 1, in image matching, while shifting (scanning) the position within the search image, a portion of the same size as the template image in the search image is compared, and the matching rate is calculated for each position. When this matching rate is equal to or higher than a predetermined threshold value, it is determined that the corresponding position in the search image is similar to the template image.

[0011] In the present embodiment, the images to be compared are grayscale images in which each pixel takes a value of 0 to 255. However, it is not limited to this, and comparison between color images may also be possible. Also, the matching rate is normalized to 0% to 100%, that is, 0.0 to 1.0, and is set to 100% when they completely match. However, the range of the matching rate is not limited to this. Also, as illustrated in FIG. 2, the matching rate is obtained by normalizing the sum of the absolute values of the pixel differences. However, it is not limited to this, and for example, a value calculated by the square root of the sum of the squares of the pixel differences, the cosine similarity of the pixel differences, etc. may also be used.

[0012] Here, the matching rate varies significantly depending on the method of cutting out from the captured image including the margin of the template image. The margin is the peripheral area of a control (operation target element such as a button, checkbox, text box, etc.) on the PC screen of the operation target, and is usually the background on the PC screen. For example, in the example shown in FIG. 3, the template image shown in FIG. 3(a) is cut out larger than the template image shown in FIG. 3(b), and the margin portion is larger. In this case, since the ratio of the different part (presence or absence of "レ") to the whole becomes relatively small, the matching rate becomes as large as 98.9%.

[0013] In addition, when the color of a control such as a button on the PC screen changes due to a mouse over or the like where the mouse cursor is placed on an element, the matching rate may change even if there is no significant change in the appearance to the human eye. For example, for "Register" in the search target image illustrated in FIG. 4, usually, as illustrated in FIG. 4(a), the matching rate with "Register" in the template image is 100%. On the other hand, as illustrated in FIG. 4(b), when the color of "Register" in the search target image changes due to a mouse over, the matching rate with "Register" in the template image decreases, and the matching rate is lower than that of "Reset" with different characters.

[0014] Therefore, the matching rate calculation device of the present embodiment detects the background area of the template image and excludes it from the contribution to the matching rate. Thereby, it is possible to exclude the margin part of the template image that has little meaning in specifying an element on the PC screen and reduce the influence on the matching rate.

[0015] In addition, an edge area that is an area with a large change compared to the surroundings is detected, and the weight of the contribution to the matching rate is increased. Thereby, the matching rate calculation device increases the contribution of a more characteristic area of the graphic design such as the edge of the control image, and can reduce the influence on the matching rate of a change that occurs in a relatively wide area of the control although the change caused by a visual effect such as a mouse over is small.

[0016] Thus, according to the matching rate calculation device, by bringing the change in the matching rate closer to the human sense and calculating the matching rate without being affected by the method of cutting out the template image or a mouse over or the like, it becomes possible to easily set the threshold for determining the matching rate in image matching.

[0017] [Configuration of Matching Rate Calculation Device] FIG. 5 is a schematic diagram illustrating the schematic configuration of the matching rate calculation device of the present embodiment. As illustrated in FIG. 5, the matching rate calculation device 10 of the present embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0018] The input unit 11 is implemented using an input device such as a keyboard or mouse, and in response to input operations by the operator, it inputs various instruction information, such as processing start, to the control unit 15. The output unit 12 is implemented using a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays the results of the match rate calculation process, which will be described later.

[0019] The communication control unit 13 is implemented using a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device that manages various types of information used in the match rate calculation process.

[0020] The memory unit 14 is implemented using semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or storage devices such as hard disks or optical discs. The memory unit 14 pre-stores processing programs for operating the match rate calculation device 10, as well as data used during the execution of the processing programs, or temporarily stores them each time processing is performed. The memory unit 14 may also be configured to communicate with the control unit 15 via the communication control unit 13.

[0021] The control unit 15 is implemented using a CPU (Central Processing Unit) or the like, and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, a background detection unit 15b, an edge detection unit 15c, and a calculation unit 15d, as illustrated in Figure 5, and performs the match rate calculation process. Note that each or part of these functional units may be implemented on different hardware. For example, the acquisition unit 15a may be implemented on different hardware than the other functional units. Furthermore, the control unit 15 may also include other functional units.

[0022] The acquisition unit 15a acquires the image to be searched and the template image that are the target of the image matching process. For example, the acquisition unit 15a acquires the image to be searched and the template image generated by screen capture via the input unit 11, or via the communication control unit 13 from a user terminal or management device.

[0023] The acquisition unit 15a may acquire the image to be processed and the template image in advance and store them in the storage unit 14, or it may transfer them immediately to the subsequent functional unit without storing them in the storage unit 14.

[0024] The background detection unit 15b detects the background region of the template image for image matching. Specifically, the background detection unit 15b detects pixels of the same color as the four outer edges of the template image as the background region.

[0025] Here, Figure 6 is a diagram illustrating the match rate calculation process. Figure 6(a) is the image to be searched for in the process described below, and Figure 6(b) is the template image. In this embodiment, both are grayscale images. Furthermore, since the image to be searched and the template image are the same size (20 pixels wide and 15 pixels high), it is not necessary to calculate the match rate multiple times while scanning the image to be searched; the match rate is calculated only once.

[0026] Figure 7 is a diagram illustrating the processing of the background detection unit. For example, as shown in Figure 7(a), the background detection unit 15b determines the background color if the color information of the pixels on all four outer edges of the template image is the same. The background detection unit 15b then detects pixels of the same color inward from the pixels on the four edges as the background region.

[0027] The edge detection unit 15c detects edge regions of the template image. Specifically, the edge detection unit 15c detects as an edge region the pixel in the template image that has the largest color difference from the surrounding pixels of the target pixel.

[0028] Here, Figure 8 is a diagram illustrating the processing of the edge detection unit. For example, as shown in Figure 8(a), the edge detection unit 15c calculates the maximum absolute value of the difference between the target pixel of interest and the surrounding pixels. Figure 8(a) illustrates a case where the central pixel is the target pixel, the color difference of the lower right pixel is 150, and the other pixels are the same color with a difference of 0. Then, as shown in Figure 8(b), the maximum difference between each pixel and the surrounding pixels is calculated, and pixels whose calculated value is greater than or equal to a predetermined threshold are identified as edge regions. In the example shown in Figure 8(b), the pixels with a difference value of 150 in the shaded area are detected as edge regions.

[0029] Next, Figures 9 to 12 are diagrams illustrating the processing of the calculation unit. First, the calculation unit 15d calculates the match rate between the template image and the image being searched, excluding the detected background area. Here, the calculation unit 15d calculates the match rate using the pixel-by-pixel difference between the template image and the image being searched.

[0030] For example, the calculation unit 15d compares the template image exemplified in Figure 9(a) with a comparison target portion of the searched image that is the same size as the template image, and calculates the difference value for each pixel, as exemplified in Figure 9(b).

[0031] The calculation unit 15d then identifies the background region of the template image from the difference values ​​for each pixel, as illustrated in Figure 10(a). The calculation unit 15d also identifies pixels in the background region of the template image whose difference value is less than or equal to a predetermined threshold (for example, 0), as background pixels, as illustrated in Figure 10(b). In Figure 10(b), the pixels with diagonal shading from the upper right to the lower left are identified as background pixels. The calculation unit 15d then excludes the identified background pixels from the comparison pixels used to calculate the match rate.

[0032] Furthermore, the calculation unit 15d calculates the match rate between the template image and the image being searched by adding predetermined weights to the edge regions. For example, as illustrated in Figure 11(a), the calculation unit 15d identifies the edge regions of the template image from the difference values ​​for each pixel, and identifies the pixels in the edge regions as weight-increasing pixels.

[0033] In this process, the calculation unit 15d excludes pixels identified as background pixels, as illustrated in Figure 10(b), from the weight increase pixels, as illustrated in Figure 11(b). In Figure 11(b), the pixels with diagonal shading from the top left to the bottom right are identified as weight increase pixels. Alternatively, instead of prioritizing background pixels, weight increase pixels may be prioritized, and pixels identified as weight increase pixels may be excluded from the background pixels.

[0034] Then, the calculation unit 15d calculates the match rate between the comparison target portion of the searched image and the template image, as illustrated in Figure 12. For example, the calculation unit 15d calculates the match rate using the ratio to the maximum possible value of the weighted sum of the difference values. That is, the calculation unit 15d calculates the match rate as 1 - (weighted sum of difference values) ÷ (maximum possible value of the weighted sum of difference values).

[0035] Specifically, as illustrated in Figure 12, the calculation unit 15d first calculates the weighted sum of the difference values ​​between the two images. In the example shown in Figure 12, a predetermined weight of 2.0 is added to the sum of the difference values ​​for the 133 weight-increasing pixels illustrated in Figure 11(b). In addition, the sum of the difference values ​​for the 26 pixels obtained by excluding the 144 background pixels illustrated in Figure 10(b) from the total of 300 pixels is calculated.

[0036] Furthermore, the calculation unit 15d calculates the maximum possible weighted sum of the difference values ​​between the two images. In the example shown in Figure 12, the maximum difference value for each pixel is set to 255, and the maximum value obtained by adding the weight of 2.0 for the 133 weight-increasing pixels is calculated as 255 × 133 × 2.0. Additionally, the maximum possible value for the 26 pixels excluding the background pixels is calculated as 255 × 26.

[0037] The calculation unit 15d then calculates the match rate as {1 - (weighted sum of difference values) ÷ (maximum possible value of the weighted sum of difference values)}. In the example shown in Figure 12, the match rate is calculated to be approximately 0.611 = 61.1%.

[0038] [Match rate calculation process] Next, with reference to Figure 13, the match rate calculation process by the match rate calculation device 10 according to this embodiment will be described. Figure 13 is a flowchart of the match rate calculation process procedure. The flowchart in Figure 13 starts, for example, when the user instructs the device to start.

[0039] First, the acquisition unit 15a acquires the image to be searched and the template image for the image matching process. Then, the background detection unit 15b detects the background region of the template image for image matching (step S1). For example, the background detection unit 15b detects pixels of the same color as the four outer edges of the template image as the background region.

[0040] Furthermore, the edge detection unit 15c detects edge regions of the template image (step S2). For example, the edge detection unit 15c detects as an edge region the pixel in the template image that has the largest color difference from the surrounding pixels of the target pixel.

[0041] Then, the calculation unit 15d calculates the match rate between the template image and the image being searched, excluding the detected background region. The calculation unit 15d also calculates the match rate between the template image and the image being searched by adding predetermined weights to the edge regions (step S3). For example, the calculation unit 15d uses the pixel-by-pixel difference values ​​between the template image and the image being searched to calculate the match rate as 1 - (weighted sum of difference values) ÷ (maximum possible value of the weighted sum of difference values). This completes the series of match rate calculation processes.

[0042] [effect] As described above, in the match rate calculation device 10 of this embodiment, the background detection unit 15b detects the background region of the template image for image matching. The calculation unit 15d excludes the detected background region and calculates the match rate between the template image and the image to be searched.

[0043] Specifically, the calculation unit 15d calculates the match rate using the pixel-by-pixel difference between the template image and the image being searched. The background detection unit 15b detects pixels of the same color as the four outer edges of the template image as the background region.

[0044] As a result, the match rate calculation device 10 can calculate the match rate by excluding the margins of the template image, which have little significance when identifying elements on the PC screen, thereby reducing their impact on the match rate.

[0045] Furthermore, the edge detection unit 15c detects the edge regions of the template image. In this case, the calculation unit 15d adds a predetermined weight to the edge regions and calculates the match rate between the template image and the image being searched.

[0046] Specifically, the edge detection unit 15c detects the pixel with the maximum color difference between the target pixel and surrounding pixels in the template image as an edge region.

[0047] As a result, the match rate calculation device 10 can calculate the match rate by emphasizing common designs such as the edges of the control buttons, thereby reducing the influence of color changes within the edge area caused by mouseover on the match rate.

[0048] Here, Figure 14 is a diagram illustrating the effect of the match rate calculation process. Conventionally, as illustrated in Figure 14(a), the match rate with the image being searched changes depending on how the template image is cropped from the captured image and the color changes due to mouseover, etc. In the example shown in Figure 14(a), the match rate between the template image "Registered" and the image being searched "Reset" changes to 95% and 98% respectively, depending on the size of the margin area of ​​the template image. Furthermore, even if the size of the margin area is the same, the match rate is calculated to be the same 95% for the image being searched "Reset" and the "Registered" image whose color has changed due to mouseover.

[0049] In contrast, the match rate calculation device 10 of this embodiment, as illustrated in Figure 14(b), makes it possible to calculate the match rate by eliminating the influence of the margins of the template image and suppressing the influence of color changes due to mouseover, etc. In the example shown in Figure 14(b), the match rate with the template image "Registered" is 99%, which is closer to human perception, than the 70% match rate with the searched image "Reset".

[0050] In this way, the match rate calculation device 10 can calculate the match rate in image matching in a way that is closer to human intuition, suppressing the influence of cropping methods and mouseovers. Therefore, it becomes easier for humans to set the match rate threshold for determining whether the searched image and the template image are similar.

[0051] [program] It is also possible to create a program that describes the processing performed by the match rate calculation device 10 according to the above embodiment in a language that can be executed by a computer. In one embodiment, the match rate calculation device 10 can be implemented by installing a match rate calculation program that performs the above match rate calculation processing as packaged software or online software on a desired computer. For example, by having the above match rate calculation program run on an information processing device, the information processing device can be made to function as the match rate calculation device 10. The information processing device referred to here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistant). Furthermore, the functions of the match rate calculation device 10 may be implemented on a cloud server.

[0052] Figure 15 shows an example of a computer running a match rate calculation program. Computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070. These components are connected by bus 1080.

[0053] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. A removable storage medium, such as a magnetic disk or optical disk, is inserted into the disk drive 1041. A serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. A video adapter 1060 is connected to, for example, a display 1061.

[0054] Here, the hard disk drive 1031 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. The information described in the above embodiment is stored, for example, in the hard disk drive 1031 or the memory 1010.

[0055] Furthermore, the match rate calculation program is stored in the hard disk drive 1031 as a program module 1093 containing instructions to be executed by the computer 1000, for example. Specifically, the program module 1093 containing instructions for each process executed by the match rate calculation device 10 described in the above embodiment is stored in the hard disk drive 1031.

[0056] Furthermore, the data used for information processing by the match rate calculation program is stored as program data 1094, for example, in the hard disk drive 1031. The CPU 1020 then reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as needed and executes the procedures described above.

[0057] Furthermore, the program module 1093 and program data 1094 related to the match rate calculation program are not limited to being stored on the hard disk drive 1031; for example, they may be stored on a removable storage medium and read by the CPU 1020 via the disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the match rate calculation program may be stored on another computer connected via a network such as a LAN or WAN (Wide Area Network) and read by the CPU 1020 via the network interface 1070.

[0058] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention. [Explanation of symbols]

[0059] 10. Match Rate Calculation Device 11 Input section 12 Output section 13 Communication Control Unit 14 Storage section 15 Control Unit 15a Acquisition part 15b Background detection unit 15c Edge detection unit 15d Calculation Unit

Claims

1. A background detection unit that detects the background region of the template image for image matching, An edge detection unit for detecting the edge region of the template image, A calculation unit that calculates the match rate between the template image and the image being searched, excluding the detected background region, It has, The matching rate calculation device is characterized in that the calculation unit adds a predetermined weight to the pixels in the edge region among the pixel-by-pixel difference values ​​between the template image and the image to be searched, and calculates the matching rate using the ratio of the weighted sum of the difference values ​​to the maximum possible value.

2. The matching rate calculation device according to claim 1, characterized in that the background detection unit detects pixels of the same color as the four outer edges of the template image as a background region.

3. The match rate calculation device according to claim 2, characterized in that the edge detection unit detects the pixel with the maximum color difference between the target pixel of the template image and the surrounding pixels as the edge region.

4. A method for calculating the match rate performed by a match rate calculation device, A background detection process for detecting the background region of a template image for image matching, An edge detection step for detecting the edge region of the template image, A calculation step of calculating the match rate between the template image and the image being searched, after excluding the detected background region, Includes, The calculation step is a method for calculating a match rate, characterized in that a predetermined weight is added to the pixels in the edge region among the pixel-by-pixel difference values ​​between the template image and the image to be searched, and the match rate is calculated using the ratio of the weighted sum of the difference values ​​to the maximum possible value.

5. A background detection step for detecting the background region of a template image for image matching, An edge detection step for detecting the edge region of the template image, A calculation step of calculating the match rate between the template image and the image being searched, after excluding the detected background region, Have the computer run it, The calculation step is a match rate calculation program characterized by assigning a predetermined weight to the pixels in the edge region among the pixel-by-pixel difference values ​​between the template image and the image to be searched, and calculating the match rate using the ratio of the weighted sum of the difference values ​​to the maximum possible value.

Images