Image processing device, program, and image processing method

Abstract

An image processing device (100) comprises: a template analysis unit (102) that identifies, as a reference pixel, a characteristic pixel in a template image, divides the template image into a plurality of regions, and selects, as a comparison pixel, a pixel in a belonging region that is a region to which the reference pixel belongs among the plurality of regions, the pixel being separated by a first distance from the boundary of the belonging region; and a search processing unit (105) that cuts out a plurality of target images from an input image by sequentially cutting out an image having the same size as the template image as a target image from the input image while changing the position, and searches for a target image most similar to the template image from among the plurality of target images using the pixel value of the comparison pixel.

Description

Image processing device, program, and image processing method

[0001] The present disclosure relates to an image processing device, a program, and an image processing method.

[0002] Template matching is a well-known method for detecting specific patterns in an image. Template matching is a technique that compares each part of an input image with a prepared template that represents the pattern to be detected, and then detects the part that most closely resembles the template.

[0003] However, because template matching involves a comparison operation of pixel values, accuracy decreases if the input image is rotated, enlarged, reduced, or deformed relative to the template.

[0004] In the prior art, a feature amount that varies little with respect to rotation and deformation is calculated, and a comparison operation is performed using the feature amount to suppress a decrease in accuracy (see, for example, Patent Document 1).

[0005] International Publication No. 2015 / 029113

[0006] However, conventional techniques require comparison calculations using feature quantities, which requires not only prior analysis of the template but also processing to calculate feature quantities for the input image during the search process, resulting in the problem of complex calculations and an increase in the amount of calculations.

[0007] Therefore, one or more aspects of the present disclosure aim to realize template matching that is simple and requires a small amount of calculation while ensuring robustness against rotation and deformation.

[0008] An image processing device according to one aspect of the present disclosure is characterized by comprising: a template analysis unit that identifies a characteristic pixel in a template image as a reference pixel, divides the template image into a plurality of regions, and selects, as comparison pixels, pixels within an belonging region, which is the region to which the reference pixel belongs, that are located a first distance from a boundary of the belonging region; and a search processing unit that extracts, from an input image, images of the same size as the template image as target images, sequentially cutting them out at different positions, and searches for a target image that is most similar to the template image from among the plurality of target images, using pixel values ​​of the comparison pixels.

[0009] A program according to one aspect of the present disclosure causes a computer to function as: a template analysis unit that identifies a characteristic pixel in a template image as a reference pixel, divides the template image into a plurality of regions, and selects, from the plurality of regions, a pixel in an belonging region to which the reference pixel belongs that is a region that is a region that is a first distance away from a boundary of the belonging region, as a comparison pixel; and a search processing unit that extracts, from an input image, images of the same size as the template image as target images, sequentially cutting out the images while changing their positions, and searches for a target image that is most similar to the template image from among the plurality of target images, using pixel values ​​of the comparison pixels.

[0010] An image processing method according to one aspect of the present disclosure includes: identifying a characteristic pixel in a template image as a reference pixel; dividing the template image into a plurality of regions; selecting, as comparison pixels, pixels within an belonging region, which is the region to which the reference pixel belongs, that are a first distance away from a boundary of the belonging region; extracting, from an input image, images of the same size as the template image as target images while changing their positions, thereby extracting a plurality of target images from the input image; and using pixel values ​​of the comparison pixels, searching for a target image that is most similar to the template image from among the plurality of target images.

[0011] According to one or more aspects of the present disclosure, it is possible to realize simple template matching with a small amount of calculation while ensuring robustness against rotation and deformation.

[0012] FIG. 1 is a block diagram schematically showing the configuration of an image processing device according to Embodiments 1 and 2. (A) to (E) are schematic diagrams for explaining distortion due to rotation of an input image. FIG. 1 is a block diagram schematically showing the configuration of a template analysis unit. (A) to (E) are schematic diagrams for explaining region division. (B) is a block diagram schematically showing the configuration of a search processing unit 105 according to Embodiment 1. (C) is a block diagram schematically showing the configuration of a PC. (D) is a flowchart showing the operation of the image processing device according to Embodiment 1. (E) is a flowchart showing the processing of the template analysis unit. (F) is a flowchart showing the processing of the search processing unit according to Embodiment 1. (F) is a block diagram schematically showing the configuration of a search processing unit according to Embodiment 2. (F) is a flowchart showing the processing of the search processing unit according to Embodiment 2. (F) is a block diagram schematically showing the configuration of an image processing device according to Embodiment 3. (F) is a flowchart showing the operation of the image processing device according to Embodiment 3.

[0013] 1 is a block diagram showing a schematic configuration of an image processing device 100 according to embodiment 1. The image processing device 100 includes a template acquisition unit 101, a template analysis unit 102, an analysis result storage unit 103, an image acquisition unit 104, a search processing unit 105, and a search result output unit 106.

[0014] First, a conventional template matching will be described. In template matching, a template image and an input image are used to detect the position of a pattern that is the same as that of the template image in the input image.

[0015] The principle of template matching is to cut out multiple regions of the same size as the template image from the input image, compare the template image with each of the cut-out regions, and output the most similar region as the matching result.

[0016] The degree to which the template image and the cut-out area overlap, in other words, the similarity, can be defined, for example, by the sum of the squares of the differences between the pixel values ​​of each pixel in the template image and each pixel in the cut-out area. The smaller this value, the higher the similarity, and in the case of a perfect match, the value will be 0. Therefore, the position of the area where this sum is smallest is detected as the matching result.

[0017] This is called a squared difference, but the definition of similarity is not limited to this example. For example, similarity may be defined by other operations such as normalized squared difference, cross-correlation, normalized cross-correlation, correlation coefficient, or normalized correlation coefficient.

[0018] Furthermore, the similarity does not need to be defined using all pixels in the template image, and may be calculated using only the pixel values ​​of a pre-selected portion of pixels. In this case, it is generally appropriate to select pixels that well represent the characteristics of the template image. For example, pixels with large gradient values, corner pixels, pixels selected using SIFT (Scale-Invariant Feature Transform) features, pixels selected using SURF (Speed-Up Robust Features) features, pixels selected using AKAZE (Accelerated KAZE) features, or pixels selected using co-occurrence probability may be selected.

[0019] This embodiment will be described below. The template acquisition unit 101 acquires a template image. The template image may be acquired from a network such as the Internet via a communication interface such as a network interface card (NIC) (not shown), or may be acquired from a storage unit such as a storage (not shown).

[0020] The template analysis unit 102 analyzes the template image acquired by the template acquisition unit 101. Here, the template analysis unit 102 identifies pixels to be used in a search by analyzing the template image. Specifically, the template analysis unit 102 identifies characteristic pixels in the template image as reference pixels. The template analysis unit 102 also divides the template image into a plurality of regions. Then, the template analysis unit 102 selects, as comparison pixels, pixels that are in an belonging region, which is a region to which the reference pixel belongs, and that are a first distance away from the boundary of the belonging region, from among the plurality of divided regions. This comparison pixel is the pixel to be used in a search.

[0021] Here, the template analysis unit 102 can divide the template image into regions consisting of pixels similar to the reference pixel so as to include the reference pixel, and set the regions as belonging regions. Note that the template analysis unit 102 can determine whether a pixel is similar to the reference pixel by, for example, comparing the difference in pixel value, feature amount, or gradient with a threshold.

[0022] The template analysis unit 102 can select, for example, a comparison pixel from pixels on a normal line at a point on the boundary of the belonging region that is closest to the reference pixel. Furthermore, the template analysis unit 102 can select, from among pixels that are a first distance away from the boundary of the belonging region, a pixel that is a second distance away from the reference pixel and has a pixel value closest to the pixel value of the reference pixel or a feature value closest to the feature value of the reference pixel.

[0023] Here, distortion due to rotation of an input image will be described. Figures 2(A) to 2(E) are schematic diagrams for explaining distortion due to rotation of an input image. When a subject depicted in input image 120 shown in Figure 2(A) is rotated in the real world around a straight line in the image plane, such as vertical line L shown at the center of input image 120, as the rotation axis, the subject in area AR in Figure 2(A) is photographed with distortion, compressed in the left-right direction, as the rotation angle g increases, as shown in the images shown in Figures 2(B) to 2(E).

[0024] As described above, template matching is a technique for comparing a template image with multiple images extracted from an input image and detecting the position of the image with the highest similarity. Therefore, if all pixels in the template image are used to calculate the similarity, there may be cases where the similarity is not necessarily high at the correct position due to such distortion. In such cases, the accuracy of matching decreases.

[0025] Therefore, the template analysis unit 102 identifies pixels that are robust to distortion and whose pixel values ​​fluctuate little when such distortion occurs. Figure 3 is a block diagram showing a schematic configuration of the template analysis unit 102. The template analysis unit 102 includes a feature calculation unit 102a, a reference pixel selection unit 102b, an area division unit 102c, and a comparison pixel selection unit 102d.

[0026] The feature calculation unit 102a calculates the feature of each pixel in the template image. The feature to be calculated may be any feature, but for example, feature such as SIFT, SURF, AKAZE, or co-occurrence probability can be used.

[0027] The reference pixel selection unit 102b determines a reference pixel that serves as a reference for selecting a pixel to be compared with the input image from among the pixels in the template pixel array based on the feature calculated by the feature calculation unit 102a. Any method for selecting a reference pixel based on the feature may be used. For example, a predetermined number of feature values ​​may be identified in descending order, descending order, descending order of the average absolute value of the difference between the feature values ​​of predetermined surrounding pixels, or descending order of the average absolute value of the difference between the feature values ​​of predetermined surrounding pixels, and a pixel corresponding to the identified feature value may be selected as the reference pixel. Alternatively, a pixel that is an extreme value or an inflection point in the feature distribution may be selected as the reference pixel.

[0028] The region dividing unit 102c divides the template image into a plurality of regions. By dividing the template image into a plurality of regions, it becomes possible to clarify to which region in the template image the reference pixel selected by the reference pixel selecting unit 102b belongs.

[0029] 4 is a schematic diagram for explaining the division of the region. The reference pixel RD shown in Fig. 4 is one of the reference pixels selected by the reference pixel selection unit 102b in the above manner.

[0030] To clarify the region to which the reference pixel belongs, the region division unit 102c, for example, analyzes pixels surrounding the reference pixel RD and determines whether the surrounding pixels belong to the same region as the reference pixel RD or a different region, thereby dividing the region around the reference pixel RD. The determination of whether they belong to the same region can be achieved, for example, by comparing the pixel values, feature values, or gradients of the reference pixel RD and the surrounding pixels with a predetermined threshold. By comparing with the threshold, pixels that can be determined to be similar to the reference pixel can be determined to belong to the same region as the reference pixel.

[0031] Furthermore, the region dividing unit 102c may divide the entire template image into regions other than the periphery of the reference pixel RD. In the example shown in Fig. 4, the template image is divided into seven regions: a foreground region indicated by regions AR1, AR2, AR3, AR4, and AR5, and a background region indicated by regions AR6 and AR7. Here, region AR2 is determined to be the region to which the reference pixel RD belongs. Such region division can be achieved, for example, by semantic segmentation, but any known division method may be used in this embodiment.

[0032] 3, the comparison pixel selection unit 102d selects comparison pixels that are pixels to be used in the comparison calculation of template matching in the search processing unit 105. Then, the comparison pixel selection unit 102d associates the pixel positions and pixel values ​​of the selected comparison pixels and stores them in the analysis result accumulation unit 103 as analysis results.

[0033] The reference pixels selected by the reference pixel selection unit 102b are calculated based on feature values, and therefore are often distributed near the boundaries of regions, and their pixel values ​​fluctuate greatly in response to distortion of the input image. Therefore, the pixel values ​​of the reference pixels are vulnerable to distortion. Therefore, the comparison pixel selection unit 102d selects pixels that are robust to distortion.

[0034] First, when selecting one comparison pixel from one reference pixel, the comparison pixel selection unit 102d selects a pixel that belongs to the same region as the reference pixel from among the multiple regions divided by the region division unit 102c and is located a predetermined distance away from the boundary of that region. The distance used as the threshold here is also referred to as the first distance.

[0035] Any method may be used to select a specific pixel from pixels located a predetermined distance from the boundary of the region. For example, the comparison pixel selection unit 102d may select, as the comparison pixel, a pixel located a predetermined distance (also referred to as a second distance) in the normal direction of the point closest to the reference pixel on the boundary of the region including the reference pixel. Alternatively, the comparison pixel selection unit 102d may select, as the comparison pixel, an optimal pixel located a predetermined distance from the boundary of the region including the reference pixel. The optimal pixel may be determined, for example, by having a pixel value closest to the reference pixel or a feature value closest to the reference pixel.

[0036] Furthermore, when selecting multiple comparison pixels from one reference pixel, the comparison pixel selection unit 102d can additionally select pixels that are a predetermined distance away from the reference pixel. In this case, the comparison pixel selection unit 102d can select one pixel as a comparison pixel and then select other comparison pixels, just as in the case of selecting one comparison pixel for one reference pixel as described above. The other comparison pixels may not be included in the same region as the reference pixel, but may be included in a different region. However, it is desirable that the comparison pixels be at least a threshold distance away from the boundary of the region.

[0037] In the example of Figure 4, among the candidate pixels D1 to D4 that are located at a predetermined distance above, below, left, and right from the reference pixel RD, pixel D2, which is included in the same area AR2 as the reference pixel RD and has a pixel value closest to that of the reference pixel RD, can be selected as the comparison pixel.

[0038] Here, the candidate pixels are not limited to the four pixels at the top, bottom, left, and right, but may be pixels located on a circumference that is a predetermined distance away from the reference pixel. The predetermined distance may also be multiple distances. The pixel information of the comparison pixels selected in this manner is stored in the analysis result storage unit 103 as the analysis result.

[0039] 1, the analysis result storage unit 103 stores the analysis results that are the results of the analysis performed by the template analysis unit 102. As described above, the analysis results are pixel information that indicates the positions and pixel values ​​of comparison pixels.

[0040] The image acquisition unit 104 acquires an input image, which is an image to be searched based on the template image.

[0041] The search processing unit 105 performs template matching on the input image based on the template image. For example, the search processing unit 105 cuts out multiple target images from the input image by sequentially cutting out images of the same size as the template image from the input image while changing their positions. Then, the search processing unit 105 uses the pixel values ​​of the comparison pixels to search for the target image that is most similar to the template image from among the multiple target images as a second similar target image.

[0042] 5 is a block diagram showing a schematic configuration of the search processing unit 105 in Embodiment 1. The search processing unit 105 includes a comparison pixel extraction unit 105a, a comparison calculation unit 105b, and a search result calculation unit 105c.

[0043] The comparison pixel extraction unit 105a receives the input image from the image acquisition unit 104 and the analysis result of the template image from the analysis result storage unit 103, and provides pixel information used for template matching to the comparison operation unit 105b.

[0044] The comparison calculation unit 105b receives pixel information used for template matching from the comparison pixel extraction unit 105a, compares pixel values, and calculates the similarity. The calculated similarity is provided to the search result calculation unit 105c.

[0045] The search result calculation unit 105c receives the similarity from the comparison calculation unit 105b as the comparison calculation result and determines the matching result of the template matching. Here, the matching result may be the position information indicating the position of the area with the highest similarity.

[0046] Returning to FIG. 1, the search result output unit 106 outputs the search result obtained by the search processing unit 105 .

[0047] The image processing device 100 described above can be realized by a computer such as a PC 10 shown in Fig. 6. The PC 10 includes a storage 11, a memory 12, a processor 13, a communication I / F 14, an input I / F 15, and a display 16.

[0048] The storage 11 is an auxiliary storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The memory 12 is a volatile or non-volatile memory. The processor 13 is an arithmetic device such as a CPU (Central Processing Unit). The communication I / F 14 is a communication device such as a NIC. The input I / F 15 is an input device such as a keyboard and a mouse. The display 16 is a display device.

[0049] For example, the template acquisition unit 101, the template analysis unit 102, the image acquisition unit 104, the search processing unit 105, and the search result output unit 106 can be realized by the processor 13 reading a program stored in the storage 11 into the memory 12 and executing the program.

[0050] The analysis result accumulation unit 103 can be realized by the storage 11 or the memory 12 .

[0051] 7 is a flowchart showing the operation of the image processing device 100 according to the first embodiment. First, the template acquisition unit 101 acquires a template image (S10). The acquired template image is provided to the template analysis unit 102. Here, one template image may be acquired, or multiple template images may be acquired.

[0052] The template analysis unit 102 analyzes the template image and stores information on pixels used in the search as the analysis result in the analysis result storage unit 103 (S11). If there are multiple template images, information on pixels used in the search is analyzed from each of the multiple template images.

[0053] The image acquisition unit 104 acquires an input image (S12). The acquired input image is provided to the search processing unit 105.

[0054] The search processing unit 105 reads the analysis results of the template image from the analysis result storage unit 103 (S13).

[0055] The search processing unit 105 then searches for a portion of the input image that is most similar to the template image (S14). As described above, the search process is determined by the similarity. In other words, an area of ​​the same size as the template image is cut out from the input image, the similarity is calculated according to a defined similarity calculation formula, and the most similar portion is searched for while changing the cut-out position. Note that if there are multiple template images, the similarity is calculated individually for each template image, and the result output indicates which template image the most similar cut-out position was compared with.

[0056] 8 is a flowchart showing the processing in the template analysis unit 102. First, the feature amount calculation unit 102a calculates the feature amounts of pixels for the acquired template image (S20).

[0057] Next, the reference pixel selection unit 102b selects reference pixels based on the calculated feature amounts (S21). At this time, the number of reference pixels selected may be a predetermined number, or may be determined based on the feature amount distribution of the template image.

[0058] The number of reference pixels to be selected may be determined based on, for example, the average value and variance of the feature amounts. Specifically, the number may be increased or decreased as the average value of the feature amounts increases. For example, if the number increases as the average value increases, the number may be determined by the following formula (1): NUM = w × ave + min (1) where NUM is the number, w is a predetermined positive coefficient, ave is the average of the feature amounts, and min is a predetermined minimum number. However, it is desirable to set an upper limit to the number NUM. The variance can also be determined in the same manner as above.

[0059] When both the mean and the variance are used, the number may be determined by averaging (for example, weighted average) the numbers obtained as described above using the mean and the variance.

[0060] On the other hand, if the number decreases as the average value increases, the number may be determined by the following formula (2): NUM = z × ave + max (2), where z is a predetermined negative coefficient and max is a predetermined maximum number. However, it is desirable to set a lower limit to the number NUM.

[0061] The number of reference pixels to be selected may be determined by the maximum and minimum feature values, or by a combination of these. In this case, the average ave in equation (1) or (2) may be the maximum feature value, the minimum feature value, or a value obtained by subtracting the minimum feature value from the maximum feature value.

[0062] The above average, maximum value, and minimum value may be calculated for the entire template image, or may be calculated for each divided region of the template image.

[0063] Next, the region dividing unit 102c divides the template image into a plurality of regions (S22).

[0064] Finally, the comparison pixel selection unit 102d determines comparison pixels, which are pixels to be used in the search, corresponding to the divided regions, based on the reference pixel (S23).

[0065] 9 is a flowchart showing the processing in the search processing unit 105 in embodiment 1. First, the comparison pixel extraction unit 105a reads the analysis results of the template image from the analysis result storage unit 103, and identifies comparison pixels that are pixels to be used for comparison (S30).

[0066] Next, the comparison operation unit 105b cuts out an area of ​​the input image having the same size as the template image as a target image (S31).

[0067] Next, the comparison operation unit 105b extracts the pixel value of a pixel located at the same position as the comparison pixel from the target image (S32).

[0068] Next, the comparison calculation unit 105b compares the extracted pixel value with the pixel value of the comparison pixel to calculate the similarity (S33).

[0069] Then, the comparison operation unit 105b determines whether or not the entire region of the input image has been cut out as the target image (S34). If the entire region of the input image has been cut out as the target image (Yes in S34), the process proceeds to step S35. If the entire region of the input image has not been cut out as the target image (No in S34), the process proceeds to step S31. In this case, in step S31, the comparison operation unit 105b cuts out the target image so as to include the region that has not been cut out from the input image.

[0070] In step S35, the search result calculation unit 105c outputs the most similar cut-out position as the search result based on the similarity.

[0071] As described above, by selecting pixels that are robust against variations in pixel values ​​and performing template matching, it is possible to perform matching with high accuracy even when the image is distorted due to rotation or the like.

[0072] Second Embodiment As shown in FIG. 1 , an image processing device 200 according to the second embodiment includes a template acquisition unit 101, a template analysis unit 102, an analysis result storage unit 103, an image acquisition unit 104, a search processing unit 205, and a search result output unit 206.

[0073] The template acquisition unit 101, the template analysis unit 102, the analysis result storage unit 103, and the image acquisition unit 104 of the image processing device 200 according to the second embodiment are similar to the template acquisition unit 101, the template analysis unit 102, the analysis result storage unit 103, and the image acquisition unit 104 of the image processing device 100 according to the first embodiment.

[0074] The search processing unit 205 performs the same processing as the search processing unit 105 in embodiment 1, and also analyzes the distribution of similarities calculated by the comparison calculation unit 105 b to calculate the rotation angle that is causing distortion of the input image. For example, the search processing unit 205 searches for the target image that is most similar to the template image from among multiple target images as the most similar target image, and uses the most similar target image to calculate the rotation angle at which the input image is distorted relative to the template image.

[0075] 10 is a block diagram schematically illustrating the configuration of the search processing unit 205 according to Embodiment 2. The search processing unit 205 includes a comparison pixel extraction unit 105a, a comparison calculation unit 105b, a search result calculation unit 105c, a comparison result analysis unit 205d, and a rotation angle calculation unit 205e.

[0076] Comparison pixel extraction unit 105a, comparison operation unit 105b, and search result calculation unit 105c in embodiment 2 are similar to comparison pixel extraction unit 105a, comparison operation unit 105b, and search result calculation unit 105c in embodiment 1. However, comparison pixel extraction unit 105a in embodiment 2 also provides information on pixels used in template matching to comparison result analysis unit 205d, and comparison operation unit 105b provides the similarity calculated using those pixels to comparison result analysis unit 205d.

[0077] The comparison result analysis unit 205d receives information about the pixels used in comparing both the input image and the template image, in other words, the pixel positions and pixel values, from the comparison pixel extraction unit 105a, and further receives the similarity, which is the comparison result, from the comparison calculation unit 105b, and analyzes these.

[0078] The rotation angle calculation unit 205e receives the analysis result from the comparison result analysis unit 205d and calculates the rotation angle of the rotation that is the cause of the distortion. The calculated rotation angle is provided to the search result output unit 206.

[0079] 1, the search result output unit 206 outputs the search result obtained by the search processing unit 205. In the second embodiment, the search result includes the rotation angle calculated by the search processing unit 205.

[0080] Fig. 11 is a flowchart showing the processing in the search processing unit 205 in embodiment 2. Note that, among the steps included in the flowchart shown in Fig. 11, steps that perform processing similar to the processing of the steps included in the flowchart shown in Fig. 9 are assigned the same reference numerals as in the flowchart in Fig. 9.

[0081] The processing in steps S30 to S33 in Fig. 11 is the same as the processing in steps S30 to S33 in Fig. 9. However, in Fig. 11, after the processing in step S33, the processing proceeds to step S40.

[0082] In step S40, the comparison result analysis unit 205d stores the similarity calculated in step S33 in association with the position from which the target image was clipped in step S31. The process then proceeds to step S34.

[0083] The processes of steps S34 and S35 in Fig. 11 are the same as the processes of steps S34 and S35 in Fig. 9. However, in Fig. 11, after the process of step S35, the process proceeds to step S41.

[0084] In step S41, the comparison result analysis unit 205d identifies the cutout position with the highest similarity from the comparison calculation unit 105b. The comparison result analysis unit 205d may also calculate the slope, average value, or variance of the distribution of similarities using the highest similarity and similarities calculated from target images with cutout positions within a predetermined range around the cutout position with the highest similarity. The analysis results from the comparison result analysis unit 205d are provided to the rotation angle calculation unit 205e.

[0085] Next, the rotation angle calculation unit 205e calculates the rotation angle of the input image based on the analysis result of the comparison result analysis unit 205d. For example, the rotation angle calculation unit 205e may calculate the rotation angle using a predetermined calculation formula based on the maximum similarity value indicating the highest similarity, or the slope, average value, or variance of the similarity distribution. Specifically, the rotation angle calculation unit 205e may calculate the rotation angle so that the higher the maximum similarity, the smaller the rotation angle.

[0086] Furthermore, when there is a bias in the distribution of similarities, in other words, when there is a difference in the slope, average value, or variance between a first direction from the cut-out position with the highest similarity and a second direction that is the opposite direction to the first direction, the rotation angle calculation unit 205e can determine that the image is rotating in the direction where the slope, average value, or variance of the first direction is larger than the slope, average value, or variance of the second direction.

[0087] In this case, for example, the rotation angle calculation unit 205e may calculate the rotation angle such that the greater the absolute value of the difference between the slope, average value, or variance in the first direction and the slope, average value, or variance in the second direction, the greater the angle in the direction in which that value is greater. For example, the rotation angle calculation unit 205e may calculate the rotation angle using the following equation (3): Ang = c × dif (3) where Ang is the rotation angle, c is a predetermined coefficient, and dif is the absolute value of the difference between the slope, average value, or variance in the first direction and the slope, average value, or variance in the second direction.

[0088] The difference between the gradient in the first direction and the gradient in the second direction is the difference between the gradient of the similarity of the target image, which has the crop position of the highest similarity as the reference, at a position a predetermined distance away in the first direction from the reference, and the gradient of the similarity of the target image, which has the crop position of the highest similarity as the reference, at a position within a predetermined range centered at a position a predetermined distance away in the first direction from the reference, and the gradient of the similarity of the target image, which has the crop position of the highest similarity as the reference, at a position within a predetermined range centered at a position a predetermined distance away in the second direction from the reference. Furthermore, the difference between the variance in the first direction and the variance in the second direction is the difference between the variance of the target image whose cut-out position is a position within a predetermined range centered at a position a predetermined distance away from the cut-out position with the highest similarity in the first direction, and the variance of the target image whose cut-out position is a position within a predetermined range centered at a position a predetermined distance away from the cut-out position in the second direction.

[0089] As described above, in the second embodiment, not only is position detection performed by template matching, but the similarity used for matching is analyzed and the rotation angle is calculated, thereby making it possible to specify or control the position and orientation taking the rotation angle into consideration in alignment control that feeds back the specified position.

[0090] 12 is a block diagram showing a schematic configuration of an image processing device 300 according to embodiment 3. The image processing device 300 includes a template acquisition unit 101, a template analysis unit 102, an analysis result storage unit 103, an image acquisition unit 104, a search processing unit 205, a search result output unit 306, a distortion correction unit 307, and a re-search processing unit 308.

[0091] The template acquisition unit 101, template analysis unit 102, analysis result storage unit 103, and image acquisition unit 104 of the image processing device 300 according to the third embodiment are similar to the template acquisition unit 101, template analysis unit 102, analysis result storage unit 103, and image acquisition unit 104 of the image processing device 100 according to the first embodiment. Furthermore, the search processing unit 205 of the image processing device 300 according to the third embodiment is similar to the search processing unit 205 of the image processing device 200 according to the second embodiment.

[0092] The distortion correction unit 307 receives the input image from the image acquisition unit 104, receives the rotation angle from the search processing unit 205, and corrects the distortion of the input image to generate a corrected input image. The generated corrected input image is provided to the re-search processing unit 308.

[0093] The re-search processing unit 308 receives the corrected input image provided from the distortion correction unit 307, receives the analysis result of the template image from the analysis result storage unit 103, and executes the search process by template matching again. In other words, the re-search processing unit 308 uses the corrected input image to perform pattern matching with the template image.

[0094] The search result output unit 306 outputs the search result obtained by the re-search processing unit 308 .

[0095] Fig. 13 is a flowchart showing the operation of image processing device 300 according to embodiment 3. Among the steps included in the flowchart shown in Fig. 13, steps that perform the same processing as the steps included in the flowchart shown in Fig. 7 are assigned the same reference numerals as in the flowchart in Fig. 7.

[0096] The processing of steps S10 to S13 in Fig. 13 is the same as the processing of steps S10 to S13 in Fig. 7. However, after step S13 in Fig. 13, the processing proceeds to step S50.

[0097] In step S50, the search processing unit 205 searches for a portion of the input image that is most similar to the template image, as described in the second embodiment, and calculates the rotation angle of the input image.

[0098] Next, the distortion correction unit 307 corrects the distortion caused by rotation of the input image using the rotation angle calculated in step S50, and generates a corrected input image that is an image without distortion (S51). The corrected input image is provided to the re-search processing unit 308.

[0099] Here, any known technology may be used as a method for correcting image distortion due to rotation based on the rotation angle. For example, the distortion correction unit 307 is geometrically formulated by the following equation (4), and therefore, this equation (4) may be used.

[0100] Rotation has three degrees of freedom, which are expressed by three rotations around three orthogonal straight lines. If the rotation angles corresponding to each rotation axis are the roll rotation angle φ, the pitch rotation angle θ, and the yaw rotation angle ψ, the three-dimensional coordinates in real space before and after the rotation are expressed by equation (4). (4) Here, the coordinates before rotation are (x, y, z), and the coordinates after rotation are (x', y', z').

[0101] By using the camera parameters obtained by prior calibration, it is possible to convert between three-dimensional coordinates in space and two-dimensional coordinates on the image. Therefore, by combining the camera parameters with equation (4), it is possible to calculate the positional relationship of each pixel on the image before and after rotation.

[0102] Once the correspondence between pixels before and after rotation is clear, an image with corrected distortion can be generated by generating an image in which each pixel of the distorted image is moved to its pre-rotation position.

[0103] Then, the re-search processing unit 308 performs the search process by template matching again using the corrected input image (S52). Here, the template matching shown in Fig. 9 may be performed, or template matching using the reference pixel may be performed.

[0104] Although the distortion correction unit 307 is supplied with the rotation angle from the search processing unit 205, the rotation angle may also be supplied from a source other than the search processing unit 205. For example, the distortion correction unit 307 may receive rotation information of the target object from another sensor when capturing an input image and generate a corrected input image based on this information, or may use both the rotation angle input from the sensor and the rotation angle supplied from the search processing unit 205. When both are used, the re-search processing unit 308 may perform template matching using two types of corrected input images corrected using the respective rotation angles, or one corrected input image may be generated using a weighted average of the rotation angle input from the sensor and the rotation angle supplied from the search processing unit 205. Furthermore, multiple corrected input images may be generated while changing the weight of the weighted average, and the re-search processing unit 308 may perform template matching on the multiple images.

[0105] As described above, in the third embodiment, the input image is corrected in accordance with the rotation angle and the search process is performed again, thereby performing template matching using an image without distortion, and enabling matching with higher position detection accuracy.

[0106] As described above, according to the first to third embodiments, template matching processing that is robust against rotation and deformation can be performed using simple and lightweight calculations.

[0107] 100, 200, 300 Image processing device, 101 Template acquisition unit, 102 Template analysis unit, 102a Feature calculation unit, 102b Reference pixel selection unit, 102c Area division unit, 102d Comparison pixel selection unit, 103 Analysis result storage unit, 104 Image acquisition unit, 105, 205 Search processing unit, 105a Comparison pixel extraction unit, 105b Comparison calculation unit, 105c Search result calculation unit, 205d Comparison result analysis unit, 205e Rotation angle calculation unit, 106, 206, 306 Search result output unit, 307 Distortion correction unit, 308 Re-search processing unit.

Claims

1. Identify characteristic pixels in the template image as reference pixels, divide the template image into a plurality of regions, and select, as comparison pixels, pixels within the region to which the reference pixel belongs among the plurality of regions and that are at a first distance from the boundary of the region. A template analysis unit; From the input image, by sequentially cutting out images of the same size as the template image as target images while changing the position, a plurality of target images are cut out from the input image, and using the pixel values of the comparison pixels, among the plurality of target images, A search processing unit that searches for a target image that is most similar to the template image. An image processing apparatus characterized by comprising:

2. The template analysis unit divides, from the template image, a region composed of pixels similar to the reference pixel so as to include the reference pixel, and sets the region as the belonging region. The image processing apparatus according to claim 1, characterized in that:

3. The template analysis unit determines whether a pixel is a pixel similar to the reference pixel by comparing the difference in pixel value, feature amount, or gradient with a threshold value. The image processing apparatus according to claim 2, characterized in that:

4. The template analysis unit selects the comparison pixel from pixels on the normal line at the point closest to the reference pixel at the boundary. The image processing apparatus according to any one of claims 1 to 3, characterized in that:

5. The template analysis unit selects, as the comparison pixel, a pixel that is at a second distance from the reference pixel among the pixels at the first distance from the boundary and that has a pixel value closest to the pixel value of the reference pixel or a feature amount closest to the feature amount of the reference pixel. The image processing apparatus according to any one of claims 1 to 3, characterized in that:

6. The search processing unit searches for a target image that is most similar to the template image among the plurality of target images as the most similar target image, and uses the most similar target image to calculate a rotation angle by which the input image is distorted with respect to the template image. The image processing apparatus according to any one of claims 1 to 5, characterized in that:

7. An image processing apparatus according to claim 6, further comprising: a distortion correction unit that generates a corrected input image by correcting distortion of the input image using the rotation angle; and a re-search processing unit that performs pattern matching with the template image using the corrected input image.

8. A program causing a computer to function as: a template analysis unit that specifies a characteristic pixel in a template image as a reference pixel, divides the template image into a plurality of regions, and selects, as comparison pixels, pixels within a belonging region, which is the region to which the reference pixel belongs among the plurality of regions, and that are at a first distance from the boundary of the belonging region; and a search processing unit that sequentially extracts a plurality of target images from the input image by extracting, while changing the position, an image of the same size as the template image from the input image as a target image, and searches for a target image most similar to the template image among the plurality of target images using the pixel values of the comparison pixels.

9. An image processing method comprising: specifying a characteristic pixel in a template image as a reference pixel, dividing the template image into a plurality of regions, and selecting, as comparison pixels, pixels within a belonging region, which is the region to which the reference pixel belongs among the plurality of regions, and that are at a first distance from the boundary of the belonging region; sequentially extracting a plurality of target images from the input image by extracting, while changing the position, an image of the same size as the template image from the input image as a target image; and searching for a target image most similar to the template image among the plurality of target images using the pixel values of the comparison pixels.

Images