Similar area detection device, similar area detection method and program

Abstract

The similar area detection device of the embodiment comprises an acquisition unit, a feature point extraction unit, a matching unit, an outermost contour extraction unit and a detection unit. The acquisition unit acquires a first image and a second image. The feature point extraction unit extracts feature points of each of the first image and the second image. The matching unit establishes a correspondence between the feature points extracted from the first image and the feature points extracted from the second image, and detects corresponding points between the images. The outermost contour extraction unit extracts an outermost contour from each of the first image and the second image. The detection unit detects local areas similar to each other in the first image and the second image, i.e., similar areas, from each of the first image and the second image based on the outermost contour and the number of corresponding points.

Description

Technical Field

[0001] The present invention relates to a similar area detection device, a similar area detection method and a program. Background Art

[0002] Template matching is known as a method for determining the similarity of images. Template matching is a technique for comparing a template image with an image of a comparison object and detecting a portion similar to the template image from the image of the comparison object. However, in template matching, although an area similar to the entire template image can be detected from the image of the comparison object, an area similar to a local area of ​​the template image cannot be detected.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-72937 Summary of the invention

[0006] Problems to be solved by the invention

[0007] An object of the present invention is to provide a similar region detection device, a similar region detection method, and a program capable of detecting, from respective images, similar regions which are local regions similar to each other.

[0008] The similar area detection device of the embodiment comprises an acquisition unit, a feature point extraction unit, a matching unit, an outermost contour extraction unit and a detection unit. The acquisition unit acquires a first image and a second image. The feature point extraction unit extracts feature points of each of the first image and the second image. The matching unit establishes a correspondence between the feature points extracted from the first image and the feature points extracted from the second image, and detects corresponding points between the images. The outermost contour extraction unit extracts an outermost contour from each of the first image and the second image. The detection unit detects local areas similar to each other in the first image and the second image, i.e., similar areas, from each of the first image and the second image based on the outermost contour and the number of corresponding points. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a block diagram showing a functional configuration example of a similar area detection device according to an embodiment.

[0010] Figure 2 This is a flowchart showing an example of the processing procedure of the similar area detection device according to the embodiment.

[0011] Figure 3 The diagrams show specific examples of the first image and the second image.

[0012] Figure 4: is a diagram showing an example of corresponding points.

[0013] Figure 5 This is a diagram showing an example of the outermost contour.

[0014] Figure 6 This is a diagram showing an example of a method for determining whether or not a corresponding point is located inside the outermost contour.

[0015] Figure 7 This is a diagram showing an example of the relationship between the outermost contour and the corresponding points.

[0016] Figure 8 is a diagram showing an example of a similar image pair.

[0017] Fig. 9 is a diagram showing an example of a similar image pair.

[0018] Fig.10 is a diagram showing an example of a similar image pair.

[0019] Fig.11 This is a diagram for explaining an example of a method of confirming the positional relationship of corresponding points.

[0020] Fig.12 This is a diagram for explaining another example of feature point matching.

[0021] Fig.13 is a diagram showing an example of a similar image pair.

[0022] Fig.14 This is a block diagram showing a hardware configuration example of a similar area detection device according to an embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, a similar region detection device, a similar region detection method, and a program according to the embodiments will be described in detail with reference to the accompanying drawings.

[0024] <Overview of Embodiment>

[0025] The similar region detection device of the embodiment is used to detect a local region similar to each other between two images, that is, a similar region from each image, and in particular, detects the similar region based on a combination of feature point matching and outermost contour extraction.

[0026] Feature point matching refers to the following technology, which extracts feature points representing the features of the images from two images respectively, and establishes a correspondence between the feature points extracted from one image and the feature points extracted from the other image, for example, based on the approximation of the local feature values ​​of each feature point. The feature points that establish a correspondence between the images are called corresponding points. Outermost contour extraction refers to the technology of extracting the outermost contour (outermost contour) of an object such as a graphic contained in an image. In this embodiment, on the assumption that an object containing a large number of corresponding points in one image is similar to any object in the other image, the area within the outermost contour containing a large number of corresponding points is detected as a similar area for each of the two images.

[0027] In addition, as a method for detecting similar areas, a method using only feature point matching can also be considered. That is, a method of detecting the area surrounded by corresponding points obtained by feature point matching from two images as a similar area. However, there is a problem in this method: instead of detecting the entire object that is similar between the two images, only the area surrounded by corresponding points in the object is detected as a similar area; or when corresponding points exist in a part of an object that is not similar between the two images, an area containing a part of the object is detected as a similar area. In contrast, in the present embodiment, since the structure for detecting similar areas is based on a combination of feature point matching and outermost contour extraction, the entire object that is similar between the two images can be properly detected as a similar area.

[0028] The similar region detection device of the present embodiment can be effectively used, for example, in the purpose of automatically generating case data (learning data) for learning (supervised learning) a feature extractor used for similar image retrieval including local similarity. In general similar image retrieval, feature quantities representing features of an image are extracted according to a query instruction, and the feature quantities of the query image are compared with the feature quantities of a registered image, thereby retrieving similar images similar to the query image. On the other hand, in similar image retrieval including local similarity, for example, region extraction is performed on both the query image and the registered image, and the extracted local regions are compared. In this way, images that are only partially similar can be retrieved. As one of the means for improving the retrieval accuracy of such similar image retrieval, there is a method of learning a feature extractor so that the feature quantities of images judged to be similar become closer to each other. In this way, similar images that could not be retrieved before learning can be retrieved.

[0029] In order to learn the similarity of such a feature extractor, a similar image pair of one image and two similar images similar to it is required. In the case where similar images contain local similarities, instead of making the whole images into a similar image pair, it is necessary to make the local images from which similar regions of both sides are extracted into a similar image pair. In the case where such a similar image pair is required, there is a method of, for example, manually comparing multiple images and indicating the parts judged as similar regions. However, for this method, it takes a huge amount of time to obtain a large amount of learning data. In contrast, if the similar region detection device of the present embodiment is used, similar image pairs of local images can be automatically generated without human intervention, and the feature extractor used for similar image retrieval containing local similarities can be efficiently learned.

[0030] <First Embodiment>

[0031] Figure 1 FIG. 1 is a block diagram showing a functional configuration example of a similar area detection device according to the present embodiment. Figure 1 As shown, the similar region detection device according to the present embodiment includes an acquisition unit 1 , a feature point extraction unit 2 , a matching unit 3 , an outermost contour extraction unit 4 , a detection unit 5 , and an output unit 6 .

[0032] The acquisition unit 1 acquires the first image and the second image to be processed from outside the device, and transmits the acquired first image and second image to the feature point extraction unit 2 , the outermost contour extraction unit 4 , and the output unit 6 .

[0033] The first image and the second image to be processed are specified, for example, by a user using the similar area detection device. That is, if the user specifies a path indicating the storage location of the first image, the acquisition unit 1 reads the first image stored in the path. Similarly, if the user specifies a path indicating the storage location of the second image, the acquisition unit 1 reads the second image stored in the path. In addition, the image acquisition method is not limited to this, and for example, images taken by the user using a camera or a scanner may be acquired as the first image and the second image.

[0034] The feature point extraction unit 2 extracts feature points of each of the first image and the second image acquired by the acquisition unit 1, calculates the local feature value of each extracted feature point, and transmits information on the feature points and local feature values ​​of each of the first image and the second image to the matching unit 3.

[0035] The extraction of feature points and the calculation of local feature quantities use a scale-invariant and rotation-invariant method such as SIFT (Scale-Invariant Feature Transform). In addition, the method of extracting feature points and calculating local feature quantities is not limited to this, and other methods such as SURF (Speeded-Up Robust Features) or AKAZE (Accelerated KAZE) may also be used.

[0036] The matching unit 3 performs feature point matching to establish a correspondence between the feature points extracted from the first image and the feature points extracted from the second image based on the approximation of the local feature values ​​of each feature point, detects the feature points that establish a correspondence between the images (hereinafter referred to as "corresponding points"), and passes the information of the corresponding points of each image to the detection unit 5.

[0037] For example, the matching unit 3 establishes a correspondence between each feature point extracted from the first image and a feature point having the closest local feature value among the feature points extracted from the second image. At this time, for a feature point among the feature points extracted from the first image that cannot be uniquely determined as a feature point having the closest local feature value among the feature points extracted from the second image, the correspondence with the feature point extracted from the second image may not be performed. Furthermore, for a feature point among the feature points extracted from the first image whose difference in local feature value between the feature point having the closest local feature value among the feature points extracted from the second image exceeds a reference value, the correspondence with the feature point extracted from the second image may not be performed.

[0038] In addition, the matching unit 3 may establish a correspondence between each feature point extracted from the first image and a feature point having the closest local feature value among the feature points extracted from the second image, instead of establishing a correspondence between each feature point extracted from the first image and a feature point having the closest local feature value among the feature points extracted from the second image. Alternatively, the matching unit 3 may establish a correspondence between each feature point extracted from the first image and a feature point having the closest local feature value among the feature points extracted from the second image, and establish a correspondence between each feature point extracted from the second image and a feature point having the closest local feature value among the feature points extracted from the first image, that is, perform a bidirectional correspondence. In the case of performing such a bidirectional correspondence, only feature points with consistent bidirectional correspondences may be detected as corresponding points.

[0039] The outermost contour extraction unit 4 extracts the outermost contours (outermost contours) of objects such as figures contained in each of the first image and the second image acquired by the acquisition unit 1 , and transmits information of each extracted outermost contour to the detection unit 5 .

[0040] For example, the outermost contour extraction unit 4 extracts contours in each of the first image and the second image, and determines the contour that is not included in the other contours among the extracted contours as the outermost contour. As a contour extraction technique, a general edge detection technique can be used.

[0041] The detection unit 5 detects mutually similar areas between the images, i.e., similar areas, from each of the first image and the second image based on the outermost contour extracted from each of the first image and the second image by the outermost contour extraction unit 4 and the number of corresponding points detected by the matching unit 3, and passes the information of the detected similar areas to the output unit 6.

[0042] For example, the detection unit 5 counts the number of corresponding points contained in the areas within each outermost contour extracted from the first image, and detects the area with the largest number of corresponding points in the areas within each outermost contour extracted from the first image as a similar area in the first image. Similarly, the detection unit 5 counts the number of corresponding points contained in the areas within each outermost contour extracted from the second image, and detects the area with the largest number of corresponding points in the areas within each outermost contour extracted from the second image as a similar area in the second image. In addition, when the number of the largest corresponding points is less than the reference value, it can be determined that a similar area does not exist. Furthermore, instead of counting the number of corresponding points contained in the area within the outermost contour, the number of corresponding points contained in the rectangular area circumscribed to the outermost contour can be counted, and the area with the largest number of corresponding points can be detected as a similar area.

[0043] The output unit 6 cuts out an image of a rectangular area circumscribing the outermost contour of the area detected as a similar area by the detection unit 5 from each of the first image and the second image acquired by the acquisition unit 1 , and outputs the image as a similar image pair.

[0044] In addition, instead of directly cutting out the rectangular area circumscribing the outermost contour of the similar area from both the first image and the second image, the rectangular area may be cut out by changing the size of the rectangle of at least one of the first image and the second image, for example, by giving a margin to the periphery of the rectangle, slightly increasing the size of the rectangle before cutting out, or conversely slightly reducing the size of the rectangle before cutting out, etc. The similar image pair output by the output unit 6 can be used as learning data used in learning a feature extractor used in the above-mentioned similar image retrieval including local similarity.

[0045] Next, a specific example of the processing of the similar area detection device according to the present embodiment will be described with reference to specific examples. Figure 2 This is a flowchart showing an example of the processing procedure of the similar area detection device according to the present embodiment.

[0046] First, the acquisition unit 1 acquires the first image and the second image (step S101). Here, it is assumed that the acquisition unit 1 acquires Figure 3 Shown is a first image Im1 and a second image Im2.

[0047] Then, the feature point extraction unit 2 extracts the feature points of each of the first image and the second image acquired by the acquisition unit 1, and calculates the local feature value of each feature point (step S102). Furthermore, the matching unit 3 performs feature point matching of the feature points of the first image and the feature points of the second image based on the approximation of the local feature value of each feature point, and detects corresponding points of the first image and the second image (step S103).

[0048] exist Figure 4 Indicates from Figure 3 The first image Im1 and the second image Im2 shown in FIG. 1 are examples of corresponding points detected by the matching unit 3. The black dots at both ends connected by a straight line in the figure represent the corresponding points of the first image Im1 and the second image Im2. Figure 4 In order to simplify the illustration, a limited number of corresponding points are illustrated, but in fact, more corresponding points are usually detected.

[0049] Then, the outermost contour extraction unit 4 extracts the outermost contour of the object included in the image from each of the first image and the second image acquired by the acquisition unit 1 (step S104 ).

[0050] Figure 5 Indicates from Figure 3 The first image Im1 and the second image Im2 shown in FIG. 1 are examples of the outermost contours extracted by the outermost contour extraction unit 4. Figure 5 In the example, the outermost contours C1a and C1b of two graphics are extracted from the first image Im1, and the outermost contours C2a and C2b of two graphics are also extracted from the second image Im2. In addition, the outermost contour C1c of the string is extracted from the first image Im1, and the outermost contour C2c of the string is also extracted from the second image Im2. In addition, when only graphics are used as the determination object of similarity, it can be configured to determine whether the object in the image is a graphic or a character, and not to extract the outermost contours C1c and C2c of the string. In addition, it can also be configured not to extract a small outermost contour whose size ratio relative to the entire image is less than a specified value.

[0051] In addition, although it is assumed here that the outermost contour extraction of step S104 is performed after the feature point extraction of step S102 and the feature point matching of step S103 are performed, it may be assumed that the feature point extraction and feature point matching are performed after the outermost contour extraction. In addition, it is also possible that the feature point extraction and feature point matching and the outermost contour extraction are not performed sequentially (one by one), but these processes are performed in parallel (in parallel).

[0052] Then, the detection unit 5 detects similar regions from each of the first image and the second image based on the outermost contour extracted from each of the first image and the second image by the outermost contour extraction unit 4 and the number of corresponding points detected by the matching unit 3 (step S105 ).

[0053] For example, the detection unit 5 counts the number of corresponding points detected in the area inside each outermost contour extracted from the first image, and detects the area with the largest number of corresponding points in the area inside each outermost contour as a similar area in the first image. Similarly, the detection unit 5 counts the number of corresponding points detected in the area inside each outermost contour extracted from the second image, and detects the area with the largest number of corresponding points detected in the area inside each outermost contour as a similar area in the second image.

[0054] As a method for determining whether the corresponding point is located inside the outermost contour, for example, the following method can be used. Figure 6 As shown, multiple directions such as up, down, left, and right are confirmed from the corresponding points. If there are pixels belonging to the same outermost contour in any direction, the corresponding point is determined to be located inside the outermost contour. In addition, when the corresponding point is located on the outermost contour, it can be regarded as being located inside the outermost contour and counted, or it can be regarded as being located outside the outermost contour and not counted.

[0055] In addition, as a method for determining whether a corresponding point exists inside the outermost contour, the following method may be used, for each outermost contour, common identification information is assigned to each pixel of the outermost contour and the area inside it, and if the coordinates of the corresponding point have been assigned identification information, it is determined that the corresponding point exists inside the outermost contour indicated by the identification information. For example, a reference image may be created, the reference image is an image of the same size as the first image or the second image, for each outermost contour, each pixel of the outermost contour and the area inside it has a common pixel value other than 0, and the pixel value of the pixel outside the outermost contour is 0, and in the reference image, if the pixel value of the pixel having the same coordinates as the corresponding point detected from the first image or the second image is not 0, it is determined that the corresponding point exists inside the outermost contour corresponding to the pixel value indicated by the reference image.

[0056] Figure 7 Indicates from Figure 3 An example of the relationship between the outermost contours C1a, C1b, C1c extracted from the first image Im1 and the outermost contours C2a, C2b, C2c extracted from the second image Im2, and the corresponding points detected in each of the first image Im1 and the second image Im2 is shown. Figure 7 In the example shown, the outermost contour C1a has the largest number of corresponding points detected on the inner side among the outermost contours C1a, C1b, and C1c extracted from the first image Im1. Moreover, the outermost contour C2a has the largest number of corresponding points detected on the inner side among the outermost contours C2a, C2b, and C2c extracted from the second image Im2. Therefore, the detection unit 5 detects the area inside the outermost contour C1a (the area of ​​the portion surrounded by the outermost contour C1a in the first image Im1) as the similar area in the first image Im1, and detects the area inside the outermost contour C2a (the area of ​​the portion surrounded by the outermost contour C2a in the second image Im2) as the similar area in the second image Im2.

[0057] Finally, the output unit 6 cuts out a rectangular area circumscribed to the outermost contour of the similar area detected by the detection unit 5 from each of the first image Im1 and the second image Im2 acquired by the acquisition unit 1, and outputs a combination of the image of the rectangular area cut out from the first image Im1 and the image of the rectangular area cut out from the second image Im2 as a similar image pair (step S106), and a series of processing related to the similar area detection device of this embodiment is completed.

[0058] Alternatively, the output unit 6 may not directly cut out the rectangular area circumscribed to the outermost contour of the similar area, but may cut out the area by changing the size of the rectangle as described above, and output the similar image pair. Furthermore, when the rectangular sizes of the two images constituting the similar image pair are different, the rectangular sizes of the two images may be made consistent by adding a margin to the smaller rectangular size or reducing the larger rectangular size.

[0059] Figure 8 FIG. 6 shows an example of a similar image pair output by the output unit 6. Figure 8 An example of this is shown in Figure 3 The image Im1' cut out of the rectangular area circumscribing the outermost contour C1a of the first image Im1 shown in FIG. Figure 3 The combination of the second image Im2 and the image Im2' cut out of the rectangular area circumscribing the outermost contour C2a is output as a similar image pair. The similar image pair output by the output unit 6 can be used as learning data for learning the feature extractor to make the feature quantities of the similar image pair closer as described above.

[0060] As described in detail by taking the specific examples above, the similar region detection device related to the present embodiment comprises: an acquisition unit 1, which acquires the first image and the second image; a feature point extraction unit 2, which extracts the feature points of each of the first image and the second image; a matching unit 3, which establishes a correspondence relationship between the feature points extracted from the first image and the feature points extracted from the second image, and detects the corresponding points between the images; an outermost contour extraction unit 4, which extracts the outermost contour from each of the first image and the second image; and a detection unit 5, which detects the local area similar to each other in the first image and the second image, i.e., the similar region, from each of the first image and the second image based on the outermost contour extracted by the outermost contour extraction unit 4 and the number of corresponding points detected by the matching unit 3. Therefore, according to the similar region detection device, it is possible to automatically detect the similar region from each of the first image and the second image without manual instruction operation or the like.

[0061] In addition, the similar region detection device according to the present embodiment further includes an output unit 6, which cuts out an image of a rectangular region circumscribed with the outermost contour of the similar region detected by the detection unit 5 from each of the first image and the second image, and outputs the image as a similar image pair. Therefore, by using the similar region detection device, a similar image pair used as learning data for learning the above-mentioned feature extractor can be automatically generated without manual work, and the feature extractor can be efficiently learned.

[0062] <Second Embodiment>

[0063] Next, the second embodiment will be described. The second embodiment is different from the first embodiment in that the method of detecting similar regions from each of the first image Im1 and the second image Im2 is different from that of the first embodiment. In addition, the basic structure and processing overview of the similar region detection device are the same as those of the first embodiment, so the description repeated with the first embodiment will be omitted below, and only the characteristic parts of this embodiment will be described.

[0064] The detection unit 5 of the first embodiment detects, for each of the first image and the second image, an area having the largest number of corresponding points in an area within the outermost contour included in each image as a similar area. In contrast, the detection unit 5 of the second embodiment detects, for each of the first image and the second image, an area having the largest number of corresponding points in an area within the outermost contour included in each image as a similar area.

[0065] For the processing of the detection unit 5 of this embodiment, refer to Figure 7 The example shown is used for specific explanation. Figure 7In the example shown, for the outermost contours C1a, C1b, and C1c extracted from the first image Im1, 30 corresponding points are detected in the outermost contour C1a, 7 corresponding points are detected in the outermost contour C1b, and 1 corresponding point is detected in the area of ​​the two characters in the outermost contour C1c. Similarly, for the outermost contours C2a, C2b, and C2c extracted from the second image Im2, 30 corresponding points are detected in the outermost contour C2a, 7 corresponding points are detected in the outermost contour C2b, and 1 corresponding point is detected in the area of ​​the two characters in the outermost contour C2c. Here, when the similarity determination threshold is set to "5", the detection unit 5 detects the area in the outermost contour C1a and the area in the outermost contour C1b of the outermost contours C1a, C1b, and C1c extracted from the first image Im1, where the number of corresponding points detected on the inner side exceeds the similarity determination threshold, i.e., "5", as similar areas in the first image Im1. Similarly, the detection unit 5 detects the area within the outermost contour C2a and the area within the outermost contour C2b where the number of corresponding points detected inside the outermost contours C2a, C2b, and C2c extracted from the second image Im2 exceeds the similarity determination threshold value of "5" as similar areas in the second image Im2.

[0066] The detection unit 5 of the first embodiment detects the area within the outermost contour having the largest number of corresponding points as a similar area for each of the first image and the second image as described above, so a plurality of similar areas cannot be detected from each of the first image and the second image. In contrast, the detection unit 5 of the present embodiment detects the area within the outermost contour having the largest number of corresponding points exceeding the similarity determination threshold as a similar area, so a plurality of similar areas can be detected from each of the first image and the second image.

[0067] In addition, when a plurality of similar regions are detected from each of the first image and the second image, the correspondence between which similar region of the first image is similar to which similar region of the second image can be determined by referring to the relationship between corresponding points in each similar region. Figure 7 In the example shown, most of the corresponding points in the outermost contour C1a of the first image Im1 correspond to the corresponding points in the outermost contour C2a of the second image Im2, and most of the corresponding points in the outermost contour C1b of the first image Im1 correspond to the corresponding points in the outermost contour C2b of the second image Im2. Therefore, it can be seen that the area in the outermost contour C1a corresponds to the area in the outermost contour C2a, and the area in the outermost contour C1b corresponds to the area in the outermost contour C2b.

[0068] In the present embodiment, when the detection unit 5 detects a plurality of similar regions from each of the first image and the second image, the output unit 6 outputs a plurality of similar image pairs. Fig. 9FIG. 6 shows an example of a plurality of similar image pairs output by the output unit 6 of the present embodiment. Fig. 9 An example of this is shown in Figure 3 The image Im1' cut out from the rectangular area circumscribed by the outermost contour C1a of the first image Im1 shown in FIG. Figure 3 The combination of the image Im2′ cut out from the rectangular area circumscribed by the outermost contour C2a of the second image Im2 shown in FIG. Figure 3 The image Im1" cut out from the rectangular area circumscribed by the outermost contour C1b of the first image Im1 shown in FIG. Figure 3 The combination of images Im2″ cut out from the rectangular area circumscribed by the outermost contour C2b of the second image Im2 shown is output as a similar image pair.

[0069] As described above, in the similar area detection device according to the present embodiment, the detection unit 5 detects, for each of the first image and the second image, an area in which the number of corresponding points contained in the area within the outermost contour exceeds the similarity determination threshold value set in advance as a similar area. Therefore, according to the similar area detection device, when the first image and the second image include a plurality of similar areas, these plurality of detected similar areas can be automatically detected from each of the first image and the second image, and a plurality of similar image pairs can be automatically generated.

[0070] <Third Embodiment>

[0071] Next, the third embodiment will be described. In the third embodiment, when the output unit 6 cuts out the image of the rectangular area circumscribed with the outermost contour of the similar area from each of the first image and the second image and outputs it as a similar image pair, the object in the background area (the area outside the outermost contour of the contour of the similar area) outside the similar area projected into the rectangular area is removed and output. In addition, the basic structure and processing outline of the similar area detection device are the same as those of the first and second embodiments, so the description repeated with the first and second embodiments is omitted below, and only the characteristic part of this embodiment is described.

[0072] For the processing of the output unit 9 of this embodiment, refer to Fig. 9 The example shown is specifically described. Fig. 9The output unit 9 of the second embodiment outputs two groups of similar image pairs, wherein an image Im1' of a rectangular area constituting one similar image pair is an image of a portion of an object having an outermost contour C1b reflected in a background area outside the similar area (the area within the outermost contour C1a). Furthermore, an image Im1'' of a rectangular area constituting another similar image pair is an image of a portion of an object having an outermost contour C1a reflected in a background area outside the similar area (the area within the outermost contour C1b), and an image Im2'' of another rectangular area constituting another similar image pair is an image of a portion of an object having an outermost contour C2a reflected in a background area outside the similar area (the area within the outermost contour C2b).

[0073] The output unit 9 of this embodiment cuts out an image ( Fig. 9 In the case of the images Im1', Im1", and Im2") shown in the figure, the objects reflected in the background area of ​​the image are removed and then output as images constituting a similar image pair. Fig.10 FIG. 9 is an example of a similar image pair output by the output unit 9 of the present embodiment. Fig.10 As shown, in this embodiment, objects reflected in the background area of ​​each image constituting the similar image pair are removed.

[0074] As described above, in the similar region detection device according to the present embodiment, when the output unit 6 cuts out the image of the rectangular region circumscribing the outermost contour of the similar region from each of the first image and the second image and outputs it as a similar image pair, the object of the background region projected into the rectangular region is removed and output. Therefore, according to the similar region detection device, a similar image pair that does not include information other than the similar region that becomes noise can be automatically generated.

[0075] <Fourth Embodiment>

[0076] Next, the fourth embodiment will be described. In the fourth embodiment, in order to reduce the detection error of the similar area by the detection unit 5, the detection unit 5 detects the similar area from each of the first image and the second image using the positional relationship of the corresponding points in addition to the outermost contours and the number of corresponding points of each of the first image and the second image. In addition, the basic structure and processing overview of the similar area detection device are the same as those of the first to third embodiments, so the description repeated with the first to third embodiments is omitted below, and only the characteristic part of this embodiment is described.

[0077] After the detection unit 5 of this embodiment infers the similar regions of the first image and the second image by the same method as the first embodiment and the second embodiment described above, it confirms the positional relationship of the corresponding points in each inferred similar region and determines whether the inferred similar region is correct. That is, it can be considered that the positional relationship of the corresponding points detected in the similar region in the first image and the similar region in the second image is also similar, so if the positional relationship of the corresponding points is not similar, it is determined that it is not a similar region. That is, the similar region in which the positional relationship of the corresponding points in the similar region inferred based on the outermost contours and the number of corresponding points of the first image and the second image is similar is detected as a similar region.

[0078] For the processing of the detection unit 5 of this embodiment, refer to Fig.11 After inferring the similar regions in the first image and the similar regions in the second embodiment, the detection unit 5 of this embodiment performs a normalization process for comparing the positional relationship of the corresponding points in the inferred similar regions. Specifically, for example, normalization is performed so that the bounding rectangle of the similar region in the first image and the bounding rectangle of the similar region in the second image become squares of the same size, and the result is as follows: Fig.11 The detection unit 5 checks the positional relationship of the corresponding points in the standardized images NI1 and NI2, respectively, and if the positional relationship of the corresponding points in the standardized image NI1 is similar to the positional relationship of the corresponding points in the standardized image NI2, it is determined that the estimated similar region is correct. On the other hand, if the positional relationship of the corresponding points in the standardized image NI1 is not similar to the positional relationship of the corresponding points in the standardized image NI2, it is determined that the estimated similar region is incorrect.

[0079] As a method for comparing the positional relationship of corresponding points, for example, the distance between two corresponding points in each of the standardized images NI1 and NI2 is calculated using the coordinates of the corresponding points in the standardized images NI1 and NI2. And, if the distance difference between the distance between the two corresponding points calculated in the standardized image NI1 and the distance between the two corresponding points calculated in the standardized image NI2 is within a threshold value, it is determined that the positional relationship of the two corresponding points is consistent with respect to the similar area inferred in the first image and the similar area inferred in the second image. And, for example, if the ratio of corresponding points determined to have a consistent positional relationship exceeds a predetermined value for the entirety of the corresponding points in each inferred similar area, it is determined that the positional relationship of the corresponding points in the similar area inferred in the first image is similar to the positional relationship of the corresponding points in the similar area inferred in the second image.

[0080] Alternatively, instead of judging whether the positional relationship between the two corresponding points is consistent based on the distance between the two corresponding points calculated using the coordinates of the corresponding points in the standardized images NI1 and NI2, for example, the positions of the two corresponding points in one of the standardized images NI1 and NI2 may be inferred based on the relative positions of the two corresponding points in the other standardized image, and judging whether the positional relationship between the two corresponding points is consistent may be based on whether the positions of the two corresponding points in the other standardized image are consistent with the inferred positions.

[0081] As described above, in the similar region detection device according to the present embodiment, the detection unit 5 detects similar regions from each of the first image and the second image using the positional relationship of the corresponding points in addition to the outermost contours and the number of corresponding points of each of the first image and the second image. Therefore, according to the similar region detection device, it is possible to reduce the detection errors of similar regions by the detection unit 5.

[0082] <Fifth Embodiment>

[0083] Next, the fifth embodiment will be described. In the fifth embodiment, when a plurality of feature points having local feature values ​​similar to those of the feature points extracted from one of the first image and the second image are extracted from the other image, the matching unit 3 establishes a correspondence between the feature points extracted from one image and the plurality of feature points extracted from the other image. In addition, the basic structure and processing overview of the similar area detection device are the same as those of the first to fourth embodiments, so the description repeated with the first to fourth embodiments is omitted below, and only the characteristic part of this embodiment is described.

[0084] In each of the above-mentioned embodiments, when the matching unit 3 performs feature point matching between the first image and the second image, a correspondence is established between the feature points of one image and the feature points of the other image whose local feature value is closest. In this method, when the other image contains multiple objects similar to the objects contained in the one image, it is possible that the corresponding points in the other image are scattered in multiple areas, and the similar area of ​​the other image cannot be properly detected.

[0085] In contrast, in the present embodiment, when a plurality of feature points having local feature values ​​similar to those of feature points extracted from one of the first and second images are extracted from the other image, the matching unit 3 performs feature point matching between the first and second images by establishing a correspondence between the feature points extracted from one image and the plurality of feature points extracted from the other image. Therefore, when a plurality of objects similar to those contained in one image are contained in the other image, the corresponding points in the other image will not be scattered in a plurality of regions, and similar regions can be detected from the other image by, for example, the same method as in the second embodiment described above, thereby appropriately detecting a plurality of similar regions from the other image. Furthermore, in the present embodiment, for an image of a rectangular region circumscribed to the outermost contour of a similar region detected from one image, images of a plurality of rectangular regions circumscribed to the outermost contours of each of the plurality of similar regions detected from the other image can be combined to generate a plurality of similar image pairs and output them.

[0086] Fig.12 FIG. 2 shows an example of feature point matching performed by the matching unit 3 of this embodiment. Fig.13 FIG. 6 shows an example of a similar image pair output by the output unit 6 of the present embodiment. Fig.12 In the example shown, a correspondence relationship is established between one feature point extracted from the first image Im11 and two feature points extracted from the second image Im12. Therefore, in the second image Im12, there are multiple corresponding points in two areas within the two outermost contours, and these two areas are detected as similar areas. As a result, the output unit 6 Fig.13 As shown, two similar image pairs are output, namely, a combination of an image Im11' of a rectangular area cut out from the first image Im11 and an image Im12' of a rectangular area cut out from the second image Im12, and a combination of an image Im11' of a rectangular area cut out from the first image Im11 and an image Im12" of a rectangular area cut out from the second image Im12.

[0087] As described above, in the similar region detection device according to the present embodiment, when a plurality of feature points having local feature quantities similar to the feature points extracted from one of the first image and the second image are extracted from the other image, the matching unit 3 establishes a correspondence between the feature points extracted from one image and the plurality of feature points extracted from the other image. Therefore, according to the similar region detection device, when a plurality of objects similar to the object included in one image are included in the other image, it is possible to effectively suppress the corresponding points in the other image from being dispersed in a plurality of regions, and appropriately detect a plurality of similar regions from the other image.

[0088] <Supplementary Notes>

[0089] The similar area detection device of each embodiment described above can be implemented, for example, by using a general-purpose computer as basic hardware. That is, the functions of each part of the similar area detection device described above can be implemented by causing one or more processors mounted on a general-purpose computer to execute a program. In this case, the program described above can be pre-installed in the computer, or the program described above recorded in a computer-readable storage medium or the program described above distributed via a network can be appropriately installed in the computer.

[0090] Fig.14 1 is a block diagram showing an example of a hardware configuration of a similar region detection device according to each of the above-mentioned embodiments. Fig.14 The hardware structure of a general computer shown in the figure has a processor 101 such as a CPU (Central Processing Unit), a memory 102 such as a RAM (Random Access Memory) and a ROM (Read Only Memory), a storage device 103 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), a device I / F 104 for connecting a display device 106 such as a liquid crystal panel and an input device 107 such as a keyboard and a pointing device, a communication I / F 105 for communicating with the outside of the device, and a bus 108 connecting these various parts.

[0091] In passing Fig.14 In the case where the hardware structure shown implements a similar area detection device of each of the above-mentioned embodiments, for example, the processor 101 reads out and executes a program stored in the storage device 103 etc. using the memory 102, thereby realizing the functions of the above-mentioned acquisition unit 1, feature point extraction unit 2, matching unit 3, outermost contour extraction unit 4, detection unit 5 and output unit 6 etc.

[0092] In addition, the functions of the various parts of the similar area detection device of each embodiment described above can also be implemented in part or in whole by dedicated hardware (not a general-purpose processor, but a dedicated processor) such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). In addition, it is also possible to use a structure that uses multiple processors to implement the functions of the various parts described above. Moreover, the similar area detection device of each embodiment described above is not limited to being implemented by one computer, but can also be implemented by distributing the functions to multiple computers.

[0093] The embodiments of the present invention are described above, but these embodiments are provided as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. These embodiments and their variations are included in the scope or spirit of the invention, and are included in the invention described in the claims and the scope of their equivalents.

[0094] Description of symbols

[0095] 1 acquisition unit; 2 feature point extraction unit; 3 matching unit; 4 outermost contour extraction unit; 5 detection unit; 6 output unit; Im1, Im11 first images; Im2, Im12 second images; C1a, C1b, C1c, C2a, C2b, C2c outermost contours.

Claims

1. A similar area detection device, comprising: An acquisition unit, which acquires a first image and a second image; A feature point extraction unit, extracting feature points of each of the first image and the second image; A matching unit establishes a correspondence between feature points extracted from the first image and feature points extracted from the second image, and detects corresponding feature points between the images; an outermost contour extraction unit, extracting one or more first outermost contours from the first image, and extracting one or more second outermost contours from the second image; A detection unit detects one or more first similar areas and one or more second similar areas, wherein the first similar area is an area in a local area of ​​the first image within the first outermost contour where the number of corresponding feature points is the largest or an area where the number of corresponding feature points exceeds a similarity determination threshold, and the second similar area is an area in a local area of ​​the second image within the second outermost contour where the number of corresponding feature points is the largest or an area where the number of corresponding feature points exceeds the similarity determination threshold.

2. The similar area detection device according to claim 1, The similar region detection device further comprises an output unit, which cuts out one or more first similar images from the first image, cuts out one or more second similar images from the second image, and outputs a similar image pair, wherein: The first similar image is an image of a rectangular area circumscribing the first similar region, the second similar image is an image of a rectangular area circumscribing the second similar region, and the similar image pair is a combination of one of the one or more first similar images and one of the one or more second similar images.

3. The similar area detection device according to claim 2, The output unit removes objects in a background area other than the first similar area in the first similar image, and removes objects in a background area other than the second similar area in the second similar image, and outputs the images.

4. The similar area detection device according to any one of claims 1 to 3, The detection unit determines whether the first similar region and the second similar region are correct based on a positional relationship between the corresponding feature points included in one or more first similar regions and a positional relationship between the corresponding feature points included in one or more second similar regions.

5. The similar area detection device according to any one of claims 1 to 3, The matching unit establishes a correspondence relationship between the feature points extracted from the first image and the feature points extracted from the second image based on the proximity of the local feature amounts of the feature points.

6. The similar area detection device according to claim 5, The matching unit establishes a correspondence relationship between the feature points extracted from one of the first image and the second image and the plurality of feature points extracted from the other image when a plurality of feature points having local feature values ​​similar to those of the feature points extracted from the other image are extracted from the other image.

7. A similar region detection method, the similar region detection method being a method performed by a similar region detection device, comprising: An acquisition step of acquiring a first image and a second image; A feature point extraction step, extracting feature points of each of the first image and the second image; A matching step, establishing a correspondence between the feature points extracted from the first image and the feature points extracted from the second image, and detecting corresponding feature points between the images; an outermost contour extraction step, extracting one or more first outermost contours from the first image, and extracting one or more second outermost contours from the second image; A detection step of detecting one or more first similar areas and one or more second similar areas, wherein the first similar area is an area in a local area of ​​the first image within the first outermost contour where the number of corresponding feature points is the largest or an area where the number of corresponding feature points exceeds a similarity determination threshold, and the second similar area is an area in a local area of ​​the second image within the second outermost contour where the number of corresponding feature points is the largest or an area where the number of corresponding feature points exceeds the similarity determination threshold.

8. A program product, comprising a program, wherein the program is used to cause a computer to implement the following functions: The acquisition unit is configured to acquire the first image and the second image; The function of the feature point extraction unit is to extract the feature points of the first image and the second image respectively; The matching unit is configured to establish a correspondence between the feature points extracted from the first image and the feature points extracted from the second image, and detect corresponding feature points between the images; The outermost contour extraction unit is configured to extract one or more first outermost contours from the first image and one or more second outermost contours from the second image; The function of the detection unit is to detect one or more first similar areas and one or more second similar areas, wherein the first similar area is an area in a local area of ​​the first image within the first outermost contour where the number of corresponding feature points is the largest or an area where the number of corresponding feature points exceeds the similarity determination threshold, and the second similar area is an area in a local area of ​​the second image within the second outermost contour where the number of corresponding feature points is the largest or an area where the number of corresponding feature points exceeds the similarity determination threshold.

Images