Matching method, computer program and information processing device

Abstract

To provide a matching method, a computer program, and an information processing device that are robust to view changes.SOLUTION: A matching method by an information processing device is executed by an arithmetic circuit that can access a storage device, and performs object matching from an image containing a three-dimensional predetermined object. The storage device stores a pixel group for comparison preselected from a reference image containing the object as pixels representing characteristics of the object. The matching method also receives a new image containing the object by the arithmetic circuit, compares the pixel group for comparison with the new image, and identifies the object from the new image. For the pixel group for comparison, pixel groups having high similarities to multiple images included in a view-change image group are used, from among multiple pixel groups selected from the reference image after comparing the multiple pixel groups arbitrarily selected from the reference image containing the object with pixel groups extracted from the view-change image group containing the multiple images showing the object with respective different view changes.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a matching method, a computer program, and an information processing device that realize matching from an image including a three-dimensional object without being affected by the position and orientation of the three-dimensional object. [Background technology]

[0002] Conventionally, template matching has been used in the field of image processing (see, for example, Patent Document 1). Template matching searches an input image for a pattern that is identical to or similar to a template, which is a feature pattern. Template matching for planar objects has recently become possible with high accuracy. However, template matching for three-dimensional objects is still not easy. For example, a three-dimensional object looks different depending on the direction. Therefore, it is difficult to prepare a template image of a three-dimensional object in advance. It is also difficult to perform high-accuracy matching using template images from different directions.

[0003] This application of template matching for three-dimensional objects has applications in the field of robotics. When a robot grasps a three-dimensional object, it is important to understand the position and orientation of the object. Currently, 3D camera images are commonly used to understand the position and orientation of three-dimensional objects. 3D cameras use a method in which light is shone on the object and the light reflected from the object is used. However, light may not be reflected depending on the properties of the object, and 3D cameras may not be able to accurately capture an image of the object. Furthermore, 3D cameras are generally larger than regular 2D cameras. Therefore, attaching a 3D camera to the hand of a robot would hinder the robot's operability. In contrast, 2D cameras are becoming increasingly miniaturized, and even if attached to the hand of a robot, they are less likely to hinder the robot's operability. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 7-235357 Summary of the Invention

[0005] However, as mentioned above, three-dimensional objects appear differently depending on the direction. Therefore, it has been difficult to match three-dimensional objects with high accuracy and speed using template matching using images captured by a 2D camera. Therefore, there has been a demand for highly accurate matching that is not affected by the position and orientation of three-dimensional objects.

[0006] Therefore, an object of the present disclosure is to provide a matching method, a computer program, and an information processing device that achieve matching without being affected by the position and orientation of a three-dimensional object. [Means for solving the problem]

[0007] A matching method according to the present disclosure is executed by an arithmetic circuit that can access a storage device, and matches a predetermined object from an image containing a three-dimensional object. The storage device stores a group of comparison pixels preselected from a reference image containing the object as pixels representing the object's characteristics. The matching method uses the arithmetic circuit to accept a new image containing the object, compare the group of comparison pixels with the new image, and identify the object from the new image. The comparison group of pixels is determined by comparing multiple pixel groups arbitrarily selected from the reference image containing the object with pixel groups extracted from a view-change image group containing multiple images showing the object with different view changes, and selecting pixel groups from the multiple pixel groups selected from the reference image that have a high similarity to the multiple images included in the view-change image group.

[0008] Such general and specific aspects may be realized by a system, a method, and a computer program, as well as a combination thereof. [Effects of the Invention]

[0009] The matching method, computer program, and information processing device according to the present disclosure provide a matching technique that is robust to view changes. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating an information processing apparatus according to an embodiment. [Figure 2A] 10 is an example of an image including a frontal direction of a matching target. [Figure 2B] 1 is an example of an image containing a matching target rotated -45 degrees. [Figure 2C] 1 is an example of an image containing a matching target rotated +45 degrees. [Figure 3] 10 is a flowchart illustrating a comparison pixel group determination process executed by the information processing device. [Figure 4A] 2B is an example of dividing the image of FIG. 2A into multiple reference regions. [Figure 4B] This is an example of extracting an arbitrary reference region from a plurality of divided reference regions. [Figure 5] 10 is a flowchart illustrating pixel group selection processing executed in determining a comparison pixel group. [Figure 6] 10A and 10B are conceptual diagrams illustrating a process for generating candidates for a comparison pixel group. [Figure 7] FIG. 1 is a schematic diagram illustrating crossover in a genetic algorithm. [Figure 8] 10 is a flowchart illustrating a matching process. [Figure 9] 10 is a graph used to calculate fitness. [Figure 10] 10 is a graph comparing the recognition rates of different pixel groups. [Figure 11] FIG. 10 is a block diagram showing an information processing device according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described using the drawings, with appropriate reference to the drawings. However, in the detailed description, unnecessary parts of the description of the prior art and substantially identical configurations may be omitted. This is for the sake of simplicity. Furthermore, the following description and the accompanying drawings are disclosed to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

[0012] The present disclosure relates to a matching method, a computer program, and an information processing device for matching an image of a three-dimensional object with an image including the three-dimensional object. In the matching method of the present disclosure, when matching a three-dimensional object from an image including the three-dimensional object using pattern matching, a comparison pixel group generated in advance from the pattern image is used, rather than a pattern image including the three-dimensional object. The comparison pixel group is not necessarily an image composed of multiple consecutive pixels, but is composed of pixels selected from the pattern image. Therefore, before executing the matching method of the present disclosure, a comparison pixel group is generated as preprocessing.

[0013] <Configuration of information processing device> The configuration of an information processing device 1 used for matching three-dimensional objects will be described with reference to Fig. 1. As shown in Fig. 1, the information processing device 1 includes an arithmetic circuit 11, an input device 12, an output device 13, a communication circuit 14, and a storage device 15. For example, the information processing device 1 may be a personal computer or the like.

[0014] The arithmetic circuit 11 is a controller that controls the entire information processing device 1. For example, the arithmetic circuit 11 performs various processes related to matching by reading and executing the programs P1 and P2 stored in the storage device 15. The arithmetic circuit 11 may be various processors such as a CPU, an MPU, a GPU, an FPGA, a DSP, an ASIC, or a dedicated hardware circuit.

[0015] The input device 12 is used for user operations and data input. The input device 12 can be, for example, an operation button, a keyboard, a mouse, a touch panel, a microphone, etc. The output device 13 is used for outputting processing results and data. The output device 13 can be, for example, output means such as a display, a speaker, etc.

[0016] The communication circuit 14 performs data communication with other devices. The data communication may be performed wired and / or wirelessly, following, for example, a known communication standard. For example, wired data communication may be performed by using, as the communication circuit 14, a communication controller of a semiconductor integrated circuit that operates in accordance with the Ethernet (registered trademark) standard and / or the USB (registered trademark) standard. Wireless data communication may be performed by using, as the communication circuit 14, a communication controller of a semiconductor integrated circuit that operates in accordance with the IEEE 802.11 standard for wireless LANs (Local Area Networks) and / or the fourth-, fifth-, and sixth-generation mobile communication systems, known as 4G, 5G, and 6G, for mobile communications.

[0017] The storage device 15 is a recording medium for recording various information. The storage device 15 is realized by, for example, RAM, ROM, flash memory, SSD (Solid State Drive), hard disk drive, other storage devices, or an appropriate combination thereof. The storage device 15 stores a generation program P1, a matching program P2, a reference image 151, a view-change image group 152, a comparison pixel group 153, an input image 154, a matching result 155, etc. The generation program P1 and the matching program P2 are computer programs executed by the arithmetic circuit 11.

[0018] When the generation program P1 is executed, the arithmetic circuit 11 becomes a generation module. Specifically, the arithmetic circuit 11 executes the generation program P1 to generate a comparison pixel group 153 to be used for matching from a reference image 151 and a view-change image group 152.

[0019] Furthermore, when the matching program P2 is executed, the arithmetic circuit 11 becomes a matching module. Specifically, the arithmetic circuit 11 executes the matching program P2 and generates a matching result 155 of pattern matching from an input image 154 using a comparison pixel group 153.

[0020] Reference image 151 is image data including a matching target. FIG. 2A shows an example of reference image 151 of target O. Reference image 151 is image data obtained by photographing target O from a predetermined direction. When target O has a direction identified as the front, reference image 151 may be image data obtained by photographing target O from the front.

[0021] The view-change image group 152 is a collection of image data including images captured from the same direction as the reference image 151 to be matched and images captured from a different direction, i.e., view-changed images. The storage device 15 also stores multiple view-change image groups 152, each showing a different view. Each image included in the view-change image group 152 is associated with information indicating the direction from which the object was captured. For example, the object O included in the reference image 151 is placed on a rotating turntable, with the turntable at 0 degrees. In this case, the view-change image group 152 may include images of the object O captured at a predetermined pitch or at any timing when the turntable is rotated from -45 degrees to +45 degrees. The view-change image group 152 may be associated with the angle of rotation of the turntable. As a result, each view-change image group 152 includes multiple different views of the object O. Figures 2B and 2C show examples of different views of the same object O as the reference image 151 in Figure 2A. 2B shows an example of an image including an object O rotated by −45 degrees when the object O in the reference image 151 is set to 0 degrees. Also, FIG. 2C shows an example of an image in which the object O is rotated by +45 degrees.

[0022] In this embodiment, an example is given in which the object is rotated ±45 degrees around the vertical axis, but the rotation axis does not have to be the vertical axis, and multiple rotation axes may be combined, and the rotation range is not limited to 45 degrees. In addition to rotation, scaling, i.e., a change in view due to enlargement or reduction of the image caused by the distance between the camera and the object, may also be combined.

[0023] For example, when the reference image 151 is a photographed image, the object O is photographed against a specific background, so that the object O (foreground) can be distinguished from the background. For example, the reference image 151 may be an image photographed in an environment where the background is a specific color (e.g., green or blue). Furthermore, each image included in the view-change image group 152 may also be an image photographed in the same environment as the reference image 151.

[0024] The comparison pixel group 153 is an arbitrary pixel group selected from the reference image 151. The method for selecting the comparison pixel group 153 will be described later. The reference image 151 is compared with each image included in the view-change image group 152, and an arbitrary pixel group with high similarity is selected as the comparison pixel group 153. Using this comparison pixel group 153 enables pattern matching that is robust to view changes. The comparison pixel group 153 includes only a portion of pixels selected from the entire reference image 151. This reduces the processing time for pattern matching. Note that the number of pixels included in the comparison pixel group 153 is not fixed. For example, the number of pixels included in the comparison pixel group 153 varies depending on the characteristics of the object O. Specifically, it varies depending on various conditions, such as the shape, color, material, and size of the object O, as well as the similarity evaluation scale used in the matching process. It also varies depending on the number of pixels in the reference image 151.

[0025] The input image 154 is image data to be matched. The arithmetic circuit 11 detects an object O from the new input image 154.

[0026] Matching result 155 is the result of pattern matching. For example, if input image 154 includes object O, matching result 155 includes information indicating the area of ​​the matched object. Also, for example, if input image 154 does not include object O, matching result 155 includes information indicating that input image 154 does not include the object.

[0027] <Processing in information processing device 1> The information processing device 1 serves as a generation module and performs a generation process of a comparison pixel group 153 using a reference image 151 and a view-change image group 152. The information processing device 1 also serves as a matching module and performs a matching process of an input image using the comparison pixel group 153. The generation process of a comparison pixel group executed in the arithmetic circuit 11 will be described below using the flowchart shown in FIG. 3. The matching process of an input image executed in the arithmetic circuit 11 will be described using the flowchart shown in FIG. 8. Note that in the present disclosure, an example will be described in which both the generation process of a comparison pixel group and the matching process are executed in the information processing device 1. However, the generation process of a comparison pixel group and the matching process may be executed in different information processing devices.

[0028] <<Extracting pixels for comparison>> 3, the arithmetic circuit 11 first acquires the reference image 151 (S01). The arithmetic circuit 11 then stores the reference image 151 in the storage device 15. The storage of the reference image 151 in the storage device 15 may be performed in advance. At this time, the arithmetic circuit 11 reads out the reference image 151 stored in advance from the storage device 15 and uses it.

[0029] The arithmetic circuit 11 divides the reference image 151 acquired in step S01 into a plurality of reference regions (S02). For example, as shown in FIG. 4A, the arithmetic circuit 11 divides the entire reference image 151 into reference regions of a predetermined size (a predetermined number of pixels).

[0030] The arithmetic circuit 11 selects an arbitrary number of reference regions from the reference regions divided in step S02 (S03). For example, as shown in FIG. 4B, the arithmetic circuit 11 selects an arbitrary reference region from the reference regions divided in step S02 that include the target O.

[0031] The arithmetic circuitry 11 acquires the view-change image group 152 (S04). The arithmetic circuitry 11 also stores the view-change image group 152 in the storage device 15.

[0032] The arithmetic circuit 11 selects one reference region from the plurality of reference regions selected in step S03 (S05).

[0033] The arithmetic circuit 11 generates candidates for a comparison pixel group from the reference region selected in step S05 (S06). Note that the flowchart in Figure 3 illustrates an example in which the arithmetic circuit 11 selects multiple reference regions in step S03 and generates candidates for a comparison pixel group for each reference region in step S06. However, the arithmetic circuit 11 may select only one reference region in step S03 and generate one comparison pixel group in step S06.

[0034] The process of generating candidates for comparison pixel groups in step S06 will be described using the flowchart in Figure 5. In the process of generating candidate pixel groups for comparison pixel groups, the arithmetic circuit 11 uses a combinatorial optimization method. Specifically, the following describes an example of a method in which the arithmetic circuit 11 uses a genetic algorithm (GA) to generate an optimal pixel group from a pixel group determined as an initial group.

[0035] The arithmetic circuit 11 generates an initial population of the genetic algorithm based on the reference region selected in step S05 (S61). Specifically, the arithmetic circuit 11 generates a predetermined number (hereinafter, 10,000) of bit strings. Each bit string contains the same number of bits as the number of pixels in the reference region. The bit string corresponds to a chromosome contained in each individual in the GA. One pixel in the reference region corresponds to one bit in the bit string. For example, if the reference region is 61 × 61 pixels, the entire length of the bit string is 3,721 bits. The arithmetic circuit 11 randomly selects multiple pixels from the reference region and assigns "1" to bits corresponding to an arbitrary number of selected pixels (hereinafter, 75) and "0" to the other bits. In this way, the selected pixel group is expressed in the form of a bit string. When 75 pixels are selected, there are 75 loci that are 1 in the gene string with a length of 3,721 (61 × 61). FIG. 6 shows an example of a conceptual diagram of an arbitrary pixel group when the reference area is an area of ​​61 × 61 pixels. In FIG. 6, pixels assigned "1" are pixels selected for the arbitrary pixel group. The selection of pixels from this reference area and assignment to bit strings is repeated 10,000 times. In this way, the arithmetic circuit 11 generates 10,000 individuals. The multiple individuals generated in this way are called an initial population. The collection of these populations is called an individual pool.

[0036] The arithmetic circuit 11 calculates the fitness of each individual generated in step S61 (S62). Fitness is a numerical value that indicates the "strength of adaptation to view changes of the pixel group represented by that individual." By generating individuals (bit strings, i.e., pixel groups) with high fitness using GA, the pixel group to be adapted to view changes is selected. The arithmetic circuit 11 calculates the fitness for each individual, associates them, and stores them in the storage device 15 as an individual pool.

[0037] A specific method for calculating fitness is described below. First, fitness is calculated using a selected pixel group (called X) represented as a bit string of one individual, i.e., a chromosome, and a view-change image group 152 prepared in advance. As described above, the view-change image group 152 includes a large number of images in which the view changes at positions corresponding to the reference image 151. For example, if there is a horizontal view change of ±45 degrees in 1-degree increments, the arithmetic circuit 11 uses the sum of squared differences (SSD) between each individual and the pixel group corresponding to each image in the view-change image group 152. In this case, the more similar the two data, the smaller the SSD value, so it is often called dissimilarity rather than similarity. In this case, if the SSD value is calculated between X and an image with a horizontal rotation angle of 0 degrees, the SSD value will theoretically be 0 regardless of which pixels are selected, since both were captured from the same direction. In contrast, the SSD value between an image with a horizontal rotation angle other than 0 degrees is not 0, and the larger the horizontal angle, the smaller the dissimilarity with X. This indicates similarity.

[0038] A dissimilarity graph showing this relationship (described later using Figure 9) tends to have a downward convex shape with a minimum value at an angle of 0 degrees, regardless of the type of pixel group X contains. However, the steepness of the convex shape of the graph is important. If the convex shape is steep, it means that the dissimilarity increases rapidly as the view angle moves away from 0 degrees, that is, the similarity decreases rapidly. In other words, it means that the X in question is vulnerable to view changes. On the other hand, if the convex shape is gentle, it means that the dissimilarity does not increase rapidly, that is, the similarity does not decrease rapidly, even when the view angle moves away from 0 degrees, and therefore the X in question is robust to view changes. Therefore, the steepness or gentleness of the convex shape of this graph represents the "strength of adaptation of the pixel group of that individual to view changes" for the X in question. Therefore, in this embodiment, the steepness of this graph is expressed as the area of ​​the region sandwiched between the horizontal axis when SSD is 0 and the graph showing the SSD value. This area is also used as the fitness in GA. Specifically, an example will be described later using FIG. 9; it is the area represented by the horizontal axis when SSD is 0, the graph showing the SSD value, the perpendicular line from the SSD value of 0 to the SSD value when the angle is the minimum value, and the perpendicular line from the SSD value of 0 to the SSD value when the angle is the maximum value. In other words, the smaller the area, the gentler the graph is determined to be, and an index that increases the fitness is used.

[0039] As mentioned above, the number of loci is not fixed because crossover and mutation are repeated. That is, even if 75 pixels are selected in step S61, the number of loci will fluctuate while the genetic algorithm is being executed. Therefore, the arithmetic circuit 11 uses the average value obtained by dividing the SSD value by the number of loci for evaluation.

[0040] The arithmetic circuit 11 selects two individuals from the initial population generated in step S61 (S63). For example, the arithmetic circuit 11 can select the two individuals using roulette wheel selection. Roulette wheel selection is a method of randomly selecting two individuals under a rule that individuals with high fitness are more likely to be selected. The two selected bit strings are called parent individuals.

[0041] The arithmetic circuit 11 randomly crosses over the two individuals selected in step S63 (S64). Because crossover is performed randomly, the arithmetic circuit 11 may or may not perform crossover. Specifically, the arithmetic circuit 11 performs crossover with the same probability each time using a uniform random number. For example, the arithmetic circuit 11 applies a method called uniform crossover. The arithmetic circuit 11 generates two new bit strings through crossover. On the other hand, if crossover is not performed, the arithmetic circuit 11 uses bit strings obtained by copying the two parent individuals as new bit strings. The two new bit strings are called child individuals. In this way, two child individuals are generated from two parent individuals. For example, as shown in FIG. 7, the arithmetic circuit 11 uses uniform crossover, which selects two parents and selects two children using a mask pattern in which 0s and 1s are randomly assigned. For example, if the mask pattern for child A is 0, it inherits the same gene as the gene locus of parent A, and if the mask pattern is 1, it inherits the same gene as the gene locus of parent B. In this way, the process of generating two children from two parents is repeated to create a new population. Note that if the number of selected pixels represented by an individual becomes, for example, less than 50 or more than 100, the gene sequence is treated as a lethal gene and crossover is interrupted, and a new crossover is performed again so that, for example, the number of selected pixels becomes greater than or equal to 50 and less than 100.

[0042] The arithmetic circuit 11 randomly mutates the two child individuals (bit strings) generated in step S64 (S65). Because mutations are performed randomly, the arithmetic circuit 11 may or may not mutate. 0s and 1s in the genes of the individuals are randomly replaced with alleles.

[0043] The arithmetic circuit 11 calculates the fitness of the two child individuals generated in step S64 or mutated in step S65 (S66).

[0044] The arithmetic circuit 11 associates the fitness values ​​calculated in step S66 with the child individuals and stores them as an individual pool in the storage device 15 (S67). Here, the arithmetic circuit 11 deletes the two individuals with the lowest fitness values ​​from each individual in the individual pool, and adds new child individuals and fitness values. As a result, although the number of individuals in the individual pool was originally 10,000, it is possible to prevent the number of individuals in the individual pool from increasing above 10,000 even after the individuals are added.

[0045] The arithmetic circuit 11 determines whether or not the convergence condition is met, and repeats the processing of steps S63 to S67 until the convergence condition is met (S68). For example, the convergence condition may be that the processing of steps S63 to S67 has been repeated a predetermined number of times. Alternatively, the convergence condition may be that the maximum value of the fitness of the individuals in the individual pool is equal to or greater than a predetermined value. Alternatively, the convergence condition may be that the difference between the maximum value and the average fitness of the individuals in the individual pool is equal to or less than a certain value.

[0046] When convergence is achieved (YES in S68), the calculation circuit 11 selects the individual with the highest fitness from the individual pool as a candidate for the comparison pixel group (S69). At this time, the number of individuals in the individual pool is 10,000. The selected individual becomes the final output of the GA. Also, since the selected individual is a bit string, the pixel group corresponding to "1" in the bit string of the individual selected in step S69 becomes the "comparison pixel group."

[0047] Returning to the flowchart shown in Fig. 3, the arithmetic circuit 11 determines whether or not candidates for comparison pixel groups have been generated for all of the arbitrary reference regions selected in step S03 (S07).

[0048] If comparison pixel group candidates have not been generated for all of the arbitrary reference areas (NO in S07), the arithmetic circuit 11 returns to step S05, selects another reference area, and repeats the process of step S06. Note that if only one arbitrary reference area is selected in step S03, the process proceeds to the next step without repeating the process.

[0049] When candidates for comparison pixel groups have been generated for all of the reference regions (YES in S07), the arithmetic circuit 11 determines all of the candidates for comparison pixel groups generated in step S06 as comparison pixel groups 153 (S08). The arithmetic circuit 11 also stores the determined comparison pixel groups 153 in the storage device 15.

[0050] <<Matching process>> 8, the arithmetic circuit 11 first acquires an input image 154 to be used for matching (S31). The arithmetic circuit 11 also stores the input image 154 in the storage device 15.

[0051] The arithmetic circuit 11 obtains feature points from the input image 154 acquired in step S31 (S32). For example, the arithmetic circuit 11 uses a feature point extraction algorithm to obtain a plurality of feature points that meet predetermined conditions.

[0052] The arithmetic circuit 11 compares the group of pixels for comparison stored in the storage device 15 with the plurality of feature points found in step S32 (S33).

[0053] The arithmetic circuit 11 uses the comparison result of step S32 to obtain the matching result (S34). For example, matching is performed based on an area that is determined to match or be similar to the group of pixels for comparison.

[0054] In this way, the information processing device 1 can perform matching with high accuracy regardless of the view of the three-dimensional object by using the comparison pixel group obtained in advance in this way.

[0055] The image size of the input image 154 is different from that of the reference image 151. For example, the image size of the input image 154 is larger than that of the reference image 151. Therefore, the arithmetic circuit 11, for example, sequentially cuts out areas of the same size as the reference image 151 from the input image 154, calculates similar elements, and performs matching processing. Such image size adjustment processing is the same as general matching processing.

[0056] In this way, the information processing device 1 can determine a comparison pixel group that is less affected by view changes of a three-dimensional object and can be effectively used for matching. Specifically, the information processing device 1 can extract a pixel group that has small density fluctuations associated with view changes of the object as the comparison pixel group. Therefore, by using the comparison pixel group determined by the information processing device 1 as described above, matching can be performed with high accuracy regardless of the view of the three-dimensional object. In other words, by using the comparison pixels determined by the information processing device 1, matching that is robust to view changes can be achieved. Furthermore, the comparison pixel group determined by the information processing device 1 has a smaller number of pixels than the region image. Therefore, the processing time required to compare the input image with the comparison pixel group for matching can be significantly reduced.

[0057] <SSD Comparison> FIG. 9 is a graph showing the SSD between pixels generated from a reference region obtained using an object whose appearance varies depending on the viewing direction and pixels generated from corresponding multiple different view-variant images. Specifically, in Fig. 9, the horizontal axis represents the angle of rotation, and the vertical axis represents SSD. Fig. 9 shows SSDs obtained using images captured while rotating the object from -35 degrees to +35 degrees in 1-degree increments around the same rotation axis, with the direction defined as the front of the object being 0 degrees. Fig. 9 shows SSDs for pixel group A and SSDs for pixel group B. As explained in step S62 of the flowchart in Fig. 5, the arithmetic circuit 11 calculates the area shown in the graph in Fig. 9 and determines the obtained area as fitness.

[0058] In FIG. 9, the SSD of both pixel group A and pixel group B tends to increase as the angle moves away from 0 degrees. This is natural because the difference between the reference image and the view-changed image group increases as the angle moves away. On the other hand, it can be seen that pixel group B has higher angle dependency than pixel group A. In other words, pixel group A is more robust to view changes than pixel group B. Therefore, in step S08, calculation circuit 11 selects a pixel group with low angle dependency of SSD. This allows calculation circuit 11 to select a pixel group that is robust to view changes.

[0059] <Comparison of recognition rates> Figure 10 is a graph comparing the recognition rates when different pixel groups are used. Specifically, the graph in Figure 10 shows the recognition rates when using different pixel groups obtained from images captured using an object whose appearance changes depending on the viewing direction and multiple different view-change image groups. Figure 10 shows the recognition rates for images captured while rotating the object from -45 degrees to +45 degrees in 5-degree increments around the same rotation axis, with 0 degrees representing the front of the object. In Figure 10, the horizontal axis represents the rotation angle, and the vertical axis represents the recognition rate. Figure 10 shows the recognition rates for a randomly selected pixel group, all pixels, and the optimal pixel group selected using the above-mentioned method. In the vicinity of 0 degrees, which is the front, the recognition rate is high in all cases. As the angle moves away from 0 degrees, the recognition rate decreases for all pixels and the randomly selected pixel group. On the other hand, the optimal pixel group selected using the above-mentioned method has a high recognition rate.

[0060] Specifically, with the optimal pixel group, a recognition rate of 70% was achieved even for view changes of about ±30 degrees. Furthermore, with the optimal pixel group, a matching success rate of 67% was achieved even for view changes of about ±40 degrees. These are improvements of 29% and 52%, respectively, compared to using all pixels in the reference region.

[0061] In the example of FIG. 10, the total number of pixels in the reference region is 3,721, while the optimal pixel group is approximately 2%, or 75 pixels. Therefore, in the example of FIG. 10, the processing time is faster when using the optimal pixel group compared to when using all pixels. Specifically, the processing time when using all pixels is 17.3 times that when using the optimal pixel group. Note that, as mentioned above, the number of pixels constituting the optimal pixel group varies depending on the repetition of the genetic algorithm. Therefore, although the number of pixels constituting the optimal pixel group is approximately 75 pixels, it is not fixed. The difference in processing time is also an example.

[0062] <Variations> An information processing device 1A according to a modified example will be described with reference to Fig. 11. After matching the target O, the information processing device 1A can estimate the position and orientation of the target O. Therefore, the information processing device 1A can store reference point data 156, an estimation result 157, and an estimation program P3 in the storage device 15.

[0063] The reference point data 156 is data relating to reference points of the object O that are used to estimate the position and orientation of the object O. For example, the reference point data 156 may include features and coordinate information of reference points that are specified in advance on the surface of the object O.

[0064] The estimation result 157 is information indicating the position and orientation of the object O estimated by the arithmetic circuit 11.

[0065] By executing the estimation program P3, the arithmetic circuit 11 becomes an estimation module that estimates the position and orientation of the object O. Specifically, by executing the estimation program P3, the arithmetic circuit 11 detects at least three of a plurality of reference points predetermined for the object O from the input image 154, using the input image 154, the matching result 155, and the reference point data 156. The arithmetic circuit 11 also estimates the position and orientation of the object O using the detected plurality of reference points.

[0066] Other variations In the above-described embodiment, the reference image 151 and the view-change image group 152 may be photographed images of the object O. However, the reference image 151 and the view-change image group 152 are not limited to photographed images. For example, they may be images generated by software or the like, including images of the object O viewed from different directions. Similarly, the input image 154 may be an image generated by software.

[0067] In the above-described embodiment, as shown in FIGS. 4A and 4B, an example has been described in which an image is divided into multiple reference areas and an arbitrary reference area is selected from among the multiple reference areas. However, the method for selecting an arbitrary reference area is not limited to this. Specifically, as long as one or more reference areas can be selected from the reference image 151, the method is not limited. For example, instead of selecting an arbitrary reference area from among the areas divided into a grid pattern, the selected reference areas may overlap. Furthermore, even when multiple reference areas are selected, the sizes of the multiple reference areas may be different.

[0068] In the above embodiment, the view-change image group 152 includes a plurality of different views obtained by coaxially rotating the object O. However, the view-change image group 152 is not limited to images obtained by coaxially rotating the object O, as long as the views are different. The view change may include one or more changes of the pan angle, tilt angle, roll angle, and scale.

[0069] In the above-described embodiment, an example was described in which a comparison pixel group is determined using an SSD value calculated from a pixel group generated from a reference image and a pixel group extracted from a view-transition image group. However, other methods can be used as long as the similarity between the pixel group generated from the reference image and the pixel group extracted from the view-transition image group can be calculated. For example, in addition to the SSD used in this embodiment, the Sum of Absolute Difference (SAD) is used as a "dissimilarity" index, i.e., the smaller the value, the more similar the pixels are. In addition, the Sum of Absolute Difference (SAD) is used as a "similarity" index, i.e., the larger the value, the more similar the pixels are. Examples of such similarity indexes include Normalized Cross Correlation (NCC) and Zero Means Normalized Cross Correlation (ZNCC), and other methods for evaluating similarity can also be used.

[0070] (1) A matching method according to the present disclosure is executed by an arithmetic circuit that can access a storage device, and is a method for matching a predetermined object that is a three-dimensional object from an image including the object, the method comprising: The storage device includes: storing a group of pixels for comparison preselected from a reference image including the object as pixels representing the characteristics of the object; The arithmetic circuit comprises: accepting a new image containing the object; comparing the comparison pixel set with the new image; Identifying the object in the new image A matching method comprising: The pixel groups for comparison are obtained by comparing a plurality of pixel groups arbitrarily selected from a reference image including the object with pixel groups extracted from a view-change image group including a plurality of images showing the object with different view changes, and among the plurality of pixel groups selected from the reference image, the pixel group that has the highest similarity to the plurality of images included in the view-change image group is used.

[0071] (2) In the matching method of (1), when comparing a plurality of pixel groups arbitrarily selected from the reference image with the pixel groups extracted from each of the plurality of images included in the view-change image group, the sum of squared errors between the pixel groups of the reference image and the pixel groups of the plurality of images included in the view-change image group is compared, and among the pixel groups selected from the reference image, the pixel group with the smallest sum of squared errors with the plurality of images included in the view-change image group can be used.

[0072] (3) In the matching methods (1) to (2), a combinatorial optimization method can be used to select an arbitrary group of pixels with high similarity from multiple groups of pixels arbitrarily selected from the reference image.

[0073] (4) In the matching method of (3), the combinatorial optimization method may be a genetic algorithm.

[0074] (5) In the matching methods (1) to (4), the view change may include one or more changes in pan angle, tilt angle, roll angle, and scale.

[0075] (6) In the matching methods (1) to (5), the comparison pixel group can include pixel groups selected from each of a plurality of reference regions obtained by dividing the reference image into a plurality of reference regions.

[0076] (7) The computer program of the present disclosure can execute the matching method according to any one of (1) to (6).

[0077] (8) An information processing device according to the present disclosure includes an arithmetic circuit that can access a storage device, and matches a predetermined object that is a three-dimensional object from an image including the object, The storage device includes: storing a group of pixels for comparison preselected from an image including the object as pixels representing the characteristics of the object; The arithmetic circuit accepting a new image containing the object; comparing the comparison pixel group with the new image; Identifying the object in the new image An information processing device, The pixel groups for comparison are obtained by comparing a plurality of pixel groups arbitrarily selected from a reference image including the object with pixel groups extracted from a view-change image group including a plurality of images showing the object with different view changes, and among the plurality of pixel groups selected from the reference image, the pixel group that has the highest similarity to the plurality of images included in the view-change image group is used. [Explanation of symbols]

[0078] 1. Information processing equipment 11 Arithmetic circuit 12 Input Devices 13 Output Devices 14 Communication Circuits 15 Storage device 151 Reference Images 152 view-changing images 153 Comparison pixel group 154 input images 155 Matching Results 156 Control Point Data 157 Estimation results P1 Generator P2 Matching Program

Claims

1. A matching method for matching a predetermined object, which is a three-dimensional object, from an image including the object, the method being executed by an arithmetic circuit that can access a storage device, the method comprising: The storage device includes: storing a group of pixels for comparison preselected from a reference image including the object as pixels representing the characteristics of the object; The matching method includes: The arithmetic circuit accepting a new image containing the object; comparing the comparison pixel set with the new image; Identifying the object in the new image A matching method comprising: The pixel groups for comparison are obtained by comparing a plurality of pixel groups arbitrarily selected from a reference image including the object with pixel groups extracted from a view-change image group including a plurality of images showing the object with different view changes, and among the plurality of pixel groups selected from the reference image, pixel groups that have a high similarity to the plurality of images included in the view-change image group are used. Matching method.

2. In comparing a plurality of pixel groups arbitrarily selected from the reference image with the pixel groups extracted from each of the plurality of images included in the view-change image group, a sum of squared errors between the pixel group of the reference image and the pixel groups of the plurality of images included in the view-change image group is compared, and a pixel group selected from the reference image with a smaller sum of squared errors between the pixel group of the plurality of images included in the view-change image group is used. The matching method according to claim 1 .

3. A combinatorial optimization method is used to select an arbitrary group of pixels having high similarity from the plurality of groups of pixels arbitrarily selected from the reference image. The matching method according to claim 1 .

4. The combinatorial optimization method is a genetic algorithm. The matching method according to claim 3 .

5. The view change includes one or more changes of a pan angle, a tilt angle, a roll angle, and a scale. The matching method according to claim 1 .

6. The comparison pixel group includes pixel groups selected from a plurality of reference regions determined from the reference image. The matching method according to claim 1 .

7. A computer program that causes an information processing device to execute the matching method according to any one of claims 1 to 6.

8. An information processing device having an arithmetic circuit that can access a storage device, and that matches a predetermined object that is a three-dimensional object from an image including the object, The storage device includes: storing a group of pixels for comparison preselected from the image including the object as pixels representing the characteristics of the object; The arithmetic circuit accepting a new image containing the object; comparing the comparison pixel group with the new image; Identifying the object in the new image An information processing device, The pixel groups for comparison are obtained by comparing a plurality of pixel groups arbitrarily selected from a reference image including the object with pixel groups extracted from a view-change image group including a plurality of images showing the object with different view changes, and among the plurality of pixel groups selected from the reference image, pixel groups that have a high similarity to the plurality of images included in the view-change image group are used. Information processing device.

Images