Information processing device

Abstract

This information processing device comprises: a feature point extraction means that extracts, from each of an image of interest and a reference image, a plurality of feature points represented by information on position, scale, direction, and local feature amount; a provisional corresponding point determination means that compares the local feature amounts of the feature points of the image of interest and the reference image to determine a plurality of provisional corresponding points; a voting means that, at each corresponding point, casts a combination of a value of magnification of scale between the feature points and a value of a rotational angle of direction for, among a plurality of intersections identified by combinations of a plurality of sections related to the magnification of scale and a plurality of sections related to the rotational angle of direction, an intersection corresponding to a combination of a section corresponding to the value of the magnification of scale and a section corresponding to the value of the rotational angle of direction, to generate a histogram representing casting values for a plurality of intersections between the plurality of sections related to the magnification of scale and the plurality of sections related to the rotational angle of direction; and a correct corresponding point determination means that determines a correct corresponding point among the provisional corresponding points on the basis of the generated histogram.

Description

Information processing device

[0001] The present invention relates to an information processing device, an information processing method, and a recording medium.

[0002] One image matching method used by information processing devices involves comparing all feature points of a target image (query image) with those of each registered image, finding pairs of feature points with the closest local feature values ​​as corresponding points, and determining the matching result as the one with the most corresponding points among the registered images. However, in an environment where local feature values ​​change significantly due to various factors such as changes in lighting and posture, many incorrect corresponding points may be obtained. Therefore, various methods have been proposed to remove incorrect corresponding points.

[0003] For example, in RANdom SAMple Consensus (RANSAC), a subset is randomly selected from a set of corresponding points to estimate geometric transformation parameters for the image, and an operation of removing outliers is repeated many times to select the optimal set of corresponding points.

[0004] Furthermore, in the method described in Patent Document 1, for each corresponding point pair, which is a combination of provisional corresponding points, a set of correct corresponding points is determined based on all of the constraints regarding the consistency of the scaling factor, the constraints regarding the consistency of the rotation angle, and the constraints regarding the consistency of the relative positional relationship, which are based on a comparison or difference of the geometric transformation parameters of the corresponding points.

[0005] In the method described in Patent Document 2, first, a geometric transformation parameter calculation unit calculates geometric transformation parameters for each of first corresponding points, which are pairs of corresponding feature points between pairs of input images. Next, a pseudo-geometric parameter calculation unit calculates pseudo-geometric transformation parameters based on the geometric transformation parameters and second corresponding points different from the first corresponding points. Next, a parameter discretization unit specifies a discretization interval of the geometric transformation space. Next, a voting histogram creation unit creates a voting histogram by voting for the geometric transformation parameters and the pseudo-geometric transformation parameters in the discretization interval of the geometric transformation space.

[0006] Another document describing a technology related to the present invention is Patent Document 3. Patent Document 3 describes accurately joining multiple image data blocks that form original image data. Specifically, an overlapping area is extracted when an original image and a reference image related to the acquired image data blocks are superimposed using a geometric transformation formula T. Next, feature points are extracted in the overlapping area of ​​the original image and the reference image. Next, corresponding points in the overlapping area of ​​the original image and the reference image corresponding to each feature point are extracted using the geometric transformation formula T. Next, a geometric transformation formula T is generated so as to maximize the degree of match between the multiple feature points and the multiple corresponding points. Next, the geometric transformation formula T is repeatedly modified while gradually increasing the resolution R until the resolution R reaches the maximum processing resolution.

[0007] Patent WO2017 / 006852, Patent Publication No. 2016-157268, Patent Publication No. 2009-044612

[0008] However, all of the above-mentioned methods for removing mismatched points have the problem of requiring a large amount of information processing. This is because RANSAC requires repeated operations to remove outliers. Furthermore, the method described in Patent Document 1 involves a huge number of pairs of all provisional corresponding points. Furthermore, the method described in Patent Document 2 involves calculation of geometric transformation parameters, parameter discretization, and voting not only for first corresponding points but also for second corresponding points that are different from the first corresponding points. Therefore, there is a demand for a simple method for removing mismatched points.

[0009] An object of the present invention is to provide an information processing device that solves the above-mentioned problems.

[0010] an information processing device according to one aspect of the present invention includes: a target image acquisition means for acquiring an image obtained by photographing a surface of an object as a target image; a reference image acquisition means for acquiring an image obtained by photographing a surface of a reference object as a reference image; a feature point extraction means for extracting a plurality of feature points represented by information on position, scale, direction, and local feature amount from each of the target image and the reference image; a tentative corresponding point determination means for determining a plurality of tentative corresponding points by comparing the local feature amounts of each feature point of the target image and the reference image; a voting means for, for each of the tentative corresponding points, voting for an intersection point corresponding to a combination of a value of a scale magnification between feature points and a value of a rotation angle of a direction among a plurality of intersection points specified by combinations of a plurality of sections related to the scale magnification and a plurality of sections related to the rotation angle of the direction, the intersection point corresponding to a combination of a section related to the value of the scale magnification and a section related to the rotation angle of the direction, and generating a histogram representing voting values ​​for a plurality of intersection points between the plurality of sections related to the scale magnification and the plurality of sections related to the rotation angle of the direction; and a correct corresponding point determination means for determining a correct corresponding point among the tentative corresponding points based on the generated histogram. The device is configured to include:

[0011] An information processing method according to another aspect of the present invention is configured to: obtain, as a target image, an image obtained by photographing a surface of an object; obtain, as a reference image, an image obtained by photographing a surface of a reference object; extract a plurality of feature points represented by information on position, scale, direction, and local feature amount from each of the target image and the reference image; determine a plurality of tentative corresponding points by comparing the local feature amounts of each feature point in the target image and the reference image; vote, for each of the tentative corresponding points, for a combination of a scale magnification value and a directional rotation angle value between feature points, among a plurality of intersection points specified by combinations of a plurality of intervals related to the scale magnification value and a plurality of intervals related to the directional rotation angle, for an intersection point corresponding to a combination of an interval related to the scale magnification value and an interval related to the directional rotation angle; generate a histogram representing voting values ​​for a plurality of intersection points between the plurality of intervals related to the scale magnification value and the plurality of intervals related to the directional rotation angle; and determine a correct corresponding point from among the tentative corresponding points based on the generated histogram.

[0012] a process for comparing the local features of each feature point in the target image and the reference image to determine a plurality of tentative corresponding points; a process for voting, for each of the tentative corresponding points, a combination of a scale magnification value and a directional rotation angle value between feature points, among a plurality of intersection points identified by combinations of a plurality of intervals related to the scale magnification value and a plurality of intervals related to the directional rotation angle, for a combination of a scale magnification value and a directional rotation angle value, for an intersection point corresponding to a combination of an interval related to the scale magnification value and an interval related to the directional rotation angle, and generating a histogram representing voting values ​​for a plurality of intersection points between the plurality of intervals related to the scale magnification value and the plurality of intervals related to the directional rotation angle; and a process for determining a correct corresponding point from the tentative corresponding points based on the generated histogram.

[0013] By having the above-described configuration, the present invention can remove mismatched points with a small amount of information processing.

[0014] FIG. 1 is a block diagram of an information processing device according to a first embodiment of the present invention. FIG. 2 is a diagram showing an example of information stored in a product database according to the first embodiment of the present invention. FIG. 3 is a schematic diagram showing an example of a 2D histogram generated by a voting unit according to the first embodiment of the present invention. FIG. 4 is a flowchart showing an example of an operation of an information processing device according to the first embodiment of the present invention. FIG. 5 is an explanatory diagram of a method for calculating a non-rigid deformation degree in an information processing device according to a second embodiment of the present invention. FIG. 6 is a schematic diagram showing another example of a 2D histogram generated by a voting unit according to the second embodiment of the present invention. FIG. 7 is a flowchart showing an example of an operation of an information processing device according to the second embodiment of the present invention. FIG. 8 is a block diagram of an information processing device according to a third embodiment of the present invention. FIG. 9 is a flowchart showing an example of an operation of an information processing device according to the third embodiment of the present invention. FIG. 10 is a block diagram of an information processing device according to a fourth embodiment of the present invention.

[0015] Next, embodiments of the present invention will be described in detail with reference to the drawings. [First Embodiment] FIG. 1 is a block diagram of an information processing device 1 according to a first embodiment of the present invention. This information processing device 1 is a portable information processing device such as a smartphone, and has a function of searching a product database based on an image obtained by photographing a product. Hereinafter, a product is referred to as a pharmaceutical package. Furthermore, a pharmaceutical package is an outer container or outer covering that further packages a primary package that comes into direct contact with a pharmaceutical product for retail sale, and its outer shape is a rectangular parallelepiped. Furthermore, the surface (plane) on which the name of the pharmaceutical product (product name) is displayed in large letters is the front of the pharmaceutical package, and its shape is rectangular. However, the present invention is not limited to pharmaceutical packages. It is also applicable to objects other than pharmaceutical packages, such as products.

[0016] Referring to FIG. 1, the information processing device 1 comprises a camera 2, a communication I / F unit 3, an operation input unit 4, a screen display unit 5, a storage unit 6, and an arithmetic processing unit 7.

[0017] The camera 2 is, for example, a color camera or a black-and-white camera equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor having a pixel capacity of several million pixels. The communication I / F unit 3 is composed of a data communication circuit and performs data communication with various external devices wirelessly or via a wired connection. The operation input unit 4 is composed of devices such as a keyboard and a mouse, and detects operations performed by the user of the information processing device 1 and outputs the detected operations to the arithmetic processing unit 7. The screen display unit 5 is composed of devices such as an LCD (Liquid Crystal Display), and displays various information on the screen in response to instructions from the arithmetic processing unit 7.

[0018] The storage unit 6 is composed of one or more storage devices such as a hard disk or memory, and stores processing information and a program 61 required for various processes in the arithmetic processing unit 7. The program 61 is a program that is read into the arithmetic processing unit 7 and executed to realize various processing units, and is read in advance from an external device or recording medium (not shown) via a data input / output function such as the communication I / F unit 3 and stored in the storage unit 6. The main processing information stored in the storage unit 6 is a product database 62.

[0019] The product database 62 is a database that accumulates various information related to products. Fig. 2 is a diagram showing an example of information stored in the product database 62. In this example, the product database 62 is made up of a plurality of pieces of product information 621. Each piece of product information 621 is made up of an ID 6211, a product name 6212, an image 6213 of a genuine product, an image 6214 of a counterfeit product, and commentary information 6215.

[0020] The ID 6211 field contains an ID such as a number that uniquely identifies the product information 621. The product name 6212 field contains the product name of the genuine product. The image 6213 field contains an image of the genuine product taken from directly in front. The image 6214 field contains an image of the fake product taken from directly in front. The explanation information 6215 field contains a diagram and / or text that explains the key points for distinguishing between the image 6213 of the genuine product and the image 6214 of the fake.

[0021] The arithmetic processing unit 7 has one or more processors such as a CPU (Central Processing Unit) and its peripheral circuits, and by reading and executing a program 61 from the storage unit 6, the above hardware and the program 61 work together to realize various processing functions. The main processing units realized by the arithmetic processing unit 7 are a target image acquisition unit 71, a reference image acquisition unit 72, a feature point extraction unit 73, a tentative corresponding point determination unit 74, a voting unit 75, a correct corresponding point determination unit 76, a similarity calculation unit 77, and a search result generation and output unit 78.

[0022] The target image acquisition unit 71 is configured to acquire an image of the front of a product from the camera 2 as a target image. For example, when a user performs an operation to capture the front of a product they are about to purchase as a subject with the camera 2, the target image acquisition unit 71 acquires the image of the front of the product captured by the camera 2 as a target image. In the image captured by the camera 2, in addition to the front of the product, the side or background of the product may also be captured. The target image acquisition unit 71 extracts the front image of the product as a target image from the image captured by the camera 2 using an arbitrary method.

[0023] The reference image acquisition unit 72 is configured to read all product information 621 from the product database 62, and acquire an image 6213 of a genuine product and an image 6214 of a fake product from each piece of product information 21 as independent reference images. The reference image acquisition unit 72 uniquely manages each reference image by assigning an image number that is the ID 6211 of the product information 21 plus a sub-number. For example, if the ID 6211 of the product information 21 is "001", the image number "001-01" is assigned to the image 6213 of a genuine product included in that product information 21, and the image number "001-02" is assigned to the image 6214 of a fake product.

[0024] The feature point extraction unit 73 is configured to extract a plurality of feature points represented by information on position, scale, orientation, and local feature amount from each of the target image acquired by the target image acquisition unit 71 and the reference image acquired by the reference image acquisition unit 72. For example, the feature point extraction unit 73 may extract a plurality of feature points represented by information on position, scale, orientation, and local feature amount from each of the target image and the reference image by processing a SIFT (Scale-Invariant Feature Transform) algorithm. However, the algorithm is not limited to SIFT, and a plurality of feature points represented by information on position, scale, orientation, and local feature amount may be extracted by processing another algorithm such as ORB (Oriented FAST and Rotated BRIEF). The feature point extraction unit 73 assigns a feature point number to each feature point extracted from the target image and the reference image, and stores information on the position, scale, direction, and local feature amount in association with the feature point number. For example, the feature point extraction unit 73 manages each feature point uniquely by assigning a number obtained by adding a subnumber to the image number described above as a feature point number. For example, the feature point extraction unit 73 assigns feature point numbers such as "001-01-001" to multiple feature points extracted from the image with image number "001-01."

[0025] The tentative corresponding point determination unit 74 is configured to, for each pair of the target image and the reference image, compare the local feature amounts of each feature point extracted by the feature point extraction unit 73 between the target image and the reference image to determine multiple tentative corresponding points. For example, the tentative corresponding point determination unit 74 calculates, for each feature point in the target image, the Euclidean distance between the local feature amounts of that feature point and all feature points in the reference image, and detects the feature points with the smallest Euclidean distance as tentative corresponding points. However, the method for determining tentative corresponding points is not limited to the above, and any method may be used. The tentative corresponding point determination unit 74 saves a list of determined tentative corresponding points for each pair of the target image and the reference image. The tentative corresponding points are identified, for example, by pairs of feature point numbers. The list of tentative corresponding points is, for example, a list of pairs of feature point numbers.

[0026] The voting unit 75 is configured to perform a voting process for each pair of a target image and a reference image to generate a two-dimensional histogram.

[0027] FIG. 3 is a schematic diagram showing an example of a two-dimensional histogram generated by the voting unit 75. In this example, the two-dimensional histogram has multiple intervals related to the scale magnification assigned to the vertical axis and multiple intervals related to the directional rotation angle assigned to the horizontal axis. Here, the scale magnification is, for example, the ratio of the scale of the feature points of the target image to the scale of the feature points of the reference image that form the tentative corresponding points. The rotation angle is, for example, the angle obtained by subtracting the direction of the feature points of the reference image from the direction of the feature points of the target image that form the tentative corresponding points. Each of the multiple intervals assigned to the vertical and horizontal axes has a defined range. For example, the minimum and maximum values ​​of the scale magnification are Smin and Smax, the width of the scale interval is h1, the minimum and maximum values ​​of the rotation angle are Rmin and Rmax, and the width of the rotation angle is h2. In this case, for example, the ranges of the multiple intervals assigned to the vertical axis, in order from closest to the minimum value Smin, are as follows: interval 11 range = Smin or more, less than Smin+h1, interval 12 = Smin+h1 or more, less than Smin+2h1, etc. Also, for example, the ranges of the multiple intervals assigned to the horizontal axis, in order from closest to the minimum value Rmin, are as follows: interval 21 range = Rmin or more, less than Rmin+h2, interval 22 = Rmin+h2 or more, less than Rmin+2h2, etc.

[0028] The two-dimensional histogram shown in Fig. 3 has a plurality of intersections specified by intervals assigned to the vertical axis and intervals assigned to the horizontal axis. For example, if the total number of intervals assigned to the vertical axis is k1 and the total number of intervals assigned to the horizontal axis is k2, then there are a total of k1 x k2 intersections. Each intersection has a counter that counts the vote value and a storage area that stores information (feature point number pairs) that specifies the tentative corresponding points voted for at that intersection. The initial values ​​of the counter and storage area are NULL.

[0029] The total number of intervals k1, k2 and the interval widths h1, h2 can be determined arbitrarily. Generally, methods such as Sturges' rule, Scott's selection, and Friedman's selection are known as methods for determining the number and width of intervals in a histogram. These known methods or improved methods may be used.

[0030] The voting unit 75 performs the following voting process on the 2D histogram for each pair of a target image and a reference image. First, the voting unit 75 focuses on one tentative corresponding point among all tentative corresponding points determined by the tentative corresponding point determination unit 74 for that pair. Next, for the tentative corresponding point under consideration, the voting unit 75 determines an intersection between a scale magnification section in which the value of the scale magnification between the feature points falls within the range and a rotation angle section in which the value of the rotation angle of the direction between the feature points falls within the range, and votes for that intersection. That is, the voting unit 75 increments a counter corresponding to the intersection and stores information about the tentative corresponding point (pair of feature point numbers) in a storage area corresponding to the intersection. Next, the voting unit 75 shifts its focus to the next tentative corresponding point and performs the same process as described above. After focusing on all tentative corresponding points determined by the tentative corresponding point determination unit 74 for that pair, the voting unit 75 associates the generated 2D histogram with the pair and stores it. The voting unit 75 repeats the same process for other pairs. This generates a two-dimensional histogram for each pair of target and reference images.

[0031] The correct corresponding point determiner 76 is configured to determine correct corresponding points for each of the two-dimensional histograms generated by the voting unit 75 based on the intersection with the largest voting value. For example, the correct corresponding point determiner 76 compares the counter values ​​for each of the intersections of the two-dimensional histograms to determine the intersection with the largest counter value. Next, the correct corresponding point determiner 76 determines the set of tentative corresponding points identified by the list of feature point numbers stored in the memory area for the determined intersection as the set of correct corresponding points. The reason for this determination is that when the target image and the reference image show the front of the same product, the scale magnification and directional rotation angle of the corresponding points are often similar.

[0032] The similarity calculation unit 77 is configured to calculate the number of virtual corresponding points included in the set of correct corresponding points determined by the correct corresponding point determination unit 76 from the two-dimensional histogram generated for each pair of target image and reference image as the similarity between the pair of target image and reference image.

[0033] The search result generation and output unit 78 is configured to select the highest similarity from the similarities for each pair of target image and reference image calculated by the similarity calculation unit 77, and determine the reference image having the selected similarity as the reference image similar to the target image. Alternatively, the search result generation and output unit 78 may determine the reference image having a similarity equal to or greater than a threshold from the similarities for each pair of target image and reference image calculated by the similarity calculation unit 77 as the reference image similar to the target image. The search result generation and output unit 78 also reads out product information 621 including the determined reference image as an image 6213 of a genuine product or an image 6214 of a counterfeit product from the product database 62. The search result generation and output unit 78 also displays the read product information 621 as a search result on the screen display unit 5 and / or transmits it to an external device via the communication I / F unit 3.

[0034] 4 is a flowchart showing an example of the operation of the information processing device 1. The operation of the information processing device 1 will be described below with reference to FIG.

[0035] First, the target image acquisition unit 71 acquires an image of the front of a product from the camera 2 as a target image (step S1). Next, the reference image acquisition unit 72 reads all product information 621 from the product database 62 and acquires, from each piece of product information 621, an image 6213 of a genuine product and an image 6214 of a fake product as reference images (step S2). Next, the feature point extraction unit 73 extracts a plurality of feature points represented by information on position, scale, direction, and local feature amount from each of the target image acquired by the target image acquisition unit 71 and the reference image acquired by the reference image acquisition unit 72 (step S3).

[0036] Next, the tentative corresponding point determination unit 74 compares the local feature amounts of each feature point extracted by the feature point extraction unit 73 between the target image and the reference image for each pair of the target image and the reference image to determine multiple tentative corresponding points (step S4). Next, the voting unit 75 performs a voting process for each pair of the target image and the reference image to generate a two-dimensional histogram (step S5). In the voting process, the voting unit 75 votes for each tentative corresponding point on an intersection point corresponding to a combination of a range related to the scale magnification and a range related to the directional rotation angle, which corresponds to a combination of the value of the scale magnification and the value of the directional rotation angle between the feature points. This generates a two-dimensional histogram representing the vote values ​​for the intersection points of multiple ranges related to the scale magnification and the range related to the directional rotation angle.

[0037] Next, the correct corresponding point determiner 76 determines correct corresponding points based on the intersection with the maximum voting value for each of the two-dimensional histograms generated by the voting unit 75 (step S6). Next, the similarity calculator 77 calculates the number of tentative corresponding points included in the set of correct corresponding points determined by the correct corresponding point determiner 76 from the two-dimensional histograms generated for each pair of the target image and the reference image as the similarity between the pair of the target image and the reference image (step S7).

[0038] Next, the search result generation and output unit 78 generates and outputs search results (step S8). In generating the search results, the search result generation and output unit 78 first selects the highest similarity from the similarities calculated by the similarity calculation unit 77 for each pair of the target image and the reference image, and determines the reference image having this selected similarity as the reference image similar to the target image. Next, the search result generation and output unit 78 reads out product information 621 including the determined reference image as an image 6213 of a genuine product or an image 6214 of a counterfeit product from the product database 62. In outputting the search results, the search result generation and output unit 78 displays the read product information 621 on the screen display unit 5 and / or transmits it to an external device via the communication I / F unit 3.

[0039] As described above, the information processing device 1 is equipped with the voting unit 75 and the correct corresponding point determination unit 76 that are configured and operate as described above, and therefore can remove incorrect corresponding points with a small amount of information processing, and ultimately can reduce the amount of information processing required for image matching by searching for corresponding points.

[0040] [Second embodiment] Next, a second embodiment of the present invention will be described. The information processing device according to the second embodiment (hereinafter referred to as information processing device 1A) is different from the information processing device 1 according to the first embodiment in the function of the voting unit 75, but is otherwise the same as the first embodiment.

[0041] The voting unit 75 in this embodiment is configured to calculate a non-rigid deformation degree that represents the degree of non-rigid deformation of the front surface of a product captured in a target image acquired by the target image acquisition unit 71. The front surface of a real product is rectangular. However, the front surface of the product in an image captured obliquely by the camera 2 is non-rigidly deformed and no longer rectangular. Compared to a rectangular target image, a non-rectangular target image exhibits a reduced similarity in the scale magnification and directional rotation angle of corresponding points when the same product is captured in a reference image. This tendency for reduction becomes more pronounced as the degree of non-rigid deformation increases. In other words, the closer the surface of the product captured in the target image is to a rectangle, the greater the similarity in the scale magnification and directional rotation angle of corresponding points when the same product is captured in a reference image.

[0042] Therefore, the voting unit 75 in this embodiment is configured to generate, by voting, a type of 2D histogram corresponding to the calculated degree of non-rigid deformation from among multiple types of 2D histograms each having different value ranges for multiple intervals related to the scale magnification and the direction rotation angle.

[0043] FIG. 5 is a diagram illustrating an example of a method for calculating the degree of non-rigid deformation. The rectangle shown in FIG. 5 is composed of four sides L1 to L4, with sides L1 and L2 opposing each other and sides L3 and L4 opposing each other. The voting unit 75 extracts the four sides that constitute the front of the product shown in the target image, calculates the ratio of the long side to the short side for each pair of opposing sides, and determines the larger ratio as the degree of non-rigid deformation of the front. For example, if L1 / L2 = 1.3 and L3 / L4 = 1.1 for the rectangle shown in FIG. 5, the degree of non-rigid deformation is determined to be 1.3. However, the method for calculating the degree of non-rigid deformation is not limited to the above. Any index that indicates the degree to which the rectangle is not rectangular may be used as the degree of non-rigid deformation.

[0044] Fig. 6 is a schematic diagram showing another example of a 2D histogram generated by the voting unit 75. The 2D histogram shown in Fig. 6 differs from the 2D histogram shown in Fig. 3 in that the scale interval width is h3, which is smaller than h1, and the rotation angle width is h4, which is smaller than h2. The minimum value Smin and maximum value Smax of the scale magnification value, and the minimum value Rmin and maximum value Rmax of the rotation angle value are the same as those in Fig. 3. Therefore, the 2D histogram shown in Fig. 6 has a larger total number of intervals on the vertical and horizontal axes than the 2D histogram shown in Fig. 3.

[0045] Next, the operation of this embodiment will be described with reference to the flowchart of FIG.

[0046] The processes from step S1 to step S4 are the same as those from step S1 to step S4 in FIG. 4 . In step S11, the voting unit 75 calculates a non-rigid deformation degree representing the degree of non-rigid deformation of the front of the product depicted in the target image acquired by the target image acquisition unit 71. Next, in step S5, the voting unit 75 generates, by voting, the 2D histogram shown in FIG. 6 with a narrower section width, of the two types of 2D histograms shown in FIGS. 3 and 6 , if the calculated non-rigid deformation degree is less than threshold T. On the other hand, if the calculated non-rigid deformation degree is equal to or greater than threshold T, the voting unit 75 generates, by voting, the 2D histogram shown in FIG. 3 with a wider section width, in step S5. The subsequent processes from step S6 to step S8 are the same as those in the first embodiment.

[0047] In the above explanation, two types of two-dimensional histograms were used, each with a different range of values ​​for multiple intervals related to the scale magnification and the rotation angle of the direction, but three or more types of two-dimensional histograms may also be used.

[0048] As described above, according to this embodiment, among multiple types of 2D histograms with different ranges of values ​​for multiple intervals related to scale magnification and multiple intervals related to directional rotation angle, a type of 2D histogram corresponding to the degree of non-rigid deformation of the front surface of the product captured in the target image is generated by voting. Therefore, for a target image obtained by a user photographing the front surface of the product from a front-on perspective, a 2D histogram with a narrow range of values ​​can be generated by voting. By fully utilizing the characteristic that the scale magnification and directional rotation angle of corresponding points are often similar when the same product is captured in the target image and the reference image, the proportion of truly correct corresponding points in the set of correct corresponding points can be increased. This is because a 2D histogram with a wide range of values ​​increases the probability that incorrect corresponding points will be voted for at the intersection with the largest voting value. On the other hand, for a target image obtained by a user photographing the front surface of the product from an oblique angle, a 2D histogram with a wide range of values ​​can be generated by voting, reducing the probability that correct corresponding points will be excluded from the set of correct corresponding points. In addition, when the same product is shown in the target image and the reference image, the probability that an incorrect corresponding point will be voted for at the intersection with the largest voting value increases in a two-dimensional histogram with a wide range of values ​​in the interval, but the probability that the correct corresponding point will be excluded from the set of correct corresponding points decreases even more, so it is possible to prevent the accuracy of the similarity from decreasing drastically.

[0049] [Third embodiment] Next, a third embodiment of the present invention will be described. Fig. 8 is a block diagram of an information processing device 1B according to the third embodiment of the present invention, in which the same reference numerals as in Fig. 1 indicate the same parts, and 79 is a rigid body transformation matrix calculation unit, and 80 is a target image correction unit.

[0050] The rigid transformation matrix calculation unit 79 is configured to calculate a rigid transformation matrix between the target image and the reference image based on the correct corresponding points determined by the correct corresponding point determination unit 76. Various methods for calculating a rigid transformation matrix between two images from a plurality of corresponding points have been proposed or put to practical use. Any of these methods may be used.

[0051] The target image correcting section 80 is configured to perform rigid transformation on the target image using the rigid transformation matrix calculated by the rigid transformation matrix calculating section 79 .

[0052] Next, the operation of this embodiment will be described with reference to the flowchart of FIG.

[0053] The processes from step S1 to step S7 are the same as those from step S1 to step S7 in FIG. 4 . Next, the search result generation and output unit 78 selects the highest similarity from the similarities calculated by the similarity calculation unit 77 for each pair of target image and reference image, and determines the reference image having the selected similarity as the reference image similar to the target image (step S21). Next, the rigid transformation matrix calculation unit 79 calculates a rigid transformation matrix between the target image and the reference image based on the corresponding points between the target image and the reference image determined in step S21 (step S22). Next, the target image correction unit 80 performs rigid transformation on the target image using the calculated rigid transformation matrix to generate a corrected target image (step S23). Next, the search result generation and output unit 78 reads product information 621 from the product database 62, including the reference image determined in step S21 as an image 6213 of a genuine product or an image 6214 of a counterfeit product (step S24). Next, the search result generation and output unit 78 displays the corrected target image as a monitor image of the target image captured by the camera, and the read-out product information 621 as a search result on the screen display unit 5, and / or transmits them to an external device via the communication I / F unit 3 (step S25).

[0054] As described above, according to this embodiment, the same effects as those of the first embodiment can be obtained, and the target image captured by the user using the camera 2 can be displayed on the monitor as a corrected image. Therefore, for example, the size and orientation of the target image captured by the user can be adjusted to match the reference image. As a result, the user can easily recognize the differences between the target image and an image of a real product or the like on the screen.

[0055] [Fourth embodiment] Next, a fourth embodiment of the present invention will be described. Fig. 10 is a block diagram of an information processing device 10 according to this embodiment. Note that this embodiment will provide an overview of the information processing device described above.

[0056] As shown in FIG. 10 , the information processing device 10 in this embodiment is composed of a target image acquisition unit 11, a reference image acquisition unit 12, a feature point extraction unit 13, a tentative corresponding point determination unit 14, a voting unit 15, and a correct corresponding point determination unit 16.

[0057] The target image acquisition unit 11 is configured to acquire an image obtained by photographing a surface of an object as the target image. The reference image acquisition unit 12 is configured to acquire an image obtained by photographing a surface of a reference object as the reference image. The feature point extraction unit 13 is configured to extract a plurality of feature points represented by information on position, scale, direction, and local feature amount from each of the target image and the reference image. The tentative corresponding point determination unit 14 is configured to determine a plurality of tentative corresponding points by comparing the local feature amounts of each feature point in the target image and the reference image. The voting unit 15 is configured to vote, for each tentative corresponding point, a combination of a scale magnification value and a direction rotation angle value between the feature points for an intersection point corresponding to a combination of a range corresponding to the scale magnification value and a range corresponding to the direction rotation angle value, among a plurality of intersection points identified by combinations of a plurality of ranges related to the scale magnification and a plurality of ranges related to the direction rotation angle, and to generate a histogram representing the voting values ​​for a plurality of intersection points between a plurality of ranges related to the scale magnification and a plurality of ranges related to the direction rotation angle. The correct corresponding point determination unit 16 is configured to determine correct corresponding points from among the tentative corresponding points based on the generated histogram.

[0058] The information processing device 10 configured as described above operates as follows. First, the target image acquisition unit 11 acquires an image obtained by photographing the surface of an object as the target image. Next, the reference image acquisition unit 12 acquires an image obtained by photographing the surface of a reference object as the reference image. Next, the feature point extraction unit 13 extracts a plurality of feature points represented by information on position, scale, orientation, and local feature amount from each of the target image and the reference image. Next, the tentative corresponding point determination unit 14 determines a plurality of tentative corresponding points by comparing the local feature amounts of each feature point in the target image and the reference image. Next, the voting unit 15 votes, for each tentative corresponding point, for an intersection point corresponding to a combination of the scale magnification value and the directional rotation angle value between the feature points, among a plurality of intersection points identified by combinations of a plurality of intervals related to the scale magnification and a plurality of intervals related to the directional rotation angle, for a combination of the scale magnification value and the directional rotation angle value, and generates a histogram representing the voting values ​​for a plurality of intersection points between a plurality of intervals related to the scale magnification and a plurality of intervals related to the directional rotation angle. Next, the correct corresponding point determiner 16 determines correct corresponding points from among the tentative corresponding points based on the generated histogram.

[0059] According to the information processing device 10 configured and operating as described above, since it is equipped with the voting unit 15 and the correct corresponding point determination unit 16 configured and operating as described above, it is possible to remove incorrect corresponding points with a small amount of information processing.

[0060] Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0061] For example, in the above, the voting unit votes for one intersection for each tentative corresponding point, but it may also be configured to vote for at least one of the intersection corresponding to the combination of the section corresponding to the scale magnification value and the section corresponding to the direction rotation angle value, and up to eight surrounding intersections that directly contact the intersection. Also, in the above, the positive corresponding point determiner calculates the similarity based on the maximum vote value, but it may also be configured to calculate the similarity based on a vote value that is equal to or greater than a threshold.

[0062] Furthermore, the information processing device may use, instead of the above-mentioned CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

[0063] The present invention can be used in general image processing such as image matching using local features.

[0064] REFERENCE SIGNS LIST 1 Information processing device 2 Camera 3 Communication I / F unit 4 Operation input unit 5 Screen display unit 6 Storage unit 7 Arithmetic processing unit 10 Information processing device 11 Target image acquisition unit 12 Reference image acquisition unit 13 Feature point extraction unit 14 Provisional corresponding point determination unit 15 Voting unit 16 Correct corresponding point determination unit

Claims

1. a target image acquisition means for acquiring an image obtained by photographing a surface of an object as a target image; a reference image acquisition means for acquiring an image obtained by photographing a surface of a reference object as a reference image; a feature point extraction means for extracting a plurality of feature points represented by information on position, scale, direction, and local feature amount from each of the target image and the reference image; a virtual corresponding point determination means for determining a plurality of virtual corresponding points by comparing the local feature amounts of each feature point of the target image and the reference image; a voting means for voting, for each of the virtual corresponding points, for a combination of a scale magnification value between feature points and a rotation angle value of a direction, for an intersection point corresponding to a combination of a section corresponding to the scale magnification value and a section corresponding to the rotation angle value of the direction, among a plurality of intersection points specified by a combination of a plurality of sections related to the scale magnification and a plurality of sections related to the rotation angle of the direction, and generating a histogram representing vote values ​​for a plurality of intersection points between the plurality of sections related to the scale magnification and the plurality of sections related to the rotation angle of the direction; a correct corresponding point determining means for determining correct corresponding points among the tentative corresponding points based on the generated histogram; An information processing device comprising:

2. the correct corresponding point determining means determines a correct corresponding point from among the tentative corresponding points based on an intersection of the plurality of sections with the largest voting value. The information processing device according to claim 1 .

3. The voting means calculating a non-rigid deformation degree representing the degree of non-rigid deformation of the surface from the target image; generating, by the voting, a histogram of a type corresponding to the calculated degree of non-rigid deformation from among a plurality of types of histograms having different value ranges for a plurality of intervals related to the scale magnification and a plurality of intervals related to the rotation angle of the direction; The information processing device according to claim 2 .

4. and further comprising a similarity calculation means for calculating a similarity between the target image and the reference image based on the determined number of correct corresponding points.

4. The information processing device according to claim 2 or 3.

5. further comprising a rigid body transformation matrix calculation means for calculating a rigid body transformation matrix between the target image and the reference image based on the determined correct corresponding points; 4. The information processing device according to claim 2 or 3.

6. and a target image correcting means for performing rigid transformation on the target image using the calculated rigid transformation matrix. The information processing device according to claim 5 .

7. An image obtained by photographing a surface of an object is acquired as a target image; An image obtained by photographing a surface of a reference object is acquired as a reference image; extracting a plurality of feature points represented by information on position, scale, direction, and local feature amount from each of the target image and the reference image; determining a plurality of virtual corresponding points by comparing the local feature amounts of each feature point of the target image and the reference image; For each of the virtual corresponding points, a combination of a scale magnification value between feature points and a rotation angle value of a direction is voted for at an intersection point corresponding to a combination of a section corresponding to the scale magnification value and a section corresponding to the rotation angle value of the direction, among a plurality of intersection points identified by a combination of a plurality of sections related to the scale magnification and a plurality of sections related to the rotation angle of the direction, and a histogram is generated that represents the vote values ​​for a plurality of intersection points between the plurality of sections related to the scale magnification and the plurality of sections related to the rotation angle of the direction; determining a correct corresponding point from among the tentative corresponding points based on the generated histogram; Information processing methods.

8. In determining the correct corresponding point, a correct corresponding point is determined from among the tentative corresponding points based on an intersection of the plurality of sections with the largest voting value. The information processing method according to claim 7.

9. In the said vote, calculating a non-rigid deformation degree representing the degree of non-rigid deformation of the surface from the target image; generating, by the voting, a histogram of a type corresponding to the calculated degree of non-rigid deformation from among a plurality of types of histograms having different value ranges for a plurality of intervals related to the scale magnification and a plurality of intervals related to the rotation angle of the direction; The information processing method according to claim 8.

10. On the computer, A process of capturing an image of a surface of an object as a target image; A process of capturing an image of a surface of a reference object as a reference image; A process of extracting a plurality of feature points represented by information on position, scale, direction, and local feature amount from each of the target image and the reference image; a process of comparing the local feature amounts of each feature point of the target image and the reference image to determine a plurality of virtual corresponding points; a process of voting, for each of the virtual corresponding points, for a combination of a scale magnification value between feature points and a rotation angle value of a direction, for an intersection point corresponding to a combination of a section corresponding to the scale magnification value and a section corresponding to the rotation angle value of the direction, among a plurality of intersection points identified by a combination of a plurality of sections related to the scale magnification and a plurality of sections related to the rotation angle of the direction, and generating a histogram representing vote values ​​for a plurality of intersection points between the plurality of sections related to the scale magnification and the plurality of sections related to the rotation angle of the direction; a process of determining a correct corresponding point from among the tentative corresponding points based on the generated histogram; A program to perform the following.