Information processing device

Abstract

This information processing device comprises: a feature point extraction means that extracts, from each of an image of interest and a reference image, a plurality of feature points represented by information on position, scale, direction, and local feature amount; a provisional corresponding point determination means that compares the local feature amounts of the feature points of the image of interest and the reference image to determine a plurality of provisional corresponding points; a voting means that, at each corresponding point, casts a combination of a value of magnification of scale between the feature points and a value of a rotational angle of direction for, among a plurality of intersections identified by combinations of a plurality of sections related to the magnification of scale and a plurality of sections related to the rotational angle of direction, an intersection corresponding to a combination of a section corresponding to the value of the magnification of scale and a section corresponding to the value of the rotational angle of direction, to generate a histogram representing casting values for a plurality of intersections between the plurality of sections related to the magnification of scale and the plurality of sections related to the rotational angle of direction; and a correct corresponding point determination means that determines a correct corresponding point among the provisional corresponding points on the basis of the generated histogram.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a recording medium.

Background Art

[0002] In one method of image matching performed by an information processing apparatus, all feature points of a target image (query image) and each registered image are compared, and a pair of feature points with the closest local feature amount is obtained as a corresponding point. Among the registered images, the one with the largest number of corresponding points is used as the matching result. However, in an environment where local feature amounts change significantly due to various factors such as changes in lighting and posture, many incorrect corresponding points may be obtained. Therefore, various methods for removing incorrect corresponding points have been proposed.

[0003] For example, in RANdom SAmple Consensus (RANSAC), a subset is randomly selected from a set of corresponding points to estimate geometric transformation parameters of an image, and the operation of removing outliers is repeated many times to select an optimal set of corresponding points.

[0004] Also, in the method described in Patent Document 1, for each corresponding point pair that is a combination of provisional corresponding points, based on all of the constraint conditions regarding the consistency of the magnification ratio, the constraint conditions regarding the consistency of the rotation angle, and the constraint conditions regarding the consistency of the relative positional relationship, which are based on the comparison or difference of the geometric transformation parameters of the corresponding points, a set of correct corresponding points is determined.

[0005] Furthermore, in the method described in Patent Document 2, first, a geometric transformation parameter calculation unit calculates a geometric transformation parameter for each of the first corresponding points, which are pairs of corresponding feature points between pairs of input images. Next, a pseudo-geometric parameter calculation unit calculates a pseudo-geometric transformation parameter based on the geometric transformation parameter and a second corresponding point that is different from the first corresponding point. Next, a parameter discretization unit identifies a discretization interval of the geometric transformation space. Finally, a voting histogram creation unit creates a voting histogram by voting on the geometric transformation parameter and the pseudo-geometric transformation parameter within the discretization interval of the geometric transformation space.

[0006] Furthermore, Patent Document 3 describes other technologies related to the present invention. Patent Document 3 accurately joins multiple image data blocks that form the original image data. Specifically, it extracts the overlapping region when the original image and reference image related to the acquired image data block are superimposed using a geometric transformation formula T. Next, feature points are extracted in the overlapping region of the original image and the reference image. Next, corresponding points in the overlapping region of the original image and the reference image corresponding to each feature point are extracted using the geometric transformation formula T. Next, a geometric transformation formula T is generated so that the degree of agreement between multiple feature points and their corresponding points is maximized. Next, the geometric transformation formula T is repeatedly modified while gradually increasing the resolution R until the resolution R becomes the maximum processing resolution. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Re-Special WO2017 / 006852 [Patent Document 2] Japanese Patent Publication No. 2016-157268 [Patent Document 3] Japanese Patent Publication No. 2009-044612 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, all of the methods described above for removing mismatched points have the problem of requiring a large amount of information processing. This is because RANSAC needs to repeat the outlier removal operation many times. Also, the method described in Patent Document 1 results in an enormous number of pairs of provisional matched points. Furthermore, the method described in Patent Document 2 calculates geometric transformation parameters, discretizes parameters, and performs voting not only for the first matched point but also for second matched points that are different from the first matched point. Therefore, there is a need for a simpler way to remove mismatched points.

[0009] The object of the present invention is to provide an information processing device that solves the above-mentioned problems. [Means for solving the problem]

[0010] An information processing device according to one embodiment of the present invention is A target image acquisition means that acquires an image obtained by photographing the surface of an object as the target image, A reference image acquisition means that acquires an image obtained by photographing the surface of a reference object as a reference image, A feature point extraction means for extracting multiple feature points from the aforementioned target image and the aforementioned reference image, each of which is represented by information on position, scale, direction, and local feature quantity. A provisional correspondence point determination means that determines a plurality of provisional correspondence points by comparing the local feature quantities of each feature point of the target image and the reference image, Voting means for each of the aforementioned provisional correspondence points, which votes for a combination of the scale magnification value and the rotation angle value between feature points, to an intersection point among a plurality of intersection points identified by a combination of a plurality of intervals relating to the scale magnification value and a plurality of intervals relating to the rotation angle in the direction, and which corresponds to the intersection point corresponding to the combination of the interval relating to the scale magnification value and the interval relating to the rotation angle in the direction, and generates a histogram representing the vote values ​​for the plurality of intersection points of the plurality of intervals relating to the scale magnification value and the plurality of intervals relating to the rotation angle in the direction, Based on the generated histogram, a correct correspondence point determination means determines the correct correspondence point among the provisional correspondence points, It is configured to include.

[0011] Information processing methods according to other embodiments of the present invention are: The image obtained by photographing the surface of an object is acquired as the target image. The image obtained by photographing the surface of the reference object is acquired as the reference image. Multiple feature points, represented by information on position, scale, direction, and local features, are extracted from the aforementioned target image and the aforementioned reference image. Multiple provisional correspondence points are determined by comparing the local feature quantities of each feature point in the target image and the reference image. For each of the aforementioned provisional correspondence points, the combination of the scale magnification value and the rotation angle value between feature points is voted for among multiple intersections identified by the combination of multiple intervals relating to the scale magnification value and multiple intervals relating to the rotation angle in the direction, and a histogram is generated representing the vote values ​​for the multiple intersections of the multiple intervals relating to the scale magnification value and multiple intervals relating to the rotation angle in the direction. Based on the generated histogram, the correct corresponding point among the provisional corresponding points is determined. It is structured in this way.

[0012] Another embodiment of the present invention is a computer-readable recording medium, On the computer, The process involves capturing an image of the surface of an object and acquiring it as the target image, The process involves capturing an image of the surface of a reference object and acquiring it as the reference image, A process to extract multiple feature points from the aforementioned target image and the aforementioned reference image, each represented by information on position, scale, direction, and local feature quantity, A process to determine multiple provisional correspondence points by comparing the local feature quantities of each feature point in the target image and the reference image, For each of the provisional corresponding points, vote for the intersection corresponding to the combination of the section corresponding to the value of the magnification of the scale between the feature points and the section corresponding to the value of the rotation angle of the direction among the plurality of intersections specified by the combination of the plurality of sections regarding the magnification of the scale and the plurality of sections regarding the rotation angle of the direction, and generate a histogram representing the voting values for the plurality of intersections of the plurality of sections regarding the magnification of the scale and the plurality of sections regarding the rotation angle of the direction. Based on the generated histogram, perform a process of determining correct corresponding points among the provisional corresponding points. It is configured to record a program for causing the above to be performed.

Advantages of the Invention

[0013] Since the present invention has the above-described configuration, incorrect corresponding points can be removed with a small amount of information processing.

Brief Description of the Drawings

[0014] [Figure 1] It is a block diagram of an information processing apparatus according to a first embodiment of the present invention. [Figure 2] It is a diagram showing an example of information stored in a product database in the first embodiment of the present invention. [Figure 3] It is a schematic diagram showing an example of a two-dimensional histogram generated by a voting unit in the first embodiment of the present invention. [Figure 4] It is a flowchart showing an example of the operation of an information processing apparatus according to a first embodiment of the present invention. [Figure 5] It is an explanatory diagram of a method for calculating a non-rigid deformation degree in an information processing apparatus according to a second embodiment of the present invention. [Figure 6] It is a schematic diagram showing another example of a two-dimensional histogram generated by a voting unit in the second embodiment of the present invention. [Figure 7] It is a flowchart showing an example of the operation of an information processing apparatus according to a second embodiment of the present invention. [Figure 8]This is a block diagram of an information processing device according to a third embodiment of the present invention. [Figure 9] A flowchart illustrating an example of the operation of an information processing apparatus according to the third embodiment of the present invention. [Figure 10] This is a block diagram of an information processing device according to a fourth embodiment of the present invention. [Modes for carrying out the invention]

[0015] Next, embodiments of the present invention will be described in detail with reference to the drawings. [First Embodiment] Figure 1 is a block diagram of an information processing device 1 according to a first embodiment of the present invention. This information processing device 1 is a portable information processing device such as a smartphone, and has the function of searching a product database based on an image obtained by photographing a product. Hereinafter, the product will be a pharmaceutical package. The pharmaceutical package is an outer container or outer covering that further packages the primary packaging that comes into direct contact with the pharmaceutical for retail purposes, and its outer shape will be a rectangular parallelepiped. The side (plane) on which the name of the pharmaceutical (product name) is prominently displayed will be the front of the pharmaceutical package, and its shape will be rectangular. However, the present invention is not limited to pharmaceutical packages. It can be applied to objects such as products other than pharmaceutical packages.

[0016] Referring to Figure 1, the information processing device 1 consists of a camera 2, a communication I / F unit 3, an operation input unit 4, a screen display unit 5, a storage unit 6, and an arithmetic processing unit 7.

[0017] Camera 2 is a color or monochrome camera equipped with, for example, a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor with a pixel capacity of several million pixels. The communication I / F unit 3 consists of a data communication circuit and performs data communication with various external devices wirelessly or via wired connection. The operation input unit 4 consists of a device such as a keyboard or mouse and detects user operations on the information processing device 1 and outputs them to the arithmetic processing unit 7. The screen display unit 5 consists of a device such as an LCD (Liquid Crystal Display) and displays various information on the screen in accordance with instructions from the arithmetic processing unit 7.

[0018] The storage unit 6 consists of one or more storage devices such as a hard disk or memory, and stores processing information and programs 61 necessary for various processes in the arithmetic processing unit 7. The programs 61 are programs that realize various processing processes when read and executed by the arithmetic processing unit 7, and are pre-read from external devices or recording media (not shown) via data input / output functions such as the communication I / F unit 3 and stored in the storage unit 6. The main processing information stored in the storage unit 6 is the product database 62.

[0019] The product database 62 is a database that stores various information about products. Figure 2 shows an example of the information stored in the product database 62. In this example, the product database 62 consists of multiple product information entries 621. Each product information entry 621 consists of an ID 6211, a product name 6212, an image of a genuine product 6213, an image of a counterfeit product 6214, and explanatory information 6215.

[0020] The ID6211 field contains an ID, such as a number that uniquely identifies the product information 621. The product name 6212 field contains the product name of the genuine product. The image of the genuine product 6213 field contains an image of the genuine product taken from directly in front. The image of the counterfeit product 6214 field contains an image of the counterfeit product taken from directly in front. The explanatory information 6215 field contains a diagram and / or text that explains the key points for distinguishing between the image of the genuine product 6213 and the image of the counterfeit product 6214.

[0021] The arithmetic processing unit 7 has one or more processors such as a CPU (Central Processing Unit) and their peripheral circuits, and by reading and executing the program 61 from the storage unit 6, it realizes various processing functions by cooperating with the hardware and the program 61. The main processing functions realized by the arithmetic processing unit 7 are the target image acquisition unit 71, the reference image acquisition unit 72, the feature point extraction unit 73, the provisional correspondence point determination unit 74, the voting unit 75, the correct correspondence point determination unit 76, the similarity calculation unit 77, and the search result generation output unit 78.

[0022] The target image acquisition unit 71 is configured to acquire an image of the front of the product from camera 2 as the target image. For example, when a user takes a picture of the front of the product they intend to purchase with camera 2, the target image acquisition unit 71 acquires the image of the front of the product taken by camera 2 as the target image. The image taken by camera 2 may also include the side or background of the product in addition to the front. The target image acquisition unit 71 extracts the image of the front of the product as the target image from the image taken by camera 2 using any method.

[0023] The reference image acquisition unit 72 is configured to read all product information 621 from the product database 62 and acquire images 6213 of genuine products and images 6214 of counterfeit products from each piece of product information 21 as independent reference images. The reference image acquisition unit 72 uniquely manages each reference image by assigning an image number to the ID 6211 of the product information 21, with a branch number added to it. For example, if the ID 6211 of the product information 21 is "001", the image number "001-01" is assigned to the image 6213 of the genuine product included in that product information 21, and the image number "001-02" is assigned to the image 6214 of the counterfeit product.

[0024] The feature point extraction unit 73 is configured to extract multiple feature points, each represented by information on position, scale, direction, and local features, from both the target image acquired by the target image acquisition unit 71 and the reference image acquired by the reference image acquisition unit 72. For example, the feature point extraction unit 73 may extract multiple feature points, each represented by information on position, scale, direction, and local features, from both the target image and the reference image by processing with the SIFT (Scale-Invariant Feature Transform) algorithm. However, it is not limited to SIFT, and multiple feature points, each represented by information on position, scale, direction, and local features, may be extracted by processing with other algorithms such as ORB (Oriented FAST and Rotated BRIEF). The feature point extraction unit 73 assigns a feature point number to each feature point extracted from the target image and the reference image, and stores the information on position, scale, direction, and local features in association with the feature point number. For example, the feature point extraction unit 73 uniquely manages each feature point by assigning a number obtained by adding a branch number to the image number mentioned above as the feature point number. For example, multiple feature points extracted from the image with image number "001-01" are assigned feature point numbers such as "001-01-001".

[0025] The provisional correspondence point determination unit 74 is configured to determine multiple provisional correspondence points for each pair of target image and reference image by comparing the local features of each feature point extracted by the feature point extraction unit 73 in both the target image and the reference image. For example, the provisional correspondence point determination unit 74 calculates the Euclidean distance between the local features of each feature point in the target image and all the feature points in the reference image, and detects the feature points with the smallest Euclidean distance as provisional correspondence points. However, the method for determining provisional correspondence points is not limited to the above, and any method may be used. The provisional correspondence point determination unit 74 stores a list of determined provisional correspondence points for each pair of target image and reference image. Provisional correspondence points are identified, for example, by pairs of feature point numbers. The list of provisional correspondence points is, for example, a list of pairs of feature point numbers.

[0026] The voting unit 75 is configured to perform a voting process for each pair of target image and reference image to generate a two-dimensional histogram.

[0027] Figure 3 is a schematic diagram showing an example of a two-dimensional histogram generated by the voting unit 75. In this example, the two-dimensional histogram assigns multiple intervals related to the scale magnification on the vertical axis and multiple intervals related to the rotation angle of direction on the horizontal axis. Here, the scale magnification is, for example, the ratio of the scale of the feature points of the target image to the scale of the feature points of the reference image that constitute the provisional correspondence points. The rotation angle is, for example, the angle obtained by subtracting the direction of the feature points of the reference image from the direction of the feature points of the target image that constitute the provisional correspondence points. Furthermore, a range of values ​​is defined for each of the multiple intervals assigned to the vertical and horizontal axes. For example, the minimum and maximum values ​​of the scale magnification are Smin and Smax, the width of the scale interval is h1, the minimum and maximum values ​​of the rotation angle are Rmin and Rmax, and the width of the rotation angle is h2. In this case, for example, the ranges of the multiple intervals assigned to the vertical axis are, in order from closest to the minimum value Smin, the range of interval 11 = Smin or greater, less than Smin+h1, interval 12 = Smin+h1 or greater, less than Smin+2h1, ... Also, for example, the ranges of the multiple intervals assigned to the horizontal axis are, in order from closest to the minimum value Rmin, the range of interval 21 = Rmin or greater, less than Rmin+h2, interval 22 = Rmin+h2 or greater, less than Rmin+2h2, ...

[0028] Furthermore, the two-dimensional histogram shown in Figure 3 has multiple intersections identified by the intervals assigned to the vertical axis and the intervals assigned to the horizontal axis. For example, if the total number of intervals assigned to the vertical axis is k1 and the total number of intervals assigned to the horizontal axis is k2, then there are a total of k1 × k2 intersections. Each intersection has a counter that counts the vote value and a memory area that stores information (feature point number pairs) that identifies the provisional corresponding point voted for at that intersection. The initial values ​​of the counter and the memory area are NULL.

[0029] The method for determining the total number of intervals k1 and k2, and the interval widths h1 and h2, is arbitrary. Generally, methods for determining the number and width of histogram intervals are known, such as Sturges' formula, Scott's choice, and Friedman's choice. These known methods or improved versions thereof may be used.

[0030] The voting unit 75 performs the following voting process on the 2D histogram for each pair of target image and reference image. First, the voting unit 75 focuses on one of the provisional correspondence points determined by the provisional correspondence point determination unit 74 for the pair. Next, the voting unit 75 determines the intersection of the interval relating to the scale multiplier, in which the value of the scale multiplier between feature points is included in the value range, and the interval relating to the rotation angle, in which the value of the rotation angle between feature points is included in the value range, for the provisional correspondence point under consideration, and votes for that intersection. That is, it increments a counter corresponding to the intersection and stores the information of the provisional correspondence point (a pair of feature point numbers) in the memory area corresponding to the intersection. Next, the voting unit 75 shifts its attention to the next provisional correspondence point and performs the same process as above. Once the voting unit 75 has finished considering all the provisional correspondence points determined by the provisional correspondence point determination unit 74 for the pair, it saves the generated 2D histogram associated with that pair. The voting unit 75 repeats the same process for other pairs. This generates a 2D histogram for each pair of target and reference images.

[0031] The correct correspondence point determination unit 76 is configured to determine the correct correspondence point for each 2D histogram generated by the voting unit 75, based on the intersection with the maximum vote value. For example, the correct correspondence point determination unit 76 compares the counter values ​​at each intersection of the 2D histogram and determines the intersection with the maximum counter value. Next, the correct correspondence point determination unit 76 determines the set of provisional correspondence points, identified by a list of feature point numbers stored in the memory area of ​​the determined intersection, as the set of correct correspondence points. The reason for this determination is that when the front of the same product is shown in the target image and the reference image, the scale magnification and rotation angle of the correspondence points are often similar.

[0032] The similarity calculation unit 77 is configured to calculate the similarity of a pair of target images and reference images by determining the number of provisional correspondence points included in the set of correct correspondence points determined by the correct correspondence point determination unit 76 from the two-dimensional histogram generated for each pair of target images and reference images.

[0033] The search result generation output unit 78 is configured to select the highest similarity from the similarity scores of each pair of target image and reference image calculated by the similarity calculation unit 77, and to determine the reference image having this selected similarity as the reference image similar to the target image. Alternatively, the search result generation output unit 78 may determine the reference image having a similarity score of above a threshold from the similarity scores of each pair of target image and reference image calculated by the similarity calculation unit 77 as the reference image similar to the target image. The search result generation output unit 78 also reads product information 621 from the product database 62, which includes this determined reference image as an image 6213 of a genuine product or an image 6214 of a counterfeit product. The search result generation output unit 78 also displays the read product information 621 as a search result on the screen display unit 5, and / or transmits it to an external device via the communication I / F unit 3.

[0034] Figure 4 is a flowchart illustrating an example of the operation of the information processing device 1. The operation of the information processing device 1 will be explained below with reference to Figure 4.

[0035] First, the target image acquisition unit 71 acquires an image of the front of the product from the camera 2 as the target image (step S1). Next, the reference image acquisition unit 72 reads all product information 621 from the product database 62 and acquires an image 6213 of a genuine product and an image 6214 of a counterfeit product from each piece of product information 621 as reference images (step S2). Next, the feature point extraction unit 73 extracts multiple feature points from the target image acquired by the target image acquisition unit 71 and the reference image acquired by the reference image acquisition unit 72, each represented by information on position, scale, direction, and local features (step S3).

[0036] Next, the provisional correspondence point determination unit 74 compares the local features of each feature point in the target image and the reference image extracted by the feature point extraction unit 73 for each pair of target image and reference image to determine multiple provisional correspondence points (step S4). Next, the voting unit 75 performs a voting process for each pair of target image and reference image to generate a two-dimensional histogram (step S5). In the above voting process, the voting unit 75 votes for each provisional correspondence point at the intersection point corresponding to the combination of the scale multiplier value and the direction rotation angle value between the feature points, which corresponds to the combination of the intervals relating to the scale multiplier and the intervals relating to the direction rotation angle. This generates a two-dimensional histogram representing the vote values ​​for the intersection points of multiple intervals relating to the scale multiplier and multiple intervals relating to the direction rotation angle.

[0037] Next, the correct correspondence point determination unit 76 determines the correct correspondence point for each 2D histogram generated by the voting unit 75 based on the intersection with the maximum vote value (step S6). Next, the similarity calculation unit 77 calculates the similarity of the target image and reference image pair by counting the number of provisional correspondence points included in the set of correct correspondence points determined by the correct correspondence point determination unit 76 from the 2D histogram generated for each target image and reference image pair (step S7).

[0038] Next, the search result generation output unit 78 generates and outputs the search results (step S8). In generating the search results, the search result generation output unit 78 first selects the highest similarity from the similarity of each pair of target image and reference image calculated by the similarity calculation unit 77, and determines the reference image with this selected similarity to the target image. Next, the search result generation output unit 78 reads product information 621 from the product database 62, which includes this determined reference image as an image 6213 of a genuine product or an image 6214 of a counterfeit product. In outputting the search results, the search result generation output unit 78 displays the read product information 621 on the screen display unit 5 and / or transmits it to an external device via the communication I / F unit 3.

[0039] As explained above, the information processing device 1 is equipped with a voting unit 75 and a correct correspondence point determination unit 76 configured and operating as described above, so that incorrect correspondence points can be removed with a small amount of information processing, and consequently the amount of information processing required for image matching by correspondence point search can be reduced.

[0040] [Second Embodiment] Next, a second embodiment of the present invention will be described. The information processing device according to the second embodiment (hereinafter referred to as information processing device 1A) differs from the information processing device 1 according to the first embodiment in that the function of the voting unit 75 is different, and otherwise it is the same as the first embodiment.

[0041] In this embodiment, the voting unit 75 is configured to calculate the degree of non-rigid deformation, which represents the degree of non-rigid deformation of the front of the product shown in the target image acquired by the target image acquisition unit 71. The front of a real product is rectangular in shape. However, the front of the product in the image obtained by taking a picture from an oblique angle with camera 2 undergoes non-rigid deformation and is no longer rectangular. Compared to a rectangular target image, a non-rectangular target image shows a decrease in the similarity of the scale magnification and rotation angle of the corresponding points when the same product is shown in the reference image. This decrease becomes more pronounced as the degree of non-rigid deformation increases. In other words, the closer the surface of the product shown in the target image is to a rectangle, the higher the similarity of the scale magnification and rotation angle of the corresponding points when the same product is shown in the reference image.

[0042] Therefore, the voting unit 75 in this embodiment is configured to generate a type of two-dimensional histogram corresponding to the calculated non-rigid deformation degree through voting, from among multiple types of two-dimensional histograms in which the range of values ​​of multiple intervals relating to the scaling factor and the range of values ​​of multiple intervals relating to the rotation angle of the direction differ.

[0043] Figure 5 shows an example of a method for calculating the degree of non-rigid deformation. The rectangle shown in Figure 5 is composed of four sides L1 to L4, with sides L1 and L2 facing each other, and sides L3 and L4 facing each other. The voting unit 75 extracts the four sides that make up the front of the product shown in the target image, calculates the ratio of the longer side to the shorter side for each pair of opposing sides, and uses the larger of these ratios as the degree of non-rigid deformation of the front. For example, in the rectangle shown in Figure 5, if L1 / L2 = 1.3 and L3 / L4 = 1.1, the degree of non-rigid deformation is set to 1.3. However, the method for calculating the degree of non-rigid deformation is not limited to the above. Any index that indicates the degree of non-rectangularity may be used as the degree of non-rigid deformation.

[0044] Figure 6 is a schematic diagram showing another example of a two-dimensional histogram generated by the voting unit 75. The two-dimensional histogram shown in Figure 6 differs from the two-dimensional histogram shown in Figure 3 in that the width of the scale interval is h3, which is smaller than h1, and the width of the rotation angle is h4, which is smaller than h2. The minimum value Smin and maximum value Smax of the scale magnification, and the minimum value Rmin and maximum value Rmax of the rotation angle are the same as in Figure 3. Therefore, the two-dimensional histogram shown in Figure 6 has a larger total number of intervals on the vertical and horizontal axes compared to the two-dimensional histogram shown in Figure 3.

[0045] Next, the operation of this embodiment will be explained with reference to the flowchart in Figure 7.

[0046] The processes from step S1 to step S4 are the same as those shown in Figure 4. In step S11, the voting unit 75 calculates the degree of non-rigid deformation, which represents the degree of non-rigid deformation of the front of the product shown in the target image acquired by the target image acquisition unit 71. Next, in step S5, the voting unit 75 generates the two-dimensional histogram shown in Figure 6, which has a narrower interval, by voting, if the calculated degree of non-rigid deformation is less than the threshold T. On the other hand, if the calculated degree of non-rigid deformation is greater than or equal to the threshold T, the voting unit 75 generates the two-dimensional histogram shown in Figure 3, which has a wider interval, by voting in step S5. The processes from step S6 to step S8 thereafter are the same as in the first embodiment.

[0047] In the above explanation, we used two types of two-dimensional histograms, each with different ranges for multiple intervals related to the scaling factor and multiple intervals related to the rotation angle of the direction. However, you may also use three or more types of two-dimensional histograms.

[0048] As explained above, according to this embodiment, from among multiple types of 2D histograms, each with different ranges of values ​​for multiple intervals related to scale magnification and multiple intervals related to rotation angles of direction, a type of 2D histogram corresponding to the degree of non-rigid deformation of the front of the product shown in the target image is generated by voting. Therefore, for target images obtained by a user taking a picture of the front of the product from directly in front, a 2D histogram with a narrow range of values ​​for the intervals can be generated by voting, and by making full use of the characteristic that the scale magnification and rotation angle of direction of corresponding points are often similar when the same product is shown in the target image and the reference image, the proportion of truly correct corresponding points in the set of correct corresponding points can be increased. This is because in a 2D histogram with a wide range of values ​​for the intervals, the probability of an incorrect corresponding point being voted for at the intersection with the highest vote value increases. On the other hand, for target images obtained by a user taking a picture of the front of the product from an oblique angle, a 2D histogram with a wide range of values ​​for the intervals can be generated by voting, and the probability of a correct corresponding point being excluded from the set of correct corresponding points can be reduced. Furthermore, if the same product is depicted in both the target image and the reference image, in a 2D histogram with a wide range of values, the probability of a mismatch being voted for at the intersection with the highest vote value increases. However, the probability of a correct match being excluded from the set of correct matches decreases even more significantly, thus preventing an extreme decrease in similarity accuracy.

[0049] [Third Embodiment] Next, a third embodiment of the present invention will be described. Figure 8 is a block diagram of the information processing apparatus 1B according to the third embodiment of the present invention, where the same reference numerals as in Figure 1 indicate the same parts, 79 is a rigid body transformation matrix calculation unit, and 80 is a target image correction unit.

[0050] The rigid body transformation matrix calculation unit 79 is configured to calculate a rigid body transformation matrix between the target image and the reference image based on the correct correspondence points determined by the correct correspondence point determination unit 76. Various methods have been proposed or put into practical use for calculating a rigid body transformation matrix between two images from multiple correspondence points. Any of these methods may be used.

[0051] The target image correction unit 80 is configured to apply a rigid body transformation to the target image using the rigid body transformation matrix calculated by the rigid body transformation matrix calculation unit 79.

[0052] Next, the operation of this embodiment will be explained with reference to the flowchart in Figure 9.

[0053] The processes from step S1 to step S7 are the same as the processes from step S1 to step S7 in Figure 4. Next, the search result generation output unit 78 selects the highest similarity from the similarity of each pair of target image and reference image calculated by the similarity calculation unit 77, and determines the reference image having this selected similarity as the reference image similar to the target image (step S21). Next, the rigid body transformation matrix calculation unit 79 calculates a rigid body transformation matrix between the target image and the reference image based on the positive correspondence points between the target image and the reference image determined in step S21 (step S22). Next, the target image correction unit 80 generates a corrected target image by applying a rigid body transformation to the target image using the rigid body transformation matrix calculated above (step S23). Next, the search result generation output unit 78 reads product information 621 from the product database 62 that includes the reference image determined in step S21 as an image 6213 of a genuine product or an image 6214 of a counterfeit product (step S24). Next, the search result generation output unit 78 displays the corrected target image as a monitor image of the target image captured by the camera, and the read product information 621 as a search result on the screen display unit 5, and / or transmits it to an external device via the communication I / F unit 3 (step S25).

[0054] As described above, this embodiment provides the same effects as the first embodiment, and also allows the user to monitor a corrected image of the target image captured by the camera 2. For example, the size and orientation of the target image captured by the user can be adjusted to match the reference image. As a result, the user can easily recognize the differences between the target image and an image of the actual product on the screen.

[0055] [Fourth Embodiment] Next, a fourth embodiment of the present invention will be described. Figure 10 is a block diagram of the information processing device 10 according to this embodiment. This embodiment will provide an overview of the information processing device described above.

[0056] As shown in Figure 10, the information processing device 10 in this embodiment consists of a target image acquisition unit 11, a reference image acquisition unit 12, a feature point extraction unit 13, a provisional correspondence point determination unit 14, a voting unit 15, and a correct correspondence point determination unit 16.

[0057] The target image acquisition unit 11 is configured to acquire an image obtained by photographing the surface of an object as the target image. The reference image acquisition unit 12 is configured to acquire an image obtained by photographing the surface of a reference object as the reference image. The feature point extraction unit 13 is configured to extract multiple feature points represented by information on position, scale, direction, and local features from both the target image and the reference image. The provisional correspondence point determination unit 14 is configured to determine multiple provisional correspondence points by comparing the local features of each feature point in the target image and the reference image. The voting unit 15 is configured to vote for each provisional correspondence point on the intersection point corresponding to the combination of the scale magnification value and the direction rotation angle value, from among multiple intersection points identified by combinations of multiple intervals related to the scale magnification value and multiple intervals related to the direction rotation angle, and to generate a histogram representing the voting values ​​for multiple intersection points of multiple intervals related to the scale magnification value and multiple intervals related to the direction rotation angle. The correct correspondence point determination unit 16 is configured to determine the correct correspondence point among the provisional correspondence points based on the generated histogram.

[0058] The information processing device 10 configured as described above operates as follows. First, the target image acquisition unit 11 acquires an image obtained by photographing the surface of an object as the target image. Next, the reference image acquisition unit 12 acquires an image obtained by photographing the surface of a reference object as the reference image. Next, the feature point extraction unit 13 extracts multiple feature points from the target image and the reference image, each represented by information on position, scale, direction, and local feature quantities. Next, the provisional correspondence point determination unit 14 determines multiple provisional correspondence points by comparing the local feature quantities of each feature point in the target image and the reference image. Next, the voting unit 15 votes for each provisional correspondence point on the intersection corresponding to the combination of the scale magnification value and the direction rotation angle value among multiple intersection points identified by the combination of multiple intervals related to the scale magnification value and multiple intervals related to the direction rotation angle, and generates a histogram representing the voting values ​​for the multiple intersection points of the multiple intervals related to the scale magnification value and multiple intervals related to the direction rotation angle. Next, the correct correspondence point determination unit 16 determines the correct correspondence point among the provisional correspondence points based on the generated histogram.

[0059] As described above, the information processing device 10, which is configured and operates as described, includes a voting unit 15 and a correct correspondence point determination unit 16, which are configured and operate as described above, and therefore can remove incorrect correspondence points with a small amount of information processing.

[0060] Although the present invention has been described above with reference to the embodiments described above, the present invention is not limited to the embodiments described above. Various modifications to the configuration and details of the present invention can be made within the scope of the present invention as can be understood by those skilled in the art.

[0061] For example, in the above, the voting unit voted for one intersection for each provisional correspondence point, but it may also vote for at least one intersection of the intersection corresponding to a combination of the interval corresponding to the scale magnification value and the interval corresponding to the rotation angle value of the direction, and up to eight surrounding intersections that are directly touching it. Also, in the above, the correct correspondence point determination unit calculated the similarity based on the maximum vote value, but it may also calculate the similarity based on a vote value above a threshold.

[0062] Furthermore, instead of the CPU mentioned above, the information processing device may use a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof. [Industrial applicability]

[0063] This invention can be used for a wide range of image processing applications, including image matching using local features. [Explanation of Symbols]

[0064] 1. Information Processing Device 2 cameras 3. Communication I / F section 4. Operation Input Section 5 Screen display section 6 Memory section 7. Arithmetic Processing Unit 10 Information Processing Devices 11 Target image acquisition unit 12 Reference Image Acquisition Unit 13 Feature point extraction unit 14. Provisional Correspondence Point Determination Section 15 Voting Department 16 Correct Corresponding Point Determination Unit

Claims

1. A target image acquisition means that acquires an image obtained by photographing the surface of an object as the target image, A reference image acquisition means that acquires an image obtained by photographing the surface of a reference object as a reference image, A feature point extraction means for extracting multiple feature points from the aforementioned target image and the aforementioned reference image, each of which is represented by information on position, scale, direction, and local feature quantity. A provisional correspondence point determination means that determines a plurality of provisional correspondence points by comparing the local feature quantities of each feature point of the target image and the reference image, Voting means for each of the aforementioned provisional correspondence points, which votes for a combination of the scale magnification value and the rotation angle value between feature points, to an intersection point among a plurality of intersection points identified by a combination of a plurality of intervals relating to the scale magnification value and a plurality of intervals relating to the rotation angle in the direction, and which corresponds to the intersection point corresponding to the combination of the interval relating to the scale magnification value and the interval relating to the rotation angle in the direction, and generates a histogram representing the vote values ​​for the plurality of intersection points of the plurality of intervals relating to the scale magnification value and the plurality of intervals relating to the rotation angle in the direction, Based on the generated histogram, a correct correspondence point determination means determines the correct correspondence point among the provisional correspondence points, Includes, The aforementioned voting method is From the aforementioned target image, the degree of non-rigid deformation representing the degree of non-rigid deformation of the surface is calculated, From among multiple types of histograms, each having different ranges of values ​​for multiple intervals relating to the scaling factor and multiple intervals relating to the rotation angle in the direction, a type of histogram corresponding to the calculated non-rigid deformation is generated by the vote. Information processing device.

2. The correct correspondence point determination means determines the correct correspondence point among the provisional correspondence points based on the intersection of the plurality of intervals where the voting value is maximum. The information processing apparatus according to claim 1.

3. The system further includes a similarity calculation means for calculating the similarity between the target image and the reference image based on the number of correct corresponding points determined above. The information processing apparatus according to claim 1 or 2.

4. The system further includes rigid transformation matrix calculation means for calculating a rigid transformation matrix between the target image and the reference image based on the determined correct correspondence points, The information processing apparatus according to claim 1 or 2.

5. The system further includes a target image correction means that applies a rigid body transformation to the target image using the rigid body transformation matrix calculated above. The information processing apparatus according to claim 4.

6. A computer, The image obtained by photographing the surface of an object is acquired as the target image. The image obtained by photographing the surface of the reference object is acquired as the reference image. Multiple feature points, represented by information on position, scale, direction, and local features, are extracted from the aforementioned target image and the aforementioned reference image. Multiple provisional correspondence points are determined by comparing the local feature quantities of each feature point in the target image and the reference image. For each of the aforementioned provisional correspondence points, the combination of the scale magnification value and the rotation angle value between feature points is voted for among multiple intersections identified by the combination of multiple intervals relating to the scale magnification value and multiple intervals relating to the rotation angle in the direction, and a histogram is generated representing the vote values ​​for the multiple intersections of the multiple intervals relating to the scale magnification value and multiple intervals relating to the rotation angle in the direction. Based on the generated histogram, the correct corresponding point among the provisional corresponding points is determined. In the aforementioned vote, From the aforementioned target image, the degree of non-rigid deformation representing the degree of non-rigid deformation of the surface is calculated, From among multiple types of histograms, each having different ranges of values ​​for multiple intervals relating to the scaling factor and multiple intervals relating to the rotation angle in the direction, a type of histogram corresponding to the calculated non-rigid deformation is generated by the vote. Information processing methods.

7. The computer In determining the correct corresponding point, the correct corresponding point among the provisional corresponding points is determined based on the intersection of the multiple intervals where the voting value is maximum. The information processing method according to claim 6.

8. On the computer, The process involves capturing an image of the surface of an object and acquiring it as the target image, The process involves capturing an image of the surface of a reference object and acquiring it as the reference image, A process to extract multiple feature points from the aforementioned target image and the aforementioned reference image, each represented by information on position, scale, direction, and local feature quantities, A process to determine multiple provisional correspondence points by comparing the local feature quantities of each feature point in the target image and the reference image, For each of the aforementioned provisional correspondence points, the combination of the scale magnification value and the rotation angle value between feature points is voted for among multiple intersections identified by the combination of multiple intervals relating to the scale magnification value and multiple intervals relating to the rotation angle in the direction, and a histogram is generated representing the vote values ​​for the multiple intersections of the multiple intervals relating to the scale magnification value and multiple intervals relating to the rotation angle in the direction. Based on the generated histogram, a process is performed to determine the correct corresponding point among the provisional corresponding points. Have them do it, In the aforementioned vote, From the aforementioned target image, the degree of non-rigid deformation representing the degree of non-rigid deformation of the surface is calculated, A program for generating, by voting, a type of histogram corresponding to the calculated non-rigid deformation, from among multiple types of histograms, each having different ranges of values ​​for multiple intervals relating to the scaling factor and multiple intervals relating to the rotation angle in the direction.

Images