Information processing device, information processing method, and program

Abstract

In generating a plane projection transformation matrix using two pairs of corresponding points, the reduction in the accuracy of the plane projection transformation matrix is ​​suppressed. [Solution] The information processing device 10 includes: a corresponding point detection unit 11 that extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the images from among the extracted feature points; a corresponding figure detection unit 12 that, for each pair of feature points, uses the position of each feature point constituting the pair and either the scale information or the angle information to detect a pair of figures derived from each feature point that corresponds between the images; and a transformation matrix generation unit 13 that uses three or more pairs of figures to generate a planar projection transformation matrix for transforming one image into the other.

Description

[Technical Field]

[0001] This disclosure relates to an image processing apparatus and an image processing method for generating a planar projection transformation matrix, and more particularly to a program for implementing these. [Background technology]

[0002] The transformation that connects multiple images obtained by photographing a plane from different angles is called a plane projection transformation. A plane projection transformation is represented by a 3x3 matrix and is a transformation that turns one quadrilateral into another. For this reason, it is known that if four different points on a plane can be observed in an image, a plane projection transformation matrix can be generated. Patent Document 1 discloses a technique for detecting four or more pairs of corresponding feature points (hereinafter simply referred to as corresponding points) between two images and applying the DLT (Direct Linear Transform) method to the detected pairs of corresponding points to create a frontal view of an overhead image of a sports scene.

[0003] Furthermore, Patent Document 2 discloses a technique for generating a planar projection transformation matrix using two or more pairs of corresponding points by using feature points that calculate scale and angle in addition to image coordinates. Such feature points are extracted by calculating SIFT (Scale Invariant Feature Transform) features or SURF (Speeded-Up Robust Features) features from an image.

[0004] According to the technology disclosed in Patent Document 2, compared to the technology disclosed in Patent Document 1, the minimum number of corresponding point pairs used for generation can be reduced from 4 to 4, which is expected to reduce the number of trials in combinatorial optimization such as RANSAC (Random Sample Consensus). [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent No. 7334058 [Patent Document 2] Japanese Patent No. 7448034 [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] However, the method for generating a planar projective transformation matrix using two sets of corresponding point pairs disclosed in Patent Document 2 has a problem of low accuracy of the planar projective transformation matrix. That is, at the feature points calculated by SIFT or SURF, the coordinates, scale, and angle each include an estimation error. However, in the technique disclosed in Patent Document 2, the respective error distributions are treated as the same, so there arises a problem of low accuracy of the planar projective transformation matrix.

[0007] For example, the scale and coordinates of the feature points extracted by SIFT feature amount are estimated in sub-pixel units by approximating the DoG (Difference of Gaussian) value around the feature points as a quadratic surface and obtaining the extreme value. On the other hand, the angle of the feature point is determined as the discrete maximum value of a histogram with a fixed number of bins. Thus, since the position, scale, and angle of the feature points are obtained by different methods, they have different error distributions. Therefore, assuming the same error distribution, the accuracy of the generated planar projective transformation matrix will decrease. As a result, even though two sets of corresponding point pairs are used, the number of trials of RANSAC is not reduced as per the theoretical value.

[0008] An example of the object of the present disclosure is to suppress a decrease in the accuracy of a planar projective transformation matrix in the generation of a planar projective transformation matrix using two sets of corresponding point pairs. [Means for Solving the Problems]

[0009] To achieve the above object, an information processing apparatus in one aspect of the present disclosure is A corresponding point detection unit extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection unit detects a pair of shapes derived from each feature point that correspond between images, using the position of each feature point constituting the pair and either the scale information or the angle information. A transformation matrix generation unit generates a planar projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images. It is characterized by having the following features.

[0010] Furthermore, in order to achieve the above objectives, the information processing method in one aspect of this disclosure is: A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection step is performed, in which, using the position of each feature point constituting the pair and either the scale information or the angle information, a corresponding pair of shapes derived from each feature point is detected between images. A transformation matrix generation step, which involves generating a planar projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images, It is characterized by having the following:

[0011] Furthermore, in order to achieve the above objectives, the program in one aspect of this disclosure is On the computer, A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection step is performed, in which, using the position of each feature point constituting the pair and either the scale information or the angle information, a corresponding pair of shapes derived from each feature point is detected between images. A transformation matrix generation step, which involves generating a planar projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images, It is characterized by causing the execution of [the specified action]. [Effects of the Invention]

[0012] As described above, according to this disclosure, when generating a plane projection transformation matrix using two pairs of corresponding points, it is possible to suppress a decrease in the accuracy of the plane projection transformation matrix. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 shows an example of corresponding feature point pairs between images. [Figure 2] Figure 2 is a schematic diagram showing the general configuration of an example of an information processing device. [Figure 3] Figure 3 is a configuration diagram that specifically shows the configuration of the first example of an information processing device. [Figure 4] Figure 4 shows an example of corresponding point pairs and corresponding figure pairs. [Figure 5] Figure 5 is a flowchart showing a first example of the operation of an information processing device. [Figure 6] Figure 6 shows an example of a pair of corresponding regions between images. [Figure 7] Figure 7 is a flowchart showing a second example of the operation of the information processing device. [Figure 8] Figure 8 is a block diagram showing an example of a computer that implements an information processing device. [Modes for carrying out the invention]

[0014] (Prerequisite explanation) First, the prerequisites for this disclosure will be explained using Figure 1. Figure 1 is a diagram showing an example of corresponding feature point pairs between images. In the example in Figure 1, Image 1 and Image 2 are obtained by photographing the same object from different directions.

[0015] Specifically, as shown in Figure 1, a point on a plane is photographed from different angles. This allows for the observation of feature point m in image 1 and feature point m' in image 2. Feature points m and m' are corresponding point pairs, representing points on the same plane observed in different images. Hereafter, these corresponding feature point pairs will also be referred to as "corresponding point pairs."

[0016] Furthermore, such feature points can be extracted from image data by calculating, for example, SIFT (Scale Invariant Feature Transform) features or SURF (Speeded-Up Robust Features) features. Feature points extracted from such features possess scale information and angular information.

[0017] Scale information indicates the region where the feature quantity lies within a set range, centered on the feature point. In the example in Figure 1, the scale information is represented by a circle with radius r centered on the feature point. Angle information indicates the direction of the feature point, obtained from the histogram of the luminance gradient direction in the region centered on the feature point. In the example in Figure 1, this is represented by θ.

[0018] As shown in Figure 1, feature point m has scale information and angle information, namely scale r and angle θ. Feature point m' corresponding to feature point m has scale information and angle information, namely scale r' and angle θ'. Line l in Image 1 i Line l' in image 2 is a straight line passing through feature point m with a slope θ'. Similarly, line l' in image 2 is a straight line passing through feature point m' with a slope θ'.

[0019] (Embodiment 1) The information processing device, information processing method, and program in Embodiment 1 will be described below with reference to Figures 2 to 5.

[0020] [Device configuration] First, we will explain the schematic configuration of an example of an information processing device using Figure 2. Figure 2 is a configuration diagram showing the schematic configuration of an example of an information processing device.

[0021] The information processing device 10 shown in Figure 2 is a device for generating a plane projection transformation matrix between images; in other words, it is a plane projection transformation matrix generation device. As shown in Figure 2, the information processing device comprises a corresponding point detection unit 11, a corresponding shape detection unit 12, and a transformation matrix generation unit 13.

[0022] The corresponding point detection unit 11 extracts feature points containing scale information and angle information from each of two images obtained by photographing the same object from different directions. Then, from the feature points extracted from each of the two images, the corresponding point detection unit 11 detects pairs of feature points that correspond between the two images (corresponding point pairs).

[0023] The corresponding shape detection unit 12 detects corresponding pairs of shapes derived from each feature point in each corresponding pair, using the position of each feature point constituting the corresponding pair and at least one of the scale information and angle information between images.

[0024] The transformation matrix generation unit 13 generates a planar projection transformation matrix for transforming one image into the other, using three or more pairs of corresponding shapes between the images.

[0025] Thus, the information processing device 10 generates a plane projection transformation matrix using not only corresponding point pairs but also corresponding figure pairs between images (hereinafter referred to as "corresponding figure pairs"). Therefore, the information processing device 10 can suppress a decrease in the accuracy of the plane projection transformation matrix when generating a plane projection transformation matrix using two sets of corresponding point pairs.

[0026] Next, the configuration and functions of the information processing device 10 will be explained in more detail using Figures 3 and 4. Figure 3 is a configuration diagram specifically showing the configuration of the first example of the information processing device. Figure 4 is a diagram showing an example of corresponding point pairs and corresponding figure pairs.

[0027] As shown in Figure 3, the information processing device 10 includes, in addition to the corresponding point detection unit 11, corresponding shape detection unit 12, and transformation matrix generation unit 13 described above, a data acquisition unit 14, a storage unit 15, and an output unit 16.

[0028] The data acquisition unit 14 acquires image data of the same object taken from different directions using external devices such as server devices, terminal devices, and imaging devices, and stores the acquired image data in the storage unit 15.

[0029] In this embodiment, the corresponding point detection unit 11 first retrieves image data of two images (image 1 and image 2) from the storage unit 15, and then extracts feature points from each of the two retrieved image data. Specifically, the corresponding point detection unit 11 extracts feature points by calculating, for example, SIFT (Scale Invariant Feature Transform) features or SURF (Speeded-Up Robust Features) features in each image.

[0030] The corresponding point detection unit 11 then detects three or more pairs of corresponding points by applying existing feature point matching techniques to the extracted feature points. Each extracted feature point has scale information indicated by the radius r and angle information indicated by the angle θ, as described above.

[0031] In this embodiment, the corresponding shape detection unit 12 identifies a virtual line passing through each feature point using the position and angle information of each feature point constituting the corresponding point pair, for each of the three or more extracted corresponding point pairs, and detects the identified virtual line for each feature point as a corresponding shape pair.

[0032] Specifically, as shown in Figures 1 and 4, the corresponding point pair is feature point mi and the feature point m' i Assuming that it is so, the corresponding figure detection unit 12 determines the line l which is the line with the inclination θ passing through the feature point m i with the inclination θ passing through the feature point m i and specifies the line l which is a straight line with the inclination θ passing through the feature point m i and the feature point m' i and specifies the line l' which is a straight line with the inclination θ' passing through the feature point m' i and specifies the line l' which is a straight line with the inclination θ' passing through the feature point m' i and sets the specified line l i and the line l' i as a corresponding figure pair. Three or more corresponding figure pairs are also detected according to the number of sets of corresponding point pairs.

[0033] The transformation matrix generation unit 13 generates a planar projective transformation matrix using the three or more detected corresponding point pairs and the three or more corresponding figure pairs as well.

[0034] The output unit 16 outputs the planar projective transformation matrix generated by the transformation matrix generation unit 13. Examples of the output destination include an image processing device. Note that the information processing device 10 may constitute a part of the image processing device. Examples of the image processing device include a device that generates three-dimensional data of an object from a plurality of images of the object using the planar projective transformation matrix.

[0035] [Device Operation] Next, the operation of the information processing device 10 will be described with reference to FIG. 5. FIG. 5 is a flowchart showing a first example of the operation of the information processing device. In the following description, FIGS. 1 to 3 will be appropriately referred to. Also, in the first embodiment, an information processing method is implemented by operating the information processing device 10. Therefore, the description of the information processing method in the first embodiment will be replaced with the following description of the operation of the information processing device 10.

[0036] As shown in FIG. 5, first, the data acquisition unit 14 acquires image data of images obtained by photographing the same object from different directions from an external device (step S1). The data acquisition unit 14 stores each acquired image data in the storage unit 15.

[0037] Next, the correspondence point detection unit 11 acquires image data of any two images (image 1 and image 2) from the storage unit 15, performs feature point matching on each image of the acquired image data, and detects three or more pairs of correspondence points (step S2).

[0038] Next, the corresponding shape detection unit 12 detects a corresponding pair of lines as a corresponding shape pair for each corresponding point pair detected in step S2, using the position and angle information of each feature point constituting that corresponding point pair (step S3).

[0039] Specifically, in step S3, the corresponding shape detection unit 12 identifies a virtual line passing through each feature point using the position and angle information of each feature point constituting the corresponding point pair, and detects the identified pair of virtual lines as a corresponding shape pair.

[0040] Next, the transformation matrix generation unit 13 generates a plane projection transformation matrix using the three or more pairs of corresponding points detected in step S2 and the pairs of corresponding figures detected in step S3 (step S4).

[0041] Subsequently, the output unit 16 outputs the plane projection transformation matrix generated in step S4 (step S5).

[0042] Furthermore, if it is necessary to generate a planar projection transformation matrix between two other images, steps S2 to S5 are executed again. In this case, in step S2, the corresponding point detection unit 11 acquires image data for two other combinations of images.

[0043] Furthermore, in step S4, the transformation matrix generation unit 13 can generate the plane projection transformation matrix multiple times while changing the corresponding figure pairs used. The transformation matrix generation unit 13 then evaluates the accuracy of the multiple generated plane projection transformation matrices and selects the plane projection transformation matrix with the highest accuracy based on the evaluation results.

[0044] Another method for selecting a highly accurate plane projection transformation matrix is, for example, as used in RANSAC, to apply the generated plane projection transformation matrix to the corresponding point pairs that were not used in the generation, measure the reprojection error between the corresponding points, and select the plane projection transformation matrix that minimizes the sum of the measured reprojection errors. In this method, instead of the reprojection error, a first-order approximation of the reprojection error, such as the Sampson Error or algebraic error, may be used.

[0045] [Specific example] Here, a specific example of processing in the information processing device 10 will be described below. In the following example, the coordinates of feature points are represented by a 3x1 homogeneous representation. For example, the coordinates of feature point m and its corresponding feature point m' are given by m=[u,v,1] T m'=[u',v',1] T It is represented by ]. Furthermore, the plane projection transformation matrix is ​​represented by a 3x3 matrix H. Although matrix H has 9 elements, it has 8 degrees of freedom due to scale indetermination.

[0046] Specifically, the plane projection transformation between a corresponding pair of feature points, feature point m and feature point m', is represented by equation 1 below. In equation 1 below, "~" indicates that both sides are equal to a constant multiple.

[0047]

number

[0048] Next, we will explain the corresponding figure pairs. As mentioned above, feature point m has angle information as angle information θ. Therefore, the line l passing through feature point m is represented by the following equation 2.

[0049]

number

[0050] Based on the above equations 1 and 2, the planar projection transformation between line l and its corresponding line l' is expressed by the following equation 3.

[0051]

number

[0052] Therefore, given N pairs of corresponding points and M pairs of corresponding figures, the matrix H can be estimated by solving the optimization problem represented by the following equation 4. In the following equation 4, "×" represents the cross product of 3D vectors.

[0053]

number

[0054] From the above number 4, there is one pair of corresponding points "m' i ×Hm i Two constraints can be obtained from this. Also, as shown in Figure 4, one pair of corresponding figures (a pair of corresponding lines) is defined for each pair of corresponding points. Line l i The feature point m is always i Because it passes through, one pair of corresponding lines "l j ×HTl' j From this, we can derive one constraint condition. Therefore, in order to solve equation 4 above, equation 5 below must be satisfied.

[0055]

number

[0056] The smallest configuration that satisfies the above equation 5 is N=3, M=2. That is, by using at least 3 pairs of corresponding points and 2 of the 3 pairs of corresponding figures defined by the 3 pairs of corresponding points, we can solve the above equation 5 and generate the matrix H. Of course, we can also use all 3 pairs of corresponding lines. Note that the above equation 4 can be reduced to the DLT (Direct Linear Transformation) method.

[0057] Here, we will explain steps S2 to S4 shown in Figure 5 in detail, step by step, following the specific examples described above.

[0058] [Step S2] In step S2, the corresponding point detection unit 11 acquires image data of image 1 and image data of image 2. Then, the corresponding point detection unit 11 performs feature point matching on image 1 and image 2 to find three or more corresponding point pairs {m i ,m' i Detects values ​​i∈{1,···,N},N≧3}. The subscript "i" indicates the corresponding number.

[0059] [Step S3] In step S3, the corresponding shape detection unit 12 detects the corresponding point pair m i ,m' i For each of these, feature points m are determined based on the above number 2. i Line l passing through i and point m' i Line l' passing through i Calculate and the corresponding shape pairs of M groups {l i ,l' i Detect i∈{1,···,N},M=N}.

[0060] [Step S4] In step S4, the transformation matrix generation unit 13 uses the detected M pairs of corresponding points (M≧3) and N pairs of corresponding figures (N≧2) to generate a plane projection transformation matrix based on the above equation 4.

[0061] [Effects in Embodiment 1] As described above, according to Embodiment 1, a plane projection transformation matrix can be generated simply by using at least three pairs of corresponding points obtained by feature point matching. The reason is as follows.

[0062] From the three pairs of corresponding points, six constraint conditions can be obtained as shown in equation 1 above. Furthermore, by using two of the pairs of corresponding figures associated with the three pairs of corresponding points, two constraint conditions can be obtained as shown in equation 2 above. Therefore, the matrix H with 8 degrees of freedom can be calculated based on equation 4 above, and the above-mentioned effect can be obtained.

[0063] Furthermore, according to Embodiment 1, the accuracy in generating the plane projection transformation matrix is ​​higher than that of the method disclosed in Patent Document 2. This is because the plane projection transformation matrix is ​​generated using only the positions of corresponding point pairs and angle information, without using scale information and angle information with different error distributions in combination.

[0064] [program] In Embodiment 1, the program can be any program that causes the computer to execute steps S1 to S5 shown in Figure 5. By installing and executing this program on the computer, the information processing device 10 and the information processing method can be realized. In this case, the computer's processor functions as a corresponding point detection unit 11, a corresponding figure detection unit 12, a transformation matrix generation unit 13, a data acquisition unit 14, and an output unit 16, and performs the processing.

[0065] Furthermore, the memory unit 15 may be implemented by a storage device such as a hard disk installed in the computer, or by a storage device of another computer. Examples of computers include general-purpose PCs, server computers, smartphones, and tablet devices.

[0066] Furthermore, in Embodiment 1, the program may be executed by a computer system (such as a cloud system) constructed by multiple computers. In this case, for example, each computer may function as one of the following: the corresponding point detection unit 11, the corresponding shape detection unit 12, the transformation matrix generation unit 13, the data acquisition unit 14, and the output unit 16.

[0067] (Embodiment 2) Next, the information processing device, information processing method, and program in Embodiment 2 will be described with reference to Figures 6 and 7.

[0068] [Device configuration] First, the configuration of the information processing device will be described. In Embodiment 2, the information processing device has the same configuration as the information processing device 10 shown in Figure 2. In the following description, Figure 2 and the reference numerals shown in Figure 2 will be used for reference.

[0069] In Embodiment 2, the information processing device also includes a corresponding point detection unit 11, a corresponding shape detection unit 12, a transformation matrix generation unit 13, a data acquisition unit 14, a storage unit 15, and an output unit 16, similar to the example shown in Figure 2. However, in Embodiment 2, the function of the corresponding shape detection unit 12 differs from that of Embodiment 1. The differences from Embodiment 1 will be explained below.

[0070] In Embodiment 2, the corresponding shape detection unit 12 detects corresponding pairs of regions between images as corresponding shape pairs, using the position and scale information of each feature point constituting the corresponding point pair for each corresponding point pair.

[0071] Specifically, the corresponding shape detection unit 12 identifies a virtual circle centered on each feature point for each corresponding point pair, using the position and scale information of each feature point constituting the corresponding point pair, and detects the region enclosed by each virtual circle as a corresponding shape pair.

[0072] Here, the function of the corresponding shape detection unit 12 will be explained in detail using Figure 6. Figure 6 is a diagram showing an example of a pair of corresponding regions between images. As shown in Figure 6, the corresponding point pair is the feature point m in image 1. i And, feature point m' in Image 2 i Let's assume that this is the case. In this case, the corresponding shape detection unit 12 uses the scale information (radius r) to find the feature point m i Radius r centered on i Identify the region enclosed by the circle and the feature point m' i Radius r' centered at iIdentify the regions enclosed by circles, and designate the two identified circular regions as corresponding shape pairs. The subscripts indicate the corresponding numbers.

[0073] Furthermore, in Embodiment 2, the transformation matrix generation unit 13 generates a plane projection transformation matrix using three or more pairs of corresponding lines and three or more pairs of corresponding figures (pairs of regions enclosed by circles) that correspond to these lines.

[0074] [Device operation] Next, the operation of the information processing device will be explained using Figure 7. Figure 7 is a flowchart showing a second example of the operation of the information processing device. In the following explanation, Figures 2 and 6 will be referred to as appropriate. In Embodiment 2, the information processing method is also implemented by operating the information processing device. Therefore, the explanation of the information processing method in Embodiment 2 will be replaced by the following explanation of the operation of the information processing device.

[0075] As shown in Figure 7, first, the data acquisition unit 14 acquires image data from an external device, obtained by photographing the same object from different directions (step S11). The data acquisition unit 14 stores each acquired image data in the storage unit 15. Step S11 is the same as step S1 shown in Figure 5.

[0076] Next, the correspondence point detection unit 11 acquires image data of any two images (image 1 and image 2) from the storage unit 15, performs feature point matching on each image of the acquired image data, and detects three or more pairs of correspondence points (step S12). Step S12 is the same as step S2 shown in Figure 5.

[0077] Next, the corresponding shape detection unit 12 detects a pair of points and corresponding circular regions as a corresponding shape pair for each corresponding point pair detected in step S12, using the position and scale information of each feature point constituting that corresponding point pair (step S13).

[0078] Specifically, in step S12, the corresponding shape detection unit 12 identifies a virtual circle centered on each feature point using the position and scale information of each feature point constituting the corresponding point pair, and detects the pair of regions (circular regions) enclosed by each virtual circle as a corresponding shape pair.

[0079] Next, the transformation matrix generation unit 13 generates a plane projection transformation matrix using the three or more pairs of corresponding points detected in step S12 and the pairs of corresponding figures detected in step S13 (step S14).

[0080] Subsequently, the output unit 16 outputs the plane projection transformation matrix generated in step S14 (step S15).

[0081] Furthermore, in Embodiment 2, if it is necessary to generate a planar projection transformation matrix between two other images, steps S12 to S15 are executed again. In this case, in step S12, the corresponding point detection unit 11 acquires image data for two other combinations of images.

[0082] Furthermore, in step S14, the transformation matrix generation unit 13 can generate the plane projection transformation matrix multiple times while changing the corresponding figure pairs used. The transformation matrix generation unit 13 then evaluates the accuracy of the multiple generated plane projection transformation matrices and selects the plane projection transformation matrix with the highest accuracy based on the evaluation results.

[0083] [Specific example] Here, a specific example of the processing of the information processing device in Embodiment 2 will be described below. First, in Embodiment 2 as well, the plane projection transformation between the corresponding point pair, feature point m and feature point m', is represented by Equation 1 above.

[0084] Next, we will explain the corresponding figure pairs. As mentioned above, feature point m has information about the radius r centered on the feature point as scale information. In Image 1, the coordinates of the center are (c u ,c vCircle C with radius r can be expressed by the following quadratic form, equation 6. In equation 6 below, point x is any point on circle C.

[0085]

number

[0086] Then, when point x is projected onto point x' by matrix H, x'~Hx, and as shown in equation 7 below, circle C is projected onto circle C'.

[0087]

number

[0088] Next, multiple circles lie on the same plane, and circle C i and C j However, each is circle C' i and C' j Corresponding to this, we obtain the following number 8.

[0089]

number

[0090] Here, since the quadratic representation of a circle has an indeterminate scale, the determinants are normalized as shown in equation 9 below so that they are equal.

[0091]

number

[0092] Therefore, given N pairs of corresponding figures, the matrix H can be estimated by solving the optimization problem represented by the following equation 10.

[0093]

number

[0094] The two pairs of corresponding figures shown in the above number 8 {C i, C j}⇔{C' i, C' j Six constraints can be obtained from}. The number of combinations of selecting 2 pairs of corresponding figures from N pairs without repetition is " N C2 = N*(N-1) / 2. Therefore, in order to solve the above equation 10, the following equation 11 must be satisfied.

[0095]

number

[0096] The smallest configuration that satisfies equation 11 above is N=3. That is, if there are at least 3 pairs of corresponding regions, we can solve equation 10 above to generate matrix H. Note that, similar to equation 4 above, equation 8 above also reduces to DLT.

[0097] Here, steps S12 to S14 shown in Figure 7 will be explained in detail, following the specific example described above.

[0098] [Step S12] In step S12, the corresponding point detection unit 11 acquires image data of image 1 and image data of image 2. Then, the corresponding point detection unit 11 performs feature point matching on image 1 and image 2 to find three or more corresponding point pairs {m i ,m' i Detect i∈{1,···,N},N≧3}.

[0099] [Step S13] In step S3, the corresponding shape detection unit 12 detects the corresponding point pair m i ,m' i For each of these, based on the above number 8, the central coordinates are given by the feature point m. i , radius of feature point m i scale r i Circle C i The central coordinates are the feature points m' i , radius of feature point m' iThe scale r' i Circle C' i N pairs of corresponding circles {C i ,C' i Detect i∈{1,···,N},N≧3}.

[0100] [Step S14] In step S14, the transformation matrix generation unit 13 generates a plane projection transformation matrix based on the above number 10 using the detected N pairs (N≧3) of corresponding figure pairs.

[0101] Unlike equation 4 above, in equation 8, it is not necessary to consider the planar projection transformation of the coordinates of the corresponding point pairs (equation 1 above). This is because, as shown in equation 6 above, the circle expressed in quadratic form contains the coordinates of its center, i.e., the position of the feature point's coordinates.

[0102] [Effects in Embodiment 2] As described above, according to Embodiment 2, a plane projection transformation matrix can be generated using only three pairs of corresponding figures. The reason is as follows.

[0103] When selecting two pairs of corresponding shapes from three pairs, the number of combinations is three, resulting in a total of 18 constraints. Therefore, the matrix H with 8 degrees of freedom can be calculated based on the above equation 10, and the above effect is obtained.

[0104] Furthermore, in Embodiment 2, as in Embodiment 1, the accuracy of generating the plane projection transformation matrix is ​​higher than that of the method disclosed in Patent Document 2. This is because the plane projection transformation matrix is ​​generated using only the positions of corresponding point pairs and scale information, without using scale information and angle information with different error distributions in combination.

[0105] [program] In Embodiment 2, the program can be any program that causes the computer to execute steps S11 to S15 shown in Figure 7. By installing and executing this program on the computer, the information processing device and the information processing method can be realized. In this case, the computer's processor functions as a corresponding point detection unit 11, a corresponding figure detection unit 12, a transformation matrix generation unit 13, a data acquisition unit 14, and an output unit 16, and performs the processing.

[0106] Furthermore, the memory unit 15 may be implemented by a storage device such as a hard disk installed in the computer, or by a storage device of another computer. Examples of computers include general-purpose PCs, server computers, smartphones, and tablet devices.

[0107] Furthermore, in Embodiment 2, the program may also be executed by a computer system (such as a cloud system) constructed by multiple computers. In this case, for example, each computer may function as one of the following: the corresponding point detection unit 11, the corresponding figure detection unit 12, the transformation matrix generation unit 13, the data acquisition unit 14, and the output unit 16.

[0108] (modified version) The embodiments 1 and 2 described above are not limited to the examples described above. A variety of modifications that can be understood by those skilled in the art can be applied to the examples described above. For example, the embodiments can also be implemented in the forms shown in the following modified examples.

[0109] For example, in Embodiments 1 and 2, if a correspondence point pair contains an error (a mismatched point or outlier), the mismatched point may be removed using RANSAC and LO-RANSAC, a derivative of RANSAC, as disclosed in Patent Document 2.

[0110] Furthermore, in random sampling of three pairs of corresponding points, as in Patent Document 2, signed areas may be calculated using three pairs of corresponding points or three pairs of corresponding figures, and mismatched points may be determined by comparing the signs of the calculated signed areas. In this case, the transformation matrix generation unit 13 determines whether or not to generate a plane projection transformation matrix based on the result of determining mismatched points. Note that RANSAC is a widely known technology, so a detailed description is omitted.

[0111] (physical configuration) Here, a computer that implements an information processing device by executing the program in Embodiment 1 or 2 will be described using Figure 8. Figure 8 is a block diagram showing an example of a computer that implements an information processing device.

[0112] As shown in Figure 8, the computer 110 comprises a CPU (Central Processing Unit) 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication.

[0113] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to, or instead of, the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0114] The CPU 111 loads the program in the embodiment, which consists of a set of codes stored in the storage device 113, into the main memory 112, and performs various calculations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0115] Furthermore, the program in this embodiment is provided stored on a computer-readable recording medium 120. The program in this embodiment may also be distributed over the internet via a communication interface 117.

[0116] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0117] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0118] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0119] Furthermore, the information processing device can be implemented not only by a computer with a program installed, but also by using hardware corresponding to each part, such as electronic circuits. Moreover, the information processing device may be partially implemented by a program and the remaining parts by hardware. In the embodiment, the computer is not limited to the computer shown in Figure 8.

[0120] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 15) described below, but are not limited to the following descriptions.

[0121] (Note 1) A corresponding point detection unit extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection unit detects a pair of shapes derived from each feature point that correspond between images, using the position of each feature point constituting the pair and either the scale information or the angle information. A transformation matrix generation unit generates a planar projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images. It is equipped with An information processing device characterized by the following:

[0122] (Note 2) The corresponding shape detection unit identifies a virtual line passing through each feature point for each pair of feature points, using the position of each feature point constituting the pair and the angle information, and detects the virtual line identified for each feature point as a corresponding pair of shapes between the images. The information processing device described in Appendix 1.

[0123] (Note 3) The corresponding shape detection unit identifies a virtual circle centered on each feature point for each pair of feature points, using the position of each feature point constituting the pair and the scale information, and detects the region enclosed by the identified virtual circle for each feature point as a corresponding pair of shapes between the images. The information processing device described in Appendix 1.

[0124] (Note 4) The transformation matrix generation unit performs the generation of the plane projection transformation matrix multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and selects one of the multiple plane projection transformation matrices generated. The information processing device described in Appendix 1.

[0125] (Note 5) The transformation matrix generation unit calculates a signed area using at least three pairs of figures, and determines whether or not to generate the plane projection transformation matrix using the sign of the calculated signed area. The information processing device described in Appendix 1.

[0126] (Note 6) A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection step is performed, in which, using the position of each feature point constituting the pair and either the scale information or the angle information, a corresponding pair of shapes derived from each feature point is detected between images. A transformation matrix generation step, which involves generating a planar projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images, Having, An information processing method characterized by the following:

[0127] (Note 7) In the corresponding shape detection step, for each pair of feature points, a virtual line passing through each feature point is identified using the position of each feature point constituting the pair and the angle information, and the virtual line identified for each feature point is detected as a corresponding pair of shapes between the images. The information processing method described in Appendix 6.

[0128] (Note 8) In the corresponding shape detection step, for each pair of feature points, a virtual circle centered on each feature point is identified using the position of each feature point constituting the pair and the scale information, and the region enclosed by the virtual circle identified for each feature point is detected as a corresponding pair of shapes between the images. The information processing method described in Appendix 6.

[0129] (Note 9) In the transformation matrix generation step, the generation of the plane projection transformation matrix is ​​performed multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and one of the multiple generated plane projection transformation matrices is selected. The information processing method described in Appendix 6.

[0130] (Note 10) In the transformation matrix generation step, a signed area is calculated using at least three pairs of figures, and the sign of the calculated signed area is used to determine whether or not to generate the plane projection transformation matrix. The information processing method described in Appendix 6.

[0131] (Note 11) On the computer, A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection step is performed, in which, using the position of each feature point constituting the pair and either the scale information or the angle information, a corresponding pair of shapes derived from each feature point is detected between images. A transformation matrix generation step, which involves generating a planar projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images, A program that executes something.

[0132] (Note 12) In the corresponding shape detection step, for each pair of feature points, a virtual line passing through each feature point is identified using the position of each feature point constituting the pair and the angle information, and the virtual line identified for each feature point is detected as a corresponding pair of shapes between the images. The program described in Appendix 11.

[0133] (Note 13) In the corresponding shape detection step, for each pair of feature points, a virtual circle centered on each feature point is identified using the position of each feature point constituting the pair and the scale information, and the region enclosed by the virtual circle identified for each feature point is detected as a corresponding pair of shapes between the images. The program described in Appendix 11.

[0134] (Note 14) In the transformation matrix generation step, the generation of the plane projection transformation matrix is ​​performed multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and one of the multiple generated plane projection transformation matrices is selected. The program described in Appendix 11.

[0135] (Note 15) In the transformation matrix generation step, a signed area is calculated using at least three pairs of figures, and the sign of the calculated signed area is used to determine whether or not to generate the plane projection transformation matrix. The program described in Appendix 11. [Industrial applicability]

[0136] As described above, this disclosure makes it possible to suppress the decrease in accuracy of the plane projection transformation matrix when generating a plane projection transformation matrix using two pairs of corresponding points. This disclosure is useful for computer systems that require image processing. [Explanation of Symbols]

[0137] 10 Information Processing Devices 11. Corresponding point detection unit 12 Corresponding shape detection unit 13 Transformation Matrix Generation Unit 14. Data Acquisition Unit 15 Storage section 16 Output section 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A corresponding point detection unit extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection unit detects a pair of shapes derived from each feature point that correspond between images, using the position of each feature point constituting the pair and either the scale information or the angle information. A transformation matrix generation unit generates a planar projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images. It is equipped with An information processing device characterized by the following:

2. The corresponding shape detection unit identifies a virtual line passing through each feature point for each pair of feature points, using the position of each feature point constituting the pair and the angle information, and detects the virtual line identified for each feature point as a corresponding pair of shapes between the images. The information processing apparatus according to claim 1.

3. The corresponding shape detection unit identifies a virtual circle centered on each feature point for each pair of feature points, using the position of each feature point constituting the pair and the scale information, and detects the region enclosed by the identified virtual circle for each feature point as a corresponding pair of shapes between the images. The information processing apparatus according to claim 1.

4. The transformation matrix generation unit performs the generation of the plane projection transformation matrix multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and selects one of the multiple plane projection transformation matrices generated. The information processing apparatus according to claim 1.

5. The transformation matrix generation unit calculates a signed area using at least three pairs of figures, and determines whether or not to generate the plane projection transformation matrix using the sign of the calculated signed area. The information processing apparatus according to claim 1.

6. A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection step is performed, in which, using the position of each feature point constituting the pair and either the scale information or the angle information, a corresponding pair of shapes derived from each feature point is detected between images. A transformation matrix generation step, which involves generating a plane projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images, Having, An information processing method characterized by the following:

7. On the computer, A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding shape detection step is performed, in which, using the position of each feature point constituting the pair and either the scale information or the angle information, a corresponding pair of shapes derived from each feature point is detected between images. A transformation matrix generation step, which involves generating a plane projection transformation matrix for transforming one of the images into the other, using three or more pairs of corresponding figures between the aforementioned images, A program that executes something.

Images