Information processing device, information processing method, and program

Abstract

This method suppresses numerical instability in the generation of a plane projection transformation matrix using two pairs of corresponding points. [Solution] The information processing device 10 includes: a corresponding point detection unit 11 that extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the images from among the feature points extracted from each image; a corresponding line detection unit 12 that detects corresponding pairs of lines between the images using the position and angle information of each feature point for each pair of feature points; a corresponding region detection unit 13 that detects corresponding pairs of regions between the images using the position and scale information of each feature point for each pair of feature points; and a transformation matrix generation unit 14 that generates a plane projection transformation matrix for transforming one image to the other using the corresponding line pairs and corresponding region pairs.

Description

[Technical Field]

[0001] This disclosure relates to an image processing apparatus and an image processing method for generating a planar projection transformation matrix, and more particularly to a program for implementing these. [Background technology]

[0002] The transformation that connects multiple images obtained by photographing a plane from different angles is called a plane projection transformation. A plane projection transformation is represented by a 3x3 matrix and is a transformation that turns one quadrilateral into another. For this reason, it is known that if four different points on a plane can be observed in an image, a plane projection transformation matrix can be generated. Patent Document 1 discloses a technique for detecting four or more pairs of corresponding feature points (hereinafter simply referred to as corresponding points) between two images and applying the DLT (Direct Linear Transform) method to the detected pairs of corresponding points to create a frontal view of an overhead image of a sports scene.

[0003] Furthermore, Non-Patent Document 1 and Patent Document 2 disclose a technique for generating a planar projection transformation matrix using two or more pairs of corresponding points by using feature points that calculate scale and angle in addition to image coordinates. Such feature points are extracted by calculating SIFT (Scale Invariant Feature Transform) features or SURF (Speeded-Up Robust Features) features from an image.

[0004] According to the technologies disclosed in Non-Patent Document 1 and Patent Document 2, the minimum number of corresponding point pairs used for generation can be reduced from four to four, compared to the technology disclosed in Patent Document 1. Therefore, it is expected that the number of trials in combinatorial optimization such as RANSAC (Random Sample Consensus) will be reduced. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent No. 7334058 [Patent Document 2] Japanese Patent No. 7448034 [Non-Patent Document]

[0006] [Non-Patent Document 1] D. Barath, Z. Kukkelova, ”Homography from two orientation-and scale-covariant features,” 2019 IEEE / CVF International Conference on Computer Vision (ICCV), pp.1091-1099, 2019. [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] However, the method for generating a planar projective transformation matrix using two sets of corresponding point pairs disclosed in Non-Patent Document 1 and Patent Document 2 has the following problems. The problems will be specifically described below.

[0008] First, the method disclosed in Non-Patent Document 1 uses, as a constraint condition, that the invariance of the scale and angle of feature points is associated by an affine transformation, which is a first-order approximation of planar projective transformation. However, in feature points such as SIFT or SURF, variable elimination is performed by setting the skew value of the affine transformation, which should be zero, to non-zero. Therefore, the method disclosed in Non-Patent Document 1 has a problem that numerical operations are unstable.

[0009] Furthermore, the method disclosed in Patent Document 2 generates additional corresponding point pairs from the scale and angle of feature points to generate a planar projection transformation matrix. However, the scale values ​​of feature points such as SIFT or SURF are generally only a few pixels to about 10 pixels, which is relatively very small compared to the distance between feature points, which is tens to hundreds of pixels. As a result, the rectangle defined using the two detected corresponding point pairs and the two additional corresponding point pairs becomes a long, thin line, approaching the so-called degeneration condition. Consequently, the method disclosed in Patent Document 2 also suffers from the problem of unstable numerical calculations.

[0010] One example of the purpose of this disclosure is to suppress instability in numerical calculations when generating a plane projection transformation matrix using two pairs of corresponding points. [Means for solving the problem]

[0011] To achieve the above objective, the information processing device in one aspect of this disclosure is: A corresponding point detection unit extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection unit detects a corresponding pair of lines between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection unit detects a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation unit generates a planar projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. It is characterized by having the following features.

[0012] Furthermore, in order to achieve the above objectives, the information processing method in one aspect of this disclosure is: A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection step is performed, in which a corresponding pair of lines is detected between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection step is performed to detect a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation step, which generates a plane projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. It is characterized by having the following:

[0013] Furthermore, in order to achieve the above objectives, the program in one aspect of this disclosure is On the computer, A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection step is performed, in which a corresponding pair of lines is detected between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection step is performed to detect a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation step, which generates a plane projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. It is characterized by causing the execution of [the specified action]. [Effects of the Invention]

[0014] As described above, according to this disclosure, instability in numerical operations can be suppressed when generating a plane projection transformation matrix using two pairs of corresponding points. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 shows an example of corresponding feature point pairs between images. [Figure 2] Figure 2 is a schematic diagram showing the general configuration of an example of an information processing device. [Figure 3] Figure 3 is a diagram illustrating the configuration of an example of an information processing device. [Figure 4] Figure 4 is a flowchart illustrating the operation of an example of an information processing device. [Figure 5] Figure 5 is a block diagram showing an example of a computer that implements an information processing device. [Modes for carrying out the invention]

[0016] (Prerequisite explanation) First, the prerequisites for this disclosure will be explained using Figure 1. Figure 1 is a diagram showing an example of corresponding feature point pairs between images. In the example in Figure 1, Image 1 and Image 2 are obtained by photographing the same object from different directions.

[0017] Specifically, as shown in Figure 1, a point on a plane is photographed from different angles. This allows for the observation of feature point m in image 1 and feature point m' in image 2. Feature points m and m' are corresponding point pairs, representing points on the same plane observed in different images. Hereafter, these corresponding feature point pairs will also be referred to as "corresponding point pairs."

[0018] Furthermore, such feature points can be extracted from image data by calculating, for example, SIFT (Scale Invariant Feature Transform) features or SURF (Speeded-Up Robust Features) features. Feature points extracted from such features possess scale information and angular information.

[0019] Scale information indicates the region where the feature quantity lies within a set range, centered on the feature point. In the example in Figure 1, the scale information is represented by a circle with radius r centered on the feature point. Angle information indicates the direction of the feature point, obtained from the histogram of the luminance gradient direction in the region centered on the feature point. In the example in Figure 1, this is represented by θ.

[0020] As shown in Figure 1, feature point m has scale information and angle information, namely scale r and angle θ. Feature point m' corresponding to feature point m has scale information and angle information, namely scale r' and angle θ'. Line l in Image 1 i Line l' in image 2 is a straight line passing through feature point m with a slope θ'. Similarly, line l' in image 2 is a straight line passing through feature point m' with a slope θ'.

[0021] (Embodiment) The information processing device, information processing method, and program in the embodiment will be described below with reference to Figures 2 to 5.

[0022] [Device configuration] First, the general configuration of the information processing device will be explained using Figure 1. Figure 2 is a configuration diagram showing the general configuration of an example of an information processing device.

[0023] The information processing device 10 shown in Figure 2 is a device for generating a planar projection transformation matrix between images; in other words, it is a planar projection transformation matrix generator. As shown in Figure 2, the information processing device comprises a corresponding point detection unit 11, a corresponding line detection unit 12, a corresponding region detection unit 13, and a transformation matrix generation unit 14.

[0024] The corresponding point detection unit 11 extracts feature points containing scale information and angle information from each of two images obtained by photographing the same object from different directions. Then, from the feature points extracted from each of the two images, the corresponding point detection unit 11 detects pairs of feature points that correspond between the two images (corresponding point pairs).

[0025] The correspondence line detection unit 12 detects corresponding line pairs (hereinafter referred to as "corresponding line pairs") between images for each corresponding point pair, using the position and angle information of each feature point constituting that corresponding point pair.

[0026] The corresponding region detection unit 13 detects corresponding region pairs (hereinafter referred to as "corresponding region pairs") between images using the position and scale information of each feature point constituting the corresponding point pair for each corresponding point pair.

[0027] The transformation matrix generation unit 14 uses the corresponding line pairs and corresponding region pairs to generate a planar projection transformation matrix for transforming one image into the other.

[0028] In this way, the information processing device 10 generates a plane projection transformation matrix using not only the corresponding point pairs, but also the corresponding line pairs and corresponding region pairs detected from each feature point constituting the corresponding point pairs. Therefore, the information processing device 10 can suppress instability in numerical calculations when generating a plane projection transformation matrix using two pairs of corresponding points.

[0029] Next, we will explain the configuration and functions of the information processing device 10 in more detail using Figure 3. Figure 3 is a configuration diagram that specifically shows the configuration of an example of an information processing device.

[0030] As shown in Figure 3, the information processing device 10 includes, in addition to the corresponding point detection unit 11, corresponding line detection unit 12, corresponding region detection unit 13, and transformation matrix generation unit 14 described above, a data acquisition unit 15, a storage unit 16, and an output unit 17.

[0031] The data acquisition unit 15 acquires image data of images obtained by photographing the same object from different directions from external devices, such as a server device, a terminal device, and an imaging device, and stores the acquired image data in the storage unit 16.

[0032] In the embodiment, the corresponding point detection unit 11 first extracts feature points from each of the image data of two images (image 1 and image 2) retrieved from the storage unit 16. Specifically, the corresponding point detection unit 11 extracts feature points by calculating, for example, SIFT (Scale Invariant Feature Transform) feature amounts or SURF (Speeded-Up Robust Features) feature amounts in each image.

[0033] Then, the corresponding point detection unit 11 applies an existing feature point matching technique to the extracted feature points to detect two or more pairs of corresponding points. Each of the extracted feature points has scale information indicated by a radius r and angle information indicated by an angle θ, as described above.

[0034] In the embodiment, the corresponding line detection unit 12 identifies a virtual line passing through each feature point using the position and angle information of each feature point constituting a pair of corresponding points for each pair of corresponding points, and detects the identified virtual lines of each feature point as a pair of corresponding lines.

[0035] Specifically, if a pair of corresponding points is feature point m i and feature point m' i as shown in FIG. 1, the corresponding line detection unit 12 identifies a line l i which is a straight line with an inclination θ i passing through feature point m, and identifies a line l' i which is a straight line with an inclination θ' i passing through feature point m', and sets the identified line l i and line l' i as a pair of corresponding lines. i i i as a pair of corresponding lines.

[0036] In this embodiment, the corresponding region detection unit 13 identifies a virtual circle centered on each feature point using the position and scale information of each feature point constituting the corresponding point pair, and detects the region enclosed by each virtual circle as a corresponding region pair.

[0037] Specifically, the corresponding point pair is feature point m i and feature point m' i If that is the case, then, as shown in Figure 1, The corresponding circle detection unit 13 detects feature point m i Radius r centered on i Identify the region enclosed by the circle and the feature point m' i Radius r' centered at i Identify the regions enclosed by the circles, and designate the two identified circular regions as a corresponding region pair.

[0038] In this embodiment, the transformation matrix generation unit 14 generates a planar projection transformation matrix using the corresponding line pair and corresponding region pair detected for each of the two or more detected corresponding point pairs.

[0039] The output unit 17 outputs the plane projection transformation matrix generated by the transformation matrix generation unit 14. An image processing device is one possible output destination. The information processing device 10 may also constitute a part of the image processing device. An example of an image processing device is one that generates 3D data of an object from multiple images of the object using the plane projection transformation matrix.

[0040] [Device operation] Next, the operation of the information processing device 10 will be explained using Figure 4. Figure 4 is a flowchart showing the operation of an example of the information processing device. In the following explanation, Figures 1 to 3 will be referred to as appropriate. In this embodiment, the information processing method is implemented by operating the information processing device 10. Therefore, the explanation of the information processing method in this embodiment will be replaced by the following explanation of the operation of the information processing device 10.

[0041] As shown in Figure 4, first, the data acquisition unit 15 acquires image data from an external device, obtained by photographing the same object from different directions (step S1). The data acquisition unit 15 stores each acquired image data in the storage unit 16.

[0042] Next, the corresponding point detection unit 11 acquires image data of any two images (image 1 and image 2) from the storage unit 16, performs feature point matching on each image of the acquired image data, and detects two or more pairs of corresponding points (step S2).

[0043] Next, the correspondence line detection unit 12 detects a correspondence line pair for each correspondence point pair detected in step S2, using the position and angle information of each feature point constituting that correspondence point pair (step S3).

[0044] Specifically, in step S3, the correspondence line detection unit 12 identifies a virtual line passing through each feature point for each correspondence point pair, using the position and angle information of each feature point constituting the correspondence point pair, and detects the identified virtual line for each feature point as a correspondence line pair.

[0045] Next, the corresponding region detection unit 13 detects a corresponding region pair for each corresponding point pair detected in step S2, using the position and scale information of each feature point constituting that corresponding point pair (step S4).

[0046] Specifically, in step S4, the corresponding region detection unit 13 identifies a virtual circle centered on each feature point for each corresponding point pair, using the position and scale information of each feature point constituting the corresponding point pair, and detects the region enclosed by each virtual circle as a corresponding region pair.

[0047] Next, the transformation matrix generation unit 14 generates a plane projection transformation matrix using the corresponding line pairs detected in step S3 and the corresponding region pairs detected in step S4 (step S5).

[0048] Subsequently, the output unit 17 outputs the plane projection transformation matrix generated in step S5 (step S6).

[0049] Furthermore, if it is necessary to generate a planar projection transformation matrix between two other images, steps S2 to S6 are executed again. In this case, in step S2, the corresponding point detection unit 11 acquires image data for two other combinations of images.

[0050] Furthermore, suppose that in step S2, three or more pairs of corresponding points are detected, and as a result, three or more pairs of corresponding lines and corresponding regions are also detected. In this case, in step S5, the transformation matrix generation unit 14 can generate the plane projection transformation matrix multiple times while changing the pairs of corresponding lines and corresponding regions used. The transformation matrix generation unit 14 then evaluates the accuracy of the multiple plane projection transformation matrices generated and selects the plane projection transformation matrix with the highest accuracy based on the evaluation results.

[0051] Another method for selecting a highly accurate plane projection transformation matrix is, for example, as used in RANSAC, to apply the generated plane projection transformation matrix to the corresponding point pairs that were not used in the generation, measure the reprojection error between the corresponding points, and select the plane projection transformation matrix that minimizes the sum of the measured reprojection errors. In this method, instead of the reprojection error, a first-order approximation of the reprojection error, such as the Sampson Error or algebraic error, may be used.

[0052] [Specific example] Here, a specific example of processing in the information processing device 10 will be described below. In the following example, the coordinates of feature points are represented by a 3x1 homogeneous representation. For example, the coordinates of feature point m and its corresponding feature point m' are given by m=[u,v,1] T m'=[u',v',1] T It is represented by ]. Furthermore, the plane projection transformation matrix is ​​represented by a 3x3 matrix H. Although matrix H has 9 elements, it has 8 degrees of freedom due to scale indetermination.

[0053] Specifically, the plane projection transformation between a corresponding pair of feature points, feature point m and feature point m', is represented by equation 1 below. In equation 1 below, "~" indicates that both sides are equal to a constant multiple.

[0054]

number

[0055] Next, we will explain the corresponding line pairs. As mentioned above, feature point m has an angle θ as angular information. Therefore, the line l passing through feature point m is represented by the following equation 2.

[0056]

number

[0057] Based on the above equations 1 and 2, the planar projection transformation between line l and its corresponding line l' is expressed by the following equation 3.

[0058]

number

[0059] Next, we will explain the corresponding region pair. Here, we assume that the corresponding region is a circle. In Image 1, the coordinates of the center are (c u ,c v Circle C with radius r can be expressed by the following quadratic form, equation 4. In equation 4 below, point x is any point on circle C.

[0060]

number

[0061] Then, when point x is projected onto point x' by matrix H, x'~Hx, and as shown in equation 5 below, circle C is projected onto circle C'.

[0062]

number

[0063] Next, multiple circles lie on the same plane, and circle C i and C j However, each is circle C' i and C' j Corresponding to this, the following relationship 6 is obtained.

[0064]

number

[0065] Here, since the quadratic representation of a circle has an indeterminate scale, the determinants are normalized as shown in Equation 7 below so that they are equal to each other.

[0066]

number

[0067] Therefore, from equations 3 and 6 above, given N pairs of corresponding points, the matrix H can be estimated by solving the optimization problem represented by equation 8 below. In equation 8 below, "×" represents the cross product of 3D vectors.

[0068]

number

[0069] Here, we will explain steps S2 to S5 shown in Figure 4 in detail, step by step, following the specific examples described above.

[0070] [Step S2] In step S2, the corresponding point detection unit 11 acquires image data of image 1 and image data of image 2. Then, the corresponding point detection unit 11 performs feature point matching on image 1 and image 2 to obtain N pairs of corresponding points {m i ,m' iDetects values ​​i∈{1,···,N},N≧2}. The subscript "i" indicates the corresponding number.

[0071] [Step S3] In step S3, the corresponding line detection unit 12 detects the corresponding point pair m i ,m' i For each of these, feature points m are determined based on the above number 2. i Line l passing through i and point m' i Line l' passing through i Calculate and N pairs of corresponding lines {l i ,l' i Detect i∈{1,···,N},N≧2}.

[0072] [Step S4] In step S4, the corresponding circle detection unit 13 detects the corresponding point pair m i ,m' i For each of these, based on the above number 6, the central coordinates are given by the feature point m. i , radius of feature point m i scale r i Circle C i The central coordinates are the feature points m' i , radius of feature point m' i The scale r' i Circle C' i N pairs of corresponding circles {C i ,C' i Detect i∈{1,···,N},N≧2}.

[0073] [Step S5] In step S5, the transformation matrix generation unit 14 generates a plane projection transformation matrix based on the above equation 8, using the detected N pairs (N≧2) of corresponding line pairs and corresponding circle pairs.

[0074] [Effects in the embodiment] As described above, according to the embodiment, instability in numerical calculations is suppressed, and a plane projection transformation matrix can be generated if there are at least two pairs of corresponding points, based on the scale information and angle information of the feature points. The reason is as follows.

[0075] As shown in Figure 1, for each pair of corresponding points, one pair of corresponding lines is defined. Since the corresponding line l always passes through the feature point m, one pair of corresponding lines [l i ×H T l' i From ], one constraint condition can be obtained. Furthermore, the two pairs of corresponding regions shown in equation 6 above [{C i ,C j}⇔{C' i ,C' j Six constraints can be obtained from}.

[0076] The number of combinations of selecting 2 pairs of corresponding domains from N pairs without repetition is as follows: N C2 = N × (N-1) / 2. The number of constraints obtained from N pairs of correspondence lines and N pairs of correspondence regions is greater than the 8 degrees of freedom of matrix H when N + 6 × N × (N-1) / 2 ≥ 8. In other words, if there are at least N = 2 pairs of correspondence points, it is possible to solve the above number 8, and thus the matrix H can be generated.

[0077] Furthermore, since equation 8 above is a linear equation for matrix H, it can be reduced to the well-known DLT method. Also, the planar projection transformation of the position coordinates of the corresponding point pairs (equation 1 above) does not need to be considered. This is because, as shown in equation 4 above, the circle expressed in quadratic form contains the coordinates of its center, i.e., the coordinate positions of the feature points.

[0078] Furthermore, according to this embodiment, the accuracy of generating the planar projection transformation matrix can be made more accurate than the method disclosed in Patent Document 2. This is because feature points on the corresponding region are not explicitly used as additional corresponding points. As shown in equation 6 above, the constraint on the corresponding region based on the scale of the feature points is C i -1 C j It is expressed as the ratio of two circles, as shown above. Therefore, since it does not assume deformation of thin linear quadrilaterals, the stability of numerical calculations is improved.

[0079] [Differentiation] The embodiments described above are not limited to the examples described above. A variety of modifications that can be understood by those skilled in the art can be applied to the examples described above. For example, the embodiments can also be implemented in the forms shown in the following modified examples.

[0080] For example, in the embodiment, if a pair of corresponding points contains an error (a mismatched point or outlier), the mismatched point may be removed using RANSAC disclosed in Patent Document 2 or a derivative of RANSAC such as LO-RANSAC. Since RANSAC is a widely known technology, a detailed description will be omitted.

[0081] Alternatively, in random sampling of two pairs of corresponding points, mismatched points may be determined by comparing the sign of the normal vector formed between the vector connecting the feature points of the corresponding point pair and the corresponding line pair. Here, we will explain "the sign of the normal vector formed between the vector connecting the feature points of the corresponding point pair and the corresponding line pair."

[0082] Let {m1, m2}⇔{m'1, m'2} be two pairs of corresponding points, and let {l1, l2}⇔{l'1, l'2} be two pairs of corresponding lines. The vector connecting the feature points of a pair of corresponding points is the vector connecting the coordinates of each feature point. Therefore, the sign of the normal vector between the vector connecting the feature points of a pair of corresponding points and the pair of corresponding lines can be the sign of the determinant of the 3x3 matrix [m1, m2, l1] and the sign of the determinant of the 3x3 matrix [m'1, m'2, l'1].

[0083] As is widely known in the estimation of the plane projection transformation matrix, identical signs of the determinants are a necessary condition for the correctness of the corresponding point pairs and corresponding line pairs. Therefore, mismatched points can be determined by comparing the signs of each determinant. Furthermore, in addition to the matrices described above, the signs of the determinants of [m1, m2, l2] and [m'1, m'2, l'2] may also be used. Adding the signs of the determinants makes the criteria more rigorous, increasing the probability that the corresponding point pairs and corresponding line pairs are correct.

[0084] [program] In this embodiment, the program can be any program that causes the computer to execute steps S1 to S6 shown in Figure 4. By installing and executing this program on the computer, the information processing device 10 and the information processing method can be realized. In this case, the computer's processor functions as a corresponding point detection unit 11, a corresponding line detection unit 12, a corresponding region detection unit 13, a transformation matrix generation unit 14, a data acquisition unit 15, and an output unit 17, and performs the processing.

[0085] Furthermore, the memory unit 16 may be implemented by a storage device such as a hard disk installed in the computer, or by a storage device in another computer. Examples of computers include general-purpose PCs, server computers, smartphones, and tablet devices.

[0086] Furthermore, in this embodiment, the program may be executed by a computer system (such as a cloud system) constructed by multiple computers. In this case, for example, each computer may function as one of the following: a corresponding point detection unit 11, a corresponding line detection unit 12, a corresponding region detection unit 13, a transformation matrix generation unit 14, a data acquisition unit 15, and an output unit 17.

[0087] [Physical configuration] Here, a computer that implements the information processing device 10 by executing the program in the embodiment will be described using Figure 5. Figure 5 is a block diagram showing an example of a computer that implements the information processing device.

[0088] As shown in Figure 5, the computer 110 comprises a CPU (Central Processing Unit) 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication.

[0089] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to, or instead of, the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0090] The CPU 111 loads the program in the embodiment, which consists of a set of codes stored in the storage device 113, into the main memory 112, and performs various calculations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0091] Furthermore, the program in this embodiment is provided stored on a computer-readable recording medium 120. The program in this embodiment may also be distributed over the internet via a communication interface 117.

[0092] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0093] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0094] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0095] Furthermore, the information processing device 10 can be implemented not only by a computer on which a program is installed, but also by using hardware corresponding to each part, such as electronic circuits. Moreover, the information processing device 10 may be partially implemented by a program and the remaining part by hardware. In the embodiment, the computer is not limited to the computer shown in Figure 5.

[0096] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 15) described below, but are not limited to the following descriptions.

[0097] (Note 1) A corresponding point detection unit extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection unit detects a corresponding pair of lines between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection unit detects a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation unit generates a planar projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. It is equipped with An information processing device characterized by the following:

[0098] (Note 2) The corresponding line detection unit identifies a virtual line passing through each feature point for each pair of feature points, using the position of each feature point constituting the pair and the angle information, and detects the identified virtual line for each feature point as a corresponding line pair between the images. The information processing device described in Appendix 1.

[0099] (Note 3) The corresponding region detection unit identifies a virtual circle centered on each feature point for each pair of feature points, using the position of each feature point constituting the pair and the scale information, and detects the region enclosed by the identified virtual circle for each feature point as a pair of corresponding regions between the images. The information processing device described in Appendix 1.

[0100] (Note 4) The transformation matrix generation unit performs the generation of the plane projection transformation matrix multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and selects one of the multiple plane projection transformation matrices generated. The information processing device described in Appendix 1.

[0101] (Note 5) The transformation matrix generation unit selects a specific pair of feature points and a corresponding pair of lines between the image detected therefrom, calculates a vector connecting one feature point and the other feature point in the selected pair of feature points, calculates the normal vector formed by the calculated vector and the selected pair of lines, and determines whether or not to generate the plane projection transformation matrix based on the sign of the components of the calculated normal vector. The information processing device described in Appendix 1.

[0102] (Note 6) A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection step is performed, in which a corresponding pair of lines is detected between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection step is performed to detect a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation step, which generates a plane projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. Having, An information processing method characterized by the following:

[0103] (Note 7) In the corresponding line detection step, for each pair of feature points, a virtual line passing through each feature point is identified using the position of each feature point constituting the pair and the angle information, and the virtual line identified for each feature point is detected as a corresponding line pair between the images. The information processing method described in Appendix 6.

[0104] (Note 8) In the corresponding region detection step, for each pair of feature points, a virtual circle centered on each feature point is identified using the position of each feature point constituting the pair and the scale information, and the region enclosed by the virtual circle identified for each feature point is detected as a pair of corresponding regions between the images. The information processing method described in Appendix 6.

[0105] (Note 9) In the transformation matrix generation step, the generation of the plane projection transformation matrix is ​​performed multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and one of the multiple generated plane projection transformation matrices is selected. The information processing method described in Appendix 6.

[0106] (Note 10) In the transformation matrix generation step, a specific pair of feature points and a corresponding pair of lines between them and the image detected are selected, a vector connecting one of the selected feature points and the other in the selected pair of feature points is calculated, the normal vector formed by the calculated vector and the selected pair of lines is calculated, and based on the sign of the components of the calculated normal vector, it is determined whether or not to generate the plane projection transformation matrix. The information processing method described in Appendix 6.

[0107] (Note 11) On the computer, A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection step is performed, in which a corresponding pair of lines is detected between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection step is performed to detect a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation step, which generates a plane projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. A program that executes something.

[0108] (Note 12) In the corresponding line detection step, for each pair of feature points, a virtual line passing through each feature point is identified using the position of each feature point constituting the pair and the angle information, and the virtual line identified for each feature point is detected as a corresponding line pair between the images. The program described in Appendix 11.

[0109] (Note 13) In the corresponding region detection step, for each pair of feature points, a virtual circle centered on each feature point is identified using the position of each feature point constituting the pair and the scale information, and the region enclosed by the virtual circle identified for each feature point is detected as a pair of corresponding regions between the images. The program described in Appendix 11.

[0110] (Note 14) In the transformation matrix generation step, the generation of the plane projection transformation matrix is ​​performed multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and one of the multiple generated plane projection transformation matrices is selected. The program described in Appendix 11.

[0111] (Note 15) In the transformation matrix generation step, a specific pair of feature points and a corresponding pair of lines between them and the image detected are selected, a vector connecting one of the selected feature points and the other in the selected pair of feature points is calculated, the normal vector formed by the calculated vector and the selected pair of lines is calculated, and based on the sign of the components of the calculated normal vector, it is determined whether or not to generate the plane projection transformation matrix. The program described in Appendix 11. [Industrial applicability]

[0112] As described above, this disclosure makes it possible to suppress instability in numerical calculations when generating a plane projection transformation matrix using two pairs of corresponding points. This disclosure is useful for computer systems that require image processing. [Explanation of Symbols]

[0113] 10 Information Processing Devices 11. Corresponding point detection unit 12 Corresponding line detection unit 13. Corresponding area detection unit 14 Transformation Matrix Generation Unit 15. Data Acquisition Unit 16 Memory section 17 Output section 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A corresponding point detection unit extracts feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detects corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection unit detects a corresponding pair of lines between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection unit detects a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation unit generates a planar projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. It is equipped with An information processing device characterized by the following:

2. The corresponding line detection unit identifies a virtual line passing through each feature point for each pair of feature points, using the position of each feature point constituting the pair and the angle information, and detects the identified virtual line for each feature point as a corresponding line pair between the images. The information processing apparatus according to claim 1.

3. The corresponding region detection unit identifies a virtual circle centered on each feature point for each pair of feature points, using the position of each feature point constituting the pair and the scale information, and detects the region enclosed by the identified virtual circle for each feature point as a pair of corresponding regions between the images. The information processing apparatus according to claim 1.

4. The transformation matrix generation unit performs the generation of the plane projection transformation matrix multiple times while changing the corresponding line pairs and corresponding region pairs between the images, and selects one of the multiple plane projection transformation matrices generated. The information processing apparatus according to claim 1.

5. The transformation matrix generation unit selects a specific pair of feature points and a corresponding pair of lines between the image detected therefrom, calculates a vector connecting one feature point and the other feature point in the selected pair of feature points, calculates the normal vector formed by the calculated vector and the selected pair of lines, and determines whether or not to generate the plane projection transformation matrix based on the sign of the components of the calculated normal vector. The information processing apparatus according to claim 1.

6. A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection step is performed, in which a corresponding pair of lines is detected between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection step is performed to detect a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation step, which generates a plane projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. Having, An information processing method characterized by the following:

7. On the computer, A correspondence point detection step involves extracting feature points having scale information and angle information from each of two images obtained by photographing the same object from different directions, and detecting corresponding pairs of feature points between the two images from among the feature points extracted from each of the two images. For each pair of feature points, a corresponding line detection step is performed, in which a corresponding pair of lines is detected between images using the position of each feature point constituting the pair and the angle information. For each pair of feature points, a corresponding region detection step is performed to detect a pair of corresponding regions between images using the positions of each feature point constituting the pair and the scale information. A transformation matrix generation step, which generates a plane projection transformation matrix for transforming one of the images into the other, using corresponding pairs of lines and corresponding pairs of regions between the images. A program that executes something.

Images