Image processing device, image processing method, and computer-readable recording medium

Abstract

An image processing device 10 comprises: a camera matrix acquisition unit 11 that acquires a camera matrix for each image when point group data of an object is generated from a plurality of images; a projection vector calculation unit 12 that selects a pair of images from the plurality of images and calculates a projection vector from one of the images in the pair to the other image by using the camera matrix of each of the images in the pair; and a feature-point pair setting unit 13 that selects a feature point in one of the images in the pair, sets a projection area, in which the selected feature point is projected in the other image, by using the selected feature point and the projection vector, specifies a feature point corresponding to the selected feature point from among feature points existing in the projection area, and sets the two feature points as a feature point pair.

Description

Image processing device, image processing method, and computer-readable recording medium

[0001] The present disclosure relates to an image processing device and an image processing method for generating point cloud data of an object from multiple images, and further to a computer-readable recording medium on which a program for realizing these is recorded.

[0002] In recent years, techniques for constructing the three-dimensional shape of an object captured in an image have been attracting attention. A representative example of such a technique is Structure from Motion (SfM) (see, for example, Patent Document 1). In SfM, a specific object is photographed multiple times while changing the viewpoint of a camera, and the three-dimensional shape of the specific object is reconstructed from the multiple images obtained (see, for example, Patent Document 1).

[0003] Three-dimensional shape reconstruction using SfM is performed according to the following steps (a) to (l): (a) For each image, its features (e.g., SIFT features, SURF features) are calculated, and robust feature points that are resistant to changes in image scaling, rotation, and illumination are extracted. (b) The extracted feature points are matched between images, and matching feature point pairs are extracted. (c) For example, the geometric relationship between feature point pairs is calculated using robust estimation, and incorrect feature point pairs are eliminated.

[0004] (d) A set of paired images is selected as an initial paired image from the multiple images from which feature points have been extracted. (e) Feature points corresponding to both of the initial paired images are used as a feature point pair, and for each of the initial paired images, the camera matrix of the camera that captured that image is calculated. (f) For each of the initial paired images, the three-dimensional coordinates of the position of the camera that captured that image and a rotation matrix are found from the camera matrix. Furthermore, the three-dimensional coordinates of the camera position and rotation matrix are used to calculate the three-dimensional coordinates of the feature point pair corresponding to both. (g) A bundle adjustment is performed on the three-dimensional coordinates and camera matrix calculated for the initial paired images to minimize the error between them.

[0005] (h) A new image is selected from among the images other than the initial pair of images, and the newly selected image and one of the initial pair of images are combined to form a new pair of images. (i) Among the feature points extracted from the newly selected image, feature points corresponding to feature points extracted in one of the initial pair of images that has become the new pair of images are identified. Then, using the feature points of one of the initial pair of images that has become the new pair of images and the identified feature points as a new feature point pair, a camera matrix of the camera that captured the newly selected image is calculated. (j) From the camera matrix, three-dimensional coordinates and a rotation matrix of the position of the camera that captured the newly selected image are obtained, and using these, three-dimensional coordinates of the feature points of the newly selected image are calculated. (k) A bundle adjustment is performed on the three-dimensional coordinates and camera matrix calculated for the new pair of images to minimize the error between them. (l) Steps (h) to (k) above are repeated until all images have been selected, generating three-dimensional point cloud data of the specific object, and as a result, the three-dimensional shape (three-dimensional point cloud data) is restored.

[0006] International Publication No. 2019 / 058487

[0007] However, in conventional SfM, missing parts may occur in the generated 3D point cloud data. When missing parts occur, the quality of the 3D point cloud data is significantly reduced. There are two reasons why missing parts occur:

[0008] The first reason is that when extracting feature point pairs from paired images, feature point pairs that should actually be determined to be inliers (matching) are mistakenly determined to be outliers (non-matching) and are excluded.

[0009] Specifically, for each feature point, the Euclidean distance between the feature vectors of other features and points is first calculated, and the feature point with the smallest Euclidean distance and the feature point with the second smallest Euclidean distance are identified. Of these, the feature point with the smallest Euclidean distance and the feature point that is the base point of the Euclidean distance are candidates for feature point pairs.

[0010] The minimum Euclidean distance is then divided by the second minimum Euclidean distance to calculate a ratio, and it is determined whether the calculated ratio is equal to or greater than a threshold. If the result of the determination is that the ratio is equal to or greater than the threshold, the minimum Euclidean distance and the second Euclidean distance are close, and therefore the aforementioned feature points pair is deemed to be potentially incompatible and is excluded from the feature points pair. However, the excluded feature points pair may be an inlier, and if it is an inlier, it will be the cause of the missing portion described above.

[0011] Furthermore, in areas with few features, such as walls, the feature amounts of each feature point are similar even if they do not match each other, so there is a high possibility that outlier feature point pairs will occur. Such feature point pairs are eliminated by bundle adjustment, but in such areas, the number of inlier feature point pairs will decrease, resulting in many missing parts.

[0012] The second reason is that, due to poor accuracy in calculating the camera matrix in the early stages of constructing paired images, feature points pairs in the early stages that should actually be determined as inliers are determined as outliers by bundle adjustment and are therefore excluded.

[0013] An example of an objective of the present disclosure is to prevent an occurrence of a situation in which an inlier feature point pair is erroneously determined to be an outlier when generating three-dimensional point cloud data from feature points of multiple images.

[0014] In order to achieve the above object, an image processing device according to one aspect of the present disclosure comprises: a camera matrix acquisition unit that, when point cloud data of an object is generated from a plurality of images including the object, acquires, for each of the plurality of images, a camera matrix of a camera that captured the image; a projection vector calculation unit that selects a pair of images from the plurality of images, and calculates a projection vector from one of the selected pair of images to the other image using the camera matrix of each of the selected pair of images; and a feature point pair setting unit that selects a feature point in one of the selected pair of images, and sets, in the other image of the selected pair of images, a projection area onto which the selected feature point is expected to be projected, using the selected feature point and the calculated projection vector, identifies a feature point that corresponds to the selected feature point from among feature points present within the set projection area, and sets the selected feature point and the identified feature point as a feature point pair.

[0015] Furthermore, in order to achieve the above object, an image processing method according to one aspect of the present disclosure is characterized by comprising: a camera matrix acquisition step of, when point cloud data of an object is generated from a plurality of images including the object, acquiring, for each of the plurality of images, a camera matrix of a camera that captured the image; a projection vector calculation step of selecting a pair of images from the plurality of images, and calculating a projection vector from one of the selected pair of images to the other of the selected pair of images using the camera matrix of each of the selected pair of images; and a feature point pair setting step of selecting a feature point in one of the selected pair of images, and setting, in the other of the selected pair of images, a projection area onto which the selected feature point is expected to be projected using the selected feature point and the calculated projection vector, identifying a feature point that corresponds to the selected feature point from among feature points present within the set projection area, and setting the selected feature point and the identified feature point as a feature point pair.

[0016] Furthermore, in order to achieve the above-mentioned object, a computer-readable recording medium according to one aspect of the present disclosure is characterized in that it records a program including instructions for causing a computer to execute the following: a camera matrix acquisition step of, when point cloud data of an object is generated from a plurality of images including the object, acquiring, for each of the plurality of images, a camera matrix of a camera that captured the image; a projection vector calculation step of selecting a pair of images from the plurality of images and calculating a projection vector from one of the selected pair of images to the other of the selected pair of images using the camera matrix of each of the selected pair of images; and a feature point pair setting step of selecting a feature point in one of the selected pair of images and setting, in the other of the selected pair of images, a projection area onto which the selected feature point is expected to be projected using the selected feature point and the calculated projection vector, identifying a feature point that corresponds to the selected feature point from among feature points present within the set projection area, and setting the selected feature point and the identified feature point as a feature point pair.

[0017] As described above, according to the present disclosure, when generating three-dimensional point cloud data from feature points of multiple images, it is possible to prevent occurrence of a situation in which an inlier feature point pair is erroneously determined to be an outlier.

[0018] FIG. 1 is a configuration diagram showing a schematic configuration of an example of an image processing device. FIG. 2 is a configuration diagram showing a more specific configuration of an example of an image processing device. FIG. 3 is a diagram showing an example of a projection vector. FIG. 4 is a diagram showing an example of an epipolar line and a projection region. FIG. 5 is a flow diagram showing an example of a process for generating initial point cloud data in the image processing device. FIG. 6 is a flow diagram showing an example of a process for setting new feature point pairs in the image processing device. FIG. 7 is a diagram showing a process for projecting feature points of one image onto another image using depth. FIG. 8 is a block diagram showing an example of a computer that realizes the image processing device.

[0019] First Embodiment In the first embodiment, an example of an image processing device, an image processing method, and a program will be described below with reference to FIGS.

[0020] [Device Configuration] First, the schematic configuration of an example of an image processing device will be described with reference to Fig. 1. Fig. 1 is a diagram showing the schematic configuration of an example of an image processing device.

[0021] 1, an image processing device 10 is a device used to generate point cloud data of an object from multiple images. As shown in FIG. 1, the image processing device 10 includes a camera matrix acquisition unit 11, a projection vector calculation unit 12, and a feature point pair setting unit 13.

[0022] When point cloud data of an object is generated from a plurality of images including the object, the camera matrix acquisition unit 11 acquires, for each of the plurality of images, the camera matrix of the camera that captured each image.

[0023] The projection vector calculation unit 12 selects a pair of images from among a plurality of images, and calculates a projection vector from one of the selected pair of images to the other image using the camera matrix of each of the selected pair of images.

[0024] The feature points pair setting unit 13 first selects a feature point in one of the selected pair of images. Then, using the selected feature point and the calculated projection vector, the feature points pair setting unit 13 sets a projection area in the other of the selected pair of images into which the selected feature point is expected to be projected. Furthermore, the feature points pair setting unit 13 identifies a feature point that corresponds to the previously selected feature point from among the feature points present in the set projection area, and sets the selected feature point and the identified feature point as a feature points pair.

[0025] In this way, after generating point cloud data of the object, the image processing device 10 calculates the projection vector and uses the calculated projection vector to reset the feature points pairs used to generate the point cloud data. Therefore, even if a feature points pair that was originally determined to be an inlier was determined to be an outlier and excluded when generating the point cloud data, it can be restored.

[0026] Therefore, according to the image processing device 10, when generating three-dimensional point cloud data from feature points of multiple images, it is possible to prevent an inlier feature point pair from being erroneously determined as an outlier. Then, if point cloud data is generated again using the newly set feature point pair, the quality of the point cloud data can be improved.

[0027] Next, the configuration and functions of the image processing device 10 will be specifically described with reference to Figures 2 to 4. Figure 2 is a configuration diagram showing the configuration of an example of an image processing device in more detail.

[0028] As shown in FIG. 2, the image processing device 10 includes a point cloud data generation unit 14 and a storage unit 15 in addition to the camera matrix acquisition unit 11, the projection vector calculation unit 12, and the feature point pair setting unit 13 described above.

[0029] The point cloud data generation unit 14 generates new point cloud data of the object using the feature points pairs set by the feature points pair setting unit 13. The point cloud data generation unit 14 can also generate initial point cloud data. The function of the point cloud data generation unit 14 will be described later.

[0030] 2 , as will be described later, when the point cloud data generation unit 14 first generates point cloud data, the camera matrix of each image is calculated, and the calculated camera matrix is ​​stored in the storage unit 15. Therefore, in the example of FIG. 2 , the camera matrix acquisition unit 11 acquires the camera matrix of each image from the storage unit 15, and sends the acquired camera matrix to the projection vector calculation unit 12.

[0031] As shown in FIG. 3, the projection vector calculation unit 12 selects a pair of images and calculates the camera matrix [R １ｃｗ |t １ｃｗ ] and the other camera matrix [R ２ｃｗ |t ２ｃｗ ] and the projection vector [R γ |t γ ] is calculated.

[0032]

[0033] 3 is a diagram showing an example of a projection vector. In FIG. 3, the projection vector [R γ |t γ ] is the feature point p(x １１ , y １１ ) to the feature point p'(x ２１ , y ２１ ) Point A (X, Y) indicates the points in real space of feature points p and p'. The coordinates of feature points p and p' are respectively coordinates in the camera coordinate system. The coordinates of point A are coordinates in the world coordinate system.

[0034] The feature point pair setting unit 13 first selects one of the feature points in one of the paired images. Furthermore, the feature point pair setting unit 13 calculates an epipolar line in the other of the selected paired images using the selected feature point and the calculated projection vector, as shown in Fig. 4. Fig. 4 is a diagram showing an example of the epipolar line and the projection region.

[0035] Specifically, the feature point pair setting unit 13 sets the feature point p ｉ and the corresponding feature point p of the other image. ｉ The Fundamental matrix (reference document) shown in the following equation 2 is calculated as a numerical matrix that expresses the geometric relationship with

[0036]

[0037] References: Richard Hartley and Andrew Zisserman, “Multiple View Geometry in Computer Vision Second Edition”, Cambridge University Press, March 2004.

[0038] Next, the feature point pair setting unit 13 sets the feature point p ｉ The epipolar line l on which １ The epipolar line l' corresponding to １ In the following equation 3, x is the feature point p ｉ indicates the points on the epipolar line where

[0039]

[0040] Then, the feature points pair setting unit 13 sets an area based on the calculated epipolar line in the other image of the selected pair of images as a projection area, as shown in Fig. 4. In the example of Fig. 4, the feature points pair setting unit 13 sets an area whose distance from the epipolar line is equal to or less than r as the projection area.

[0041] Once the projection area is set in this manner, the feature point pair setting unit 13 performs a matching process to identify a feature point that corresponds to the previously selected feature point from among the feature points present in the set projection area, and sets the selected feature point and the identified feature point as a feature point pair. Note that the matching here is performed by comparing the feature amounts of the respective target feature points, similar to the feature point extraction unit 21 described below.

[0042] The projection vector calculation unit 12 sequentially selects all paired images used to generate the point cloud data and calculates the above-mentioned projection vector for each of the paired images. The feature point pair setting unit then sequentially selects all feature points of one of the images for each of the paired images and sets feature point pairs.

[0043] Here, we will specifically explain the functions of the point cloud data generation unit 14. As shown in Fig. 2, the point cloud data generation unit 14 includes a feature point extraction unit 21, a filtering unit 22, a camera matrix calculation unit 23, a three-dimensional coordinate calculation unit 24, and a bundle adjustment unit 25. The point cloud data generation unit 14 generates initial point cloud data by executing the above-mentioned steps (a) to (l) using these units.

[0044] The feature point extraction unit 21 first acquires multiple image data of an object from an external device, such as an imaging device, a terminal device, or a storage device that stores image data. The feature point extraction unit 21 then calculates, for example, SIFT features or SURF features for each image data to identify feature points on the image. Furthermore, the feature point extraction unit 21 compares the features between feature points to identify corresponding feature point pairs between the images and extracts the identified feature point pairs. The feature point extraction unit 21 also stores information indicating the extracted feature point pairs (hereinafter referred to as "feature point pair information") in the storage unit 15.

[0045] The filtering unit 22 calculates the geometric relationship between the feature points for each combination of feature points, identifies erroneous combinations of feature points based on the calculation results, and eliminates the identified combinations of feature points. The filtering process by the filtering unit 22 is performed using robust estimation, as in conventional SfM.

[0046] The camera matrix calculation unit 23 calculates a camera matrix for each image from which feature points have been extracted, and stores information indicating each calculated camera matrix (hereinafter referred to as “camera matrix information”) in the storage unit 15.

[0047] Specifically, the camera matrix calculation unit 23 first selects one pair of images as an initial pair of images from among the plurality of images from which feature points have been extracted.

[0048] Next, the camera matrix calculation unit 23 calculates the feature point pairs (m １ ~m ５ , m' １ ~m' ５ ) to calculate the camera matrix P of the camera that captured one image and the camera matrix P' of the camera that captured the other image. If the position of the camera that captured one image is taken as the origin, the camera matrices P and P' can be expressed by the following equations 4 and 5, respectively.

[0049]

[0050]

[0051] In the above equation 4, I is the rotation matrix of the camera 41. As shown in FIG. 7, the position of the camera that captured one image is the origin, so I=(1,1,1). Also, in the above equation 5, R is the rotation matrix of the camera that captured the other image (R=(R ｘ ,R ｙ , R ｚ )) t is a translation matrix and corresponds to the three-dimensional coordinates of the position of the camera that captured the other image (t = (t ｘ ,t ｙ , t ｚ )).

[0052] Then, the camera matrix calculation unit 23 calculates R and t by solving the equations shown in the following formulas 6 to 8 using the coordinates of each feature point. In formulas 6 to 8, m hat is m(m １ ~m ５ ) is the coordinate on one image obtained by normalizing m' (m' １ ~m' ５ ) are the coordinates on the other image obtained by normalizing E, the essential matrix, and K, the camera calibration matrix.

[0053]

[0054]

[0055]

[0056] The calibration matrix K can be calculated from the following equations 9 and 10. ｘ , c ｙ is the center coordinate of the camera.

[0057]

[0058]

[0059] Next, the three-dimensional coordinate calculation unit 24 calculates the three-dimensional coordinates of the position of each camera, and calculates the three-dimensional coordinates M (M １ ~M ５Specifically, the three-dimensional coordinate calculation unit 24 calculates the three-dimensional coordinate M by solving the following equation 11. Furthermore, the matrix A in equation 11 is shown in equation 12. In equation 12, p ｉＴ are the rows of the camera matrix P, and p' ｉＴ are the rows of the camera matrix P′.

[0060]

[0061]

[0062] The bundle adjustment unit 25 performs bundle adjustment to minimize the errors of the three-dimensional coordinates M and the camera matrices P and p′ calculated for the initial pair of images.

[0063] When bundle adjustment is performed on the initial pair of images, the camera matrix calculation unit 23 newly selects one image from among images other than the initial pair of images, from which feature points have been extracted, and sets the newly selected image and one of the initial pair of images as a new pair of images.

[0064] Then, the camera matrix calculation unit 23 calculates the feature points (m'') of the newly selected image corresponding to the feature points of the image selected from the initial pair of images. １ ~m'' ３ ) and sets feature point pairs using these feature points. Then, the camera matrix calculation unit 23 calculates the camera matrix P ｎ Calculate the camera matrix P ｎ can be expressed by the following equation 13.

[0065]

[0066] Specifically, the camera matrix calculation unit 23 calculates the camera matrix P of the camera that captured the newly selected image by solving the equation shown in the following Expression 14 using the specified feature points of the image selected from the initial pair of images. ｎ R ｎ and t ｎ Calculate.

[0067]

[0068] In Equation 14, Mｉ is the three-dimensional coordinate of the feature point in the newly selected image that is common to the image selected from the initial pair of images. ｉ The coordinates of the feature points in the newly selected image are normalized. The coordinates of the feature points in the newly selected image are the camera and m that captured the newly selected image, as shown in the following equation (15). ｉ Indicates the distance to the hat.

[0069]

[0070] Next, the three-dimensional coordinate calculation unit 24 calculates the camera matrix P ｎ R ｎ and t ｎ By solving the above equation 11 using ｉ Furthermore, when new three-dimensional coordinates are calculated, the bundle adjustment unit 25 performs bundle adjustment on the newly calculated three-dimensional coordinates and camera matrix.

[0071] The point cloud data generator 14 repeatedly performs the above-described process to calculate the three-dimensional coordinates of the object, and a set of the calculated three-dimensional coordinates becomes point cloud data. The point cloud data generator 14 stores the generated point cloud data in the storage unit 15.

[0072] Furthermore, when a new feature points pair is set by the feature points pair setting unit 13, the filtering unit 22, the camera matrix calculation unit 23, the three-dimensional coordinate calculation unit 24, and the bundle adjustment unit 25 execute the processing again to generate new point cloud data. Note that in this case, the camera matrix calculated when the initial point cloud data was generated may be used without executing the processing by the camera matrix calculation unit 23.

[0073] [Device Operation] Next, an example of the operation of the image processing device will be described using Figures 5 and 6. In the following description, Figures 1 to 4 will be referenced as appropriate. In addition, in the first embodiment, an image processing method is implemented by operating the image processing device 10. Therefore, the description of the image processing method will be replaced by the following description of the operation of the image processing device 10.

[0074] First, the process of generating the initial point cloud data will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing an example of the process of generating the initial point cloud data of the image processing device.

[0075] As shown in FIG. 5, first, the feature point extraction unit 21 acquires a plurality of image data of an object from an external device, calculates the feature amount for each image data, and extracts feature points (step A1).

[0076] Furthermore, the feature point extraction unit 21 executes matching of the feature points extracted in step A1 between the image data to extract feature point pairs (step A2).

[0077] Next, the filtering unit 22 calculates the geometric relationship between the feature points for each combination of feature points, identifies erroneous feature point pairs based on the calculation results, and eliminates the identified feature point pairs (step A3).

[0078] Next, after the processing of step A3 is performed, the camera matrix calculation unit 23 selects one pair of images as an initial pair of images from among the plurality of images from which feature points have been extracted (step A4).

[0079] Next, the camera matrix calculation unit 23 identifies feature point pairs from the feature points extracted from each of the initial pair of images, and calculates the camera matrix P of the camera that captured one of the images and the camera matrix P' of the camera that captured the other image using the identified feature point pairs (step A5).

[0080] Next, for each initial pair of images, the three-dimensional coordinate calculation unit 24 calculates the three-dimensional coordinates of the camera that captured the image using the camera matrix calculated in step A5. Then, the three-dimensional coordinate calculation unit 24 calculates the three-dimensional coordinates of the feature points pairs using the calculated three-dimensional coordinates and the rotation matrix calculated when calculating the camera matrix in step A5 (step A6).

[0081] Next, the bundle adjustment unit 25 performs bundle adjustment to minimize the errors in the three-dimensional coordinates and each camera matrix calculated for the initial pair of images (step A7).

[0082] Next, the camera matrix calculation unit 23 newly selects one image from among the images other than the initial pair of images from which feature points have been extracted in step A1, and sets the newly selected image and one of the initial pair of images as a new pair of images (step A8).

[0083] Next, the camera matrix calculation unit 23 identifies feature point pairs for the new pair of images, and calculates the camera matrix P ｎ is calculated (step A9).

[0084] Next, for the newly selected image, the three-dimensional coordinate calculation unit 24 calculates the three-dimensional coordinates of the camera that captured the image using the camera matrix calculated in step A9. Then, the three-dimensional coordinate calculation unit 24 calculates the three-dimensional coordinates of the feature points pair identified in step A9 using the calculated three-dimensional coordinates and the rotation matrix calculated when calculating the camera matrix in step A9 (step A10).

[0085] Next, the bundle adjustment unit 25 performs bundle adjustment to minimize the errors in the three-dimensional coordinates and each camera matrix calculated for the pair of images selected in step A8 (step A11).

[0086] Next, the camera matrix calculation unit 23 determines whether or not the processing has been completed for all the image data acquired in step A1 (step A12).

[0087] If the result of the determination in step A12 is that processing has not been completed for all of the image data acquired in step A1, the camera matrix calculation unit 23 executes step A8 again. On the other hand, if the result of the determination in step A12 is that processing has been completed for all of the image data acquired in step A1, the point cloud data is complete. Therefore, the point cloud data generation unit 14 ends processing and stores the generated point cloud data in the storage unit 15.

[0088] Next, a process for generating new point cloud data will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing an example of a process for setting new feature point pairs in an image processing device.

[0089] First, the camera matrix acquisition unit 11 acquires the camera matrix of each image from the camera matrix information stored in the storage unit 15 (step B1).

[0090] Next, the projection vector calculation unit 12 selects a pair of images and calculates the camera matrix [R １ｃｗ |t １ｃｗ ] and the other camera matrix [R ２ｃｗ |t ２ｃｗ ] and the projection vector [R γ |t γ ] is calculated (step B2).

[0091] Next, the feature point pair setting unit 13 selects one of the feature points of one of the pair of images (step B3).

[0092] Next, the feature point pair setting unit 13 uses the selected feature points and the calculated projection vector to calculate an epipolar line in the other image of the pair of images, and sets an area based on the epipolar line in the other image of the pair of images as a projection area (step B4).

[0093] Furthermore, the feature points pair setting unit 13 performs a matching process to identify, from among the feature points present in the set projection area, a feature point that corresponds to the feature point selected in step B3, and sets the selected feature point and the identified feature point as a feature points pair (step B5).

[0094] Next, the feature points pair setting unit 13 determines whether or not the processing has been completed for all feature points in one of the paired images (step B6).

[0095] If it is determined in step B6 that processing has not been completed for all feature points in one of the paired images, the feature points pair setting unit 13 executes step B3 again.

[0096] On the other hand, if the result of the determination in step B6 shows that processing has been completed for all feature points in one of the paired images, the feature points pair setting unit 13 updates the feature points pair information stored in the memory unit 15 and determines whether processing has been completed for all of the image data acquired in step A1 (step B7).

[0097] If it is determined in step B7 that processing has not been completed for all image data acquired in step A1, step B2 is executed again.

[0098] On the other hand, if the result of the determination in step B7 is that processing has been completed for all image data acquired in step A1, the point cloud data generator 14 generates new point cloud data of the object using the feature points pairs set by the feature points pair setting unit 13 (step B8). Upon completion of step B8, the processing in the image processing device 10 ends.

[0099] As described above, after steps A1 to A12 are executed to generate point cloud data of the object, steps B1 to B5 are executed to reset feature points pairs using the projection vector, and then new point cloud data is generated. Therefore, when generating the new point cloud data, even if feature points pairs that would normally be determined to be inliers have been determined to be outliers and excluded, they can be restored. As a result, the accuracy of the point cloud data is greatly improved.

[0100] [Program] The program may be any program that causes a computer to execute steps B1 to B8 shown in Fig. 6. The program may also cause a computer to further execute steps A1 to A12 shown in Fig. 5. By installing and executing the program on a computer, the image processing device 10 and the information processing method can be realized. In this case, the processor of the computer functions as a camera matrix acquisition unit 11, a projection vector calculation unit 12, a feature point pair setting unit 13, and a point cloud data generation unit 14 and performs processing.

[0101] In the present embodiment, the storage unit 15 may be realized by storing the data files constituting the storage unit 15 in a storage device such as a hard disk provided in the computer, or may be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0102] The program may also be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the camera matrix acquisition unit 11, the projection vector calculation unit 12, the feature point pair setting unit 13, and the point cloud data generation unit 14.

[0103] Second Embodiment Hereinafter, in a second embodiment, another example of an image processing device, an image processing method, and a program will be described with reference to FIG.

[0104] In the second embodiment, the image processing device has the same configuration as the image processing device 10 shown in Fig. 2. In the following, Fig. 2 will be referred to. However, in the second embodiment, the image processing device 10 differs from that in the first embodiment in terms of the function of the feature points pair setting unit 13. The following description will focus on the differences.

[0105] The feature point pair setting unit 13 first selects feature points of one of the paired images selected by the projection vector calculation unit 12. Next, the feature point pair setting unit 13 calculates the depth of the selected feature points using the point cloud data generated initially.

[0106] Next, the feature point pair setting unit 13 projects the selected feature point onto the other image of the selected pair of images using the projection vector calculated by the projection vector calculation unit 12 and the calculated depth. Then, the feature point pair setting unit 13 sets an area centered on the projected point as a projection area.

[0107] Here, the processing in the feature points pair setting unit 13 will be specifically described with reference to Fig. 7. Fig. 7 is a diagram showing the processing of projecting feature points of one image onto another image using depth.

[0108] 7, the depth is the distance from a feature point on an image to a feature point in real space. Therefore, the feature point pair setting unit 13 calculates the depth Z shown in FIG. 7 from the three-dimensional coordinates of the camera position calculated by the three-dimensional coordinate calculation unit 24 and the three-dimensional coordinates of feature point A. ｃ１ Calculate.

[0109] Then, the feature point pair setting unit 13 uses the following equations 16 and 17 to set the feature point p １ (x １１ , y １１ ) onto the other image, point C ２１ coordinates (x ２１ , y ２１ ) is calculated. In addition, the feature point pair setting unit 13 calculates the projected point C ２１ A region with a radius r as the center is set as the projection region.

[0110]

[0111]

[0112] In the above formulas 16 and 17, f ｘ and f ｙ indicates the focal length of the camera that captured one of the images. ｘ and c ｙ indicates the principal point (center coordinates of the camera) of the camera that captured one of the images.

[0113] Also in the second embodiment, steps A1 to A12 shown in Fig. 5 and steps B1 to B8 shown in Fig. 6 are executed. Any program may be used as long as it causes a computer to execute these steps.

[0114] However, in the second embodiment, the content of step B4 differs from that of the first embodiment. In the second embodiment, in step B4, the feature points pair setting unit 13 first calculates the depth of the selected feature point using the point cloud data generated initially. Next, the feature points pair setting unit 13 projects the selected feature point onto the other image of the selected pair of images using the projection vector and the depth. Then, the feature points pair setting unit 13 sets an area centered on the projected point as a projection area.

[0115] In this way, in the second embodiment, as in the first embodiment, feature points pairs are reset using projection vectors, and then new point cloud data is generated. Therefore, even when the second embodiment is used, when generating new point cloud data, even if a feature points pair that should have been determined to be an inlier has been determined to be an outlier and excluded, it can be restored. As a result, the accuracy of the point cloud data is greatly improved.

[0116] [Physical Configuration] A computer that implements the image processing device 10 by executing the programs in the first and second embodiments will now be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of a computer that implements the image processing device.

[0117] 8, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0118] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0119] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0120] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0121] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0122] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0123] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0124] The image processing device 10 can be realized not by a computer with a program installed, but by hardware corresponding to each part, such as an electronic circuit. Furthermore, the image processing device 10 may be realized in part by a program and in the remaining part by hardware. In the embodiment, the computer is not limited to the computer shown in FIG. 8.

[0125] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 15) described below, but are not limited to the following descriptions.

[0126] (Supplementary Note 1) An image processing device comprising: a camera matrix acquisition unit that, when point cloud data of an object is generated from a plurality of images including the object, acquires, for each of the plurality of images, a camera matrix of a camera that captured the image; a projection vector calculation unit that selects a pair of images from the plurality of images, and calculates a projection vector from one of the selected pair of images to the other of the selected pair of images using the camera matrix of each of the selected pair of images; and a feature point pair setting unit that selects a feature point in one of the selected pair of images, and sets, in the other of the selected pair of images, a projection area onto which the selected feature point is expected to be projected using the selected feature point and the calculated projection vector, identifies a feature point that corresponds to the selected feature point from among feature points present within the set projection area, and sets the selected feature point and the identified feature point as a feature point pair.

[0127] (Supplementary Note 2) The image processing device according to Supplementary Note 1, wherein the feature point pair setting unit calculates an epipolar line in the other image of the selected pair of images using the selected feature point and the calculated projection vector, and further sets an area based on the epipolar line as the projection area.

[0128] (Supplementary Note 3) The image processing device according to Supplementary Note 1, wherein the feature point pair setting unit calculates a depth of the selected feature point using the point cloud data, projects the selected feature point onto the other image of the selected pair of images using the calculated projection vector and the calculated depth, and sets an area centered on the projected point as the projection area.

[0129] (Supplementary Note 4) The image processing device according to Supplementary Note 1, wherein the projection vector calculation unit sequentially selects all paired images used to generate the point cloud data and calculates the projection vector for each of the paired images, and the feature point pair setting unit sequentially selects all of the feature points on one side of the paired images for each of the paired images and sets the feature point pairs.

[0130] (Supplementary Note 5) The image processing device according to Supplementary Note 4, further comprising: a point cloud data generation unit that generates new point cloud data of the object using the set feature point pairs.

[0131] (Supplementary Note 6) An image processing method comprising: a camera matrix acquisition step of, when point cloud data of an object is generated from a plurality of images including the object, acquiring, for each of the plurality of images, a camera matrix of a camera that captured the image; a projection vector calculation step of selecting a pair of images from the plurality of images and calculating a projection vector from one of the selected pair of images to the other of the selected pair of images using the camera matrix of each of the selected pair of images; and a feature point pair setting step of selecting a feature point in one of the selected pair of images, and setting, in the other of the selected pair of images, a projection area onto which the selected feature point is expected to be projected using the selected feature point and the calculated projection vector, identifying a feature point that corresponds to the selected feature point from among feature points present in the set projection area, and setting the selected feature point and the identified feature point as a feature point pair.

[0132] (Supplementary Note 7) The image processing method according to Supplementary Note 6, wherein in the feature point pair setting step, an epipolar line in the other image of the selected pair of images is calculated using the selected feature point and the calculated projection vector, and further, an area based on the epipolar line is set as the projection area.

[0133] (Supplementary Note 8) The image processing method according to Supplementary Note 6, wherein in the feature point pair setting step, a depth of the selected feature point is calculated using the point cloud data, the selected feature point is projected onto the other image of the selected pair of images using the calculated projection vector and the calculated depth, and an area centered on the projected point is set as the projection area.

[0134] (Supplementary Note 9) The image processing method according to Supplementary Note 6, wherein in the projection vector calculation step, all paired images used in generating the point cloud data are selected in order, and the projection vector is calculated for each of all paired images; and in the feature point pair setting step, all of the feature points on one side are selected in order for each of all paired images, and the feature point pair setting is performed.

[0135] (Supplementary Note 10) The image processing method according to Supplementary Note 9, further comprising a point cloud data generating step of generating new point cloud data of the object using the set feature point pairs.

[0136] (Supplementary Note 11) A computer-readable recording medium having recorded thereon a program including instructions for causing a computer to execute the following steps: a camera matrix acquisition step of, when point cloud data of an object is generated from a plurality of images including the object, acquiring, for each of the plurality of images, a camera matrix of a camera that captured the image; a projection vector calculation step of selecting a pair of images from the plurality of images and calculating a projection vector from one of the selected pair of images to the other of the selected pair of images using the camera matrix of each of the selected pair of images; and a feature point pair setting step of selecting a feature point in one of the selected pair of images, and setting, in the other of the selected pair of images, a projection area onto which the selected feature point is expected to be projected using the selected feature point and the calculated projection vector, identifying a feature point that corresponds to the selected feature point from among feature points present in the set projection area, and setting the selected feature point and the identified feature point as a feature point pair.

[0137] (Supplementary Note 12) The computer-readable recording medium according to Supplementary Note 11, wherein in the feature point pair setting step, an epipolar line in the other image of the selected pair of images is calculated using the selected feature point and the calculated projection vector, and further, an area based on the epipolar line is set as the projection area.

[0138] (Supplementary Note 13) The computer-readable recording medium according to Supplementary Note 11, wherein in the feature point pair setting step, a depth of the selected feature point is calculated using the point cloud data, the selected feature point is projected onto the other image of the selected pair of images using the calculated projection vector and the calculated depth, and an area centered on the projected point is set as the projection area.

[0139] (Supplementary Note 14) The computer-readable recording medium according to Supplementary Note 11, wherein in the projection vector calculation step, all paired images used in generating the point cloud data are selected in order, and the projection vector is calculated for each of all paired images; and in the feature point pair setting step, all of the feature points on one side are selected in order for each of all paired images, and the feature point pairs are set.

[0140] (Supplementary Note 15) The computer-readable recording medium according to Supplementary Note 14, wherein the program further includes instructions for causing the computer to execute a point cloud data generation step of newly generating point cloud data of the object using the set feature point pairs.

[0141] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0142] This application claims priority based on Japanese Patent Application No. 2023-137199, filed on August 25, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0143] The present disclosure is useful in the field of image processing for reconstructing the three-dimensional shape of an object from image data.

[0144] REFERENCE SIGNS LIST 10 Image processing device 11 Camera matrix acquisition unit 12 Projection vector calculation unit 13 Feature point pair setting unit 14 Point cloud data generation unit 15 Storage unit 21 Feature point extraction unit 22 Filtering unit 23 Camera matrix calculation unit 24 Three-dimensional coordinate calculation unit 25 Bundle adjustment unit 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. An image processing apparatus comprising: a camera matrix acquisition means for, when point cloud data of an object is generated from a plurality of images including the object, acquiring, for each of the plurality of images, a camera matrix of a camera that captured the image; a projection vector calculation means for selecting a pair of images from the plurality of images, and calculating a projection vector from one of the selected pair of images to the other image using the camera matrix of each of the selected pair of images; and a feature point pair setting means for selecting a feature point in one of the selected pair of images, and setting, in the other image of the selected pair of images, a projection area into which the selected feature point is expected to be projected, using the selected feature point and the calculated projection vector, identifying a feature point corresponding to the selected feature point from among feature points present within the set projection area, and setting the selected feature point and the identified feature point as a feature point pair.

2. The image processing device according to claim 1, wherein the feature point pair setting means uses the selected feature points and the calculated projection vector to calculate an epipolar line in the other image of the selected pair of images, and further sets an area based on the epipolar line as the projection area.

3. The image processing device according to claim 1, wherein the feature point pair setting means calculates the depth of the selected feature point using the point cloud data, projects the selected feature point onto the other image of the selected pair of images using the calculated projection vector and the calculated depth, and sets an area centered on the projected point as the projection area.

4. The image processing device according to claim 1, wherein the projection vector calculation means sequentially selects all paired images used in generating the point cloud data and calculates the projection vector for each of the paired images, and the feature point pair setting means sequentially selects all of the feature points of one of the paired images for each of the paired images and sets the feature point pairs.

5. The image processing device according to claim 4, further comprising a point cloud data generating means for generating new point cloud data of the object using the set feature point pairs.

6. An image processing method comprising the steps of: when point cloud data of an object is generated from a plurality of images including the object, acquiring, for each of the plurality of images, a camera matrix of a camera that captured the image; selecting a pair of images from the plurality of images, and calculating a projection vector from one of the selected pair of images to the other image using the camera matrix of each of the selected pair of images; selecting a feature point in one of the selected pair of images, and setting, in the other image of the selected pair of images, a projection area into which the selected feature point is expected to be projected, using the selected feature point and the calculated projection vector; identifying a feature point corresponding to the selected feature point from among feature points present within the set projection area; and setting the selected feature point and the identified feature point as a feature point pair.

7. The image processing method according to claim 6, wherein in setting the feature points pair, an epipolar line in the other image of the selected pair of images is calculated using the selected feature points and the calculated projection vector, and further, a region based on the epipolar line is set as the projection region.

8. The image processing method according to claim 6, wherein in setting the feature point pair, the point cloud data is used to calculate a depth of the selected feature point, the calculated projection vector and the calculated depth are used to project the selected feature point onto the other image of the selected pair of images, and an area centered on the projected point is set as the projection area.

9. The image processing method according to claim 6, wherein in calculating the projection vector, all paired images used in generating the point cloud data are selected in order, and the projection vector is calculated for each of the paired images; and in setting the feature points pairs, all of the feature points of one of the paired images are selected in order, for each of the paired images, and the feature points pairs are set.

10. The image processing method according to claim 9, further comprising generating new point cloud data of the object using the set feature point pairs.

11. A computer-readable recording medium having recorded thereon a program including instructions to: cause a computer to, when point cloud data of an object is generated from multiple images including the object, acquire, for each of the multiple images, a camera matrix of the camera that captured the image; select a pair of images from among the multiple images, and calculate a projection vector from one of the selected pair of images to the other image using the camera matrix of each of the selected pair of images; select a feature point in one of the selected pair of images, and set, in the other image of the selected pair of images, a projection area into which the selected feature point is expected to be projected, using the selected feature point and the calculated projection vector; identify a feature point corresponding to the selected feature point from among feature points present within the set projection area; and set the selected feature point and the identified feature point as a feature point pair.

12. The computer-readable recording medium according to claim 11, wherein the program causes the computer to: calculate an epipolar line in the other image of the selected pair of images using the selected feature points and the calculated projection vector in setting the feature point pair; and further set an area based on the epipolar line as the projection area.

13. The computer-readable recording medium according to claim 11, wherein the program causes the computer to: calculate the depth of the selected feature point using the point cloud data in setting the feature point pair; project the selected feature point onto the other image of the selected pair of images using the calculated projection vector and the calculated depth; and set an area centered on the projected point as the projection area.

14. A computer-readable recording medium as described in claim 11, wherein the program causes the computer to: in calculating the projection vector, sequentially select all paired images used in generating the point cloud data, and calculate the projection vector for each of all paired images; and in setting the feature point pairs, in sequentially select all of the feature points of one of the paired images for each of all paired images, and execute the setting of the feature point pairs.

15. The computer-readable recording medium according to claim 14, wherein the program further includes instructions for causing the computer to newly generate point cloud data of the object using the set feature point pairs.

Images