Three-dimensional position calculation device, three-dimensional position calculation method, and three-dimensional position calculation program
The method calculates three-dimensional positions of image features from a single photograph by generating a virtual camera image and associating pixels with a three-dimensional point cloud, addressing the inefficiency of conventional techniques that require multiple images.
Patent Information
- Application Number
- PCT/JP2023/019974
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2024-12-05
AI Technical Summary
Conventional techniques for calculating the three-dimensional position of image features require multiple photographs taken from different positions and orientations, which is inefficient and time-consuming.
A device and method that calculate the three-dimensional position of image features using a single photograph by generating a virtual camera image and associating pixels with a three-dimensional point cloud, allowing for the determination of corresponding points without the need for multiple images.
Enables the calculation of three-dimensional positions of image features from a single photograph, reducing the effort and time required to capture multiple images, while accurately associating image features with their corresponding points in a three-dimensional space.
Smart Images

Figure JP2023019974_05122024_PF_FP_ABST
Abstract
Description
Three-dimensional position calculation device, three-dimensional position calculation method, and three-dimensional position calculation program
[0001] The present invention relates to a three-dimensional position calculation device, a three-dimensional position calculation method, and a three-dimensional position calculation program.
[0002] 2. Description of the Related Art In the field of three-dimensional measurement of a subject, there is a technique for calculating the three-dimensional position of image features of a subject captured using two or more photographs.
[0003] These techniques use two or more photographic images of a subject, detect image features from each image, and calculate corresponding points between the images by feature matching. Using the corresponding points and information on the shooting position and orientation of each image, the 3D coordinate position of the image features is calculated.
[0004] Japanese Patent Application Publication No. 2014-52977
[0005] Conventional techniques for calculating the three-dimensional position of image features require the preparation of at least two or more photographs of the same part taken from different photographing positions and orientations.
[0006] The present invention has been made in light of the above circumstances, and its object is to provide a technique for calculating the three-dimensional position of an image feature without requiring two or more photographs.
[0007] One aspect of the present invention is a three-dimensional position calculation device that calculates the three-dimensional positions of image features, which are points in a three-dimensional point cloud corresponding to image feature points in a photographic image. The three-dimensional position calculation device includes a photographic image / three-dimensional point cloud correspondence point calculation unit that receives, as input, photographic image information for one photographic image, image feature points and image feature amounts of the image feature points, and three-dimensional point cloud information for a three-dimensional point cloud related to the photographic image, and calculates the three-dimensional positions of the image feature points based on the photographic image information, image feature points, image feature amounts, and three-dimensional point cloud information.
[0008] One aspect of the present invention is a three-dimensional position calculation method for calculating the three-dimensional positions of image features, which are points in a three-dimensional point cloud corresponding to image feature points in a photographic image. The first three-dimensional position calculation method includes the steps of receiving, as input, photographic image information for one photographic image, image feature points of the photographic image and image feature values for each image feature point, and three-dimensional point cloud information for a three-dimensional point cloud related to the photographic image, generating a virtual camera image by projectively transforming the three-dimensional point cloud, and calculating the three-dimensional positions of the image feature by associating pixels of the photographic image with pixels of the virtual camera image at the same positions as those of the photographic image.
[0009] The second three-dimensional position calculation method includes the steps of receiving as input photographic image information for one photographic image, image feature points of the photographic image and image feature quantities for each image feature point, and three-dimensional point cloud information for a three-dimensional point cloud related to the photographic image; generating a virtual camera in the three-dimensional point cloud space at the same position and orientation as the camera that captured the photographic image, and drawing a virtual camera screen that projects the three-dimensional point cloud space using the virtual camera; calculating corresponding pixels between the photographic image and the virtual camera screen; and calculating the three-dimensional position of the image feature quantities by associating the three-dimensional point cloud at pixels of the virtual camera screen that are at the same positions as pixels of the photographic image.
[0010] One aspect of the present invention is a three-dimensional position calculation program that calculates the three-dimensional position of an image feature from a corresponding point in a three-dimensional point cloud to an image feature point in a photographic image. The three-dimensional position calculation program causes a computer having a processor and a storage device to execute at least some of the functions of the components of the three-dimensional position calculation device.
[0011] According to the present invention, a technique is provided for calculating the three-dimensional position of an image feature without requiring two or more photographs.
[0012] FIG. 1 is a block diagram showing the functional configuration of a three-dimensional position calculation device according to an embodiment. FIG. 2 is a flowchart showing the flow of processing performed by the three-dimensional position calculation device according to an embodiment. FIG. 3 is a block diagram showing the functional configuration of a three-dimensional position calculation device having a photographic image / three-dimensional point cloud correspondence point calculation unit according to a first configuration example. FIG. 4 is a flowchart showing the flow of processing performed by the photographic image / three-dimensional point cloud correspondence point calculation unit according to the first configuration example. FIG. 5 is a diagram schematically showing the concept of processing performed by the photographic image / three-dimensional point cloud correspondence point calculation unit according to the first configuration example. FIG. 6 is a block diagram showing the functional configuration of a three-dimensional position calculation device having a photographic image / three-dimensional point cloud correspondence point calculation unit according to a second configuration example. FIG. 7 is a flowchart showing the flow of processing performed by the photographic image / three-dimensional point cloud correspondence point calculation unit according to the second configuration example. FIG. 8 is a diagram schematically showing the concept of processing performed by the photographic image / three-dimensional point cloud correspondence point calculation unit according to the second configuration example. FIG. 9 is a diagram schematically showing optimal correspondence point determination processing performed by the photographic image / three-dimensional point cloud correspondence point calculation unit. Fig. 10 is a diagram showing a hidden point removal process from a virtual camera viewpoint executed by a photographic image / three-dimensional point cloud corresponding point calculation unit. Fig. 11 is a diagram showing a hidden point removal process from a virtual camera viewpoint executed by a photographic image / three-dimensional point cloud corresponding point calculation unit. Fig. 12 is a block diagram showing a hardware configuration of a three-dimensional position calculation device according to an embodiment.
[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0014] (Basic Functional Configuration) First, the basic functional configuration of a three-dimensional position calculation device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the basic functional configuration of a three-dimensional position calculation device 10 according to an embodiment. The three-dimensional position calculation device 10 according to an embodiment is a device that calculates three-dimensional coordinate positions corresponding to pixels of image feature points in a photographic image from photographic image information about one photographic image and a three-dimensional point cloud, which is data of a set of points in three-dimensional space.
[0015] The three-dimensional position calculation device 10 has a photographic image information acquisition unit 20, a three-dimensional point cloud information acquisition unit 30, an image feature calculation unit 40, a photographic image / three-dimensional point cloud correspondence point calculation unit 50, and a three-dimensional position information database (DB) 70 of image features.
[0016] The photographic image information acquisition unit 20 acquires photographic image information for one photographic image. The photographic image information is provided from an external source. The photographic image information includes a camera model, internal parameters, and external parameters for the camera that captured the photographic image. The camera model represents where three-dimensional space is projected on the photographic image plane. For example, the camera model includes a pinhole camera model. The internal parameters are parameters that represent the characteristics of the camera's optical system. The internal parameters are used for image correction, distortion correction, etc. For example, the internal parameters include the focal length, optical center, aspect ratio, etc. The external parameters are parameters that represent the position and orientation of the camera. For example, the external parameters include the camera position, camera orientation, camera tilt, etc. The photographic image information acquisition unit 20 transmits the photographic image information to the image feature amount calculation unit 40. Hereinafter, the camera model for the camera will also be simply referred to as the camera.
[0017] The three-dimensional point cloud information acquisition unit 30 acquires three-dimensional point cloud information. The three-dimensional point cloud information is data of a set of points in three-dimensional space related to a photographic image. The three-dimensional point cloud information is provided from an external source. The three-dimensional point cloud information includes three-dimensional origin information of the point cloud. The three-dimensional point cloud information has been calibrated with the external parameters of the photographic image information. In other words, the position and orientation of the camera that captured the photographic image in the three-dimensional space of the point cloud are known. The three-dimensional point cloud information acquisition unit 30 transmits the three-dimensional point cloud information to the photographic image / 3D point cloud corresponding point calculation unit 50.
[0018] The image feature amount calculation unit 40 calculates image feature points of the photographic image and image feature amounts of each image feature point from the photographic image information received from the photographic image information acquisition unit 20. Image feature points are points that represent prominent parts of an image. Image feature amounts are used for image processing, pattern recognition, and the like. For example, image feature amounts include color, shape, texture, and the like. The image feature points and image feature amounts may be calculated using SIFT (Scale-Invariant Feature Transform) or AKAZE (Accelerated-KAZE), or may be extracted by deep learning. The image feature amount calculation unit 40 transmits the calculated image feature points and image feature amounts to the photographic image / 3D point cloud correspondence point calculation unit 50, along with the camera model, internal parameters, and external parameters of the camera.
[0019] The photographic image / 3D point cloud correspondence point calculation unit 50 generates a virtual camera viewpoint and a virtual camera in the 3D point cloud space according to the camera model, external parameters, and internal parameters. The virtual camera viewpoint includes information such as the position, direction, and field of view of the virtual camera. Furthermore, the photographic image / 3D point cloud correspondence point calculation unit 50 calculates each point in the 3D point cloud corresponding to each image feature point in the 2D screen coordinates of the virtual camera. In other words, the photographic image / 3D point cloud correspondence point calculation unit 50 calculates the 3D position of the image feature of the photographic image. In this way, the photographic image / 3D point cloud correspondence point calculation unit 50 matches the original photographic image with the 3D point cloud. Specific matching methods will be described later. It is preferable to reduce mismatches. Techniques for reducing mismatches include a method of determining the center of gravity of the N nearest points as the optimal matching point, or a method of determining and removing 3D point clouds that should not be visible from the virtual camera. Specific methods will be described later. The photographic image / 3D point cloud corresponding point calculation unit 50 transmits the 3D position information of the corresponding points of the 3D point cloud to the image feature points of the photographic image together with the image feature amounts to the 3D position information DB 70 of image features.
[0020] The image feature three-dimensional position information DB 70 stores three-dimensional position information of image feature amounts of photographic images, that is, three-dimensional position information and image feature amounts of points in the three-dimensional point cloud corresponding to image feature points of the photographic images.
[0021] (Basic Operation) Next, the basic operation of the three-dimensional position calculation device 10 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of processing executed by the three-dimensional position calculation device 10 according to the embodiment.
[0022] In step S1, the photographic image information acquisition unit 20 acquires photographic image information for one photographic image. The details of the photographic image information are as described above. The three-dimensional point cloud information acquisition unit 30 also acquires three-dimensional point cloud information.
[0023] In step S2, the image feature amount calculation unit 40 receives the photographic image information as an input from the photographic image information acquisition unit 20, and calculates the image feature points of the photographic image and the image feature amounts of each image feature point from the photographic image information. The details of the image feature points and the image feature amounts are as described above.
[0024] In step S3, the photographic image / 3D point cloud correspondence point calculation unit 50 receives as input the 3D point cloud information from the 3D point cloud information acquisition unit 30, and receives as input the image feature points and image feature amounts from the image feature amount calculation unit 40, and calculates correspondence points between the photographic image and the 3D point cloud. Specifically, the photographic image / 3D point cloud correspondence point calculation unit 50 calculates the three-dimensional positions of the correspondence points of the 3D point cloud with respect to the image feature points of the photographic image. In other words, the photographic image / 3D point cloud correspondence point calculation unit 50 calculates the three-dimensional positions of the image feature amounts of the photographic image. Details of the calculation of the three-dimensional positions of the correspondence points of the 3D point cloud with respect to the image feature points of the photographic image will be described later.
[0025] In step S4, the image feature 3D position information DB 70 receives and stores the 3D position information of the image feature of the photographic image, i.e., the 3D position information and image feature of the corresponding points in the 3D point cloud to the image feature points of the photographic image, from the photographic image / 3D point cloud corresponding point calculation unit 50.
[0026] (First Configuration Example of Photographic Image / 3D Point Cloud Correspondence Point Calculation Unit 50) Next, a first configuration example of the photographic image / 3D point cloud correspondence point calculation unit 50 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of a three-dimensional position calculation device 10 having a photographic image / 3D point cloud correspondence point calculation unit 50 according to the first configuration example. Only the photographic image / 3D point cloud correspondence point calculation unit 50 will be described below.
[0027] The photographic image / 3D point cloud corresponding point calculation unit 50 according to the first configuration example calculates the 3D coordinate positions of corresponding pixels using the 3D point cloud as a starting point. To this end, the photographic image / 3D point cloud corresponding point calculation unit 50 performs a process of projectively transforming the 3D point cloud onto the virtual camera image to calculate the 3D coordinate positions.
[0028] The photographic image / 3D point cloud correspondence point calculation unit 50 according to the first configuration example includes a point cloud identification ID assignment unit 51, a virtual camera viewpoint hidden point removal unit 52, a virtual camera image generation unit 53, a virtual camera image / image feature amount correspondence point calculation unit 54, and a 3D point cloud / image feature amount correspondence point calculation unit 56. The virtual camera image / image feature amount correspondence point calculation unit 54 also includes an optimal correspondence point determination unit 55.
[0029] The point cloud identification ID assigning unit 51 assigns a point cloud identification ID to each point of the three-dimensional point cloud received from the three-dimensional point cloud information acquiring unit 30 and records the three-dimensional point cloud coordinate position.
[0030] The virtual camera viewpoint hidden point removal unit 52 determines and removes points (hidden points) that would not normally be visible from the virtual camera viewpoint.
[0031] The virtual camera image generation unit 53 performs projective transformation on the 3D point cloud using the camera model and information on the external and internal parameters of the camera to generate a virtual camera image. The virtual camera image is a virtual image, and the pixels of the virtual camera image can be said to be virtual pixels. For this reason, hereinafter, the pixels of the virtual camera image will be referred to as virtual pixels.
[0032] The virtual camera image / image feature quantity corresponding point calculation unit 54 applies the same two-dimensional screen coordinate system as that of the photographic image to the virtual camera image, and associates pixels of the two-dimensional screen coordinate system of the photographic image with pixels of the two-dimensional screen coordinate system of the virtual camera image at the same positions. In this way, the virtual camera image / image feature quantity corresponding point calculation unit 54 calculates corresponding points of the virtual camera image for image feature points of the photographic image.
[0033] The optimal corresponding point determination unit 55 determines the optimal corresponding point from a plurality of point groups in order to reduce erroneous correspondence between virtual pixels of the virtual camera image and pixels of the photographic image, which is determined by the virtual camera image / image feature amount corresponding point calculation unit 54.
[0034] The 3D point cloud / image feature quantity corresponding point calculation unit 56 calculates the 3D positions of the corresponding points of the 3D point cloud with respect to the image feature points of the photographic image. The 3D point cloud / image feature quantity corresponding point calculation unit 56 also associates the image feature quantities of the photographic image with the information on the 3D positions of the calculated corresponding points. In this way, the 3D positions of the image feature quantities of the photographic image are calculated.
[0035] (Example of Operation of Photographic Image / 3D Point Cloud Correspondence Point Calculation Unit 50 According to First Configuration Example) Next, the processing executed by the photographic image / 3D point cloud correspondence point calculation unit 50 according to the first configuration example will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a sub-flowchart showing the flow of the processing for calculating corresponding points between a photographic image and a 3D point cloud by the photographic image / 3D point cloud correspondence point calculation unit 50 according to the first configuration example. Fig. 5 is a diagram schematically showing the processing for calculating corresponding points between a photographic image and a 3D point cloud by the photographic image / 3D point cloud correspondence point calculation unit 50 according to the first configuration example.
[0036] In step S11, the point cloud identification ID assigning unit 51 assigns a point cloud identification ID to each point in the three-dimensional point cloud and records the three-dimensional point cloud coordinate position (x, y, z), as shown on the left side of Fig. 5. For example, the point cloud identification ID assigning unit 51 records a point cloud identification ID (000001) and a three-dimensional point cloud coordinate position (x1, y1, z1) for one point in the three-dimensional point cloud. By performing the same process for each point in the three-dimensional point cloud, a data set of the point cloud identification ID and the three-dimensional point cloud coordinate position (x, y, z) is obtained.
[0037] In step S12, the virtual camera viewpoint hidden point removal unit 52 of the virtual camera vc performs a process of determining and removing points (hidden points) that would not normally be visible from the virtual camera viewpoint. The hidden point removal process from the virtual camera viewpoint will be described later.
[0038] In step S13, the virtual camera image generation unit 53 performs projective transformation of the 3D point cloud onto a virtual camera image using information on the external and internal parameters of the camera, as shown in the center of Fig. 5, to generate the virtual camera image. At this time, the point cloud identification ID is not changed, but the coordinate position is updated. For example, the virtual camera image generation unit 53 changes the 3D point cloud coordinate position (x1, y1, z1) to the virtual pixel coordinate position (xa_1, ya_1).
[0039] Projective transformation is performed using general image processing based on the camera model, internal parameters, and external parameters.
[0040] In step S14, the virtual camera image / image feature quantity correspondence point calculation unit 54 applies the same two-dimensional screen coordinate system as the photographic image shown on the right side of Fig. 5 to the virtual camera image, as shown in the center of Fig. 5. Furthermore, the virtual camera image / image feature quantity correspondence point calculation unit 54 associates pixels in the two-dimensional screen coordinate system of the photographic image with virtual pixels in the two-dimensional screen coordinate system of the virtual camera image that are at the same positions. In this way, the virtual camera image / image feature quantity correspondence point calculation unit 54 calculates correspondence points in the virtual camera image for image feature points in the photographic image.
[0041] For this reason, the virtual camera image / image feature correspondence point calculation unit 54 assigns a pixel identification ID to each pixel of the photographic image and records the two-dimensional pixel coordinate position, as shown on the right side of FIG. 5 . For example, the virtual camera image / image feature correspondence point calculation unit 54 assigns a pixel identification ID (000111) to one pixel of the photographic image and records its two-dimensional pixel coordinate position (xb_111, yb_111). In this example, the virtual pixel coordinate position (xa_1, ya_1) in the two-dimensional screen coordinate system of the virtual camera image matches the pixel coordinate position (xb_111, yb_1111) in the two-dimensional screen coordinate system of the photographic image. The virtual camera image / image feature correspondence point calculation unit 54 assigns a point cloud identification ID (000001) to the pixel with the pixel identification ID (000111) and the pixel coordinate position (xb_111, yb_1111).
[0042] In the above-described matching process, the correspondence may be determined using RGB information and semantic information of each pixel of the photographic image and the virtual camera image. That is, even if the pixels have the same coordinates, if the RGB information is equal to or greater than a threshold or if the semantic information is different, the correspondence is not performed.
[0043] The above-mentioned matching process only needs to be performed on pixels of image feature points among the pixels of the photographic image.
[0044] In the above-described correspondence processing, the best corresponding point determination unit 55 determines the best corresponding point from a plurality of point groups in order to reduce erroneous correspondence between virtual pixels of the virtual camera image and pixels of the photographic image, which is performed by the virtual camera image / image feature amount correspondence point calculation unit 54. The best corresponding point determination processing will be described later.
[0045] In step S15, the three-dimensional point cloud / image feature amount corresponding point calculation unit 56 calculates the three-dimensional positions of the corresponding points of the three-dimensional point cloud with respect to the image feature points of the photographic image.
[0046] For example, the 3D point cloud / image feature corresponding point calculation unit 56 searches for a point cloud identification ID that matches the point cloud identification ID assigned to a pixel of the photographic image by the virtual camera image / image feature corresponding point calculation unit 54 from the data set of point cloud identification IDs and 3D point cloud coordinate positions (x, y, z) obtained by the point cloud identification ID assigning unit 51. The 3D point cloud / image feature corresponding point calculation unit 56 associates the pixel of the image feature point of the photographic image that has the matching point cloud identification ID with the point cloud coordinate positions (x, y, z) of the point in the 3D point cloud. This calculates the three-dimensional position of the corresponding point in the 3D point cloud for the image feature point of the photographic image.
[0047] The 3D point cloud / image feature corresponding point calculation unit 56 also associates the image feature with the information on the 3D position of the calculated corresponding point, thereby calculating the 3D position of the image feature.
[0048] (Second Configuration Example of Photographic Image / 3D Point Cloud Correspondence Point Calculation Unit 50) Next, a first configuration example of the photographic image / 3D point cloud correspondence point calculation unit 50 will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the functional configuration of a three-dimensional position calculation device 10 having a photographic image / 3D point cloud correspondence point calculation unit 50 according to the second configuration example. Only the photographic image / 3D point cloud correspondence point calculation unit 50 will be described below.
[0049] The photographic image / 3D point cloud corresponding point calculation unit 50 according to the second configuration example calculates the point cloud coordinate positions of the corresponding points in the 3D point cloud, starting from the pixel of the photographic image. For this purpose, the calculation process is performed by drawing the screen (field of view) of the virtual camera.
[0050] The photographic image / 3D point cloud correspondence point calculation unit 50 according to the second configuration example includes a point cloud identification ID assignment unit 61, a virtual camera viewpoint hidden point removal unit 62, a virtual camera screen drawing unit 63, a virtual camera / photographic image correspondence pixel calculation unit 64, a virtual camera pixel / 3D point cloud correspondence point calculation unit 65, and a 3D point cloud / image feature quantity correspondence point calculation unit 67. The virtual camera pixel / 3D point cloud correspondence point calculation unit 65 also includes an optimal correspondence point determination unit 66.
[0051] The functions of the point cloud identification ID assignment unit 61, the hidden point removal unit from a virtual camera viewpoint 62, the optimal corresponding point determination unit 66, and the 3D point cloud / image feature corresponding point calculation unit 67 in the photographic image / 3D point cloud corresponding point calculation unit 50 according to the second configuration example are the same as those of the point cloud identification ID assignment unit 51, the hidden point removal unit from a virtual camera viewpoint 52, the optimal corresponding point determination unit 55, and the 3D point cloud / image feature corresponding point calculation unit 56 in the photographic image / 3D point cloud corresponding point calculation unit 50 according to the first configuration example, respectively.
[0052] In the following, only the different parts, that is, the virtual camera screen drawing unit 63, the virtual camera / photographic image corresponding pixel calculation unit 64, and the virtual camera pixel / 3D point cloud corresponding point calculation unit 65 will be described.
[0053] The virtual camera screen drawing unit 63 generates a virtual camera in the three-dimensional point cloud space according to the camera model and internal parameters of the camera that took the photographic image, places the virtual camera according to the position and orientation of the camera's external parameters, and draws a virtual camera screen that projects the three-dimensional point cloud space using the virtual camera.
[0054] The virtual camera / photographic image corresponding pixel calculation unit 64 applies the same two-dimensional screen coordinate system as that of the photographic image to the virtual camera screen, and associates pixels of the two-dimensional screen coordinate system of the photographic image with pixels of the two-dimensional screen coordinate system of the virtual camera screen at the same positions. In this way, the virtual camera / photographic image corresponding pixel calculation unit 64 calculates corresponding pixels between the photographic image and the virtual camera screen.
[0055] The virtual camera pixel / 3D point cloud correspondence calculation unit 65 projects a straight line from the camera lens position of the virtual camera to each virtual pixel on the virtual camera screen and extracts a point cloud that lies on the line. Furthermore, the virtual camera pixel / 3D point cloud correspondence calculation unit 65 associates the point cloud identification ID of the extracted point cloud with each virtual pixel on the virtual camera screen. This associates each image feature point of the photographic image with the three-dimensional position of each point of the three-dimensional point cloud. In other words, the three-dimensional position of the image feature of the photographic image is calculated.
[0056] (Example of Operation of Photographic Image / 3D Point Cloud Correspondence Point Calculation Unit 50 According to Second Configuration Example) Next, the processing executed by the photographic image / 3D point cloud correspondence point calculation unit 50 according to the second configuration example will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a sub-flowchart showing the flow of the processing for calculating corresponding points between a photographic image and a 3D point cloud by the photographic image / 3D point cloud correspondence point calculation unit 50 according to the second configuration example. Fig. 8 is a diagram schematically showing the processing for calculating corresponding points between a photographic image and a 3D point cloud by the photographic image / 3D point cloud correspondence point calculation unit 50 according to the second configuration example.
[0057] In step S21, the point cloud identification ID assigning unit 61 assigns a point cloud identification ID to each point of the three-dimensional point cloud and records the three-dimensional point cloud coordinate position (x, y, z), as shown on the left side of Fig. 8. This results in a data set of the point cloud identification ID and the three-dimensional point cloud coordinate position (x, y, z).
[0058] In step S22, the hidden point removal unit 62 for the virtual camera vc performs a process of determining and removing points (hidden points) that would not normally be visible from the virtual camera viewpoint. The hidden point removal process for the virtual camera viewpoint will be described later.
[0059] In step S23, the virtual camera screen rendering unit 63 generates a virtual camera vc in the three-dimensional point cloud space according to the internal parameters of the camera that captured the photographic image, as shown on the left side of Fig. 8. Furthermore, the virtual camera screen rendering unit 63 places the virtual camera vc in accordance with the position and orientation of the camera's external parameters, and renders a virtual camera screen that captures the three-dimensional point cloud space using the virtual camera, as shown in the center of Fig. 8.
[0060] In step S24, the virtual camera / photographic image corresponding pixel calculation unit 64 applies the same two-dimensional screen coordinate system as the photographic image shown on the right side of Fig. 8 to the virtual camera screen, as shown in the center of Fig. 8. Furthermore, the virtual camera / photographic image corresponding pixel calculation unit 64 associates pixels of the screen coordinates of the photographic image with virtual pixels of the screen coordinate system of the virtual camera screen at the same positions. In this way, the virtual camera / photographic image corresponding pixel calculation unit 64 calculates corresponding pixels between the virtual camera screen and the photographic image.
[0061] For this reason, the virtual camera / photographic image corresponding pixel calculation unit 64 assigns a pixel identification ID to each pixel of the photographic image, records the two-dimensional pixel coordinate position, and assigns a virtual pixel identification ID, as shown on the right side of Fig. 8. Furthermore, the virtual camera / photographic image corresponding pixel calculation unit 64 assigns a virtual pixel identification ID to each virtual pixel of the virtual camera screen, records the virtual pixel coordinate position, and assigns a point cloud identification ID, as shown in the center of Fig. 8.
[0062] For example, the virtual camera / photographic image corresponding pixel calculation unit 64 assigns a pixel identification ID (000014) to one pixel of the photographic image, records the two-dimensional pixel coordinate position (xc_14, yc_14), and assigns a virtual pixel identification ID (K_000014), as shown on the right side of Fig. 8. Furthermore, the virtual camera / photographic image corresponding pixel calculation unit 64 assigns a virtual pixel identification ID (K_000014) to a virtual pixel at the same position as the pixel with the pixel identification ID (000014), and records the two-dimensional virtual pixel coordinate position (xd_14, yd_14).
[0063] In step S25, the virtual camera pixel / 3D point cloud correspondence calculation unit 65 projects a straight line from the camera lens position of the virtual camera vc to each virtual pixel of the virtual camera screen vcs, and extracts a point cloud on the straight line. Furthermore, the virtual camera pixel / 3D point cloud correspondence calculation unit 65 associates a point cloud identification ID of the extracted point cloud with each virtual pixel of the virtual camera screen vcs.
[0064] For example, the virtual camera pixel / 3D point cloud correspondence point calculation unit 65 projects a straight line passing through the two-dimensional virtual pixel coordinate position (xd_14, yd_14) on the virtual camera screen vcs of the virtual pixel identification ID (K_000014) as shown on the left side of Fig. 8, and extracts the point cloud of the point cloud pixel identification ID (0000111) on the straight line. Furthermore, the virtual camera pixel / 3D point cloud correspondence point calculation unit 65 assigns the point cloud identification ID (0000111) to the virtual pixel of the virtual pixel identification ID (K_000014) as shown in the center of Fig. 8.
[0065] In the above-described matching process, the correspondence may be determined using the RGB information and semantic information of each pixel of the photographic image and the virtual camera screen vcs. In other words, even if the pixels have the same coordinates, if the RGB information is equal to or greater than a threshold or if the semantic information is different, the correspondence is not performed.
[0066] The above-mentioned matching process only needs to be performed on pixels of image feature points among the pixels of the photographic image.
[0067] In the above-described correspondence processing, the optimum corresponding point determination unit 66 determines the optimum corresponding point from the multiple point clouds in order to reduce erroneous correspondence between the virtual pixels of the virtual camera screen vcs and the pixels of the photographic image, which is performed by the virtual camera pixel / 3D point cloud correspondence point calculation unit 65. The optimum corresponding point determination processing will be described later.
[0068] In step S26, the three-dimensional point cloud / image feature amount corresponding point calculation unit 67 calculates the three-dimensional positions of the corresponding points of the three-dimensional point cloud with respect to the image feature points of the photographic image.
[0069] For this reason, the 3D point cloud / image feature corresponding point calculation unit 67 searches for the point cloud identification ID of the point cloud extracted by the virtual camera pixel / 3D point cloud corresponding point calculation unit 65 from the data set of the point cloud identification ID and the 3D point cloud coordinate position (x, y, z) obtained by the point cloud identification ID assignment unit 61. As a result, the 3D point cloud / image feature corresponding point calculation unit 67 obtains the point cloud coordinate position (x, y, z) of the 3D point cloud corresponding point of the pixel of the image feature point of the photographic image.
[0070] For example, the three-dimensional point cloud / image feature corresponding point calculation unit 67 searches for the point cloud identification ID (0000111) from the data set of the point cloud identification ID and the three-dimensional point cloud coordinate position (x, y, z) obtained by the point cloud identification ID assignment unit 61, and obtains the three-dimensional point cloud coordinate position (x222, y222, z222) of the point cloud with the point cloud identification ID (0000111).
[0071] The 3D point cloud / image feature corresponding point calculation unit 67 associates the point cloud coordinate positions (x, y, z) of the acquired 3D point cloud points with the pixels of the image feature points of the photographic image, thereby calculating the 3D positions of the corresponding points of the 3D point cloud to the image feature points of the photographic image.
[0072] The 3D point cloud / image feature corresponding point calculation unit 67 also associates the image feature with the information on the 3D position of the calculated corresponding point, thereby calculating the 3D position of the image feature.
[0073] (Optimal corresponding point determination process) In the photographic image / 3D point cloud corresponding point calculation unit 50 according to the first configuration example, when the resolution of the photographic image is low or the point cloud density of the 3D point cloud is high, one virtual pixel in the 2D screen coordinate system of the virtual camera image may contain multiple points of the point cloud.
[0074] In the photographic image / 3D point cloud correspondence point calculation unit 50 according to the second configuration example, when the resolution of the photographic image is low or the point cloud density of the 3D point cloud is high, one virtual pixel in the 2D screen coordinate system of the virtual camera screen may contain multiple points of the point cloud.
[0075] In either case, it is desirable to determine the optimal corresponding point from multiple points in the point cloud and reduce erroneous correspondences. Therefore, it is advisable to determine the optimal point from multiple points in the point cloud using attribute information of the photographic image (RGB information, semantic information, etc.) and attribute information of the point cloud (3D coordinate position, RGB information, reflection intensity, semantic information, etc.).
[0076] The best corresponding point determination process for reducing the above-mentioned erroneous pixel correspondences will be described below with reference to Fig. 9. Fig. 9 is a diagram schematically illustrating the best corresponding point determination process executed by the photographic image / 3D point cloud corresponding point calculation unit 50. This best corresponding point determination process is a method of extracting any multiple points close to the center position when one pixel in a two-dimensional screen coordinate system is subdivided, and selecting or newly generating the best corresponding point using attribute information of each point.
[0077] When one pixel PX in the two-dimensional screen coordinate system is subdivided, N points pi (i=1, 2, . . . , N) that are at a distance within a threshold value s from the center position q are extracted.
[0078] If no point pi is extracted, it is determined that there is no corresponding point.
[0079] If there is one extracted point pi, that point is selected as the corresponding point.
[0080]
[0081] Alternatively, the RGB information of each point and the RGB information of the pixel of the photographic image may be used as the weight. Alternatively, the three-dimensional coordinate position of each point may be referenced, and the number of points within a certain distance may be used as the weight.
[0082] The above process allows optimal corresponding points to be selected or newly generated, thereby reducing mismatches between pixels in the photographic image and virtual pixels in the virtual camera image or virtual camera screen.
[0083] (Hidden point removal process from the virtual camera viewpoint) In the photographic image / 3D point cloud correspondence point calculation unit 50 according to the first configuration example, when projective transformation is performed onto a virtual camera image, the virtual camera image may contain points that should not be visible from the virtual camera.
[0084] In the photographic image / 3D point cloud corresponding point calculation unit 50 according to the second configuration example, there are cases where the virtual camera screen includes points that should not be visible from the virtual camera.
[0085] In either case, it is desirable to remove points that should not be visible to the virtual camera, to reduce mismatches.
[0086] Hereinafter, the hidden point removal process from the virtual camera viewpoint for reducing the aforementioned pixel mismatch will be described with reference to Figures 10 and 11. Figures 10 and 11 are diagrams that schematically show the hidden point removal process from the virtual camera viewpoint executed by the photographic image / 3D point cloud correspondence point calculation unit 50. This hidden point removal process from the virtual camera viewpoint is a method of processing the 3D point cloud in advance. This is an extension to the target space of the method of removing points that should not be visible from the virtual camera v.
[0087] For example, determining parts that are not visible from the virtual camera's viewpoint can be done by meshing the point cloud. Meshing of point clouds can be done using technologies such as open3D and CloudCompare. However, meshing of point clouds requires a lot of calculation time. Therefore, here we will show an example of a method that does not involve meshing.
[0088] Conventional techniques for removing points that are not visible from the virtual camera include Direct Visibility of Point Sets and Open3D's Hidden Point Removal. Direct Visibility of Point Sets is disclosed in the document "S. Katz, A. Tal, and R. Basri, "Direct Visibility of Point Sets," ACM Transactions on Graphics, Vol. 26, No. 3, Article 24."
[0089] However, these conventional technologies are targeted at point clouds of desks and objects that exist in space, and when targeting a space with a large depth, all point clouds that are more than a certain distance from the virtual camera's viewpoint are excluded. Below, we will show a method that eliminates this problem and removes parts that are not visible from the virtual camera's viewpoint.
[0090] First, as shown in Figure 10, the original three-dimensional point cloud PG0 of the target space is scaled in the line of sight direction of the virtual camera vc so that it fits inside the virtual sphere vss, for example, to become a three-dimensional point cloud PG1. For example, as shown in Figure 11, the target space TS is compressed into a thick spherical shape with the virtual camera viewpoint vcp at the center. This allows the target space TS to be treated as a spherical object. For example, the target space TS is compressed into a spherical shape as follows:
[0091]
[0092] The compression rate may be changed according to the distance from the virtual camera viewpoint vcp. The compression rate can be changed using, for example, the following formula.
[0093]
[0094] For example, by using a sigmoid function, it is possible to compress the data so that closer points are denser and farther points are sparser.
[0095] The subsequent processing can be performed by treating the target space TS as a spherical object and using the conventional technique of direct visibility of point sets mentioned above.
[0096] This series of processes makes it possible to remove points that should not be visible to the virtual camera v (hidden points), thereby reducing mismatches between pixels in the photographic image and virtual pixels in the virtual camera image or virtual camera screen.
[0097] In the above-described example of operation of the photographic image / 3D point cloud correspondence point calculation unit 50, an example has been described in which both the optimum correspondence point determination process and the hidden point removal process from the virtual camera viewpoint are performed in order to reduce erroneous correspondences between pixels of a photographic image and virtual pixels of a virtual camera image or virtual camera screen. However, it is also possible to perform only one of the optimum correspondence point determination process and the hidden point removal process from the virtual camera viewpoint.
[0098] (Hardware Configuration) Next, the hardware configuration of the three-dimensional position calculation device 10 will be described with reference to Fig. 12. For example, the three-dimensional position calculation device 10 is configured by a computer 80. Fig. 12 is a block diagram showing the hardware configuration of the computer 80 that configures the three-dimensional position calculation device 10.
[0099] 2 shows the hardware configuration of a computer 80 that constitutes the three-dimensional position calculation device 10. The computer 80 has a processor 81, a read-only memory (ROM) 82, a random access memory (RAM) 83, an auxiliary storage device 84, an input device 85, and an output device 86.
[0100] The processor 81 , ROM 82 , RAM 83 , auxiliary storage device 84 , input device 85 , and output device 86 are electrically connected to one another via a bus 87 , and can send and receive information to and from one another via the bus 87 .
[0101] The processor 81 is configured by a general-purpose hardware processor including, for example, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), etc. The processor 81 controls the ROM 82, the RAM 83, the auxiliary storage device 84, the input device 85, and the output device 86 as a whole.
[0102] The ROM 82 is a non-volatile memory that constitutes part of the main storage device. The ROM 82 non-temporarily stores a startup program required when the processor 81 is started. The processor 81 starts up by executing the program in the ROM 82. The ROM 82 is configured, for example, by an EPROM (Erasable Programmable Read Only Memory), and stores various startup settings in addition to the startup program.
[0103] The RAM 83 is a volatile memory that constitutes part of the main storage device. The RAM 83 temporarily stores programs required for processing by the processor 81 and data required for executing the programs. The processor 81 executes the programs in the RAM 83 to perform calculations on the data in the RAM 83 and store the calculation results in the RAM 83.
[0104] The auxiliary storage device 84 is configured with non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The auxiliary storage device 84 non-temporarily stores programs executed by the processor 81 and data required for executing the programs. The processor 81 loads the programs and data in the auxiliary storage device 84 into the RAM 83 and executes the programs to perform various functions. The auxiliary storage device 84 also constitutes the three-dimensional position information DB 70 of image features.
[0105] The input device 85 may include a keyboard, a mouse, a touch panel, a receiving device, a disk drive, an input port, etc. The input device 85 is not limited to these and may include any other input device. The output device 86 may include a display, a printer, a transmitting device, a disk drive, an output port, etc. The output device 86 is not limited to these and may include any other output device. The input device 85 and the output device 86 may be configured as an input / output device having the functions of both devices. The input / output device may include a touch panel, a transmitting / receiving device, a disk drive, an input / output port, etc. The receiving device, transmitting device, and transmitting / receiving device may be wired or wireless. Through the input device 85, photographic image information is input to the photographic image information acquisition unit 20, and 3D point cloud information is input to the 3D point cloud information acquisition unit 30.
[0106] The program non-temporarily stored in the auxiliary storage device 84 is provided to the computer 80, for example, via a non-temporary computer-readable recording medium 88. The non-temporary computer-readable recording medium 88 includes disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), and magneto-optical disks (MO, etc.), as well as semiconductor memories.
[0107] The programs non-temporarily stored in the auxiliary storage device 84 include a three-dimensional position calculation program. The three-dimensional position calculation program is a program that causes the computer 80 to execute at least some of the functions of the components of the three-dimensional position calculation device 10.
[0108] If the non-transitory computer-readable recording medium 88 is a disk, the program is non-temporarily stored in the auxiliary storage device 84 via, for example, a disk drive which is the input device 85, or if the non-transitory computer-readable recording medium 88 is a semiconductor memory, via an input port which is the input device 85. Alternatively, the program may be stored in a server on a network, downloaded from the server, and non-temporarily stored in the auxiliary storage device 84.
[0109] When the computer 80 starts up, the processor 81 executes a program in the ROM 82 and loads and starts the OS into the RAM 83. Under control of the OS, the processor 81 monitors input instructions, connections to external devices, and the like. Under control of the OS, the processor 81 also sets up a program area and a data area in the RAM 83. In response to an instruction to start the three-dimensional position calculation device 10, the processor 81 loads a three-dimensional position calculation program from the auxiliary storage device 84 into the program area of the RAM 83 and loads data required for executing the three-dimensional position calculation program from the auxiliary storage device 84 into the data area of the RAM 83. The processor 81 calculates data in the data area according to the three-dimensional position calculation program and writes the calculation results to the data area. Through these operations, the processor 81, RAM 83, and auxiliary storage device 84 work together to execute at least some of the functions of the components of the three-dimensional position calculation device 10. For example, the processor 81, RAM 83, and auxiliary storage device 84 work together to perform the functions of the photographic image information acquisition unit 20, the three-dimensional point cloud information acquisition unit 30, the image feature calculation unit 40, and the photographic image / three-dimensional point cloud correspondence point calculation unit 50.
[0110] (Effects) According to the embodiment, by combining one photograph of the target space with a three-dimensional point cloud space, it is possible to calculate the three-dimensional position of an image feature from a single photograph. This makes it possible to calculate the position and orientation of an object and the position and orientation of the shooting camera. In other words, a technology is provided for calculating the three-dimensional position of an image feature without requiring two or more photographs. In other words, because only one photograph is required to calculate the three-dimensional position of an image feature, it is possible to save the effort and time required for photographing.
[0111] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.
[0112] 10... 3D position calculation device 20... Photographic image information acquisition unit 30... 3D point cloud information acquisition unit 40... Image feature calculation unit 50... Photographic image / 3D point cloud corresponding point calculation unit 51... Point cloud identification ID assignment unit 52... Hidden point removal unit from virtual camera viewpoint 53... Virtual camera image generation unit 54... Virtual camera image / image feature corresponding point calculation unit 55... Optimal corresponding point determination unit 56... 3D point cloud / image feature corresponding point calculation unit 61... Point cloud identification ID assignment unit 62... Hidden point removal unit from virtual camera viewpoint 63... Virtual camera screen drawing unit 64... Virtual camera / photographic image corresponding pixel calculation unit 65... Virtual camera pixel / 3D point cloud corresponding point calculation unit 66... Optimal corresponding point determination unit 67... 3D point cloud / image feature corresponding point calculation unit 70... 3D position information database of image features 80... Computer 81... Processor 82... ROM 83... RAM 84: Auxiliary storage device 85: Input device 86: Output device 87: Bus 88: Non-transitory computer-readable recording medium
Claims
1. A three-dimensional position calculation device having a photographic image / three-dimensional point cloud corresponding point calculation unit that receives as input photographic image information for one photographic image, image feature points and image feature amounts of the image feature points of the photographic image, and three-dimensional point cloud information for a three-dimensional point cloud related to the photographic image, and calculates the three-dimensional positions of the image feature amounts, which are corresponding points of the three-dimensional point cloud to the image feature points, based on the photographic image information, the image feature points, the image feature amounts, and the three-dimensional point cloud information.
2. The three-dimensional position calculation device according to claim 1, wherein the photographic image / three-dimensional point cloud corresponding point calculation unit generates a virtual camera image by projectively transforming the three-dimensional point cloud, and calculates corresponding points between the virtual camera image and the photographic image, thereby calculating the three-dimensional positions of the corresponding points.
3. The three-dimensional position calculation device described in claim 1, wherein the photographic image / three-dimensional point cloud corresponding point calculation unit generates a virtual camera in the three-dimensional point cloud space at the same position and orientation as the camera that took the photographic image, draws a virtual camera screen that projects the three-dimensional point cloud space using the virtual camera, calculates corresponding pixels between the photographic image and the virtual camera screen, and calculates corresponding points in the three-dimensional point cloud of the photographic image from the corresponding pixels, thereby calculating the three-dimensional position of the image feature.
4. The three-dimensional position calculation device according to claim 1, wherein the photographic image / three-dimensional point cloud corresponding point calculation unit generates a virtual image corresponding to the photographic image, and determines the corresponding point of the three-dimensional point cloud to the image feature point based on a weighted average of multiple points of the point cloud contained in one pixel of the virtual image.
5. The three-dimensional position calculation device according to claim 1, wherein the photographic image / three-dimensional point cloud corresponding point calculation unit generates a virtual camera in a three-dimensional point cloud space related to the photographic image, and removes points that should not be visible from the viewpoint of the virtual camera from the three-dimensional point cloud information.
6. A three-dimensional position calculation method comprising the steps of: receiving as input photographic image information for one photographic image, image feature points and image feature quantities of the image feature points of the photographic image, and three-dimensional point cloud information for a three-dimensional point cloud related to the photographic image; generating a virtual camera image by projecting the three-dimensional point cloud onto a virtual camera image; and calculating the three-dimensional positions of the image feature quantities, which are corresponding points of the three-dimensional point cloud to the image feature points, by matching pixels of the virtual camera image at the same positions as pixels of the photographic image.
7. A three-dimensional position calculation method comprising the steps of: receiving as input photographic image information for one photographic image, image feature points and image feature quantities of the photographic image, and three-dimensional point cloud information for a three-dimensional point cloud related to the photographic image; generating a virtual camera in three-dimensional point cloud space at the same position and orientation as the camera that captured the photographic image, and drawing a virtual camera screen that projects the three-dimensional point cloud space using the virtual camera; calculating corresponding pixels between the photographic image and the virtual camera screen; and calculating the three-dimensional positions of the image feature quantities, which are corresponding points of the three-dimensional point cloud to the image feature points, by calculating corresponding points of the three-dimensional point cloud of the photographic image from the corresponding pixels.
8. A three-dimensional position calculation program that causes a computer having a processor and a storage device to execute at least a part of the functions of the components of the three-dimensional position calculation device according to any one of claims 1 to 5.
Citation Information
Patent Citations
Image measurement processor, image measurement processing method and image measurement processing program by single camera
JP2012027000A
Robot, control device, and control method
JP2016218561A
Location information recording method and device
JP2019032218A
