Coordinate calculation device, coordinate calculation method, and computer-readable recording medium

Abstract

This coordinate calculation device 10 comprises: a designation reception unit 11 that receives a designation of a specific point on a two-dimensional image obtained by imaging an object; a combination extraction unit 12 that compares a first feature point extracted from the two-dimensional image with a second feature point included in three-dimensional point group data of the object to extract a combination of the first feature point and the second feature point; an external parameter calculation unit 13 that calculates external parameters of a camera by using two-dimensional coordinates of the first feature point and three-dimensional coordinates of the second feature point; and a coordinate calculation unit 14 that calculates three-dimensional coordinates of an intersection point of a virtual line segment passing through the camera center and the specific point and the three-dimensional point group data of the object by using the external parameters, and sets the intersection point as the specific point.

Description

Coordinate calculation device, coordinate calculation method, and computer-readable recording medium

[0001] The present disclosure relates to a coordinate calculation device and a coordinate calculation method for calculating the coordinates of a specific location of an object on point cloud data, and further to a computer-readable recording medium on which a program for realizing these is recorded.

[0002] Patent Document 1 discloses an apparatus for identifying the position of a specific point on a three-dimensional model of an object from a two-dimensional image of the specific point on the object. The apparatus disclosed in Patent Document 1 is useful, for example, for identifying the position of a defect that appears on the exterior of a structure such as a bridge.

[0003] Specifically, suppose a defect is detected in some of a large number of images of a structure. In this case, the device disclosed in Patent Document 1 compares feature points in the image in which the defect was detected with feature points in a 3D model to identify the 3D coordinates of the defect and displays the identified 3D coordinates on a screen. Therefore, by using the device disclosed in Patent Document 1, it is possible to immediately identify the location of the defect detected from the image in the structure, and to quickly carry out repair work.

[0004] International Publication No. 2019 / 046420

[0005] However, the device disclosed in Patent Document 1 has a problem in that the accuracy of identifying the position decreases under certain conditions. This problem will be specifically described below.

[0006] First, a three-dimensional model of an object is generally point cloud data created using a technique called SfM (Structure from Motion). In SfM, feature points are first extracted from each of a large number of two-dimensional images of the object taken from different viewpoints, and combinations of feature points common to the images are then identified. Next, camera matrices are calculated for each combination of two images for which combinations of feature points have been identified, and the three-dimensional coordinates of the feature points are calculated using the two camera matrices thus calculated. Thereafter, the above-mentioned point cloud data is generated using the feature points whose three-dimensional coordinates have been calculated.

[0007] Incidentally, point cloud data created by SfM in this manner is called sparse point cloud data, and its accuracy as a 3D model is low. For this reason, a technique called Multi-View Stereo may be used to interpolate between points in the sparse point cloud data using the camera pose [R|t] determined by SfM and 3D point cloud information, thereby generating dense point cloud data. Furthermore, as a further process for this dense point cloud data, a triangle (mesh) is constructed for every three neighboring points, and a texture (image) is applied to the constructed mesh. This process results in a 3D model that is closer to the real thing.

[0008] However, in the device disclosed in Patent Document 1, the 2D image of the specific location is an image used to generate a coarse point cloud. Therefore, there may be a large error between the coordinates of the specific location on this 2D image and the coordinates of the specific location on the texture that is applied based on the dense point cloud data. In such cases, the estimated positions of the specific location on the 2D image and the specific location on the texture also differ, resulting in the problem of reduced positional accuracy as described above.

[0009] Furthermore, in the device disclosed in Patent Document 1, the problem of reduced positional accuracy occurs even when the two-dimensional image showing the specific location is not the two-dimensional image used in SfM.

[0010] One example of the objective of the present disclosure is to improve the accuracy of position identification when identifying the position of a specific location on a three-dimensional model from a two-dimensional image that captures the specific location of an object, without setting any constraints on the two-dimensional image or the three-dimensional model.

[0011] In order to achieve the above object, a coordinate calculation device according to one aspect of the present disclosure comprises: a designation receiving unit that receives designation of a specific point on a two-dimensional image of an object; a combination extraction unit that extracts a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; an external parameter calculation unit that calculates external parameters of a camera used to photograph the object, using two-dimensional coordinates on the two-dimensional image of the first feature point extracted as a combination and three-dimensional coordinates corresponding to the second feature point extracted as a combination; and a coordinate calculation unit that calculates three-dimensional coordinates of an intersection point between a virtual line segment passing through a camera center of the camera and the specific point and the three-dimensional point cloud data of the object, using the calculated external parameters, and sets the intersection point whose three-dimensional coordinates are calculated as the specific point whose designation was accepted.

[0012] Furthermore, in order to achieve the above object, a coordinate calculation method according to one aspect of the present disclosure is characterized by comprising: a designation receiving step of receiving designation of a specific point on a two-dimensional image of an object; a combination extraction step of extracting a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; an external parameter calculation step of calculating external parameters of a camera used to photograph the object, using two-dimensional coordinates on the two-dimensional image of the first feature point extracted as a combination and three-dimensional coordinates corresponding to the second feature point extracted as a combination; and a coordinate calculation step of calculating, using the calculated external parameters, three-dimensional coordinates of an intersection point between a virtual line segment passing through the camera center and the specific point and the three-dimensional point cloud data of the object, and setting the intersection point whose three-dimensional coordinates are calculated as the specific point whose designation was accepted.

[0013] Furthermore, in order to achieve the above object, a computer-readable recording medium according to one aspect of the present disclosure has recorded thereon a program including instructions for causing a computer to execute the following: a designation receiving step of receiving designation of a specific point on a two-dimensional image of an object; a combination extraction step of extracting a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; an external parameter calculation step of calculating external parameters of a camera used to photograph the object, using two-dimensional coordinates on the two-dimensional image of the first feature point extracted as a combination and three-dimensional coordinates corresponding to the second feature point extracted as a combination; and a coordinate calculation step of calculating three-dimensional coordinates of an intersection point between a virtual line segment passing through the camera center and the specific point and the three-dimensional point cloud data of the object, using the calculated external parameters, and setting the intersection point whose three-dimensional coordinates are calculated as the specific point whose designation was accepted.

[0014] As described above, according to the present disclosure, when identifying the position of a specific location on a three-dimensional model from a two-dimensional image that captures the specific location of an object, the accuracy of identifying the location can be improved without setting constraints on the two-dimensional image and the three-dimensional model.

[0015] FIG. 1 is a diagram showing a schematic configuration of a first example of a coordinate calculation device. FIG. 2 is a diagram showing the configuration of the first example of the coordinate calculation device. FIG. 3 is a diagram showing a process of extracting a combination of a first feature point and a second feature point. FIG. 4 is a diagram showing an example of a camera position and orientation calculated from two-dimensional coordinates and three-dimensional coordinates on a specified two-dimensional image. FIG. 5 is a diagram showing an example of an intersection point between a virtual line segment and three-dimensional point cloud data. FIG. 6 is a diagram showing an example of expressing an intersection point using centroid coordinates. FIG. 7 is a flow chart showing a first example of the operation of the coordinate calculation device. FIG. 8 is a diagram showing the configuration of a modified example of the coordinate calculation device. FIG. 9 is a diagram showing the configuration of a second example of the coordinate calculation device. FIG. 10 is a diagram showing the configuration of a third example of the coordinate calculation device. FIG. 11 is a block diagram showing an example of a computer that realizes the coordinate calculation device.

[0016] First Embodiment In the first embodiment, a coordinate calculation device, a coordinate calculation method, and a program, each of which is a first example, will be described below with reference to FIGS.

[0017] [Device Configuration] First, the schematic configuration of an example of a coordinate calculation device will be described with reference to Fig. 1. Fig. 1 is a diagram showing the schematic configuration of a first example of a coordinate calculation device.

[0018] 1 is a device for calculating the coordinates of a specific location of an object on point cloud data, and includes a designation receiving unit 11, a combination extraction unit 12, an external parameter calculation unit 13, and a coordinate calculation unit 14.

[0019] The designation receiving unit 11 receives designation of specific points on a two-dimensional image of an object. The combination extracting unit 12 extracts matching combinations of first feature points extracted from the two-dimensional image of the object and second feature points derived from three-dimensional point cloud data of the object.

[0020] The external parameter calculation unit 13 calculates the external parameters of the camera used to photograph the object, using the two-dimensional coordinates on the image of the first feature points extracted as a combination and the three-dimensional coordinates corresponding to the second feature points extracted as a combination.

[0021] The coordinate calculation unit 14 uses the calculated external parameters to calculate the three-dimensional coordinates of the intersection point between a virtual line segment passing through the camera center of the camera used to photograph the object and the specific point and the three-dimensional point cloud data of the object. Then, the coordinate calculation unit 14 sets the intersection point whose coordinates have been calculated as the specific point whose designation has been accepted.

[0022] In this way, the coordinate calculation device 10 determines the coordinates of the specific point as the coordinates of the point (intersection point) where a virtual line segment passing from the camera center through the specific point on the two-dimensional image intersects with the three-dimensional point cloud data (three-dimensional model) of the object. Therefore, even if a texture is applied to dense point cloud data, the position of the specific point can be identified with high accuracy. The coordinate calculation device 10 can improve the accuracy of position identification without imposing constraints on the two-dimensional image and the three-dimensional model.

[0023] Next, the configuration and functions of an example of the coordinate calculation device 10 will be specifically described with reference to Figures 2 to 6. Figure 2 is a configuration diagram showing the configuration of a first example of the coordinate calculation device.

[0024] As shown in FIG. 2 , in the first embodiment, the coordinate calculation device 10 includes a three-dimensional model display unit 15 and a storage unit 20 in addition to the above-mentioned designation receiving unit 11, combination extraction unit 12, external parameter calculation unit 13, and coordinate calculation unit 14.

[0025] The storage unit 20 stores three-dimensional point cloud data 50 of the object and image data (hereinafter referred to as "creation two-dimensional image data") 51 of a two-dimensional image (hereinafter referred to as "creation two-dimensional image") used to create the three-dimensional point cloud data. In the example of FIG. 2, the storage unit 20 is provided in the coordinate calculation device 10, but this is not limiting. The storage unit 20 may be provided in another device connected to the coordinate calculation device 10 via a network. In the first embodiment, the three-dimensional point cloud data 50 stored in the storage unit 20 is created using the creation two-dimensional image data 51 by, for example, a technique known as SfM (Structure from Motion). Furthermore, the three-dimensional point cloud data 50 stored in the storage unit 20 may be subjected to mesh construction and texture application.

[0026] The designation receiving unit 11 receives data specifying a specific point on a two-dimensional image via the input device 30, thereby accepting the designation of the specific point. Specifically, the designation receiving unit 11 receives, for example, image data of a two-dimensional image of the object used for the designation (hereinafter referred to as a "designated two-dimensional image") and coordinate data of the specific point on the two-dimensional image. The input device 30 is a keyboard, a mouse, a touch panel, or the like. The input device 30 may also be a device separate from the coordinate calculation device 10, such as a computer, a smartphone, or a tablet terminal. Furthermore, the specific point may be manually designated on the input device 30, or may be designated by a machine such as a computer without manual intervention.

[0027] Furthermore, in the first embodiment, the designated two-dimensional image is a two-dimensional image different from the creation two-dimensional image used to create the three-dimensional point cloud data of the object, and is captured by a camera different from the camera that captured the creation two-dimensional image. Note that the designated two-dimensional image may be an image used as the creation two-dimensional image. Furthermore, the camera that captured the designated two-dimensional image may be the same as the camera that captured the creation two-dimensional image. The designated two-dimensional image may be any two-dimensional image obtained by capturing an image of the object.

[0028] The combination extraction unit 12 first extracts first feature points from the specified 2D image. Then, the combination extraction unit 12 compares the extracted first feature points with second feature points to extract matching combinations of the first feature points and the second feature points. In the first embodiment, the second feature points are feature points of the creation 2D image.

[0029] The extraction of combinations of first feature points and second feature points by the combination extraction unit 12 will be specifically described with reference to FIG. 3 . FIG. 3 is a diagram showing the process of extracting combinations of first feature points and second feature points. As shown in FIG. 3 , the feature points (second feature points) of the creation two-dimensional image 60 are the same as the point cloud data included in the three-dimensional point cloud data 50 (indicated by ● in the figure). Therefore, the combination extraction unit 12 extracts feature points (second feature points) from the creation two-dimensional image 60, compares the extracted second feature points with the feature points of the specified two-dimensional image 61 (indicated by x: first feature points in the figure), and compares the feature amounts of each to extract three or more matching combinations of first feature points and second feature points.

[0030] Here, for the second feature points extracted as a combination, corresponding three-dimensional coordinates have already been calculated. Therefore, in the first embodiment, the external parameter calculation unit 13 uses the three-dimensional point cloud data to identify three-dimensional coordinates (X, Y, Z) corresponding to the second feature points extracted as a combination. Then, the external parameter calculation unit 13 calculates the external parameters of the camera used to capture the specified two-dimensional image using the two-dimensional coordinates (x, y) of the three or more first feature points extracted as a combination and the identified three-dimensional coordinates (X, Y, Z).

[0031] Specifically, as shown in Fig. 4, the external parameter calculation unit 13 calculates the position and orientation of the camera for converting two-dimensional coordinates from the camera coordinates of the camera to world coordinates, that is, the rotation matrix R and the translation matrix t ([R|t]). The position of the camera is represented by the translation matrix t, and the orientation of the camera is represented by the rotation matrix R. The calculation of the rotation matrix R and the translation matrix t is performed by using three-dimensional coordinates (X, Y, Z) and the position of a specific point a １ ~a ３ (In the case of three) two-dimensional coordinates (u １ , v １ ) ~ (u ３ , v ３ ) to solve a PnP (Perspective-n-Point) problem. Fig. 4 is a diagram showing an example of the position and orientation of a camera calculated from two-dimensional coordinates and three-dimensional coordinates on a specified two-dimensional image.

[0032] As shown in FIG. 5, the coordinate calculation unit 14 first calculates the distance between the center t of the camera and the specific point a by using the external parameters [R|t] of the camera used to capture the specified two-dimensional image 61. １ A virtual line segment passing through the line segment is calculated, and the intersection point A between the calculated line segment and the three-dimensional point cloud data is calculated. １ The coordinates of the specified two-dimensional image 61 are calculated. Note that the center of the camera that captured the specified two-dimensional image 61 is equal to the translation matrix t, so the center of the camera is set to "t". Figure 5 is a diagram showing an example of an intersection point between a virtual line segment and three-dimensional point cloud data.

[0033] This method of calculating intersection points is called Ray Casting, and is performed using the algorithm of Tomas Moller (reference information: http: / / www.graphics.cornell.edu / pubs / 1997 / MT97.pdf).

[0034] A method for calculating the coordinates of the intersection point will be specifically described below. First, as shown in FIG. 5, a mesh for applying texture is constructed in the three-dimensional point cloud data 50. １ is assumed to exist in one of the meshes. Also, the world coordinates of the camera center t are (x ｃ , y ｃ , z ｃ), from the center of the camera to intersection point A １ The distance (depth) to the intersection point A is d, and the unit direction vector of the imaginary line segment is γ. １ is expressed by the following equation 1.

[0035]

[0036] Also, as shown in FIG. １ is the vertex of the mesh V ０ , V １ , V ２ Then, it can also be expressed by barycentric coordinates as shown in the following equation 2. Fig. 6 is a diagram showing an example in which the intersection point is expressed by barycentric coordinates.

[0037]

[0038] Here, since 1=w+u+v holds in the barycentric coordinates, the above equation 2 can also be expressed as the following equation 3. The equation 4 is a condition for the equation 3 to hold.

[0039]

[0040]

[0041] Then, the following equation 5 is derived from the above equations 1 and 3. Furthermore, equation 5 can be expressed as equation 6.

[0042]

[0043]

[0044] Therefore, under the condition that the above equation 4 is satisfied, the unit direction vector γ is calculated, and by using the calculated unit direction vector γ, the depth d can be derived from the above equation 6. As a result, the intersection point A can be calculated from the above equation 1. １ can be calculated.

[0045] Here, the world coordinates of the camera center t are (x ｃ , y ｃ , z ｃ ), specific point a １ The two-dimensional coordinates of (u １ , v １ ) when the depth d is z ｃIn this case, the following equation 7 is derived.

[0046]

[0047] The intersection point when the depth d=1 is A' １ and its world coordinates are (x ｗ１ , y ｗ１ , z ｗ１ ) Depth d = z ｃ = 1, the intersection point is A' １ The coordinates of the intersection point A in the camera coordinate system are as shown in the following equation 8. １ ' is "A' Ｃ１ " is expressed as ".

[0048]

[0049] Intersection point A' １ The coordinates in the camera coordinates of can be converted into world coordinates using the external parameters [R|t] of the camera, as shown in the following equation 9.

[0050]

[0051] Here, "A' １ -t" is the distance from the center t of the camera to the intersection point A １ Therefore, "A'" indicates the vector of the line segment that passes through １ The unit direction vector γ is calculated by finding "−t". The unit direction vector γ is as shown in the following equations 10 and 11.

[0052]

[0053]

[0054] Therefore, as described above, by applying the unit direction vector γ obtained from the above equations 10 and 11 to the above equation 1, the intersection point A １ The coordinate calculation unit 14 calculates the coordinates of the intersection point A １ is a specific point on the three-dimensional point cloud data.

[0055] When the intersection point is calculated, the three-dimensional model display unit 15 uses the three-dimensional point cloud data of the object to display a three-dimensional model of the object on the screen of the display device 40. Furthermore, the three-dimensional model display unit 15 displays the intersection point, the coordinate of which is calculated by the coordinate calculation unit 14, as a specified specific point on the three-dimensional model.

[0056] [Device Operation] Next, an example of the operation of the coordinate calculation device 10 will be described with reference to FIG. 7. FIG. 7 is a flow diagram showing a first example of the operation of the coordinate calculation device. In the following description, reference will be made to FIGS. 1 to 6 as appropriate. In addition, in the first embodiment, a coordinate calculation method is implemented by operating the coordinate calculation device 10. Therefore, the description of the coordinate calculation method will be substituted for the description of the operation of the coordinate calculation device 10 below.

[0057] 7 , first, the designation receiving unit 11 receives a designation of a specific point in a designated two-dimensional image via the input device 30 (step A1). Specifically, the designation receiving unit 11 receives data that identifies the specific point on the two-dimensional image, image data of the designated two-dimensional image, and coordinate data of the specific point on the two-dimensional image.

[0058] Next, the combination extraction unit 12 extracts first feature points from the designated two-dimensional image used to designate the specific points in step A1 (step A2).

[0059] Next, the combination extraction unit 12 extracts matching combinations of the extracted first feature points and second feature points included in the 3D point cloud data (step A3). Specifically, the combination extraction unit 12 compares the extracted first feature points with the feature points (second feature points) of the 2D image for creation, and extracts matching combinations of the first feature points and the second feature points.

[0060] The external parameter calculation unit 13 uses the three-dimensional point cloud data to identify three-dimensional coordinates (X, Y, Z) corresponding to the second feature points extracted as a combination in step A3 (step A4).

[0061] Then, using the two-dimensional coordinates (x, y) of the first feature point extracted as a combination in step A3 and the identified three-dimensional coordinates (X, Y, Z), the external parameters [R|t] of the camera used to capture the specified two-dimensional image are calculated (step A5).

[0062] Next, the coordinate calculation unit 14 calculates a virtual line segment passing through the center of the camera used to capture the specified 2D image and the specific point using the external camera parameters [R|t] calculated in step A5, and calculates the coordinates of the intersection of the calculated line segment and the 3D point cloud data (step A6). The calculated coordinates become the coordinates of the specific point accepted in step A1.

[0063] Next, the 3D model display unit 15 uses the 3D point cloud data of the object to display a 3D model of the object on the screen of the display device 40, and further displays the intersection point whose coordinates were calculated in step A6 as a specified specific point on the 3D model (step A7).

[0064] [Effects of First Embodiment] In the first embodiment, the coordinates of the point (intersection point) where a virtual line segment that passes through a specific point on a two-dimensional image from the center of the camera intersects with the three-dimensional point cloud data (three-dimensional model) of the object are calculated as the three-dimensional coordinates of the specific point. In other words, in the first embodiment, unlike conventional methods, the three-dimensional position of the specific point is not calculated based only on a comparison between the feature points of the two-dimensional image and the feature points of the three-dimensional point cloud data.

[0065] Therefore, even if a texture is applied to the three-dimensional point cloud data, and even if the two-dimensional image used to specify the specific point is different from the image used to create the three-dimensional point cloud data, the position of the specific point can be identified with high accuracy.

[0066] [Modification] Next, a modification of the coordinate calculation device 10 will be described with reference to Fig. 8. Fig. 8 is a configuration diagram showing the configuration of the modification of the coordinate calculation device.

[0067] In the example of FIG. 8 , the coordinate calculation device 10 includes a point cloud data generation unit 16 in addition to the above-mentioned specification reception unit 11, combination extraction unit 12, external parameter calculation unit 13, coordinate calculation unit 14, 3D model display unit 15, and memory unit 20.

[0068] The point cloud data generation unit 16 acquires multiple two-dimensional images of the object, i.e., multiple two-dimensional images for creation 60, and performs SfM using these to generate three-dimensional point cloud data 50, and stores the generated three-dimensional point cloud data 50 in the memory unit 20.

[0069] Specifically, the point cloud data generation unit 16 first acquires a plurality of creation two-dimensional images 60 and stores the acquired creation two-dimensional images 60 in the storage unit 20. The point cloud data generation unit 16 also extracts feature points from each of the acquired creation two-dimensional images 60. Next, the point cloud data generation unit 16 sets two or more pair images from the plurality of creation two-dimensional images 60.

[0070] Furthermore, the point cloud data generation unit 16 extracts, for each pair of images, combinations of feature points that match each other from each of the pair of images. Furthermore, for each pair of images, the point cloud data generation unit 16 uses the camera matrix of each of the pair of images to calculate, for each pair of images, external parameters of the position of the camera used to capture the image, i.e., the three-dimensional coordinates (world coordinates) of the camera position and the rotation matrix.

[0071] Thereafter, the point cloud data generation unit 16 calculates the three-dimensional coordinates of the extracted feature points as a combination for each pair of images using the obtained three-dimensional coordinates of the positions of each camera and the rotation matrix. Furthermore, the point cloud data generation unit 16 creates three-dimensional point cloud data 50 using the feature points whose three-dimensional coordinates have been calculated.

[0072] In this manner, in the modified example, the coordinate calculation device 10 itself can create the three-dimensional point cloud data 50. Also in the modified example, the storage unit 20 may be provided in another device connected to the coordinate calculation device 10 via a network.

[0073] In the modified example, before steps A1 to A7 shown in FIG. 7 are executed, the point cloud data generator 16 executes a process of generating three-dimensional point cloud data in advance.

[0074] [Program] The program in the first embodiment may be any program that causes a computer to execute steps A1 to A7 shown in Fig. 7. By installing and executing this program in a computer, the coordinate calculation device 10 and the coordinate calculation method in the first embodiment can be realized. In this case, the processor of the computer functions as the designation receiving unit 11, the combination extraction unit 12, the external parameter calculation unit 13, the coordinate calculation unit 14, and the 3D model display unit 15 and performs the processing.

[0075] In the first embodiment, the storage unit 20 is implemented by a storage device such as a hard disk provided in the computer. Alternatively, the storage unit 20 may be implemented by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0076] The program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the designation receiving unit 11, the combination extracting unit 12, the external parameter calculating unit 13, the coordinate calculating unit 14, and the 3D model display unit 15.

[0077] Second Embodiment Next, in a second embodiment, a second example of a coordinate calculation device, a coordinate calculation method, and a program will be described with reference to FIG.

[0078] [Device Configuration] First, the configuration of another example of a coordinate calculation device will be described with reference to Fig. 9. Fig. 9 is a configuration diagram showing the configuration of a second example of a coordinate calculation device.

[0079] 9, the coordinate calculation device 80 is a device for calculating the coordinates of a specific location of an object on point cloud data, similar to the coordinate calculation device 10 shown in FIGS. 1 and 2. As shown in FIG. 9, the coordinate calculation device 80, like the coordinate calculation device 10, includes a designation receiving unit 11, a combination extraction unit 12, an external parameter calculation unit 13, a coordinate calculation unit 14, a three-dimensional model display unit 15, and a storage unit 20.

[0080] However, in the second embodiment, the three-dimensional point cloud data is output from a sensor device 70. The sensor device 70 is composed of a depth sensor 71 and an image camera 72. Specifically, the depth sensor 71 is a LiDAR. With this configuration, when the depth sensor 71 performs scanning, the image camera 72 also performs imaging. Therefore, the sensor device 70 outputs image data (hereinafter referred to as "captured two-dimensional image data") 52 of the captured two-dimensional image (hereinafter referred to as "captured two-dimensional image") together with the three-dimensional point cloud data 50 of the object.

[0081] When the sensor device 70 outputs the three-dimensional point cloud data 50 and the image data 52 , the coordinate calculation device 80 receives the three-dimensional point cloud data 50 and the captured two-dimensional image data 52 and stores both in the memory unit 20 .

[0082] Furthermore, each point constituting the three-dimensional point cloud data 50 is linked to a feature point extracted from the captured two-dimensional image. The feature point extracted from the captured two-dimensional image is a second feature point derived from the three-dimensional point cloud data of the object. Here, the linking of each point constituting the three-dimensional point cloud data 50 to the feature point extracted from the captured two-dimensional image will be described below.

[0083] First, in the sensor device 70, a rotation matrix R from the position of the depth sensor 71 to the position of the image camera 72 is calculated by calibration. Ｌ２Ｃ and the translation matrix t Ｌ２Ｃ The sensor device 70 calculates a rotation matrix R from the origin of the coordinate system of the sensor device 70 to the position of the depth sensor 71 by using SLAM (Simultaneous Localization and Mapping). Ｌ and the translation matrix t Ｌ is calculated.

[0084] Next, in the sensor device 70, a rotation matrix R Ｌ and the translation matrix t Ｌ and a rotation matrix R Ｌ２Ｃ and the translation matrix t Ｌ２ＣAs a result, a rotation matrix R from the origin of the coordinate system of the sensor device 70 to the position of the image camera 72 is obtained. Ｃ and the translation matrix t Ｃ is calculated.

[0085] Then, in the sensor device 70, the rotation matrix R Ｃ and the parallel matrix t Ｃ and the angle of view information of the image camera 72, a three-dimensional point group appearing in each captured two-dimensional image is extracted. Then, the extracted three-dimensional point group is subjected to a rotation matrix R Ｃ and the translation matrix t Ｃ The points constituting the three-dimensional point cloud data 50 are reprojected onto the captured two-dimensional image using the rotation matrix R. Ｌ , translation matrix t Ｌ , rotation matrix R Ｃ , translation matrix t Ｃ , an optimization process for minimizing each error may be performed as necessary.

[0086] In the second embodiment, the combination extraction unit 12 retrieves the captured two-dimensional image data 52 from the storage unit 20 and extracts second feature points from the captured two-dimensional image. Thereafter, similar to the first embodiment, the combination extraction unit 12 compares the extracted second feature points with the first feature points of the specified two-dimensional image, compares the feature amounts of each, and extracts three or more matching combinations of the first feature points and the second feature points (see FIG. 3 ).

[0087] In the second embodiment, the processes performed by the external parameter calculation unit 13, the coordinate calculation unit 14, and the three-dimensional model display unit 15 are the same as those in the first embodiment.

[0088] [Device Operation] The operation of the coordinate calculation device 80 is performed in accordance with steps A1 to A7 shown in FIG. 7, as in the first embodiment. Also in the second embodiment, a coordinate calculation method is implemented by operating the coordinate calculation device 80. The explanation of the coordinate calculation method is the same as in the first embodiment, and will be omitted in the second embodiment. Details of the processing in step A2 are as described above.

[0089] [Program] As with the first embodiment, the program in the second embodiment may be a program that causes a computer to execute steps A1 to A7 shown in Fig. 7. By installing and executing this program in a computer, the coordinate calculation device 80 and the coordinate calculation method in the second embodiment can be realized. In this case, the processor of the computer functions as the designation receiving unit 11, the combination extraction unit 12, the external parameter calculation unit 13, the coordinate calculation unit 14, and the three-dimensional model display unit 15 and performs the processes.

[0090] Also in the second embodiment, the storage unit 20 is realized by a storage device such as a hard disk provided in the computer. Alternatively, the storage unit 20 may be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0091] The program in the second embodiment may also be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the designation receiving unit 11, combination extracting unit 12, external parameter calculating unit 13, coordinate calculating unit 14, and 3D model display unit 15.

[0092] Third Embodiment Next, in a third embodiment, a third example of a coordinate calculation device, a coordinate calculation method, and a program will be described with reference to FIG.

[0093] [Device Configuration] First, the configuration of another example of a coordinate calculation device will be described with reference to Fig. 10. Fig. 10 is a configuration diagram showing the configuration of a third example of a coordinate calculation device.

[0094] 1 and 2, the coordinate calculation device 90 is a device for calculating the coordinates of a specific location of an object on point cloud data. As shown in Fig. 10, like the coordinate calculation device 10, the coordinate calculation device 90 includes a designation receiving unit 11, a combination extraction unit 12, an external parameter calculation unit 13, a coordinate calculation unit 14, a three-dimensional model display unit 15, and a storage unit 20.

[0095] However, in the third embodiment, the storage unit 20 stores only the three-dimensional point cloud data 50. In the third embodiment, the storage unit 20 does not store image data of two-dimensional images. Therefore, in the third embodiment, the combination extraction unit 12 retrieves the three-dimensional point cloud data 50 from the storage unit 20 and extracts second feature points from the three-dimensional point cloud data 50.

[0096] Specifically, in the third embodiment, the combination extraction unit 12 extracts combinations of matching first and second feature points without specifying characteristic parts. In this case, the combination extraction unit 12 first performs edge extraction processing on the three-dimensional point cloud data 50, and further extracts a quadrangle formed by the extracted edges. Next, the combination extraction unit 12 also performs edge extraction processing on the periphery of a specific point in the specified two-dimensional image, and further extracts a quadrangle formed by the extracted edges.

[0097] Next, the combination extraction unit 12 sets quadrilateral combinations using each quadrilateral extracted from the 3D point cloud data 50 and each quadrilateral extracted from the specified 2D image, and calculates a homography matrix for each set quadrilateral combination.The combination extraction unit 12 then identifies quadrilateral combinations whose calculated matrix is ​​within a set range.

[0098] Next, for each identified combination, the combination extraction unit 12 solves the PnP problem using the vertices of each quadrangle constituting the combination to calculate the camera position and orientation for converting two-dimensional coordinates from the camera coordinates to world coordinates, i.e., the rotation matrix R and the translation matrix t ([R|t]). Then, for each identified combination, the combination extraction unit 12 reprojects the vertices of the quadrangle on the three-dimensional point cloud data onto the specified two-dimensional image using the calculated [R|t], and calculates the error between the reprojected coordinates and the coordinates of the vertices of the quadrangle on the specified two-dimensional image. Thereafter, the combination extraction unit 12 designates the combination with the smallest calculated error as the "combination of the first feature point and the second feature point."

[0099] In the third embodiment, the processes performed by the external parameter calculation unit 13, the coordinate calculation unit 14, and the three-dimensional model display unit 15 are the same as those in the first embodiment. [Device Operation] The operation of the coordinate calculation device 90 is performed in accordance with steps A1 to A7 shown in FIG. 7, as in the first embodiment. Also in the third embodiment, a coordinate calculation method is implemented by operating the coordinate calculation device 90. The explanation of the coordinate calculation method is the same as in the first embodiment, and will be omitted in the third embodiment. The details of the process in step A2 are as described above.

[0100] [Program] As with the first embodiment, the program in the third embodiment may be a program that causes a computer to execute steps A1 to A7 shown in Fig. 7. By installing and executing this program in a computer, the coordinate calculation device 90 and the coordinate calculation method in the third embodiment can be realized. In this case, the processor of the computer functions as the designation receiving unit 11, the combination extraction unit 12, the external parameter calculation unit 13, the coordinate calculation unit 14, and the three-dimensional model display unit 15 and performs the processes.

[0101] Also in the third embodiment, the storage unit 20 is realized by a storage device such as a hard disk provided in the computer. Alternatively, the storage unit 20 may be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0102] The program in the third embodiment may also be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the designation receiving unit 11, combination extracting unit 12, external parameter calculating unit 13, coordinate calculating unit 14, and 3D model display unit 15.

[0103] [Physical Configuration] A computer that implements the coordinate calculation device 10 by executing the programs in the first to third embodiments will now be described with reference to Fig. 9. Fig. 11 is a block diagram showing an example of a computer that implements the coordinate calculation device.

[0104] 11, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0105] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0106] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0107] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0108] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0109] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0110] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0111] The coordinate calculation device 10 can be realized not by a computer with a program installed, but by hardware corresponding to each unit, such as an electronic circuit. Furthermore, the coordinate calculation device 10 may be partially realized by a program and the remaining unit by hardware. In the embodiment, the computer is not limited to the computer shown in FIG. 9 .

[0112] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 18) described below, but are not limited to the following descriptions.

[0113] (Supplementary Note 1) A coordinate calculation device comprising: a designation receiving means for receiving designation of a specific point on a two-dimensional image of an object; a combination extracting means for extracting a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; an external parameter calculating means for calculating external parameters of a camera used to photograph the object, using two-dimensional coordinates on the two-dimensional image of the first feature point extracted as a combination and three-dimensional coordinates corresponding to the second feature point extracted as a combination; and a coordinate calculating means for calculating three-dimensional coordinates of an intersection point of a virtual line segment passing through the camera center of the camera and the specific point and the three-dimensional point cloud data of the object, using the calculated external parameters, and setting the intersection point whose three-dimensional coordinates are calculated as the specific point whose designation was accepted.

[0114] (Supplementary Note 2) The coordinate calculation device according to Supplementary Note 1, further comprising a three-dimensional model display means for displaying a three-dimensional model of the object on a screen using three-dimensional point cloud data of the object, and further displaying the intersection point, whose three-dimensional coordinates have been calculated, on the three-dimensional model as the designated specific point.

[0115] (Supplementary Note 3) The coordinate calculation device according to Supplementary Note 1, wherein the combination extraction means compares the first feature points with the second feature points and extracts the combinations using a result of the comparison.

[0116] (Supplementary Note 4) The coordinate calculation device according to Supplementary Note 1, wherein the combination extraction means extracts the combination in response to an instruction input from outside.

[0117] (Supplementary Note 5) The coordinate calculation device according to Supplementary Note 1, comprising a point cloud data generation means which acquires a plurality of two-dimensional images of the object, extracts feature points from each of the acquired plurality of two-dimensional images, sets two or more pair images from the plurality of two-dimensional images, extracts, for each pair of images, combinations of feature points that match each other from each of the pair of images, and further uses a camera matrix of each of the pair of images to determine, for each of the pair of images, three-dimensional coordinates of the position of the camera used for shooting and a rotation matrix, calculates three-dimensional coordinates of the feature points extracted as the combinations using the determined three-dimensional coordinates of the positions of the cameras and the rotation matrix, and creates the three-dimensional point cloud data using the feature points whose three-dimensional coordinates have been calculated.

[0118] (Supplementary Note 6) The coordinate calculation device according to Supplementary Note 1, wherein the three-dimensional point cloud data is data output from a depth sensor.

[0119] (Supplementary Note 7) A coordinate calculation method comprising: accepting a designation of a specific point on a two-dimensional image of an object; extracting a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; calculating extrinsic parameters of a camera used to photograph the object using two-dimensional coordinates on the two-dimensional image of the first feature point extracted as a combination and three-dimensional coordinates corresponding to the second feature point extracted as a combination; calculating three-dimensional coordinates of an intersection point of a virtual line segment passing through the camera center of the camera and the specific point and the three-dimensional point cloud data of the object using the calculated extrinsic parameters; and setting the intersection point whose three-dimensional coordinates are calculated as the specific point whose designation was accepted.

[0120] (Supplementary Note 8) The coordinate calculation method according to Supplementary Note 7, further comprising: displaying a three-dimensional model of the object on a screen using three-dimensional point cloud data of the object; and further displaying the intersection point, the three-dimensional coordinates of which have been calculated, on the three-dimensional model as the designated specific point.

[0121] (Supplementary Note 9) The coordinate calculation method according to Supplementary Note 7, wherein in extracting the combinations, the first feature points and the second feature points are compared, and the combinations are extracted using a result of the comparison.

[0122] (Supplementary Note 10) The coordinate calculation method according to Supplementary Note 7, wherein the extraction of the combinations comprises extracting the combinations in accordance with an instruction input from outside.

[0123] (Supplementary Note 11) The coordinate calculation method according to Supplementary Note 7 further comprises: acquiring a plurality of two-dimensional images of the object; extracting feature points from each of the acquired plurality of two-dimensional images; setting two or more pair images from the plurality of two-dimensional images; extracting, for each pair of images, combinations of feature points that match each other from each of the pair of images; using a camera matrix of each of the pair of images, determining, for each of the pair of images, three-dimensional coordinates of the position of the camera used to take the image and a rotation matrix; calculating three-dimensional coordinates of the feature points extracted as the combinations using the determined three-dimensional coordinates of the position of each of the cameras and the rotation matrix; and creating the three-dimensional point cloud data using the feature points whose three-dimensional coordinates have been calculated.

[0124] (Supplementary Note 12) The coordinate calculation method according to Supplementary Note 7, wherein the three-dimensional point cloud data is data output from a depth sensor.

[0125] (Supplementary Note 13) A computer-readable recording medium having recorded thereon a program including instructions, the program comprising: a designation receiving step of receiving designation of a specific point on a two-dimensional image of an object; causing a computer to extract a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; calculating extrinsic parameters of a camera used to photograph the object using two-dimensional coordinates on the two-dimensional image of the first feature point extracted as a combination and three-dimensional coordinates corresponding to the second feature point extracted as a combination; calculating three-dimensional coordinates of an intersection point between a virtual line segment passing through the camera center of the camera and the specific point and the three-dimensional point cloud data of the object using the calculated extrinsic parameters; and setting the intersection point whose three-dimensional coordinates are calculated as the specific point whose designation was accepted.

[0126] (Supplementary Note 14) The computer-readable recording medium according to Supplementary Note 13, wherein the program further includes instructions to cause the computer to display a three-dimensional model of the object on a screen using the three-dimensional point cloud data of the object, and to display the intersection point, the three-dimensional coordinates of which have been calculated, on the three-dimensional model as the designated specific point.

[0127] (Supplementary Note 15) The computer-readable recording medium according to Supplementary Note 13, wherein in extracting the combinations, the first feature points and the second feature points are compared, and the combinations are extracted using a result of the comparison.

[0128] (Supplementary Note 16) The computer-readable recording medium according to Supplementary Note 13, wherein the extraction of the combinations is performed in accordance with an instruction input from an external device.

[0129] (Supplementary Note 17) The computer-readable recording medium according to Supplementary Note 13, wherein the program further includes instructions to cause the computer to acquire a plurality of two-dimensional images of the object, extract feature points from each of the acquired plurality of two-dimensional images, set two or more paired images from the plurality of two-dimensional images, extract combinations of feature points that match each other from each of the paired images, use a camera matrix of each of the paired images to determine, for each of the paired images, three-dimensional coordinates of the position of the camera used to capture the image and a rotation matrix, calculate three-dimensional coordinates of the feature points extracted as the combinations using the determined three-dimensional coordinates of the positions of the cameras and the rotation matrix, and create the three-dimensional point cloud data using the feature points whose three-dimensional coordinates have been calculated.

[0130] (Supplementary Note 18) The computer-readable recording medium according to Supplementary Note 13, wherein the three-dimensional point cloud data is data output from a depth sensor.

[0131] This application claims priority based on Japanese Patent Application No. 2024-009736, filed January 25, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0132] According to the present disclosure, when identifying the position of a specific location on a 3D model from a 2D image of the specific location of an object, the accuracy of the location identification can be improved without imposing constraints on the 2D image and the 3D model. The present disclosure can be used for maintenance and management of structures, searches in specific areas, etc.

[0133] DESCRIPTION OF SYMBOLS 10 Coordinate calculation device (first embodiment) 11 Designation reception unit 12 Combination extraction unit 13 External parameter calculation unit 14 Coordinate calculation unit 15 Three-dimensional model display unit 16 Point cloud data generation unit 20 Storage unit 30 Input device 40 Display device 50 Three-dimensional point cloud data 60 Two-dimensional image for creation 61 Designated two-dimensional image 70 Sensor device 71 Depth sensor 72 Image camera 80 Coordinate calculation device (second embodiment) 90 Coordinate calculation device (third embodiment) 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. A coordinate calculation device comprising: a designation reception means for receiving a designation of a specific point on a two-dimensional image obtained by photographing an object; a combination extraction means for extracting a matching combination between a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; an external parameter calculation means for calculating external parameters of a camera used for photographing the object using the two-dimensional coordinates of the first feature point extracted as a combination on the two-dimensional image and the three-dimensional coordinates corresponding to the second feature point extracted as a combination; and a coordinate calculation means for calculating three-dimensional coordinates of an intersection point between a virtual line segment passing through the camera center of the camera and the specific point and the three-dimensional point cloud data of the object using the calculated external parameters, and setting the intersection point for which the three-dimensional coordinates have been calculated as the specific point for which the designation has been received.

2. The coordinate calculation device according to claim 1, further comprising a three-dimensional model display means for displaying a three-dimensional model of the object on a screen using the three-dimensional point cloud data of the object, and further displaying the intersection point for which the three-dimensional coordinates have been calculated as the designated specific point on the three-dimensional model.

3. The coordinate calculation device according to claim 1, wherein the combination extraction means compares the first feature point and the second feature point and extracts the combination using the result of the comparison.

4. The coordinate calculation device according to claim 1, wherein the combination extraction means extracts the combination in response to an instruction input from the outside.

5. The coordinate calculation device according to claim 1, further comprising a point cloud data generation means, wherein the point cloud data generation means acquires a plurality of two-dimensional images obtained by photographing the object, extracts feature points from each of the acquired plurality of two-dimensional images, sets two or more pairs of images from the plurality of two-dimensional images, extracts a combination of matching feature points from each image of each pair of images, further obtains the three-dimensional coordinates of the position of the camera used for photographing and the rotation matrix for each of the pair of images using the camera matrix of each of the pair of images, calculates the three-dimensional coordinates of the feature points extracted as the combination using the obtained three-dimensional coordinates of the position of each camera and the rotation matrix, and creates the three-dimensional point cloud data using the feature points for which the three-dimensional coordinates have been calculated.

6. The coordinate calculation device according to claim 1, wherein the three-dimensional point cloud data is data output from a depth sensor.

7. A coordinate calculation method, comprising: receiving a designation of a specific point on a two-dimensional image obtained by photographing an object; extracting a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; calculating external parameters of a camera used for photographing the object using the two-dimensional coordinates of the first feature point extracted as a combination on the two-dimensional image and the three-dimensional coordinates corresponding to the second feature point extracted as a combination; calculating the three-dimensional coordinates of an intersection point between a virtual line segment passing through the camera center and the specific point of the camera and the three-dimensional point cloud data of the object using the calculated external parameters; and setting the intersection point for which the three-dimensional coordinates have been calculated as the specific point for which the designation has been received.

8. The coordinate calculation method according to claim 7, wherein a three-dimensional model of the object is displayed on a screen using the three-dimensional point cloud data of the object, and the intersection point for which the three-dimensional coordinates have been calculated is displayed on the three-dimensional model as the designated specific point.

9. The coordinate calculation method according to claim 7, wherein in the extraction of the combination, the first feature point and the second feature point are compared, and the combination is extracted using the result of the comparison.

10. The coordinate calculation method according to claim 7, wherein in the extraction of the combination, the combination is extracted according to an instruction input from the outside.

11. Further, a plurality of two-dimensional images obtained by photographing the object are acquired, feature points are extracted from each of the acquired plurality of two-dimensional images, two or more pairs of images are set from the plurality of two-dimensional images, for each pair of images, a combination of feature points that match each other is extracted from each of the images of the pair of images, and further, using the camera matrices of each of the pair of images, for each of the pair of images, the three-dimensional coordinates and the rotation matrix of the position of the camera used for photographing are obtained, and the three-dimensional coordinates of the feature points extracted as the combination are calculated using the obtained three-dimensional coordinates and the rotation matrix of the position of each of the cameras, and the three-dimensional point cloud data is created using the feature points for which the three-dimensional coordinates have been calculated.

12. The coordinate calculation method according to claim 7, wherein the three-dimensional point cloud data is data output from a depth sensor.

13. A computer-readable recording medium storing a program including an instruction for causing a computer to perform: a designation reception step of receiving a designation of a specific point on a two-dimensional image obtained by photographing an object; extracting a matching combination of a first feature point extracted from the two-dimensional image and a second feature point derived from three-dimensional point cloud data of the object; calculating external parameters of a camera used for photographing the object using the two-dimensional coordinates of the first feature point extracted as a combination on the two-dimensional image and the three-dimensional coordinates corresponding to the second feature point extracted as a combination; calculating three-dimensional coordinates of an intersection point between a virtual line segment passing through the camera center of the camera and the specific point and the three-dimensional point cloud data of the object using the calculated external parameters; and designating the intersection point for which the three-dimensional coordinates have been calculated as the specific point for which the designation has been received.

14. The computer-readable recording medium according to claim 13, wherein the program further includes an instruction for causing the computer to display a three-dimensional model of the object on a screen using the three-dimensional point cloud data of the object, and display the intersection point for which the three-dimensional coordinates have been calculated as the designated specific point on the three-dimensional model.

15. The computer-readable recording medium according to claim 13, wherein in extracting the combination, the first feature point and the second feature point are compared, and the combination is extracted using the result of the comparison.

16. The computer-readable recording medium according to claim 13, wherein in extracting the combination, the combination is extracted in response to an instruction input from the outside.

17. The program further causes the computer to acquire a plurality of two-dimensional images of the object, extract feature points from each of the acquired plurality of two-dimensional images, set two or more pairs of images from the plurality of two-dimensional images, for each pair of images, extract a combination of feature points that match each other from each image of the pair of images, and further, using the camera matrix of each of the pair of images, for each of the pair of images, obtain the three-dimensional coordinates and rotation matrix of the position of the camera used for imaging, and use the obtained three-dimensional coordinates of the position of each of the cameras and the rotation matrix to calculate the three-dimensional coordinates of the feature points extracted as the combination, and create the three-dimensional point cloud data using the feature points for which the three-dimensional coordinates have been calculated. The computer-readable recording medium according to claim 13, further comprising an instruction.

18. The computer-readable recording medium according to claim 13, wherein the three-dimensional point cloud data is data output from a depth sensor.

Images