Three-dimensional point group data processing device, three-dimensional point group data processing method, and three-dimensional point group data processing program

Abstract

A three-dimensional point group data processing device according to one embodiment of the present invention comprises: a color information application unit that applies color information to each of a plurality of first points which are included in three-dimensional point group information about a first space and which correspond to positions in the first space; an invisible point group exclusion unit that excludes an invisible point group that cannot be seen from a first viewpoint among the plurality of first points in the first space; a two-dimensional point group projection unit that, by means of conversion processing using a visible point group which, among the plurality of first points, excludes the invisible point group and includes a plurality of second points, generates two-dimensional point group information which includes a plurality of third points corresponding respectively to the plurality of second points; a corresponding point acquisition unit that acquires a plurality of fourth points selected from the plurality of third points on the basis of the two-dimensional point group information; and a corresponding point output unit that outputs a plurality of fifth points corresponding to the plurality of fourth points, from among the plurality of second points. The plurality of third points each include the color information applied to one of the plurality of second points corresponding to the relevant point.

Description

Three-dimensional point cloud data processing device, three-dimensional point cloud data processing method, and three-dimensional point cloud data processing program

[0001] The embodiments relate to a three-dimensional point cloud data processing device, a three-dimensional point cloud data processing method, and a three-dimensional point cloud data processing program.

[0002] In sensor fusion (combined positioning) using a camera and LiDAR (Light Detection and Ranging), external calibration is required to calculate the relative positions of the camera and LiDAR in terms of their positions and attitudes (angles). Hereinafter, external calibration is also simply referred to as calibration. Calibration can be performed using multiple correspondences between pixels in the camera image and points in the LiDAR point cloud.

[0003] There are two types of calibration for a camera and a LiDAR: one that uses a target such as a calibration board, and one that does not use a target. Hereinafter, calibration that uses a target will be referred to as a target-based method. Calibration that does not use a target will be referred to as a targetless method.

[0004] In the target-based method, a known target such as that described above is used to obtain corresponding points from a LiDAR point cloud. Corresponding points are points included in the LiDAR point cloud that correspond to pixels in a camera image. In the target-based method, the known target must be measured simultaneously using a camera and a LiDAR. Furthermore, such measurements must be performed multiple times by changing the measurement position. In the target-based method described above, for example, when performing calibration on data that has already been measured, it may be difficult to measure the known target using the camera and LiDAR because the camera and LiDAR used for the measurement cannot be identified. In such cases, a targetless method is used.

[0005] In targetless methods, as described in, for example, Patent Document 1, corresponding points are estimated based on feature points in a camera image and a LiDAR point cloud. The feature points correspond to edges, for example. More specifically, in Patent Document 1, an object is extracted from a three-dimensional point cloud representing a real-world scene, and missing portions of the extracted object are repaired. Furthermore, image edges are extracted from a captured image taken within the scene. Then, perspective projection is performed on the reconstructed object using position and orientation information indicating the position and orientation of the camera when the captured image was taken within the scene, thereby extracting model edges corresponding to the object's contours. The position and orientation information is then corrected according to the amount of deviation between the extracted image edges and the extracted model edges.

[0006] However, the amount of feature points and the similarity between the camera image and the LiDAR point cloud depend on the space. Therefore, the accuracy of the estimation of the corresponding points varies depending on the space in which the corresponding points are estimated. Therefore, the robustness of the calibration decreases, and accurate calibration may be difficult. In such cases, for example, the corresponding points are manually selected to perform accurate calibration.

[0007] Furthermore, LiDAR measurement uses a technique called LiDAR-SLAM (Simultaneous Localization And Mapping). LiDAR-SLAM is a technique for measuring the position and orientation of a LiDAR device while moving and correcting it in real time. By measuring with LiDAR-SLAM and correcting the LiDAR point cloud measured while moving, a highly accurate three-dimensional point cloud (3D point cloud, or three-dimensional point cloud) can be obtained. This has the advantage of reducing calibration errors.

[0008] Japanese Patent Application Laid-Open No. 2019-192299

[0009] If it is possible to accurately select points corresponding to pixels in camera images in a wide-area 3D point cloud such as that measured by LiDAR-SLAM, the accuracy of calibration can be improved.

[0010] However, because a 3D point cloud is a collection of points arranged three-dimensionally, it has the characteristic that points behind a certain point are visible through the cloud. This poses a problem in that it is difficult to select corresponding points from a two-dimensional screen (2D screen). More specifically, when selecting a position (corresponding point) in three-dimensional space from a 2D screen, a point closer to or further from the correct corresponding point may be mistakenly selected. In other words, a point displaced in the depth direction may be selected as the corresponding point.

[0011] Furthermore, there is a problem that the operation of selecting corresponding points from the 2D screen takes time. More specifically, in order to find and select corresponding points from the 3D point cloud, the operation of changing the display position and orientation of the 3D point cloud on the 2D screen may take time.

[0012] The present invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a means for facilitating the operation of selecting corresponding points from a point group in three-dimensional space while suppressing erroneous selection of corresponding points.

[0013] In one embodiment, a three-dimensional point cloud data processing device includes a color information assignment unit that assigns color information to each of a plurality of first points included in three-dimensional point cloud information for a first space, which is a three-dimensional space, and which correspond to a position in the first space; an impossible viewpoint group exclusion unit that excludes an impossible viewpoint group from the plurality of first points in the first space that cannot be seen from a first viewpoint; a 2D point cloud projection unit that generates two-dimensional point cloud information for a two-dimensional space, including a plurality of third points corresponding to each of the plurality of second points, by a conversion process using a visible point group that includes a plurality of second points from the plurality of first points excluding the impossible viewpoint group; a corresponding point acquisition unit that acquires a plurality of fourth points selected from the plurality of third points based on the two-dimensional point cloud information; and a corresponding point output unit that outputs a plurality of fifth points from the plurality of second points that correspond to the plurality of fourth points, and each of the plurality of third points has the color information assigned to one of the plurality of second points corresponding to it.

[0014] According to the embodiment, it is possible to provide a means for preventing erroneous selection of corresponding points while facilitating the operation of selecting corresponding points from a point group in a three-dimensional space.

[0015] FIG. 1 is a block diagram showing an example of the hardware configuration of a 3D point cloud data processing device according to an embodiment. FIG. 2 is a block diagram showing an example of the functional configuration of the 3D point cloud data processing device according to an embodiment. FIG. 3 is a flowchart showing an example of the operation of the 3D point cloud data processing device according to an embodiment. FIG. 4 is a diagram for explaining the elimination of impossible viewpoints in the operation of the 3D point cloud data processing device according to an embodiment. FIG. 5 is a diagram for explaining the elimination of impossible viewpoints in the operation of the 3D point cloud data processing device according to an embodiment. FIG. 6 is a diagram for explaining the elimination of impossible viewpoints in the operation of the 3D point cloud data processing device according to an embodiment. FIG. 7 is a diagram for explaining the elimination of impossible viewpoints in the operation of the 3D point cloud data processing device according to an embodiment. FIG. 8 is a diagram showing an example of a display presented in the operation of the 3D point cloud data processing device according to an embodiment. FIG. 9 is a diagram showing an example of a display presented in the operation of the 3D point cloud data processing device according to a modified example.

[0016] Hereinafter, several embodiments will be described with reference to the drawings. In the following description, components having the same functions and configurations will be given the same reference numerals.

[0017] The three-dimensional point cloud data processing device described below generates two-dimensional point cloud data (2D point cloud) based on the 3D point cloud data, for example, when acquiring corresponding points of the 3D point cloud data corresponding to pixels of a captured image. The three-dimensional point cloud data processing device is configured to present the 2D point cloud generated as described above to a user, allowing the user to select points included in the 2D point cloud. The three-dimensional point cloud data processing device also acquires corresponding points of the 3D point cloud data based on the user's selection. The acquired corresponding points are used, for example, in external calibration to calculate the relative positions of the camera and LiDAR in sensor fusion using a camera and LiDAR.

[0018] 1. Embodiment A three-dimensional point cloud data processing device according to an embodiment will be described.

[0019] 1.1 Hardware Configuration First, the hardware configuration of a 3D point cloud data processing apparatus 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the hardware configuration of a 3D point cloud data processing apparatus according to an embodiment.

[0020] The three-dimensional point cloud data processing device 10 includes a control circuit 11 , a storage 12 , and a user interface 13 .

[0021] The control circuit 11 is a circuit that controls the overall components of the three-dimensional point cloud data processing device 10. The control circuit 11 includes a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). The ROM of the control circuit 11 stores programs and the like used in various processes in the three-dimensional point cloud data processing device 10. The CPU of the control circuit 11 controls the entire three-dimensional point cloud data processing device 10 in accordance with the programs stored in the ROM of the control circuit 11. The RAM of the control circuit 11 is used as a working area for the CPU of the control circuit 11.

[0022] The storage 12 can store information used in various processes in the 3D point cloud data processing device 10.

[0023] The user interface 13 is an interface that manages communication between a user and the control circuit 11. The user interface 13 includes input devices and output devices, such as a mouse, a keyboard, a touch panel, and a display.

[0024] 1.2 Functional Configuration The functional configuration of the 3D point cloud data processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the 3D point cloud data processing device according to the embodiment.

[0025] The CPU of the control circuit 11 loads a program stored in the ROM of the control circuit 11 or the storage 12 into the RAM of the control circuit 11. The CPU of the control circuit 11 then interprets and executes the program loaded into the RAM of the control circuit 11. As a result, the 3D point cloud data processing device 10 realizes the functions of a measurement data acquisition unit 21, a parameter acquisition unit 22, a color information assignment unit 23, an invisible viewpoint group exclusion unit 24, a 2D point cloud projection unit 25, a memory unit 26, a corresponding point acquisition unit 27, and a corresponding point output unit 28.

[0026] The measurement data acquisition unit 21 acquires measurement data, for example, from outside the device. The measurement data includes, for example, 3D point cloud data and captured image data for a certain real space. The 3D point cloud data includes a point cloud measured by LiDAR at multiple different measurement positions in LiDAR-SLAM, etc. Each point in the 3D point cloud data includes, for example, position information in three-dimensional space. The file format of the 3D point cloud data may be any format that contains three-dimensional position information of the point cloud. The file extension of the 3D point cloud data is, for example, ".e57" or ".ply." The captured image data corresponds to the camera image used to select corresponding points. The file format of the captured image data is, for example, a format that contains the two-dimensional positions of pixels and RGB information (RGB values ​​in the RGB color space). The file extension of the captured image data is, for example, ".jpg" or ".png." Note that, hereinafter, the captured image data will also be simply referred to as the camera image.

[0027] The parameter acquisition unit 22 acquires parameter information, for example, from outside the device. The parameter information includes internal parameter information and external parameter information related to the camera that captured the camera image acquired by the measurement data acquisition unit 21. The internal parameter information is numerical information related to the focal length of the camera lens, distortion coefficient, size of the camera image, etc. The external parameter information is three-dimensional position information and orientation information related to the camera when capturing the camera image. The orientation information is information that allows calculation of the orientation of the camera in three-dimensional space. The orientation information is, for example, one or more pieces of information such as Euler angles and quaternions. The external parameter information may be information about the approximate position and orientation of the camera in a 3D point cloud space. For example, the information may be information obtained by placing a virtual object in the 3D point cloud space, or information measured using an inertial measurement unit or the like when capturing an image with a camera in real space. In external calibration, accurate camera external parameters are calculated based on the camera external parameter information acquired by the parameter acquisition unit 22 as described above.

[0028] The color information assigning unit 23 determines color information for each point of the 3D point cloud data acquired by the measurement data acquiring unit 21. The color information is, for example, RGB values. The color information assigning unit 23 calculates HSV values ​​in an HSV color space, for example, based on normal information related to the 3D point cloud data. The color information assigning unit 23 converts the calculated HSV values ​​into RGB values. The color information assigning unit 23 also calculates RGB values ​​based on, for example, intensity information for each point of the 3D point cloud data measured by LiDAR. The color information assigning unit 23 also determines RGB values ​​for each point of the 3D point cloud data, for example, based on an RGB gradation proportional to the value of any one of the X coordinate, Y coordinate, and Z coordinate in three-dimensional space, or a category based on segmentation.

[0029] The color information assigning unit 23 assigns the color information of each point of the 3D point cloud data determined as described above to the point. That is, the color information assigning unit 23 associates each point of the 3D point cloud data with the color information corresponding to the point.

[0030] The impossible viewpoint group exclusion unit 24 excludes point groups (impossible viewpoint groups) that cannot be seen from the position and orientation in three-dimensional space (shooting viewpoint) when the camera image was captured, from the 3D point cloud data acquired by the measurement data acquisition unit 21. The impossible viewpoint group exclusion unit 24 determines impossible viewpoints included in the impossible viewpoint group based on external parameter information acquired by the parameter acquisition unit 22. The impossible viewpoint group exclusion unit 24 also generates a visible point group composed of the point group after the impossible viewpoint group has been excluded from the 3D point cloud data.

[0031] The 2D point cloud projection unit 25 uses the internal parameters and external parameters acquired by the parameter acquisition unit 22 to perform projection transformation (perspective projection transformation) of the visible point cloud generated by the invisible viewpoint group exclusion unit 24 into a 2D point cloud in two-dimensional space. The 2D point cloud projection unit 25 performs projection transformation, for example, by matrix calculation. The 2D point cloud projection unit 25 transforms the 2D point cloud so that each point included in the 2D point cloud corresponds to a point included in the visible point cloud. During the projection transformation, the 2D point cloud projection unit 25 assigns multiple identifiers IDs to each of the points included in the 2D point cloud. The multiple identifiers IDs are configured to associate each point in the 2D point cloud with its position in three-dimensional space (the point in the 3D point cloud data corresponding to that point). The 2D point cloud projection unit 25 stores information associating the identifiers IDs of the 2D point cloud with positions in three-dimensional space in the storage unit 26. The 2D point cloud inherits the assigned identifiers IDs. That is, in processing the 2D point cloud after projection transformation, the identifier ID assigned to each point of the 2D point cloud can be used.

[0032] During projection transformation, the 2D point cloud projection unit 25 performs processing so that each point in the 2D point cloud retains the color information of the corresponding point in the 3D point cloud data, which is assigned by the color information assignment unit 23. In other words, the 2D point cloud generated by the 2D point cloud projection unit 25 can be said to be a point cloud whose visibility is improved by the color information. Furthermore, in the 2D point cloud, points do not overlap in the depth direction in each line of sight from the shooting viewpoint due to the elimination of invisible viewpoints by the invisible viewpoint group elimination unit 24. As a result, the 2D point cloud can be said to be a point cloud that suppresses erroneous selection in the depth direction by using the 2D point cloud when selecting corresponding points from the 3D point cloud data.

[0033] The storage unit 26 stores, for example, a 3D point cloud position information DB. The 3D point cloud position information DB is information that associates the identifiers IDs of 2D point clouds, which are stored in the storage unit 26 by the 2D point cloud projection unit 25 as described above, with positions in three-dimensional space.

[0034] The corresponding point acquisition unit 27 presents the camera image acquired by the measurement data acquisition unit 21 and the 2D point cloud generated by the 2D point cloud projection unit 25 to the user, for example, by color display using a display or the like. The corresponding point acquisition unit 27 is also configured to allow the user to select multiple points from the displayed camera image and 2D point cloud, for example, by operating a mouse or touching a touch panel. Each of the multiple points on the 2D point cloud selected by the user in this manner is, for example, a point in the 2D point cloud corresponding to a corresponding point in the 3D point cloud data. The corresponding point acquisition unit 27 may be configured to display the visible point cloud generated by the invisible viewpoint group exclusion unit 24 alongside the camera image and the 2D point cloud.

[0035] Furthermore, for each point selected by the user when the 2D point cloud is presented, the corresponding point acquisition unit 27 stores a combination of the order of selection and the identifier ID corresponding to the point. Although not shown, the corresponding point acquisition unit 27 may store the combination in the storage unit 26.

[0036] The corresponding point output unit 28 acquires positions in three-dimensional space (points in the 3D point cloud data) corresponding to each point in the point cloud based on the identifiers ID of the points stored by the corresponding point acquisition unit 27 and the 3D point cloud position information DB in the storage unit 26. The corresponding point output unit 28 outputs a plurality of points in the 3D point cloud data acquired in this manner as corresponding point information. That is, each point in the 3D point cloud data acquired as described above is output as a corresponding point. In the corresponding point information, each corresponding point is associated with, for example, the order in which the points in the 2D point cloud corresponding to the corresponding point were selected by the user.

[0037] 1.3 Operation Next, the operation of the three-dimensional point cloud data processing apparatus 10 according to the embodiment will be described.

[0038] 1.3.1 Overall Flow First, the overall flow of the operation of the 3D point cloud data processing apparatus 10 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the operation of the 3D point cloud data processing apparatus according to the embodiment.

[0039] When the operation of the three-dimensional point cloud data processing device starts (Start), the measurement data acquisition unit 21 acquires measurement data (St1). Also, in St1, the parameter acquisition unit 22 acquires parameter information. Then, the processing proceeds to St2.

[0040] The color information assigning unit 23 determines color information for each point of the 3D point cloud data acquired by the measurement data acquiring unit 21. The color information assigning unit 23 also assigns the determined color information for each point of the 3D point cloud data (St2). Then, the processing proceeds to St3.

[0041] The impossible viewpoint group excluding unit 24 excludes the impossible viewpoint group from the 3D point cloud data by setting the position determined by the external parameters as the position of the projection viewpoint corresponding to the shooting viewpoint (St3). The impossible viewpoint group excluding unit 24 also generates a visible point group from the 3D point cloud data with the impossible viewpoint group excluded. Details of the exclusion of the impossible viewpoint group will be described later. Then, the processing proceeds to St4.

[0042] The 2D point cloud projection unit 25 converts the visible point cloud, to which color information has been assigned and from which invisible viewpoints have been removed, into a 2D point cloud by projection transformation (St4). Each point in the 2D point cloud generated by projection transformation corresponds to a point in the visible point cloud. Furthermore, each point in the 2D point cloud has color information of the corresponding point in the 3D point cloud data, which was assigned by the color information assignment unit 23 in the processing of St2. Then, the processing proceeds to St5.

[0043] The 2D point cloud projection unit 25 also assigns an identifier ID to each point in the 2D point cloud. The 2D point cloud projection unit 25 then stores a 3D point cloud position information DB that associates the 2D point cloud identifier ID with a position in three-dimensional space in the storage unit 26 (St5). Then, the processing proceeds to St6.

[0044] The corresponding point acquisition unit 27 presents the 2D point cloud converted by the 2D point cloud projection unit 25 and the camera image captured near the corresponding viewpoint to the user in color on a display or the like (St6). This allows the user to select multiple points from the displayed 2D point cloud and camera image by mouse operation or touch operation on the touch panel. Each of the multiple points corresponds to a corresponding point. Details of the presentation of the 2D point cloud will be described later. The process then proceeds to St7.

[0045] The corresponding point acquisition unit 27 also stores, for each point selected by the user, a combination of the order in which the points were selected and the corresponding identifier ID (Step 7).Then, the processing proceeds to Step 8.

[0046] The corresponding point output unit 28 acquires the position in three-dimensional space for each point stored by the corresponding point acquisition unit 27, by referring to the 3D point cloud position information DB based on the identifier ID. The corresponding point output unit 28 outputs corresponding point information including the multiple points in the 3D point cloud data acquired in this manner as multiple corresponding points (St8).

[0047] With the above processing, the operation of the 3D point cloud data processing apparatus 10 according to the embodiment is completed.

[0048] 1.3.2 Exclusion of Impossible Viewpoints Next, a method for excluding impossible viewpoints in the processing of St3 will be described with reference to FIGS. 4, 5, 6, and 7. FIGS. 4, 5, 6, and 7 are diagrams for explaining the exclusion of impossible viewpoints in the operation of the 3D point cloud data processing device according to the embodiment. FIG. 4A is a schematic diagram showing a point cloud in space before compression. FIG. 4B is a schematic diagram showing a point cloud in a cross section taken along the dotted line in FIG. 4A. FIG. 5 is a schematic diagram for explaining the visible point cloud and impossible viewpoints in the cross section taken along the dotted line in FIG. 4B before compression. FIG. 6A is a schematic diagram showing a point cloud in space after compression. FIG. 6B is a schematic diagram showing a point cloud in a cross section taken along the dotted line in FIG. 6A. FIG. 7 is a schematic diagram for explaining the visible point cloud and impossible viewpoints in the cross section taken along the dotted line in FIG. 6B after compression.

[0049] The impossible viewpoint group exclusion unit 24 according to the embodiment excludes the projection viewpoint P corresponding to the shooting viewpoint when the camera image is captured. ｐ The space is compressed by treating the 3D point cloud data as a spherical shell-shaped object with the projection viewpoint P at its center. That is, the impossible viewpoint group elimination unit 24 compresses the 3D point cloud data into a spherical shell shape without changing the diameter of the sphere depending on the size of the 3D point cloud data. More specifically, the impossible viewpoint group elimination unit 24 compresses the 3D point cloud data into a spherical shell shape without changing the diameter of the sphere depending on the size of the 3D point cloud data. ｐ The invisible viewpoint group excluding unit 24 compresses the visible point group into a spherical shell shape so that each point of the visible point group remains in the same line of sight as before compression. In other words, the invisible viewpoint group excluding unit 24 ｐ and each point of the visible point cloud are reduced on a line connecting these two points, thereby compressing the 3D point cloud data into a spherical shell shape. ｐ It is possible to compress the space without changing the view from the outside.

[0050] Position vector (coordinates) v' of each point in the compressed point cloud ｉ is, for example, the position vector v of the point before compression. ｉ In the following description, the projection viewpoint P ｐ The point P closest to ｍｉｎ The distance between ｍｉｎ In addition, the projection viewpoint P ｐ The farthest point P from ｍａｘ The distance between ｍａｘ In addition, the point cloud after compression is expressed as ｐ The point P' closest to ｍｉｎ The distance between ｍｉｎ In addition, the point cloud after compression is expressed as ｐ The farthest point P' from ｍａｘ The distance between ｍａｘ In equation (1), the position vector v ｉ and v' ｉ , and the unit vector e have a "→" symbol on each of them.

[0051]

[0052] The invisible viewpoint group excluding unit 24 may, for example, determine the distance R ｍｉｎ and distance R ｍａｘ By adjusting the ratio, it is possible to adjust the ratio of invisible viewpoints to be excluded relative to the number of points in the 3D point cloud data before compression.

[0053] The impossible viewpoint group exclusion unit 24 also excludes the projection viewpoint P ｐ The compression ratio may be changed depending on the distance from the point to the point in the visible point cloud. In this case, the position vector v′ of each point in the compressed point cloud is ｉ is expressed by the position vector of the following formula (2), for example. In formula (2), the function f(v ｉ ) is the position vector v ｉ is a function that takes as an argument and returns the compression ratio. ｉ ) is, for example, a sigmoid function. In this case, the projection viewpoint P ｐ The closer the visible point cloud is to the projection viewpoint P, the denser it becomes. ｐ The visible points are compressed so that the more distant they are, the sparser they become.

[0054]

[0055] The above-described method for excluding invisible viewpoints reduces erroneous determinations of visible points and invisible viewpoints. To further explain, when dealing with a large-scale point cloud such as space, it is necessary to fit the point cloud within a sphere of a predetermined radius. Therefore, the diameter of the sphere may be increased, or the point cloud may be scaled without changing the diameter of the sphere. However, when the diameter of the sphere is increased, the increase in the diameter of the sphere may result in invisible viewpoints being erroneously determined as visible points. Furthermore, when scaling a point cloud without changing the diameter of the sphere, the positional relationship between the projection viewpoint and each point changes, which may result in invisible viewpoints not being removed or visible points being removed.

[0056] According to the method for excluding an impossible viewpoint group according to the embodiment, the impossible viewpoint group excluding unit 24 can compress the 3D point cloud data into a spherical shell shape without changing the diameter of the sphere depending on the size of the 3D point cloud data. ｐThe space is compressed without changing the positional relationship between the projection viewpoint and each point. Therefore, by increasing the diameter of the sphere that contains the point cloud, it is possible to prevent invisible viewpoints from being erroneously determined to be visible points. It is also possible to prevent invisible viewpoints from being unable to be removed or visible points from being removed due to changes in the positional relationship between the projection viewpoint and each point. It is also possible to prevent points that would normally be considered visible points from being removed because they are located farther away than the diameter of a specified sphere from the projection viewpoint.

[0057] 1.3.3 Presentation of 2D Point Cloud Next, the presentation of the 2D point cloud to the user in the processing of St6 will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a display presented during the operation of the 3D point cloud data processing device according to the embodiment.

[0058] In the processing of St6, the correspondence point acquisition unit 27 presents the 2D point cloud converted by the 2D point cloud projection unit 25 and the camera image to the user on a display, for example. The correspondence point acquisition unit 27 also displays, for example, a visible point cloud, which is a 3D point cloud. As shown in Fig. 8, the correspondence point acquisition unit 27 may display the visible point cloud alongside the 2D point cloud and the camera image. Note that each point in the 2D point cloud shown in Fig. 8 has color information assigned by the color information assignment unit 23.

[0059] By presenting the corresponding point acquisition unit 27 in this manner, the user can sequentially select corresponding points from the 2D point cloud and the camera image by operating a mouse or touching a touch panel. In FIG. 8 , as an example, five points s1, s2, s3, s4, and s5 on the 2D point cloud selected by the user are shown at corresponding positions in the 2D point cloud. Furthermore, five points t1, t2, t3, t4, and t5 on the camera image selected by the user, which correspond to points s1, s2, s3, s4, and s5, respectively, are shown at corresponding positions in the camera image. Furthermore, in the visible point cloud of FIG. 8 , each of points s1, s2, s3, s4, and s5 is shown at a position corresponding to the point in the visible point cloud. The number of points selected by the user is not limited to five, and may be two or more.

[0060] 1.4 Effects of the Embodiment According to the embodiment, it is possible to provide a means for preventing erroneous selection of corresponding points while facilitating the operation of selecting corresponding points from a point cloud in a three-dimensional space. The effects of this are described below.

[0061] The 3D point cloud data processing device 10 according to the embodiment includes a color information assigning unit 23, an invisible viewpoint group excluding unit 24, a 2D point cloud projection unit 25, a corresponding point acquisition unit 27, and a corresponding point output unit 28. The color information assigning unit 23 assigns color information to each point included in 3D point cloud data for a real space. The invisible viewpoint group excluding unit 24 excludes invisible viewpoint groups from the points included in the 3D point cloud data that cannot be seen from the shooting viewpoint of the captured image for the real space. The 2D point cloud projection unit 25 generates a 2D point cloud for a two-dimensional space by transformation using a visible point group that excludes the invisible viewpoint groups from the points included in the 3D point cloud data. Each of the multiple points included in the 2D point cloud corresponds to a multiple point included in the visible point group. Furthermore, each point in the 2D point cloud has color information assigned to the corresponding point in the visible point cloud. The corresponding point acquisition unit 27 acquires multiple points selected from the 2D point cloud based on the 2D point cloud with color information. The corresponding point output unit 28 outputs corresponding point information including a plurality of points of the 3D point cloud data corresponding to the selected plurality of points of the 2D point cloud.

[0062] With the above-described configuration, the 3D point cloud data processing device 10 according to the embodiment allows the user to select points in the color-displayed 2D point cloud when selecting corresponding points from 3D point cloud data that correspond to pixels in a captured image. This eliminates the need to manipulate the viewpoint in 3D space when selecting corresponding points. This facilitates the selection of corresponding points. Furthermore, the 3D point cloud processing device 10 according to the embodiment generates the 2D point cloud presented to the user based on a visible point cloud, excluding points from the 3D point cloud data that are not visible from the camera's viewpoint. In other words, the 2D point cloud does not include points in the depth direction of the 3D point cloud data that correspond to points that are not visible from the camera's viewpoint. This reduces erroneous selection of points in the depth direction of the viewpoint when selecting corresponding points. This improves the accuracy of the selection of corresponding points.

[0063] 2. Modifications In the above embodiment, the user selects multiple points from the 2D point cloud presented by the corresponding point acquisition unit 27 in the process of St6, but this is not limiting. The 3D point cloud data processing device 10 may be configured to allow the user to select a line instead of selecting a point.

[0064] In the 3D point cloud data processing device 10 according to the modified example, the correspondence point acquisition unit 27 is configured to be able to perform feature amount matching on, for example, a 2D point cloud generated by the 2D point cloud projection unit 25. For example, the correspondence point acquisition unit 27 performs feature amount matching on feature points on a line for a set of a camera image and a 2D point cloud, or a set of a camera image, a 2D point cloud, and a visible point cloud. With this configuration, the user can select one or more lines from the 2D point cloud based on the result of feature amount matching performed by the correspondence point acquisition unit 27. Note that the feature amount matching may be any method that enables selection of a line from a 2D point cloud.

[0065] In the above example, the corresponding point acquisition unit 27 is configured to be able to perform feature amount matching, but this is not limiting. The 3D point cloud data processing device 10 according to the modified example may further include a functional configuration for performing feature amount matching in addition to the measurement data acquisition unit 21, the parameter acquisition unit 22, the color information assignment unit 23, the invisible viewpoint group exclusion unit 24, the 2D point cloud projection unit 25, the storage unit 26, the corresponding point acquisition unit 27, and the corresponding point output unit 28.

[0066] Furthermore, for each line selected by the user, the corresponding point acquisition unit 27 stores a combination of the order in which the lines were selected and the identifiers IDs corresponding to the points included in the line. Although not shown, the corresponding point acquisition unit 27 may store the combination in the storage unit 26.

[0067] The corresponding point output unit 28 acquires positions in three-dimensional space (points of the 3D point cloud data) corresponding to each point constituting the line based on the identifiers ID of the points constituting the line stored by the corresponding point acquisition unit 27 and the 3D point cloud position information DB of the storage unit 26. In the corresponding point information, each corresponding point is associated with, for example, the order in which the line corresponding to the corresponding point was selected by the user. Note that the corresponding point output unit 28 may be configured to select one or more points from the points constituting the line stored by the corresponding point acquisition unit 27. In this case, points of the 3D point cloud data corresponding to each selected point are set as corresponding points.

[0068] The configurations of the measurement data acquisition unit 21, parameter acquisition unit 22, color information assignment unit 23, invisible viewpoint group exclusion unit 24, 2D point cloud projection unit 25, and memory unit 26 of the three-dimensional point cloud data processing device 10 relating to the modified example are the same as those of the three-dimensional point cloud data processing device 10 relating to the embodiment.

[0069] The operation of the 3D point cloud data processing device 10 according to the modified example will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of a display presented during the operation of the 3D point cloud data processing device according to the modified example. Note that each point of the 2D point cloud shown in Fig. 9 has color information assigned by the color information assigning unit 23, similar to the 2D point cloud shown in Fig. 8.

[0070] The processes of St1 to St5 of the modified example are similar to those of the embodiment.

[0071] In the process of St6, the corresponding point acquisition unit 27 performs feature matching on, for example, the camera image, the 2D point cloud, and the visible point cloud. Furthermore, the corresponding point acquisition unit 27 presents the camera image, the 2D point cloud, and the visible point cloud to the user based on, for example, the result of feature matching. As a result, as shown in FIG. 9 , one or more lines may be selected from the displayed 2D point cloud by the user's mouse operation or touch operation on the touch panel. In FIG. 9 , as an example, three lines s6, s7, and s8 on the 2D point cloud selected by the user are displayed at corresponding positions in the 2D point cloud. Furthermore, three lines t6, t7, and t8 on the camera image selected by the user, which correspond to lines s6, s7, and s8, respectively, are displayed at corresponding positions in the camera image. Furthermore, in the visible point cloud of FIG. 9 , each of the lines s6, s7, and s8 is displayed at a position corresponding to the line in the visible point cloud.

[0072] In the process of St7, the corresponding point acquisition unit 27 stores, for example, for each line selected by the user, a combination of the order in which the lines were selected and the identifiers IDs corresponding to the points included in the line.

[0073] In the processing of St8, the corresponding point output unit 28 outputs, for example, corresponding point information including, as a plurality of corresponding points, a plurality of points of the 3D point cloud data corresponding to one or more lines selected by the user.

[0074] With the above processing, the operation of the 3D point cloud data processing apparatus 10 according to the modified example is completed.

[0075] The modified example also provides the same effects as the embodiment.

[0076] According to a modified example, the 3D point cloud data processing device 10 is configured to be able to acquire corresponding points based on the result of feature matching by the corresponding point acquisition unit 27. This makes it easier to select points in the 2D point cloud that correspond to corresponding points. This configuration also reduces the erroneous selection of corresponding points. Furthermore, by allowing the user to select lines, corresponding points can be efficiently selected.

[0077] 3. Others: The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0078] 10... Three-dimensional point cloud data processing device 11... Control circuit 12... Storage 13... User interface 21... Measurement data acquisition unit 22... Parameter acquisition unit 23... Color information assignment unit 24... Unacceptable viewpoint group elimination unit 25... 2D point cloud projection unit 26... Storage unit 27... Corresponding point acquisition unit 28... Corresponding point output unit DB... 3D point cloud position information

Claims

1. A three-dimensional point cloud data processing device comprising: a color information assignment unit that assigns color information to each of a plurality of first points included in three-dimensional point cloud information about a first space, which is a three-dimensional space, and which correspond to a position in the first space; an invisible viewpoint group exclusion unit that excludes an invisible viewpoint group from among the plurality of first points in the first space that cannot be seen from a first viewpoint; a 2D point cloud projection unit that generates two-dimensional point cloud information about a two-dimensional space, which includes a plurality of third points corresponding to each of the plurality of second points, by a conversion process using a visible point cloud that includes a plurality of second points excluding the invisible viewpoint group from among the plurality of first points; a corresponding point acquisition unit that acquires a plurality of fourth points selected from the plurality of third points based on the two-dimensional point cloud information; and a corresponding point output unit that outputs a plurality of fifth points from the plurality of second points that correspond to the plurality of fourth points, wherein each of the plurality of third points has the color information assigned to one of the plurality of second points corresponding to it.

2. A three-dimensional point cloud data processing device according to claim 1, further comprising a measurement data acquisition unit that acquires the three-dimensional point cloud information, wherein the corresponding point acquisition unit presents the two-dimensional point cloud information to a user by displaying it in color.

3. A three-dimensional point cloud data processing device as described in claim 2, wherein the corresponding point acquisition unit performs feature matching for feature points using the two-dimensional point cloud information and the captured image of the first space, and each of the multiple fourth points is a point selected using the result of the feature matching.

4. A three-dimensional point cloud data processing device according to claim 1, wherein the 2D point cloud projection unit assigns a plurality of identifiers to each of the plurality of third points, which associate the plurality of second points with the plurality of third points, and the corresponding point output unit determines the plurality of fifth points based on the plurality of identifiers.

5. A three-dimensional point cloud data processing device comprising: a corresponding point acquisition unit that presents to a user two-dimensional point cloud information for a two-dimensional space, including a plurality of second points corresponding to each of a plurality of first points in three-dimensional point cloud information for a first space, which is a three-dimensional space, and a captured image of the first space; wherein the plurality of first points are a point cloud from which invisible viewpoints that cannot be seen from a first viewpoint have been removed; and the user is configured to be able to select points in three-dimensional space that correspond to pixels of the captured image based on the presented two-dimensional point cloud information and the captured image.

6. The three-dimensional point cloud data processing device according to claim 5, wherein the corresponding point acquisition unit presents the plurality of first points to the user in addition to the two-dimensional point cloud information and the captured image.

7. A three-dimensional point cloud data processing method comprising: assigning color information to each of a plurality of first points included in three-dimensional point cloud information for a first space and corresponding to a position in the first space; excluding a group of invisible points in the first space that cannot be seen from a first viewpoint from the plurality of first points; generating two-dimensional point cloud information for a two-dimensional space including a plurality of third points corresponding to each of the plurality of second points by a conversion process using a visible point cloud including a plurality of second points excluding the group of invisible points from the plurality of first points; obtaining a plurality of fourth points selected from the plurality of third points based on the two-dimensional point cloud information; and outputting a plurality of fifth points from the plurality of second points that correspond to the plurality of fourth points, wherein generating the two-dimensional point cloud information is performed so that each of the plurality of third points holds the color information assigned to one of the plurality of second points corresponding to the point.

8. A three-dimensional point cloud data processing method comprising: presenting to a user two-dimensional point cloud information for a two-dimensional space, including a plurality of second points corresponding to each of a plurality of first points in three-dimensional point cloud information for a first space, which is a three-dimensional space, and a captured image of the first space; wherein the plurality of first points is a point cloud from which invisible viewpoints that cannot be seen from a first viewpoint have been removed; and the user is able to select points in the three-dimensional space that correspond to pixels of the captured image based on the presented two-dimensional point cloud information and the captured image.

9. A program for causing each unit of the three-dimensional point cloud data processing device according to any one of claims 1 to 6 to function as such.

Images