Point cloud data processing apparatus, point cloud data processing method, and recording medium

Abstract

The present application provides a point cloud data processing device, a point cloud data processing method and a program, which can obtain the correctness of the determination result when determining the region of the object as the detection object in the image, and can effectively utilize the determination result. In the point cloud data processing device (11), the processor acquires first shape information representing the shape feature of the first object, determines the object region of the second object corresponding to the first shape information according to the image discrimination, selects the second object point cloud data corresponding to the object region in the point cloud data based on the object region, acquires second shape information representing the shape feature of the second object based on the second object point cloud data, compares the first shape information and the second shape information, and determines whether the second object is the first object.

Description

TECHNICAL FIELD

[0001] The present application relates to a point cloud data processing apparatus, a point cloud data processing method, and a program. BACKGROUND

[0002] It is known that a technique of acquiring point cloud data composed of a large number of points having three-dimensional information of a surface of an object by reflection on the surface of the object by a laser scanner or the like. Further, each point data of the point cloud data constituting an object acquired by a laser scanner or the like can correspond to each pixel constituting an image obtained by photographing the same object.

[0003] In Patent Literature 1, a technique is described in which a mark region is determined in an image photographed by an imaging device, point cloud data corresponding to the region is extracted, and the position of the mark photographed on the image and the size of the mark are determined on the basis of the extracted point cloud data.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent Application Publication No. 2012-83157 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] Here, a case where a region in which an object of detection (a mark in the technique of Patent Literature 1) is photographed is determined in an image as described in Patent Literature 1 is considered. In a case where an object of detection is determined in an image, since the image is two-dimensional information, it is sometimes difficult to accurately determine a region in which an object that is a desired object of detection is photographed. For example, in an image that is two-dimensional information, information in a depth direction in three-dimensional information is lost, so it is sometimes difficult to distinguish a cylinder and a cube by a viewpoint of the image. In Patent Literature 1, a problem in determination of a region in an image is not mentioned.

[0009] The present application is made in view of such a situation, and an object thereof is to provide a point cloud data processing apparatus, a point cloud data processing method, and a program that can obtain correctness of a determination result when a region of an object that is an object of detection is determined in an image and can effectively utilize the determination result.

[0010] MEANS FOR SOLVING THE PROBLEMS

[0011] As an embodiment of the present invention for achieving the above-mentioned objectives, a point cloud data processing apparatus includes a memory and a processor. The memory stores in association an image of a first object as the subject and point cloud data representing three-dimensional information including at least a large number of points on the surface of the first object. The memory also stores in association the positions of pixels constituting the image and the point data constituting the point cloud data. The processor acquires first shape information representing the shape of the first object, determines the object region of a second object identified from the image, i.e., the second object corresponding to the first shape information, selects second object point cloud data corresponding to the object region from the point cloud data based on the object region, acquires second shape information representing the shape of the second object based on the second object point cloud data, compares the first shape information and the second shape information, and determines whether the second object is the first object.

[0012] According to this method, first shape information representing the shape features of a first object representing a detection target can be obtained. Additionally, second object point cloud data corresponding to the object region of a second object can be selected, and second shape information can be obtained based on the second object point cloud data. The second object corresponds to the first shape information determined from the image. Furthermore, this method compares the first and second shape information to determine whether the second object identified in the image is the same as the first object being detected. Therefore, it can determine the accuracy of the region determination result and effectively utilize the region determination result.

[0013] Preferably, the processor assigns attributes to the second object point cloud data based on the determination result.

[0014] Preferably, the processor causes the display unit to show the determination result.

[0015] Preferably, the first shape information is at least one of information related to the type of shape of the first object, information related to the surface of the first object, and information related to the size of the first object. Preferably, the shape of the first object is cylindrical.

[0016] Preferably, the processor acquires second shape information based on the second object's point cloud data through at least one of Hough transform processing, RANSAC algorithm, and a detector that has undergone machine learning. Preferably, the first object is a pipe.

[0017] Another aspect of the present invention is a point cloud data processing method using a point cloud data processing device. The point cloud data processing device includes a memory and a processor. The memory associates an image of a first object as the subject with point cloud data representing three-dimensional information including at least a large number of points on the surface of the first object. It also associates the positions of pixels constituting the image with the point data constituting the point cloud data. The point cloud data processing method includes: a step of acquiring first shape information representing the shape of the first object using the processor; a step of determining an object region of a second object identified from the image, i.e., a second object corresponding to the first shape information; a step of selecting second object point cloud data corresponding to the object region from the point cloud data based on the object region; a step of acquiring second shape information representing the shape of the second object based on the second object point cloud data; and a step of comparing the first shape information and the second shape information to determine whether the second object is the first object.

[0018] As another aspect of the present invention, a program enables a point cloud data processing apparatus to perform a point cloud data processing method. The point cloud data processing apparatus includes a memory and a processor. The memory stores, in association, an image of a first object as the subject and point cloud data representing three-dimensional information including at least a large number of points on the surface of the first object. It also stores, in association, the positions of pixels constituting the image and the point data constituting the point cloud data. The program causes the processor to perform the following steps: acquiring first shape information representing the shape of the first object; determining, based on the image, the object region of a second object corresponding to the first shape information; selecting, based on the object region, second object point cloud data corresponding to the object region from the point cloud data; acquiring, based on the second object point cloud data, second shape information representing the shape of the second object; and comparing the first shape information and the second shape information to determine whether the second object is the first object.

[0019] Invention Effects

[0020] According to the present invention, by comparing the first shape information and the second shape information, it is determined whether the region identified in the image is the first object to be detected, so it is possible to obtain the correctness of the region determination result and to effectively utilize the region determination result. Attached Figure Description

[0021] Figure 1 It is a diagram that represents images of an object taken from different viewpoints.

[0022] Figure 2 This is a block diagram illustrating the functional structure of a point cloud data processing device.

[0023] Figure 3 This is a conceptual diagram representing a method of acquiring images and point cloud data from a 3D measuring instrument and inputting them into a point cloud data processing device.

[0024] Figure 4 This is a concept diagram illustrating a laser scanner and photographic device mounted on a 3D measuring instrument.

[0025] Figure 5 This is a conceptual diagram illustrating the corresponding positional relationship between images and point cloud data.

[0026] Figure 6 This is a flowchart illustrating the point cloud data processing method.

[0027] Figure 7 It is a diagram that shows the situation where the area of ​​an object has been defined.

[0028] Figure 8 It is a diagram that illustrates the selected point cloud data based on the object region.

[0029] Figure 9 This figure illustrates an example of cylinder detection performed by the second shape information acquisition unit.

[0030] Figure 10 This diagram illustrates cylinder detection using the RANSAC algorithm. Detailed Implementation

[0031] The preferred embodiments of the point cloud data processing apparatus, point cloud data processing method and program involved in the present invention will now be described with reference to the accompanying drawings.

[0032] First, an example of the effective application of the present invention will be described. In cases where an object is identified as a detection target with a three-dimensional shape from an image that is two-dimensional data, false detections sometimes occur due to dimensional degradation in the image.

[0033] Figure 1 This diagram shows images U1 and U2 obtained by taking pictures of an object V with a cuboid shape and an object W with a cylindrical shape from different viewpoints. In the following description, for example, the case of detecting an object with a cylindrical shape, such as a pipe (the first object), will be explained.

[0034] exist Figure 1 In (A), an image U1 obtained by photographing objects V and W in the XZ plane is shown. Figure 1 (B) shows the image U2 obtained by taking pictures of objects V and W in the XY plane.

[0035] exist Figure 1In image U1 shown in (A), although objects V and W have different shapes, they are captured as the same rectangular region (object region V(1) and object region W(2)). Therefore, when judging objects V and W manually or automatically based on image U1, object region V(1) and object region W(2) are sometimes judged as regions of objects with the same shape. However, in Figure 1 In the image U2 shown in (B), it can be seen that the object region V(2) and the object region W(2) have different shapes, and the object V and the object W have different shapes.

[0036] In detecting cylindrical objects, sometimes objects V and W are identified as having a cylindrical shape based on image U1, and regions (object region V(1) and object region W(1)) are determined separately in image U1. In this case, since object W has a cylindrical shape, object region W(1) is a valid detection, but since object V has a cuboid shape, the determination of object region V(1) becomes an incorrect detection. Thus, it is not possible to effectively utilize such determination results, such as the incorrect determination of regions that include objects that are not the target of detection. Therefore, in this invention, a technique is proposed to determine the correctness of the determined regions based on such image U1.

[0037] Figure 2 This is a block diagram illustrating an example of the functional structure of the point cloud data processing apparatus 11 according to the present invention. The point cloud data processing apparatus 11 is, for example, composed of a computer. The point cloud data processing apparatus 11 can be mounted on a three-dimensional measuring instrument 1 (see reference 1). Figure 3 It can also be composed of a computer separate from the three-dimensional measuring instrument 1.

[0038] The point cloud data processing device 11 includes an image acquisition unit 13, a point cloud data acquisition unit 15, a control unit 17, a display control unit 19, an operation unit 23, and a storage unit (memory) 21. Furthermore, although not shown in the figures, the point cloud data processing device 11 possesses the hardware structure of a typical computer.

[0039] Image acquisition unit 13 acquires image 5 of the first object to be detected as the subject. Image acquisition unit 13 is configured as a computer data input unit. Additionally, point cloud data acquisition unit 15 acquires point cloud data representing three-dimensional information of a large number of points on the surface of the first object to be detected. Image acquisition unit 13 and point cloud data acquisition unit 15 are configured as a computer data input unit.

[0040] Here, the positions of each pixel constituting image 5 correspond to the three-dimensional information of each point data constituting point cloud data 7. Point cloud data 7 is composed of a large number of points, each of which has three-dimensional information (three-dimensional coordinates) of the part reflected from the surface of an object. Furthermore, the correspondence between image 5 and point cloud data 7 will be explained in detail later.

[0041] The control unit 17 executes, for example, a program stored in the storage unit 21 by a CPU (Central Processing Unit) (not shown) mounted on a computer.

[0042] The control unit 17 includes a first shape information acquisition unit 31, an object region determination unit 33, a point cloud data selection unit 35, a second shape information acquisition unit 37, a determination unit 39, and an attribute assignment unit 41.

[0043] The display control unit 19 is implemented by the CPU executing a program stored in the storage unit 21. The display control unit 19 controls the display to the display unit 9. For example, point cloud data or images are displayed on the display unit 9. In addition, the display unit 9 can display the determination result made by the determination unit 39 and notify the user.

[0044] The storage unit 21 stores the image 5 acquired by the image acquisition unit 13 and the point cloud data 7 acquired by the point cloud data acquisition unit 15. In addition, in order to realize the various functions of the point cloud data processing device 11, it stores various programs executed by the CPU.

[0045] The operation unit 23 consists of a keyboard, a mouse (for example, a pointing device), etc. The user inputs various commands into the point cloud data processing device 11 via the operation unit 23.

[0046] Figure 3 This is a conceptual diagram illustrating a method by which an image 5 and point cloud data 7 are acquired by a 3D measuring instrument 1 and input into a point cloud data processing device 11. Additionally, Figure 4 This is a concept diagram illustrating the laser scanner 101 and the photographic device (camera device) 102 mounted on the 3D measuring instrument 1.

[0047] like Figure 3 As shown, the 3D measuring instrument 1 acquires point cloud data 7 of the object A being measured and an image 5 of the object A.

[0048] The 3D measuring instrument 1 is a measuring instrument in the following manner, namely, from the mounted laser scanner 101 (refer to...) Figure 4 The laser scanner 101 emits a laser pulse and measures the distance using a time-of-flight method, which measures the time until the laser pulse is reflected from the surface of the object A being measured and returns. Additionally, the laser scanner 101 acquires angular information about the emission direction of the laser pulse, with the center of rotation of the laser scanner 101 as a reference.Figure 3 The data contains information about the horizontal direction α and the vertical direction β. Then, based on the measured distance and the angle information of the laser pulse's emission direction, the three-dimensional information of the laser pulse's reflection point is obtained. The laser scanner 101 acquires three-dimensional information of a large number of points by changing the horizontal direction α and the vertical direction β of the laser pulse's emission direction at predetermined intervals. The laser scanner 101 is not limited to time-of-flight laser scanners; it can also acquire point cloud data using phase difference methods or triangulation methods.

[0049] The three-dimensional measuring instrument 1 uses a camera 102 with an optical axis L (see reference). Figure 4 Image 5 is acquired. The imaging device 102 includes an imaging element such as a CCD (Charge Coupled Device) (not shown). The imaging element has multiple pixels composed of photoelectric switching elements arranged in two dimensions along the x-direction (horizontal direction) and y-direction (vertical direction). On the upper surface of the multiple pixels, there are color filters (not shown) arranged in a two-dimensional Bayer pattern, such as R (red), G (green), and B (blue) filters. Moreover, image 5, a color image, is acquired by the imaging device 102. Furthermore, image 5 acquired by the imaging device 102 is not limited to a color image; it can also be a black and white image.

[0050] exist Figure 4 In the example shown, a laser scanner 101 and a photographic device 102 are configured with the optical axis L aligned using a semi-transparent reflector 103. Furthermore, the positional relationship between the laser scanner 101 and the photographic device 102 is not particularly limited, as long as it is known. For example, the photographic device 102 can be mounted on the laser scanner 101 without aligning the optical axis. Additionally, the 3D measuring instrument 1 can simultaneously or sequentially acquire point cloud data 7 and image 5. Moreover, since the pixels constituting image 5 and the point data constituting point cloud data 7 have a corresponding positional relationship, the laser scanner 101 and the photographic device 102 are not limited to... Figure 4 The arrangement of the laser scanner 101 and the imaging device 102 shown can be arranged as long as the positional relationship is known in advance. For example, the semi-transparent reflector 103 may not be provided, and the imaging device 102 may be arranged on the upper part of the laser scanner 101 with the same imaging direction.

[0051] Figure 5 This is a conceptual diagram illustrating the corresponding positional relationship between image 5 and point cloud data 7. Image 5 is image data composed of pixels P arranged in a two-dimensional array. Because image 5 is a color image, pixel P has a value for each of R, G, and B. Figure 5The image 5 shows pixel P with coordinates (Px, Py) and point data Q corresponding to pixel P. Multiple point data Q constitute point cloud data 7, which includes three-dimensional coordinates (x, y, z) as positional information, the R, G, and B values ​​of the corresponding pixel P, and information on the reflection intensity (I) of the laser from the laser scanner 101. Furthermore, as in... Figure 4 As explained, since the positional relationship between the laser scanner 101 and the imaging device 102 is known, it is possible to obtain the correspondence between pixel P and point data Q.

[0052] As explained above, the pixel P constituting the image 5 and the point data Q constituting the point cloud data 7 have a corresponding positional relationship. The image 5 and the point cloud data 7 maintain the aforementioned positional relationship and are stored in the storage unit 21 of the point cloud data processing device 11.

[0053] Next, the point cloud data processing method using the point cloud data processing device 11 (and the procedure for executing the point cloud data processing method using the point cloud data processing device 11) will be described. Furthermore, the image 5 and the point cloud data 7 are pre-stored in the storage unit 21 of the point cloud data processing device 11.

[0054] Figure 6 This is a flowchart illustrating a point cloud data processing method. First, the series of steps constituting the point cloud data processing method are explained, followed by a detailed explanation of each step.

[0055] First shape information is acquired by the first shape information acquisition unit 31 (first shape information acquisition step: step S10). The first shape information is information representing the shape features of the first object being detected. Next, the object region determination unit 33 determines the object region of the second object in the image (object region determination step: step S11). The object region of the second object refers to the region on the image 5 on which the second object, as determined from image 5, has the shape features represented by the first shape information, is captured. Then, the point cloud data selection unit 35 selects the second object point cloud data corresponding to the object region from the point cloud data 7 based on the object region (point cloud data selection step: step S12). Next, the second shape information acquisition unit 37 acquires second shape information representing the shape features of the second object based on the second object point cloud data (second shape information acquisition step: step S13). Then, the determination unit 39 compares the first shape information and the second shape information to determine whether the second object is the first object (determination step: step S14). Then, the attribute assignment unit 41 assigns attributes to the second object point cloud data based on the determination result. If the second object is the first object, the attribute assignment unit 41 assigns an attribute to the selected point cloud data (attribute assignment step: step S15); if the second object is not the first object, the attribute is not assigned to the selected point cloud data (attribute assignment step: step S16).

[0056] Next, the steps described above will be explained. In the following explanation, the steps taken through... Figure 1 The image U1, which is explained, illustrates the case of detecting cylindrical objects from objects V and W.

[0057] (First shape information acquisition step)

[0058] The first shape information acquisition step is performed by the first shape information acquisition unit 31. The first shape information is information representing the shape features of a first object that is the object to be detected. Since the object to be detected is determined before detection, the first shape information has already been determined. The first shape information acquisition unit 31 acquires the first shape information stored in the storage unit 21, or acquires the first shape information input by the user via the operation unit 23.

[0059] The first shape information is not particularly limited; it can be any information representing the shape features of the first object being detected. Furthermore, the first shape information is preferably information representing the three-dimensional features of the shape of the first object. For example, the first shape information includes at least one of information related to the shape type of the first object, information related to the surface of the first object, and information related to the size of the first object. Information related to the shape type of the first object, for example, refers to information such as the first object having a cylindrical shape, a cuboid shape, or a sphere shape. Information related to the surface of the first object includes, for example, the surface roughness of the first object. Information related to the size of the first object, for example, in the case of a cylindrical shape, is information related to dimensions representing the shape features of the first object, such as maximum curvature, radius, height, and volume.

[0060] In the example of detecting an object V with a cylindrical shape described below, "cylindrical shape" is obtained as the first shape information representing the shape feature of object V.

[0061] (Steps for determining the object region)

[0062] The object region determination step is performed by the object region determination unit 33. The object region determination unit 33 determines the object region identified in the image U1 that contains the object (second object) corresponding to the first shape information.

[0063] Figure 7 This is a diagram showing the situation where object region V(1) and object region W(1) are determined in image U1 by the first shape information "cylindrical shape".

[0064] The object region determination unit 33 determines object regions using various methods. For example, the object region determination unit 33 is composed of a detector trained using deep learning, which determines object regions V(1) or W(1) in the image U1 by segmentation. Furthermore, existing models such as FCN (Fully Convolutional Network), SegNet, and Pix2Pix can be used for segmentation, or a model matching the object region determination method performed by the object region determination unit 33 can be created separately. Additionally, the learning of the detector constituting the object region determination unit 33 can utilize deep learning, or machine learning can be used in a more general sense.

[0065] Alternatively, the object region determination unit 33 can also determine the object region V(1) and the object region W(1) based on the instructions input by the user via the operation unit 23. For example, in the image U1, the user may input an instruction to fill the object region V(1) and the object region W(1) using the operation unit 23, and the object region determination unit 33 may determine the object region V(1) and the object region W(1) based on that instruction. Alternatively, the user may specify the interior of the object region V(1) and the object region W(1) in the image U1 via the operation unit 23, and the object region determination unit 33 may determine the object region V(1) and the object region W(1) based on that specification and by determining the texture in the image U1 through image processing.

[0066] (Point cloud data selection steps)

[0067] The point cloud data selection step is performed by the point cloud data selection unit 35. Each pixel constituting the image U1 corresponds to the point data constituting the point cloud data. Therefore, the point cloud data selection unit 35 selects the point cloud data corresponding to the pixels constituting the object region V(1) and object region W(1) determined by the object region determination unit 33.

[0068] Figure 8 This is a diagram that illustrates the selected point cloud data based on the object region. Figure 8 The image shows the object region V(1) and object region W(1) determined on image U1 during the object region determination step. Additionally, in... Figure 8 The figure shows point cloud data K corresponding to object region V(1) and representing three-dimensional information on the surface of object V, and point cloud data J corresponding to object region W(1) and representing three-dimensional information on the surface of object W.

[0069] Each pixel constituting object region V(1) and object region W(1) has a positional relationship with each point data constituting point cloud data.

[0070] Specifically, each pixel constituting the object region V(1) has a positional relationship corresponding to each point data constituting the point cloud data K. In addition, each pixel constituting the object region W(1) has a positional relationship corresponding to each point data constituting the point cloud data J.

[0071] (Second shape information acquisition step)

[0072] The second shape information acquisition step is performed by the second shape information acquisition unit 37. The second shape information acquisition unit 37 acquires shape information (second shape information) corresponding to the object region V(1) and the object region W(1) based on the selected point cloud data K and point cloud data J.

[0073] The second shape information acquisition unit 37 performs cylinder detection on point cloud data K and point cloud data J based on the previously acquired first shape information "cylinder shape" to acquire second shape information. An example of cylinder detection performed by the second shape information acquisition unit 37 will be described below.

[0074] Figure 9 This diagram illustrates an example of cylinder detection performed by the second shape information acquisition unit 37. The second shape information acquisition unit 37 selects an arbitrary point from both point cloud data K and point cloud data J. Figure 9 In the case shown, point data K(1) is selected from point cloud data K, and point data J(1) is selected from point cloud data J. Then, the second shape information acquisition unit 37 calculates the principal curvatures (maximum curvature and minimum curvature) for point data K(1) and point data J(1). Then, the second shape information acquisition unit 37 extracts points where the minimum curvature is 0 and the directions of the two principal curvatures are orthogonal. Figure 9 In the case shown, since point data K(1) is point data of the surface of a cuboid, the direction of maximum curvature and the direction of minimum curvature are not orthogonal, and the minimum curvature is not 0. On the other hand, for point data J(1), the direction of maximum curvature g and the direction of minimum curvature h are orthogonal, and the minimum curvature is 0. Thus, the second shape information acquisition unit 37 acquires second shape information such as "point cloud data K does not have three-dimensional information of the surface of a cylindrical object" and "point cloud data J has three-dimensional information of the surface of a cylindrical object". Furthermore, the above-described method for detecting cylinders is one example, and the second shape information acquisition unit 37 can detect cylinders using other methods. For example, the second shape information acquisition unit 37 can also be configured with a detector that has undergone machine learning, and the cylinder can be detected by this detector. In addition, the second shape information acquisition unit 37 can also perform robust estimation to improve detection accuracy. For example, the second shape information acquisition unit 37 can use the RANSAC (Random Sample Consensus) algorithm to improve detection accuracy.

[0075] Figure 10 This diagram illustrates cylinder detection using the RANSAC algorithm.

[0076] exist Figure 10 In the case shown, the second shape information acquisition unit 37 performs cylinder detection using the RANSAC algorithm on the point cloud data of the three-dimensional information of the surfaces of objects L1 and L2, which have cylindrical shapes. The second shape information acquisition unit 37 extracts multiple point data points as described above, where the maximum curvature direction is orthogonal to the minimum curvature direction and the minimum curvature direction is 0. The extracted point data thus possess characteristics such as similar maximum curvature (similar cylinder radius), similar minimum curvature direction (similar cylinder axis direction), or connectivity (connected if there are nearby points with the same properties).

[0077] In addition to the methods described above, the second shape information acquisition unit 37 can also use a method that utilizes the Hough transform or a method that combines the RANSAC algorithm and the Hough transform to perform cylinder detection.

[0078] (Judgment Steps)

[0079] The determination step is performed by the determination unit 39. By comparing the first shape information obtained in the first shape information step with the second shape information obtained in the second shape information acquisition step, it is determined whether the second object is the first object. That is, the determination unit 39 determines the area of ​​the object to be detected, or determines the correctness of the determination of the object area V(1) and the object area W(1).

[0080] Specifically, the determination unit 39 determines whether the first shape information "cylindrical shape" acquired in the first shape information acquisition step is consistent with the second shape information acquired in point cloud data K and point cloud data J. The determination unit 39 compares the first shape information "cylindrical shape" with the second shape information "three-dimensional information of the surface of an object with a cylindrical shape in point cloud data K" in point cloud data K, and determines that the second object corresponding to object region V(1) is not the first object to be detected. On the other hand, the determination unit 39 compares the first shape information "cylindrical shape" with the second shape information "three-dimensional information of the surface of an object with a cylindrical shape in point cloud data J" in point cloud data J, and determines that the second object corresponding to object region W(1) is the first object to be detected. For example, the determination result of the determination unit 39 is displayed on the display unit 9 and reported to the user under the control of the display control unit 19. Thus, the user can identify whether the region determined by the object region determination unit 33 is the region corresponding to the object to be detected.

[0081] (Steps for assigning attributes)

[0082] The attribute assignment step is performed by the attribute assignment unit 41. The attribute assignment unit 41 assigns attributes to the point cloud data J of the object region that the determination unit 39 determines corresponds to the object being detected. On the other hand, the attribute assignment unit 41 does not assign attributes to the point cloud data K of the object region that the determination unit 39 determines does not correspond to the object being detected. For example, the attribute assignment unit 41 assigns an attribute such as "cylindrical shape" to the point data constituting the point cloud data J. Therefore, the attribute assignment unit 41 can accurately assign attributes to the point cloud data of the object being detected.

[0083] As explained above, the object is identified as a cylindrical object based on image U1, and the correctness of the determination results is judged for the determined object regions V(1) and W(1). Thus, the determination results of object regions V(1) and W(1) in image U1 can be effectively utilized.

[0084] <Other>

[0085] In the above embodiments, the similar hardware structures of the control unit 17 (processing unit) and the display control unit 19 that perform various processes are various processors as shown below. These processors include general-purpose processors that execute software (programs) and function as various processing units, such as CPUs (Central Processing Units); processors whose circuit structure can be modified after manufacturing, such as FPGAs (Field Programmable Gate Arrays), which are programmable logic devices (PLDs); and processors with specially designed circuit structures for performing specific processes, such as ASICs (Application Specific Integrated Circuits), which are dedicated circuits.

[0086] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of CPU and FPGA). Alternatively, a single processor can constitute multiple processing units. As examples of a single processor constituting multiple processing units, firstly, there are computers such as client computers or servers, where a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Secondly, there are systems-on-a-chip (SoCs), which use a single integrated circuit (IC) chip to implement the overall system functionality including multiple processing units. In this way, various processing units are constructed using one or more of the aforementioned processors as hardware structures.

[0087] Furthermore, more specifically, the similar hardware structure of these various processors is a circuit composed of circuit elements such as semiconductor components.

[0088] The aforementioned structures and functions can be suitably implemented using any hardware, software, or a combination of both. For example, the present invention can also be applied to a program that causes a computer to perform the aforementioned processing steps (processing sequence), a computer-readable recording medium (non-transitory recording medium) that records such a program, or a computer capable of installing such a program.

[0089] The examples of the present invention have been described above, but the present invention is not limited to the above embodiments. Needless to say, various modifications can be made without departing from the spirit of the present invention.

[0090] Symbol Explanation

[0091] 1: 3D measuring instrument

[0092] 5: Image

[0093] 7: Point cloud data

[0094] 9: Display Section

[0095] 11: Point cloud data processing device

[0096] 13: Image Acquisition Department

[0097] 15: Point Cloud Data Acquisition Department

[0098] 17: Control Department

[0099] 19: Display Control Unit

[0100] 21: Storage Department

[0101] 23: Operations Department

[0102] 31: First Shape Information Acquisition Unit

[0103] 33: Object Region Determination Section

[0104] 35: Point Cloud Data Selection Department

[0105] 37: Second Shape Information Acquisition Unit

[0106] 39: Judgment Department

[0107] 41: Attribute Assignment Department

[0108] 101: Laser Scanner

[0109] 102: Photographic Installation

[0110] 103: Semi-transparent reflector

Claims

1. A point cloud data processing apparatus comprising a storage and a processor, the storage storing, in association with each other, an image of a first object as a subject to be detected and point cloud data representing three-dimensional information of a large number of points on a surface of at least the first object, and storing, in association with each other, positions of pixels constituting the image and point data constituting the point cloud data, wherein the processor acquires first shape information representing a feature of a shape of the first object, determines an object region of a second object discriminated from the image, that is, the second object corresponding to the first shape information, selects second object point cloud data corresponding to the object region in the point cloud data based on the object region, acquires second shape information representing a feature of a shape of the second object based on the second object point cloud data, compares the first shape information and the second shape information, and performs a determination of whether the second object is the first object, and acquires the second shape information by selecting an arbitrary one point from the second object point cloud data and calculating a maximum curvature and a minimum curvature with respect to the one point, and acquiring the second shape information based on whether directions of the maximum curvature and the minimum curvature are orthogonal and whether the minimum curvature is 0. 2.The point cloud data processing apparatus according to claim 1, wherein the shape of the first object is a cylindrical shape. 3.The point cloud data processing apparatus according to claim 2, wherein the processor acquires the second shape information based on the second object point cloud data by at least one of a Hough transform process, a RANSAC algorithm, and a detector subjected to machine learning. 4.A point cloud data processing apparatus comprising a storage and a processor, the storage storing, in association with each other, an image of a first object as a subject to be detected and point cloud data representing three-dimensional information of a large number of points on a surface of at least the first object, and storing, in association with each other, positions of pixels constituting the image and point data constituting the point cloud data, wherein the processor acquires first shape information representing a feature of a shape of the first object, determines an object region of a second object discriminated from the image, that is, the second object corresponding to the first shape information, selects second object point cloud data corresponding to the object region in the point cloud data based on the object region, acquires second shape information representing a feature of a shape of the second object based on the second object point cloud data, compares the first shape information and the second shape information, and performs a determination of whether the second object is the first object, and the processor acquires the second shape information based on the second object point cloud data by at least one of a Hough transform process, a RANSAC algorithm, and a detector subjected to machine learning. 5.The point cloud data processing apparatus according to any one of claims 1 to 4, wherein the processor attributes a property to the second object point cloud data based on a result of the determination. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 6. The point cloud data processing apparatus according to any one of claims 1 to 4, wherein the processor causes a display section to display a result of the determination.

7. The point cloud data processing apparatus according to any one of claims 1 to 4, wherein the first shape information is at least one of information related to a type of a shape of the first object, information related to a surface of the first object, and information related to a size of the first object.

8. The point cloud data processing apparatus according to claim 4, wherein the shape of the first object is a cylindrical shape.

9. The point cloud data processing apparatus according to any one of claims 1 to 4, wherein the first object is a pipe.

10. A point cloud data processing method, which is a point cloud data processing method using a point cloud data processing apparatus that includes a storage and a processor, the storage stores, in association with each other, an image that takes a first object that is a detection target as a subject and point cloud data that represents three-dimensional information of a large number of points on a surface of at least the first object, and stores, in association with each other, positions of pixels that constitute the image and each point data that constitutes the point cloud data, wherein The point cloud data processing method includes: by the processor a step of acquiring first shape information representing a feature of a shape of the first object; a step of determining an object region of a second object, which is determined from the image, corresponding to the first shape information; a step of selecting, based on the object region, second object point cloud data corresponding to the object region in the point cloud data; a step of acquiring second shape information representing a feature of a shape of the second object based on the second object point cloud data; and a step of comparing the first shape information and the second shape information to determine whether the second object is the first object, in the step of acquiring the second shape information, one point is selected from the second object point cloud data, and a maximum curvature and a minimum curvature are calculated for the one point, and the second shape information is acquired based on whether a direction of the maximum curvature and a direction of the minimum curvature are orthogonal and whether the minimum curvature is 0.

11. A point cloud data processing method, which is a point cloud data processing method using a point cloud data processing apparatus that includes a storage and a processor, the storage stores, in association with each other, an image that takes a first object that is a detection target as a subject and point cloud data that represents three-dimensional information of a large number of points on a surface of at least the first object, and stores, in association with each other, positions of pixels that constitute the image and each point data that constitutes the point cloud data, wherein The point cloud data processing method includes: by the processor a step of acquiring first shape information representing a feature of a shape of the first object; a step of determining an object region of a second object, which is determined from the image, corresponding to the first shape information; a step of selecting, based on the object region, second object point cloud data corresponding to the object region in the point cloud data; a step of acquiring second shape information representing a feature of a shape of the second object based on the second object point cloud data; and a step of comparing the first shape information and the second shape information to determine whether the second object is the first object, in the step of acquiring the second shape information, the second shape information is acquired based on the second object point cloud data by at least one of a Hough transform process, a RANSAC algorithm, and a detector subjected to machine learning.

12. A recording medium recording a program causing a point cloud data processing apparatus to perform a point cloud data processing method, the point cloud data processing apparatus including a storage and a processor, the storage storing an image in which a first object that is a detection target is a subject and point cloud data representing three-dimensional information of a large number of points on a surface of at least the first object in association with each other, and storing positions of pixels constituting the image and point data constituting the point cloud data in association with each other, wherein The program causes the processor to perform: a step of acquiring first shape information representing a feature of a shape of the first object; a step of determining an object region of a second object discriminated from the image, that is, the second object corresponding to the first shape information; a step of selecting, based on the object region, second object point cloud data corresponding to the object region in the point cloud data; a step of acquiring second shape information representing a feature of a shape of the second object based on the second object point cloud data; a step of comparing the first shape information and the second shape information, and performing determination of whether the second object is the first object, In the step of acquiring the second shape information, one point is selected from the second object point cloud data, and a maximum curvature and a minimum curvature are calculated with respect to the one point, and the second shape information is acquired based on whether the direction of the maximum curvature and the direction of the minimum curvature are orthogonal and whether the minimum curvature is 0.

13. A recording medium recording a program causing a point cloud data processing apparatus to perform a point cloud data processing method, the point cloud data processing apparatus having a storage and a processor, the storage storing an image of a first object that is a detection target as a subject and point cloud data representing three-dimensional information of a large number of points on a surface of at least the first object in association with each other, and storing positions of pixels constituting the image and each point data constituting the point cloud data in association with each other, wherein the program causes the processor to perform: a step of acquiring first shape information representing a feature of a shape of the first object; a step of determining an object region of a second object discriminated from the image, that is, the second object corresponding to the first shape information; a step of selecting, based on the object region, second object point cloud data corresponding to the object region in the point cloud data; a step of acquiring second shape information representing a feature of a shape of the second object based on the second object point cloud data; a step of comparing the first shape information and the second shape information, and performing determination of whether the second object is the first object, in the step of acquiring the second shape information, the second shape information is acquired based on the second object point cloud data by at least one of a Hough transform process, a RANSAC algorithm, and a detector in which machine learning is performed. ​ ​

Images