Point cloud information processing device, point cloud information processing method, and point cloud information processing program

Abstract

To provide a point cloud information processing device and the like, capable of enhancing robustness in aligning positions of a plurality of pieces of point cloud information.SOLUTION: A device includes: an image analyzer 31 for analyzing image information captured from different viewpoints, recognizing different regions in each image, assigning a label to each of the regions, and generating labeled image information; a point cloud labeling unit 32 for assigning, to each point of point cloud information in the different viewpoints, a label of a corresponding region in the labeled image information based on positional information of each point, thereby generating labeled point cloud information; and a point cloud integration unit 33 for aligning positions of the labeled point cloud information by using a label common in the plurality of labeled point cloud information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for processing point cloud information. [Background technology]

[0002] Laser scanners are known as surveying devices that acquire 3D data of surveyed objects. Laser scanners scan with laser light, which is a distance measurement light, to acquire point cloud information of the surveyed object. Point cloud information captures the object as a collection of points, and is data that includes position information (3D coordinates) of each point.

[0003] Point cloud information does not include areas that are shadowed (blind spots) from the viewpoint of the laser scanner. This is called occlusion. Therefore, to create a 3D model without occlusion, point clouds are acquired from multiple different viewpoints and then integrated. This requires the process of matching the point clouds acquired from different viewpoints.

[0004] In response to this, an object recognition device has been disclosed that automatically extracts a point cloud corresponding to the target object based on measured point cloud data and supports the alignment (also called registration) of point clouds by matching only point clouds that have the same shape attributes (see Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] WO2014 / 155715A1 publication Summary of the Invention [Problem to be solved by the invention]

[0006] However, the method of extracting a point cloud corresponding to the surveyed object based solely on point cloud data, as in Patent Document 1, has the problem that alignment cannot be performed properly when it includes point clouds corresponding to objects with few point cloud features, such as thin walls, or objects such as leaves fluttering in the wind, or moving objects such as automobiles.

[0007] The present invention has been made to solve these problems, and its purpose is to provide a point cloud information processing device, a point cloud information processing method, and a point cloud information processing program that can improve the robustness of aligning multiple point cloud information using image information and point cloud information from multiple different viewpoints using a surveying device. [Means for solving the problem]

[0008] In order to achieve the above object, a point cloud information processing apparatus according to the present invention is provided. Including surveyed objects The first image information and the second image information captured from the second viewpoint Including the survey object an image analysis unit that analyzes each of the second image information, recognizes different regions in each image, labels each region, and generates first labeled image information and second labeled image information; and a group of points including position information scanned from the first viewpoint. Including the survey object a first point cloud information and a group of points including position information scanned from the second viewpoint; Including the survey object The apparatus includes a point cloud labeling unit that acquires second point cloud information, generates first labeled point cloud information by assigning to each point of the first point cloud information a label of a corresponding area in the first labeled image information based on position information of each point, and generates second labeled point cloud information by assigning to each point of the second point cloud information a label of a corresponding area in the second labeled image information based on position information of each point, and a point cloud integration unit that aligns the first labeled point cloud information and the second labeled point cloud information using labels common to the first labeled point cloud information and the second labeled point cloud information.

[0009] Furthermore, in the point cloud information processing device, the point cloud integration unit may perform random sampling on point clouds to which a common label is assigned in the first labeled point cloud information and the second labeled point cloud information, and align the first labeled point cloud information with the second labeled point cloud information based on position information of the randomly sampled point clouds.

[0010] Furthermore, the point cloud information processing device includes: an image analysis unit that analyzes first image information captured from a first viewpoint and second image information captured from a second viewpoint, respectively, recognizes different regions in each image, labels each region, and generates first labeled image information and second labeled image information; and an image analysis unit that acquires first point cloud information consisting of a group of points including position information scanned from the first viewpoint and second point cloud information consisting of a group of points including position information scanned from the second viewpoint, and calculates the first labeled image information and the second labeled image information for each point in the first point cloud information based on the position information of each point. a point cloud labeling unit that generates first labeled point cloud information by assigning a label of a corresponding area in one labeled image information, and generates second labeled point cloud information by assigning a label of a corresponding area in the second labeled image information to each point of the second point cloud information based on position information of the point; and a point cloud integration unit that aligns the first labeled point cloud information and the second labeled point cloud information using a label common to the first labeled point cloud information and the second labeled point cloud information, The point cloud integration unit may generate a representative point of an object from a group of points forming at least a part of the object according to the labels in the first labeled point cloud information and the second labeled point cloud information, perform random sampling from among the plurality of objects using a common label, and align the first labeled point cloud information with the second labeled point cloud information based on the representative point of the randomly sampled object.

[0011] In addition, in the point cloud information processing device, the image analysis unit uses an image analysis model that has been machine-learned in advance to analyze the first image information and The aforementioned analyzing the second image information to recognize different regions within each image and label each region; The aforementioned First labeled image information; and The aforementioned Second labeled image information may be generated.

[0012] In addition, in the point cloud information processing device, the point cloud integration unit may be capable of displaying the first labeled point cloud information and the second labeled point cloud information in different ways depending on the labels assigned to each point cloud.

[0013] In order to achieve the above object, a point cloud information processing method according to the present invention includes: Including surveyed objects The first image information and the second image information captured from the second viewpoint Including the survey objectan image analysis step of analyzing each of the second image information, recognizing different regions in each image, labeling each region, and generating first labeled image information and second labeled image information; and a point cloud labeling unit generating a first labeled image information and a second labeled image information, each of which includes position information scanned from the first viewpoint. Including the survey object a first point cloud information and a group of points including position information scanned from the second viewpoint; Including the survey object Acquiring second point cloud information, and generating first labeled point cloud information by assigning a label of a corresponding area in the first labeled image information to each point of the first point cloud information based on position information of the point, and No. a point cloud labeling step of generating second labeled point cloud information by assigning a label of a corresponding area in the second labeled image information to each point of the second point cloud information based on position information of the point; and a point cloud integration step of aligning the first labeled point cloud information and the second labeled point cloud information by a point cloud integrating unit using a label common to the first labeled point cloud information and the second labeled point cloud information.

[0014] In order to achieve the above object, the point cloud information processing program according to the present invention is Including surveyed objects The first image information and the second image information captured from the second viewpoint Including the survey object an image analysis step of analyzing each of the second image information, recognizing different regions in each image, labeling each region, and generating first labeled image information and second labeled image information; and Including the survey object a first point cloud information and a group of points including position information scanned from the second viewpoint; Including the survey object Acquiring second point cloud information, and generating first labeled point cloud information by assigning a label of a corresponding area in the first labeled image information to each point of the first point cloud information based on position information of the point, and No.The program causes a computer to execute a point cloud labeling step of generating second labeled point cloud information by assigning a label of a corresponding area in the second labeled image information to each point of the second point cloud information based on position information of the point, and a point cloud integration step of aligning the first labeled point cloud information with the second labeled point cloud information using a label common to the first labeled point cloud information and the second labeled point cloud information. [Effects of the Invention]

[0015] According to the present invention using the above means, it is possible to improve the robustness of aligning a plurality of pieces of point cloud information by using image information and point cloud information from a plurality of different viewpoints using a surveying device. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a configuration of a point cloud information processing system including a point cloud information processing apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a top view showing a survey object and survey points of the point cloud information processing system. [Figure 3] 3 is a diagram showing an example of first image information P1 captured from the first measurement point L1 in FIG. 2. FIG. [Figure 4] 3 is a diagram showing an example of first point cloud information Q1 obtained by scanning from the first measurement point L1 in FIG. 2. FIG. [Figure 5] 3 is a diagram showing an example of second image information P2 captured from the second measurement point L2 in FIG. 2. FIG. [Figure 6] 3 is a diagram showing an example of second point cloud information Q2 scanned from the second measurement point L2 in FIG. 2. FIG. [Figure 7] FIG. 10 is a diagram showing an example of first labeled image information p1. [Figure 8] FIG. 10 is a diagram showing an example of second labeled image information p2. [Figure 9] FIG. 10 is a diagram showing an example of first labeled point cloud information q1. [Figure 10]FIG. 10 is a diagram showing an example of second labeled point cloud information q2. [Figure 11] 10 is a flowchart showing an analysis processing operation executed by an analysis processing unit 30 of the point cloud information processing apparatus 10 according to the present embodiment. [Figure 12] This is an example of the display of point cloud integration information (3D model) that has failed to be aligned. [Figure 13] FIG. 1 is a schematic block diagram showing the configuration of a computer according to an embodiment of the present invention. [Figure 14] FIG. 10 is a top view of the first labeled point cloud information q1 in FIG. 9. [Figure 15] FIG. 11 is a top view of the second labeled point cloud information q2 in FIG. [Figure 16] FIG. 10 is a top view showing an example of fitting an object model to the first labeled point cloud information q1. [Figure 17] FIG. 10 is a top view showing an example of fitting an object model to second labeled point cloud information q2. [Figure 18] FIG. 10 is a top view showing an example of sampling of objects that are inconsistent between pieces of labeled point cloud information. [Figure 19] FIG. 19 is a top view showing an example of alignment using representative points of the object sampled in FIG. 18. [Figure 20] 10 is a top view showing an example of sampling of objects that match between labeled point cloud information. [Figure 21] FIG. 21 is a top view showing an example of alignment using representative points of the object sampled in FIG. 20. [Figure 22] FIG. 10 is a block diagram showing a configuration of a point cloud information processing system 1′ including a point cloud information processing device 10′ in a second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] <Configuration> 1 is a schematic diagram of a point cloud information processing system including a point cloud information processing device of the present invention. The configuration of a point cloud information processing system 1 including a point cloud information processing device 10 of this embodiment will be described below with reference to these drawings.

[0019] As shown in FIG. 1, the point cloud information processing system 1 of this embodiment is composed of a point cloud information processing device 10 and a surveying device 20. The point cloud information processing device 10 is also connected to the surveying device 20 so that they can communicate with each other via wired or wireless communication. In this embodiment, the surveying device 20 is installed at a plurality of different surveying points, and measures and captures an object or area that is the survey target from each surveying point to obtain image information and three-dimensional point cloud information (hereinafter simply referred to as point cloud information). Note that, in this embodiment, the surveying device 20 is described as being a ground-based type, but it may also be a type that is mounted on a UAV or the like and performs measurement and surveying from the air.

[0020] The point cloud information processing device 10 has a function of calculating the positional relationship between multiple point cloud information of a survey object acquired from two or more different viewpoints and performing registration, and a function of integrating the registered point cloud information to create a three-dimensional model of the survey object. The point cloud information processing device 10 is a dedicated computer or a general-purpose computer with software installed, and is equipped with an input unit 11 such as a keyboard, mouse, or touch panel, an output unit 12 capable of displaying images such as an LCD display, a memory unit 13, and a communication unit 14. The input unit 11, output unit 12, and memory unit 13 may be provided separately so as to be able to communicate with the point cloud information processing device 10 via wired or wireless communication.

[0021] The surveying device 20 is, for example, a three-dimensional laser scanner, and includes an imaging unit 21, a scanner unit 22, and a GNSS (Global Navigation Satellite System) 23. The surveying device 20 may also include a surveying control unit that controls the imaging unit 21, the scanner unit 22, and the GNSS 23.

[0022] The scanner unit 22 scans by rotating horizontally while emitting a laser beam (distance measurement beam) back and forth within a certain range in the vertical direction, acquiring a point cloud including the object to be surveyed, and generating point cloud information. Specifically, the scanner unit 22 measures the relative distance from its own position to the measurement point of the object to be surveyed based on the time it takes for the laser beam to be emitted, reflected off the object, and returned. The scanner unit 22 also detects the direction of the laser beam (horizontal angle and vertical angle) to measure the relative angle of the measurement point. The scanner unit 22 calculates the three-dimensional coordinates of each point from the measured relative distance and relative angle.

[0023] The imaging unit 21 is, for example, a camera, which captures an image of the surveying object and generates two-dimensional image information including at least RGB intensity for each pixel. The image information also includes information on the horizontal angle with respect to the surveying instrument 20 at the time of capturing the image, and it is also possible to calculate angle information relative to the irradiation direction of the laser light in the scanner unit 22. Therefore, for example, it is possible to associate the position of each point acquired by the scanner unit 22 with a position in the image captured by the imaging unit 21.

[0024] The GNSS 23 is, for example, a Global Navigation Satellite System (GPS) and can acquire information on global position coordinates. In other words, the GNSS 23 can acquire position information of the surveying device 20, i.e., viewpoint position information of the surveying device 20. The viewpoint position information is the laser beam irradiation base point for the scanner unit 22 and the imaging base point for the imaging unit 21. Note that, when the surveying device 20 is installed at a known point, the surveying device 20 can acquire position information through an input operation by an operator without using the GNSS 23. The scanner unit 22 can include viewpoint position information (laser beam irradiation base point) in point cloud information based on the absolute position information of the surveying device 20, and can convert the relative position information of the point cloud acquired by laser beam scanning into absolute position information. Furthermore, the imaging unit 21 can include viewpoint position information (imaging base point) in image information based on the absolute position information of the surveying device 20.

[0025] The point cloud information processing device 10 includes an analysis processing unit 30 that analyzes image information captured by an imaging unit 21 of the surveying device 20 and point cloud information acquired by a scanner unit 22 .

[0026] In detail, the analysis processing unit 30 has an image analysis unit 31, a point cloud label assignment unit 32, and a point cloud integration unit 33. The analysis processing unit 30 can also communicate with the input unit 11, output unit 12, storage unit 13, communication unit 14, etc. provided in the point cloud information processing device 10. In addition, the analysis processing unit 30 can store image information and point cloud information acquired from the surveying device 20 in the storage unit 13.

[0027] The image analysis unit 31 has a function of analyzing each of the multiple pieces of image information acquired by the analysis processing unit 30. Specifically, the image analysis unit 31 has a function of automatically recognizing areas corresponding to different objects or spaces in each image using an image analysis model that has been machine-learned in advance and labeling each recognized area, which is called semantic segmentation processing. That is, the image analysis unit 31 can recognize surveying objects appearing in the acquired image information and assign labels corresponding to the surveying objects. For example, the image analysis unit 31 assigns a building label to an area of ​​a building, a pole label to an area of ​​a pole, a sky label to the sky, and a ground label to the ground. The image analysis unit 31 also assigns an unrecognizable label to an area that cannot be recognized, and assigns labels to all areas in the image, for example, for each pixel. The image analysis unit 31 performs semantic segmentation processing using the image analysis model for each piece of acquired image information and generates labeled image information for each piece. The generated labeled image information is stored in the storage unit 13.

[0028] The point cloud label assignment unit 32 has the function of acquiring multiple pieces of labeled image information generated by the image analysis unit 31 and multiple pieces of point cloud information generated by the scanner unit 22 of the surveying device 20, and generating labeled point cloud information from the labeled image information and point cloud information from the same viewpoint.

[0029] Specifically, the point cloud labeling unit 32 extracts point cloud information and labeled image information having the same viewpoint from the storage unit 13 based on viewpoint position information included in the point cloud information and labeled image information, respectively. Then, the point cloud labeling unit 32 generates labeled point cloud information by labeling each point of the extracted point cloud information with a corresponding area in the labeled image information having the same viewpoint based on the position information of the point. In other words, the point cloud information is overlaid on each labeled area of ​​the labeled image information, and the labels of the overlapping areas are reflected in each point, thereby labeling each point (so-called point cloud annotation). The generated labeled point cloud information is stored in the storage unit 13.

[0030] The point cloud integration unit 33 has a function of aligning (registering) the point clouds based on the labels of each point for each piece of labeled point cloud information generated by the point cloud labeling unit 32, and generating point cloud integrated information by integrating the point clouds. The point cloud integrated information becomes a so-called three-dimensional model.

[0031] In detail, the point cloud integration unit 33 extracts labeled point cloud information having different viewpoint position information from the storage unit 13, extracts point clouds with labels commonly included in each labeled point cloud information by sampling (e.g., random sampling), and acquires position information of the extracted point clouds. Then, the point clouds for which the position information has been acquired are aligned (registered) using a point cloud matching method such as ICP (Iterative Closest Point). The point cloud integration unit 33 then generates point cloud integrated information by integrating the point cloud information by aligning the labeled point cloud information with each other. This point cloud integrated information is stored in the storage unit 13.

[0032] The point cloud information processing device 10 can output image information, labeled image information, point cloud information, labeled point cloud information, point cloud integration information, etc. stored in the memory unit 13 to the output unit 12, edit it via the input unit 11, or transmit it to the outside via the communication unit 14.

[0033] <Example of analysis procedure> Next, an example of a process from acquiring image information and point cloud information to creating integrated point cloud information in the point cloud information processing system 1 including the point cloud information processing device 10 will be specifically described with reference to FIGS. 2 to 10. FIG. 2 is a top view showing a survey object and survey points of the point cloud information processing system 1. FIG. 3 is a diagram showing an example of first image information P1 captured from the first survey point L1 of FIG. 2. FIG. 4 is a diagram showing an example of first point cloud information Q1 scanned from the first survey point L1 of FIG. 2. FIG. 5 is a diagram showing an example of second image information P2 captured from the second survey point L2 of FIG. 2. FIG. 6 is a diagram showing an example of second point cloud information Q2 scanned from the second survey point L2 of FIG. 2. FIG. 7 is a diagram showing an example of first labeled image information p1 generated by image analysis of the first image information P1. FIG. 8 is a diagram showing an example of second labeled image information p2 generated by image analysis of the second image information P2. Fig. 9 is a diagram showing an example of first labeled point cloud information q1 in which the labels of first labeled image information p1 are reflected in first point cloud information Q1. Fig. 10 is a diagram showing an example of second labeled point cloud information q2 in which the labels of second labeled image information p2 are reflected in second point cloud information Q2. For simplicity of explanation, it is assumed that the first image information P1 and the second image information P2, and the first point cloud information Q1 and the second point cloud information Q2 have the same scale and accuracy.

[0034] 2, a case will be described in which, as a surveying object, for example, a building 41 and poles 42a to 42e installed adjacent to the building 41 are measured from the ground using the surveying device 20. Note that the top of the paper in FIG. 2 is north, and the height of the poles 42a to 42e is lower than the height of the building 41.

[0035] First, the surveying device 20 is installed at a first survey point L1 (first viewpoint) located southwest of the building 41, and the imaging unit 21 and the scanner unit 22 generate first image information and first point cloud information from the same viewpoint that includes at least a portion of the building 41 and the poles 42a to 42e. At this time, the poles 42c to 42e are obscured by the building 41 and do not appear in the image from the first survey point L1. Therefore, the image captured by the imaging unit 21 of the surveying device 20 becomes, for example, the first image information P1 shown in FIG. 3. Similarly, the point cloud generated by the scanner unit 22 of the surveying device 20 becomes, for example, the first point cloud information Q1 shown in FIG. 4. Each circle in FIG. 4 represents one point cloud, and each point is associated with RGB intensity and three-dimensional coordinates. In Figure 4, the size of the point cloud is shown to become smaller as it goes further back to create a sense of three-dimensional depth, but due to the nature of laser light, the longer the measurement distance, the larger the spot diameter (spot size), which is the laser beam width, becomes. In reality, however, the longer the measurement distance, the wider the data interval between point clouds becomes.

[0036] Next, the surveying device 20 is installed at a second survey point L2 (second viewpoint) located north of the building 41 in Fig. 2, and second image information and second point cloud information are generated that include at least a portion of the building 41 and the poles 42a to 42e, as in the case of the first survey point L1. At this time, all of the poles 42a to 42e located in front of the building 41 appear in the image from the second survey point L2, and the image information captured by the imaging unit 21 of the surveying device 20 becomes, for example, the second image information P2 shown in Fig. 5. Similarly, the point cloud generated by scanning with the scanner unit 22 of the surveying device 20 becomes, for example, the second point cloud information Q2 shown in Fig. 6.

[0037] The first image information P1 and second image information P2 and the first point cloud information Q1 and second point cloud information Q2 generated by the surveying instrument 20 are transmitted by wire or wirelessly to the point cloud information processing device 10. Note that in this embodiment, for the sake of simplicity, processing is performed on the image information P1, P2 and point cloud information Q1, Q2 acquired from two viewpoints L1, L2, but processing can also be performed on image information and point cloud information acquired from three or more different viewpoints.

[0038] Next, the image analysis unit 31 of the point cloud information processing device 10 performs semantic segmentation processing on the first image information P1 using an image analysis model that has been machine-learned in advance, to generate first labeled image information p1 as shown in Fig. 7. In Fig. 7, the image analysis unit 31 automatically recognizes an image area A1 as an area corresponding to, for example, a building 41. Similarly, the image analysis unit 31 automatically recognizes, in the first image information P1, an image area A2 corresponding to a pole 42a, an image area A3 corresponding to a pole 42b, and an image area A0 corresponding to space.

[0039] Then, the point cloud labeling unit 32 assigns a label corresponding to a building 41 (hereinafter referred to as a building label) to each pixel in the image area A1 shown in Fig. 7. Similarly, the point cloud labeling unit 32 assigns a label corresponding to a pole (hereinafter referred to as a pole label) to each pixel in the image areas A2 and A3. For each pixel in other image areas in which no object is recognized, a label corresponding to space is assigned. In this way, labels are assigned to all pixels of the first image information, and first labeled image information p1 is generated.

[0040] Furthermore, the image analysis unit 31 performs semantic segmentation processing on the second image information P2 to generate second labeled image information p2 of Fig. 8. In Fig. 8, the image analysis unit 31 automatically recognizes an image area A4 as an area corresponding to the building 41. Similarly, the image analysis unit 31 automatically recognizes image areas A5 to A9 corresponding to each of the poles 42a to 42e.

[0041] The point cloud labeling unit 32 then assigns a label corresponding to a building to image region A4 shown in Fig. 8, a label corresponding to a pole to image regions A5 to S9, and a label corresponding to space to image region A10, to each pixel in each image region. In this way, labels are assigned to all pixels of the second image information, and second labeled image information p2 is generated. Note that although the labels are indicated by different hatching in Figs. 7 and 8, the differences in labels may be expressed, for example, by using different colors depending on the label.

[0042] Next, the point cloud labeling unit 32 overlays the first point cloud information Q1 on the first labeled image information p1 based on the respective viewpoint position information of the first labeled image information p1 and the first point cloud information Q1, and reflects the labels assigned to the overlapping pixels for each point on that point. Note that in this embodiment, although this depends on the model and shooting conditions, the pixel resolution of the camera of the surveying instrument 20 is, for example, 2 mm / pixel when the measurement distance is 10 m, and the resolution (data interval) of a laser scanner for the same measurement distance is, for example, 0.3 mm. Since the pixel resolution of the imaging unit 21 is greater than the resolution of the scanner unit 22 for the same measurement distance, multiple points of the first point cloud information Q1 will overlap one pixel of the first labeled image information p1. Similarly, for the second labeled image information p2 and the second point cloud information Q2, the labels of the pixels at corresponding overlapping positions are reflected to each point. As a result, first labeled point cloud information q1 shown in FIG. 9 is generated in which the labels of each pixel of the first labeled image information p1 are reflected to each point of the first point cloud information Q1, and second labeled point cloud information q2 shown in FIG. 10 is generated in which the labels of each pixel of the second labeled image information p2 are reflected to each point of the second point cloud information Q2.

[0043] Next, the point cloud integrating unit 33 performs random sampling of the point cloud information on points labeled with buildings and poles, which are the objects to be surveyed and are included in both the first labeled point cloud information q1 and the second labeled point cloud information q2, in order to speed up processing. For example, in the first labeled point cloud information of FIG. 9, point α1 from the point cloud labeled with a building and point β1 from the point cloud labeled with a pole are extracted by random sampling. In the second labeled point cloud information, points α2 and β2 in FIG. 10 are points that match or are close to the position information of points α1 and β1. Note that the number and positions of points extracted by random sampling are not limited to these. Random sampling may be performed only from the point cloud labeled with buildings or only from the point cloud labeled with poles.

[0044] Then, the point cloud integration unit 33 and the point cloud position calculation unit 34 perform matching based on the position information of a portion of the point cloud (e.g., point α1, point β1) of the randomly sampled first labeled point cloud information q1 and a portion of the point cloud (e.g., point α2, point β2) of the second labeled point cloud information q2, and perform alignment (registration), thereby generating point cloud integration information of the first labeled point cloud information q1 and the second labeled point cloud information q2.

[0045] The point cloud integration unit 33 performs the same processing on point cloud information from other different viewpoints and integrates the information into integrated point cloud information to generate a three-dimensional model.

[0046] Next, the analysis processing operation executed by the analysis processing unit 30 of the point cloud information processing apparatus 10 according to this embodiment will be described with reference to the flowchart shown in FIG.

[0047] In step S101, the analysis processing unit 30 acquires image information captured by the imaging unit 21 from multiple viewpoints and point cloud information scanned by the scanner unit 22 from multiple viewpoints from the surveying device 20 via the communication unit 14 or from the storage unit 13. For example, the analysis processing unit 30 acquires first image information P1 and first point cloud information Q1 at the above-mentioned first survey point (first viewpoint) L1, and second image information P2 and second point cloud information Q2 at the second survey point (second viewpoint) L2.

[0048] In step S102, the image analysis unit 31 of the analysis processing unit 30 generates labeled image information by performing semantic segmentation processing on the acquired image information (image analysis step). For example, the image analysis unit 31 performs semantic segmentation processing on the first image information P1 and the second image information P2 described above to recognize different regions in each image, label each region as a building, a pole, a space, etc., and generate first labeled image information p1 and second labeled image information p2.

[0049] In step S103, the point cloud labeling unit 32 of the analysis processing unit 30 generates labeled point cloud information from the labeled image information and point cloud information of the same viewpoint (point cloud labeling step). For example, the point cloud labeling unit 32 generates the first labeled point cloud information by labeling each point of the above-mentioned first point cloud information Q1 with a label of a corresponding area in the first labeled image information p1 based on the position information of each point. Similarly, the point cloud labeling unit 32 generates the second labeled point cloud information q2 by labeling each point of the second point cloud information with a label of a corresponding area in the second labeled image information p2 based on the position information of each point.

[0050] In step S104, the point cloud integration unit 33 of the analysis processing unit 30 performs random sampling on point clouds to which common labels have been assigned in each piece of labeled point cloud information, and acquires position information of the randomly sampled point clouds. For example, the point cloud integration unit 33 performs random sampling on each of the point clouds to which the building and pole labels common to the first labeled point cloud information q1 and the second labeled point cloud information q2 have been assigned, and acquires position information of a part of the point clouds of the building and pole (for example, point α1, point β1).

[0051] Next, in step S105, the point cloud integration unit 33 generates point cloud integrated information by matching points in labeled point cloud information from different viewpoints based on the position information of the randomly sampled point clouds and performing registration. For example, the point cloud integration unit 33 matches a part of the point cloud of a building and a pole in the randomly sampled first labeled point cloud information q1 (point α1, point β1) with the point cloud of a building and a pole in the second labeled point cloud information q2 whose position information matches or is close to the part, and performs registration, thereby generating point cloud integrated information of the first labeled point cloud information q1 and the second labeled point cloud information q2.

[0052] In step S106, the point cloud integration unit 33 stores the generated point cloud integration information in the storage unit 13. At this time, the point cloud integration unit 33 may display the generated point cloud integration information on the output unit 12 in a form that can be visually recognized by the user.

[0053] In step S107, the analysis processing unit 30 determines whether or not to terminate the analysis processing. For example, if all of the point cloud information to be processed stored in the storage unit 13 has been integrated or if a user has performed a stop operation, the determination result becomes true (Y), and the analysis processing is terminated. On the other hand, if there is no user stop operation and point cloud information to be processed remains in the storage unit 13, the determination result becomes false (N), and the process returns to step S101, and the above-described processing is repeated for other image information and point cloud information.

[0054] As described above, the point cloud information processing device 10 of this embodiment generates labeled image information by labeling each region in an image of image information, assigns labels to point cloud information based on the labeled image information to generate labeled point cloud information, and performs registration between the labeled point cloud information using the common labels.

[0055] By aligning the point clouds based on the labels assigned to the point cloud information in this way, even objects with few point cloud features, such as thin walls, can be easily recognized because they have been assigned labels corresponding to the walls. Also, by assigning labels such as tree or moving object to objects such as leaves fluttering in the wind and moving objects such as automobiles, the point cloud information processing device 10 can easily exclude these objects from the survey target.

[0056] Therefore, the point cloud information processing device 10 can improve the robustness in aligning a plurality of pieces of point cloud information by using the surveying device 20 to obtain image information and point cloud information from a plurality of different viewpoints.

[0057] Furthermore, the point cloud integration unit 33 of the point cloud information processing device 10 performs random sampling on point clouds that have been assigned common labels in the labeled point cloud information, and aligns the point clouds based on the position information of the randomly sampled point clouds. By narrowing down the points by label in this way and then performing random sampling, alignment can be performed more efficiently.

[0058] In addition, the image analysis unit 31 of the point cloud information processing device 10 can easily recognize and label different areas within an image by performing so-called semantic segmentation processing using an image analysis model that has been machine-learned in advance.

[0059] Furthermore, even if the alignment fails and, for example, a discrepancy occurs between the point clouds, the integrated point cloud information, which aligns the labeled point clouds using labels, displays the point clouds differently for each label, making it easy to identify the misalignment. Specifically, FIG. 12 shows an example of a display of point cloud integrated information (3D model) that has failed to align. In this figure, the point clouds corresponding to the first wall 41a and the second wall 41b of a building 41 are integrated with a discrepancy between them. However, the first wall 41a and the second wall 41b are each labeled with a building label and displayed in the same display (e.g., color), allowing the user to easily detect the discrepancy between the first wall 41a and the second wall 41b. For example, the point cloud integration unit may automatically detect discrepancies between point clouds with the same label from the generated point cloud integrated information and perform correction processing to eliminate the discrepancy.

[0060] <Program> Here, the program for realizing each function constituting the point cloud information processing apparatus 10 according to this embodiment will be described in detail.

[0061] The point cloud information processing device 10 is implemented in a computer 801 shown in Fig. 13. The operation of each component of the point cloud information processing device 10 is stored in the form of a program in an auxiliary storage device 804 or an external server or the like that can communicate via wired or wireless communication, and the program is available for use. The CPU 802 reads the program from the auxiliary storage device 804, loads it into the main storage device 803, and executes the above-mentioned processing in accordance with the program. The CPU 802 also allocates a storage area in the main storage device 803 that corresponds to the above-mentioned storage unit 13 in accordance with the program.

[0062] Specifically, the program causes the computer 801 to perform an image analysis step of analyzing first image information captured from a first viewpoint and second image information captured from a second viewpoint, recognizing different regions in each image, labeling each region, and generating first labeled image information and second labeled image information; and acquiring first point cloud information consisting of a group of points including position information scanned from the first viewpoint and second point cloud information consisting of a group of points including position information scanned from the second viewpoint, and calculating the position information of each point for each point in the first point cloud information. and a point cloud labeling step of generating second labeled point cloud information by labeling corresponding areas in the second labeled image information based on position information of each point of the second point cloud information, and a point cloud integration step of aligning the first labeled point cloud information and the second labeled point cloud information using labels common to the first labeled point cloud information and the second labeled point cloud information.

[0063] The auxiliary storage device 804 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, and a semiconductor memory connected via an interface. In addition, when this program is distributed to the computer 801 via a network, the computer 801 that receives the program may load the program into the main storage device 803 and execute the above processing.

[0064] The program may be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in the auxiliary storage device 804.

[0065] Although the description of the embodiments of the present invention has been completed above, the aspects of the present invention are not limited to these embodiments.

[0066] In the above embodiment, the point cloud integration unit 33 extracts labeled point cloud information having different viewpoint position information from the storage unit 13, extracts point clouds with labels commonly included in each labeled point cloud information by sampling (for example, random sampling), obtains position information of the extracted point clouds, and performs registration on the point clouds whose position information has been obtained using a point cloud matching method such as ICP (Iterative Closest Point).However, the sampling and registration methods are not limited to this.For example, registration may be performed by calculating a representative value indicating the position of an object from the point clouds.

[0067] Hereinafter, with reference to FIGS. 14 to 21, a first modified example will be described in which the point cloud integration unit 33 generates representative points from a group of points forming at least a part of an object according to the labels in the first labeled point cloud information q1 and the second labeled point cloud information q2, performs random sampling from among a plurality of objects using a common label, and aligns the first labeled point cloud information q1 and the second labeled point cloud information q2 based on the representative points of the randomly sampled objects.

[0068] FIG. 14 is a top view of the first labeled point cloud information q1 in FIG. 9. FIG. 15 is a top view of the second labeled point cloud information q2 in FIG. 10. FIG. 16 is a top view showing an example of fitting an object model to the first labeled point cloud information q1. FIG. 17 is a top view showing an example of fitting an object model to the second labeled point cloud information q2. FIG. 18 is a top view showing an example of sampling of mismatched objects between labeled point cloud information. FIG. 19 is a top view showing an example of alignment using representative points of the objects sampled in FIG. 18. FIG. 20 is a top view showing an example of sampling of matching objects between labeled point cloud information. FIG. 21 is a top view showing an example of alignment using representative points of the objects sampled in FIG. 20.

[0069] 14 and 15, the outlines of the real images of the building 41 and the poles 42a-42e are indicated by dashed lines, and the points included in the point cloud information are indicated by solid circles. As shown in FIG. 14, in the first labeled point cloud information q1, a group of points q11 arranged in a rectangular shape along the south and west wall surfaces of the building 41 is generated. In addition, in the first labeled point cloud information q1, a group of points q12a and q12b arranged in a semi-cylindrical shape on the south surface of the poles 42a and 42b is generated. Similarly, as shown in FIG. 15, in the second labeled point cloud information q2, a group of points q21 arranged in a rectangular shape along the north and west wall surfaces of the building 41 and a group of points q22a-q22e arranged in a semi-cylindrical shape along the north surfaces of the poles 42a-42e are generated. The point cloud integration unit 33 fits a corresponding object model to each group of point clouds that form at least a portion of one object, and performs interpolation processing for parts that cannot be obtained by scanning from the measurement point L1 or L2. The object models are stored in advance in the storage unit 13, and representative shapes corresponding to the object are stored as object models, such as a cylindrical shape for a pole or a cubic shape for a building. Note that the object model only needs to reflect the approximate size and position of the object to be fitted, so it may simply be a cylinder, an elliptical cylinder, or the like.

[0070] Specifically, the point cloud integrating unit 33 fits a cubic building model to the building 41 in the first labeled point cloud information q1 shown in Fig. 14, and fits cylindrical pole models to the poles 42a and 42b, thereby generating the respective complemented point clouds q11', q12a', and q12b' shown by the diagonal lines in Fig. 16. Note that the complemented parts are not limited to point cloud information, and may be in any format that allows the positional information of the outline of an object to be determined.

[0071] The point cloud integrating unit 33 then generates a representative point based on the group of points of each object and their complementary point groups. For example, the representative point may be the center of the group of points calculated. Specifically, the point cloud integrating unit 33 calculates a center point g12a based on the three-dimensional position information of the group of points q12a corresponding to the pole 42a in FIG. 16 and its complementary point group q12a'. Similarly, the point cloud integrating unit 33 calculates center points g12b and g11 for the pole 42b and the building 41. The point cloud integrating unit 33 then sets these center points g11, g12a, and g12b as representative points of each object, associates them with the building label or pole label corresponding to each object, and stores them in the storage unit 13.

[0072] Similarly, for the second labeled point cloud information q2, as shown in Figure 17, the point cloud integration unit 33 calculates center points g21, g22a to g22e, sets these as representative points of each object, and stores them in the memory unit 13 in association with a building label or a pole label.

[0073] Next, the point cloud integration unit 33 samples each of the labeled point cloud information q1 and q2 for each label and aligns them using the representative value of the sampled objects. For example, to simplify the explanation, we will explain the case where the number of sampling points per label is 1, that is, one sample is taken from the pole label and one sample is taken from the building label.

[0074] 18, from the first labeled point cloud information q1, a pole 42a is selected as a sample in the pole label, and a building 41 is selected as a sample in the building label. On the other hand, from the second labeled point cloud information q2, a pole 42b is selected as a sample in the pole label, and a building 41 is selected as a sample in the building label.

[0075] The representative point g12a of the pole 42a and the representative point g11 of the building 41 in the first labeled point cloud information q1 are aligned with the representative point g22b of the pole 42b and the representative point g21 of the building 41 in the second labeled point cloud information q2. This alignment is performed by, for example, overlaying the point cloud information so that the distance between representative points with the same label is minimized. In other words, as shown in Figure 19, the objects in the first labeled point cloud information q1 and the second labeled point cloud information q2 do not match, and a gap occurs between the representative points.

[0076] 20, from the first labeled point cloud information q1, a pole 42a is selected as a sample in the pole label, and a building 41 is selected as a sample in the building label. Also from the second labeled point cloud information q2, a pole 42a is selected as a sample in the pole label, and a building 41 is selected as a sample in the building label.

[0077] When the representative point g12a of the pole 42a and the representative point g11 of the building 41 in the first labeled point cloud information q1 are aligned with the representative point g22a of the pole 42a and the representative point g21 of the building 41 in the second labeled point cloud information q2, the objects in the first labeled point cloud information q1 and the second labeled point cloud information q2 will almost match, and the positions of the representative points will also almost match, as shown in Fig. 21. In this way, sampling is repeated, including for objects that have not been sampled, to perform alignment, until the positions of the representative points of the objects match or approach within a predetermined range.

[0078] In this way, sampling from objects with the same label allows for more efficient alignment. Also, by calculating the representative points of each object and then aligning them, a rough alignment can be achieved. Furthermore, by fitting an object model according to the label to calculate the representative points, the positional accuracy of the representative points can be improved.

[0079] In the above embodiment, the point cloud information processing device 10 was provided separately from the surveying device 20, but it may be mounted on the surveying device 20 and displayed and edited in real time on the output unit of the surveying device 20 or on the output unit of a tablet used to operate the surveying device 20. This allows for editing on the spot to remove such noise from the point cloud information while viewing the display on the output unit, and then generating an integrated point cloud. In other words, accurate point cloud information can be generated at the surveying site.

[0080] Furthermore, a point cloud information processing system 1' including a point cloud information processing device 10' according to a second modification as shown in FIG. 22 may be provided, in which a point cloud feature recognition unit 34 is added to the analysis processing unit 30 of the above embodiment to form an analysis processing unit 30'. The point cloud feature recognition unit 34 extracts areas where the normal vectors and RGB intensities obtained from the position information of adjacent point clouds are uniform, recognizes the surveying object, and performs simple point cloud segmentation. The point cloud segmentation information can be used by the image analysis unit 31 for post-segmentation correction (noise removal, etc.) of the image information. This results in more accurate image segmentation processing results and improves overall alignment accuracy. The original image before segmentation and the corrected results after segmentation processing can also be used as a machine learning dataset for a new image recognition model.

[0081] Furthermore, while the imaging unit 21 in the above embodiment is a visible light camera that generates two-dimensional image information including RGB intensity, it may be an infrared camera that generates two-dimensional image information including infrared information. In this case, the point cloud associates infrared intensity and three-dimensional coordinates with each point. This makes it possible to conduct surveys even at night when the field of view is dark.

[0082] Furthermore, although the point cloud labeling unit 32 in the above embodiment assigns the same "pole label" to multiple poles, it may also be possible to individually identify the poles 42a, 42b, etc. and assign them different labels, such as "first pole," "second pole," etc. This improves the accuracy of alignment. [Explanation of symbols]

[0083] 1: Point cloud information processing system 10: Point cloud information processing device 11: Input section 12: Output section 13: Storage section 14: Communications Department 20: Surveying equipment 30: Analysis processing section 31: Image analysis unit 32: Point cloud labeling section 33: Point cloud integration unit 41: Building 42a~42e: Paul

Claims

1. an image analysis unit that analyzes first image information captured from a first viewpoint and second image information captured from a second viewpoint, respectively, recognizes different regions in each image, labels each region, and generates first labeled image information and second labeled image information; a point cloud labeling unit that acquires first point cloud information consisting of a group of points including position information scanned from the first viewpoint and second point cloud information consisting of a group of points including position information scanned from the second viewpoint, generates first labeled point cloud information by assigning to each point of the first point cloud information a label of a corresponding area in the first labeled image information based on the position information of the each point, and generates second labeled point cloud information by assigning to each point of the second point cloud information a label of a corresponding area in the second labeled image information based on the position information of the each point; a point cloud integration unit that aligns the first labeled point cloud information and the second labeled point cloud information using a label that is common to the first labeled point cloud information and the second labeled point cloud information; Equipped with The point cloud integration unit generates a representative point of an object from a group of points forming at least a part of the object according to the labels in the first labeled point cloud information and the second labeled point cloud information, performs random sampling from among a plurality of the objects using a common label, and aligns the first labeled point cloud information with the second labeled point cloud information based on the representative point of the randomly sampled object. Point cloud information processing device.

2. The image analysis unit analyzes the first image information and the second image information using an image analysis model that has been machine-learned in advance, recognizes different regions in each image, labels each region, and generates the first labeled image information and the second labeled image information. The point cloud information processing device according to claim 1 .

3. The point cloud integration unit is capable of displaying the first labeled point cloud information and the second labeled point cloud information in different ways for each label assigned to each point cloud. The point cloud information processing device according to claim 1 or 2.

Images