Point cloud identification device, point cloud identification method, and point cloud identification program

Abstract

Provided is a point cloud identification device comprising: an inference unit that inputs three-dimensional point cloud data to be inferred among a plurality of three-dimensional point cloud data, said data being acquired by, and obtained from, a sensor that measures an object, into a trained model that has undergone machine learning using labeled three-dimensional point cloud data for training, and causes the three-dimensional point cloud data to be inferred to be outputted with a label from the trained model, thereby inferring an object corresponding to the three-dimensional point cloud data to be inferred; and a characteristics correction unit that applies a correction to reduce differences in characteristics of the plurality of three-dimensional point cloud data at least before the machine learning or before the inference by the inference unit.

Description

Point cloud identification device, point cloud identification method, and point cloud identification program

[0001] The disclosed technology relates to a point cloud identification device, a point cloud identification method, and a point cloud identification program.

[0002] BACKGROUND ART Technology development is underway to use three-dimensional point cloud data acquired by measurement media equipped with cameras or laser scanners (LiDAR: Light Detection and Ranging) to identify and recognize buildings, utility poles, and various other objects in real space, and to utilize this data in a variety of fields, including autonomous driving, maintenance or preservation of social infrastructure, environmental conservation, disaster prevention, and so on.

[0003] Labeling measured point clouds for object identification is a costly task. Therefore, to reduce costs, labeling by learning-based segmentation using a deep neural network (DNN) is becoming more common. For example, one method involves preparing three-dimensional point cloud data to which correct labels have been assigned in advance through annotation, training a DNN using the data, obtaining a learning model, and then using the obtained learning model to estimate the labels of each point in unknown three-dimensional point cloud data.

[0004] As disclosed in Non-Patent Document 1, labeling using DNN involves stepwise feature extraction based on the relative coordinates of neighboring points for a given point in a point cloud, and the resulting feature quantities of the object shape at multiple distance scales are used to label each point.

[0005] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, B. Chen, "PointCNN: Convolution On X -Transformed Points", pp.828-838, 2018.

[0006] In labeling using learning-based segmentation, the accuracy of labels inferred for unknown 3D point cloud data depends on the features of the learned 3D point cloud data, which poses a problem that the accuracy of inference for 3D point cloud data in a domain with different features from the 3D point cloud data used for training may be reduced. Factors that result in a different domain include, for example, differences in the physical position of the LiDAR device due to the mounting direction or angle of the LiDAR device during point cloud measurement, and differences in point cloud density or reflection intensity due to the performance or specifications of the measuring device itself.

[0007] The technology described in Non-Patent Document 1 performs labeling based on feature values ​​obtained from multiple distance measures, which leads to a decrease in accuracy when inferring 3D point cloud data with different inter-point densities due to the different feature values. Similarly, when reflectance intensity information is incorporated into learning, this also leads to a decrease in inference accuracy when inferring 3D point cloud data with different reflectance intensity distribution trends.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide a point cloud classification device, a point cloud classification method, and a point cloud classification program that suppress a decrease in object classification accuracy.

[0009] A first aspect of the present disclosure is a point cloud identification device that includes an inference unit that inputs three-dimensional point cloud data of an inference target in the plurality of three-dimensional point cloud data obtained by a sensor that measures an object into a trained model that has been machine-learned using labeled training three-dimensional point cloud data, and causes the trained model to output the labeled three-dimensional point cloud data of the inference target, thereby inferring an object that corresponds to the three-dimensional point cloud data of the inference target, and a characteristic correction unit that performs correction to reduce differences in characteristics of the plurality of three-dimensional point cloud data at least either before the machine learning or before inference by the inference unit.

[0010] A second aspect of the present disclosure is a point cloud identification method, in which a processor inputs three-dimensional point cloud data of an inference target among multiple three-dimensional point cloud data obtained by a sensor that measures an object into a trained model that has been machine-learned using labeled training three-dimensional point cloud data, and causes the trained model to output the labeled three-dimensional point cloud data of the inference target, thereby inferring an object that corresponds to the three-dimensional point cloud data of the inference target, and performs a process of making corrections to reduce differences in characteristics of the multiple three-dimensional point cloud data at least either before the machine learning or before the inference.

[0011] A third aspect of the present disclosure is a point cloud identification program that causes a computer to input three-dimensional point cloud data of an inference target among multiple three-dimensional point cloud data obtained by a sensor that measures an object into a trained model that has been machine-learned using labeled training three-dimensional point cloud data, and to output the labeled three-dimensional point cloud data of the inference target from the trained model, thereby inferring an object that corresponds to the three-dimensional point cloud data of the inference target, and to perform a process of performing correction to reduce differences in characteristics of the multiple three-dimensional point cloud data at least either before the machine learning or before the inference.

[0012] According to the disclosed technology, it is possible to provide a point cloud classification device, a point cloud classification method, and a point cloud classification program that suppress a decrease in object classification accuracy.

[0013] 1 is a block diagram showing a hardware configuration of a point cloud identification device according to an embodiment of the disclosed technology. FIG. 2 is a block diagram showing an example of a functional configuration of the point cloud identification device. FIG. 3 is a block diagram showing a hardware configuration of a learning device according to an embodiment of the disclosed technology. FIG. 4 is a block diagram showing an example of a functional configuration of the learning device. FIG. 5 is a block diagram showing an example of a functional configuration of the point cloud identification device. FIG. 6 is a block diagram showing an example of a functional configuration of the learning device. FIG. 7 is a flowchart showing a flow of point cloud identification processing by the point cloud identification device. FIG. 8 is a flowchart showing a flow of learning processing by the learning device. FIG. 9 is a block diagram showing an example of a functional configuration of the point cloud identification device. FIG. 10 is a block diagram showing an example of a functional configuration of the learning device. FIG. 11 is a flowchart showing a flow of point cloud identification processing by the point cloud identification device. FIG. 12 is a flowchart showing a flow of learning processing by the learning device. FIG. 13 is a block diagram showing an example of a functional configuration of the point cloud identification device. FIG. 14 is a block diagram showing an example of a functional configuration of the learning device. FIG. 15 is a flowchart showing a flow of point cloud identification processing by the point cloud identification device. FIG. 16 is a flowchart showing a flow of learning processing by the learning device.

[0014] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0015] FIG. 1 is a block diagram showing the hardware configuration of a point cloud identification device 10 according to an embodiment of the disclosed technique.

[0016] 1 , the point cloud identification device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0017] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above-described components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a point cloud identification program for identifying buildings, utility poles, and various other objects in real space using three-dimensional point cloud data acquired by a measurement medium or the like.

[0018] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0019] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0020] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0021] The communication interface 17 is an interface for communicating with other devices, and the communication may use, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0022] Next, the functional configuration of the point cloud identification device 10 will be described.

[0023] FIG. 2 is a block diagram showing an example of the functional configuration of the point cloud identification device 10.

[0024] 2, the point cloud identification device 10 has, as functional components, a characteristic correction unit 101 and an inference unit 102. Each functional component is realized when the CPU 11 reads out a point cloud identification program stored in the ROM 12 or the storage 14, expands it in the RAM 13, and executes it.

[0025] The characteristic correction unit 101 performs preprocessing to reduce differences in features between domains on three-dimensional point cloud data of an inference target, which is acquired by a sensor that measures an object, such as a 3D camera or LiDAR. Examples of preprocessing to reduce differences in features between domains will be described in detail later using several examples. For example, the characteristic correction unit 101 performs preprocessing to grasp the distribution trend of reflection intensity from input three-dimensional point cloud data using the reflection intensity of certain three-dimensional point cloud data as a reference, and corrects the three-dimensional point cloud data for inference to conform to the grasped distribution trend of reflection intensity. The characteristic correction unit 101 also performs preprocessing, for example, to upsample or downsample the three-dimensional point cloud data for inference so that the point cloud density trend is uniform. The characteristic correction unit 101 also performs preprocessing, for example, to apply the characteristics of the reflection intensity distribution and point cloud density distribution of the three-dimensional point cloud data for learning, measured by a measurement device, to the three-dimensional point cloud data for inference.

[0026] The inference unit 102 performs inference processing on the three-dimensional point cloud data of the inference target that has been preprocessed by the characteristic correction unit 101, using a trained model 30 that has been generated in advance by machine learning. As the inference processing, the inference unit 102 inputs the three-dimensional point cloud data of the inference target into the trained model 30, and causes the trained model 30 to output labeled three-dimensional point cloud data of the inference target, thereby performing processing to infer an object corresponding to the three-dimensional point cloud data of the inference target. The object corresponding to the three-dimensional point cloud data is an object included in an object measured by a 3D camera, LiDAR, or the like, such as a building, a utility pole, or a manhole. The trained model 30 is a model that has been generated in advance by machine learning so as to label and output the three-dimensional point cloud data when the three-dimensional point cloud data is input.

[0027] Here, a learning device for generating the trained model 30 will be described.

[0028] FIG. 3 is a block diagram showing the hardware configuration of the learning device 20 according to an embodiment of the disclosed technology.

[0029] 3, the learning device 20 includes a CPU 21, a ROM 22, a RAM 23, a storage 24, an input unit 25, a display unit 26, and a communication interface (I / F) 27. Each component is connected to each other via a bus 29 so as to be able to communicate with each other.

[0030] The CPU 21 is a central processing unit that executes various programs and controls each component. That is, the CPU 21 reads the programs from the ROM 22 or the storage 24 and executes the programs using the RAM 23 as a work area. The CPU 21 controls the above-described components and performs various arithmetic processing in accordance with the programs stored in the ROM 22 or the storage 24. In this embodiment, the ROM 22 or the storage 24 stores a learning program for generating a trained model 30 that identifies buildings, utility poles, and various other objects in real space using three-dimensional point cloud data acquired by a measurement medium or the like.

[0031] The ROM 22 stores various programs and various data. The RAM 23 serves as a working area and temporarily stores programs or data. The storage 24 is configured with a storage device such as an HDD or SSD, and stores various programs including an operating system and various data.

[0032] The input unit 25 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0033] The display unit 26 is, for example, a liquid crystal display, and displays various information. The display unit 26 may be a touch panel type and function as the input unit 25.

[0034] The communication interface 27 is an interface for communicating with other devices, and the communication may use, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0035] Next, the functional configuration of the learning device 20 will be described.

[0036] FIG. 4 is a block diagram showing an example of the functional configuration of the learning device 20.

[0037] 4, the learning device 20 has, as its functional components, a characteristic correction unit 201 and a learning unit 202. Each functional component is realized by the CPU 21 reading out a learning program stored in the ROM 22 or the storage 24, expanding it in the RAM 23, and executing it.

[0038] The characteristic correction unit 201 performs preprocessing on the 3D point cloud data to be learned, reducing differences in features between domains. Examples of preprocessing for reducing differences in features between domains will be described in detail later using several examples. For example, the characteristic correction unit 201 performs preprocessing to grasp the distribution trend of reflection intensity using the reflection intensity of certain 3D point cloud data as a reference and correct the 3D point cloud data for learning to conform to the grasped distribution trend of reflection intensity. The characteristic correction unit 201 also performs preprocessing, for example, to upsample or downsample the 3D point cloud data for learning so that the point cloud density trend becomes uniform. The characteristic correction unit 201 also performs preprocessing to extract features of the reflection intensity distribution and point cloud density distribution of the 3D point cloud data for learning, measured by a measurement device, in order to apply them to the 3D point cloud data for inference.

[0039] The learning unit 202 generates a trained model 30 by machine learning for the labeled training 3D point cloud data that has been preprocessed by the characteristic correction unit 201. In this embodiment, the learning unit 202 generates the trained model 30 by deep learning using a DNN.

[0040] The point cloud identification device 10 and the learning device 20 may be configured as the same device. That is, the configuration of the learning device 20 may be included in the point cloud identification device 10. Furthermore, the trained model 30 generated by the learning device 20 may be stored in the point cloud identification device 10 or the learning device 20, or may be stored in a device different from the point cloud identification device 10 and the learning device 20.

[0041] (First Example) As a first example, an example will be described in which, as a preprocessing step for reducing the difference in features between domains, the distribution trend of the reflection intensity is grasped using the reflection intensity of certain three-dimensional point cloud data as a reference, and preprocessing is performed to correct the three-dimensional point cloud data used for training so that it follows the distribution trend of the reflection intensity grasped.

[0042] FIG. 5 is a block diagram showing an example of the functional configuration of the point cloud identification device 10.

[0043] 5, the point cloud identification device 10 has, as its functional components, a characteristic correction unit 101 and an inference unit 102. The characteristic correction unit 101 has a reflection intensity feature grasping unit 111 and a reflection intensity correction unit 112. Each functional component is realized when the CPU 11 reads out a point cloud identification program stored in the ROM 12 or storage 14, expands it in the RAM 13, and executes it.

[0044] The reflection intensity feature grasping unit 111 extracts the reflection intensity feature distribution of the reference three-dimensional point cloud data from the input three-dimensional point cloud data. The reflection intensity feature grasping unit 111 extracts the reflection intensity feature distribution of the reference three-dimensional point cloud data from the input three-dimensional point cloud data as follows.

[0045] For example, when determining a reference feature, the reflection intensity feature grasping unit 111 uses the three-dimensional point cloud data X ｉ and the three-dimensional point cloud data P of the domain B to be corrected ｉ The reflection intensities of the features are selected as a normal distribution, for example, to obtain a feature distribution D ｉ The feature distribution D is extracted. ｉ For example, the average μ ｉ , standard deviation σ ｉ The parameters are used in the subsequent reflectance transformation.

[0046] Here, the three-dimensional point cloud data X of the reference domain A ｉ and the three-dimensional point cloud data P of domain B to be corrected. ｉThe correspondence relationship may be, for example, a correspondence relationship between the three-dimensional point cloud data of the learning domain and the three-dimensional point cloud data of the inference domain. ｉ and the three-dimensional point cloud data P of domain B to be corrected. ｉ The correspondence relationship may be, for example, a correspondence relationship between the three-dimensional point cloud data of the inference domain and the three-dimensional point cloud data of the learning domain. ｉ and the three-dimensional point cloud data P of domain B to be corrected. ｉ The correspondence relationship may be, for example, a correspondence relationship between 3D point cloud data other than the learning and inference domains and 3D point cloud data in the learning or inference domains. However, the algorithms used in extracting the feature distribution and the parameters included therein are not limited to this.

[0047] The reflection intensity feature grasping unit 111 uses the three-dimensional point cloud data X of the reference domain A. ｉ In addition, the three-dimensional point cloud data X of the reference domain A ｉ The reference point information may be used to extract the reflection intensity feature distribution of the reference 3D point cloud data from the input 3D point cloud data. Here, the reference point information is information consisting of a set of point numbers, material information, and reflection intensity values. Table 1 shows an example of reference point information.

[0048]

[0049] The reflection intensity correction unit 112 corrects the reflection intensity distribution of the three-dimensional point cloud data to be corrected by approximating it to the reflection intensity feature distribution of the reference three-dimensional point cloud data, based on the reflection intensity feature distribution of the reference three-dimensional point cloud data obtained by the reflection intensity feature grasping unit 111. The reflection intensity correction unit 112 corrects the reflection intensity distribution of the three-dimensional point cloud data to approximate it to the reflection intensity feature distribution of the reference three-dimensional point cloud data, based on the reflection intensity feature distribution of the reference three-dimensional point cloud data obtained by the reflection intensity feature grasping unit 111. ｉ When correcting the reflection intensity distribution of the three-dimensional point cloud data to be corrected using ｉ The entire reflection intensity distribution is subjected to linear transformation by adding, subtracting, multiplying, and dividing the reflection intensity so as to match the mean and variance, and the reflection intensity-corrected three-dimensional point cloud data P ＳHowever, the algorithm used for the reflection intensity conversion is not limited to this.

[0050] The reflection intensity feature grasping unit 111 obtains the three-dimensional point cloud data P of the domain B to be corrected. ｉ In addition, the three-dimensional point cloud data P of the domain B to be corrected ｉ Using the reference point information of the above, the reflection intensity feature distribution of the three-dimensional point cloud data to be corrected is converted into the reflection intensity feature distribution D of the reference three-dimensional point cloud data. ｉ Here, the reference point information is information consisting of a set of a point number, material information, and reflection intensity value.

[0051] The inference unit 102 inputs the three-dimensional point cloud data, for which the reflection intensity has been corrected by the reflection intensity correction unit 112, to the trained model 30, and labels and outputs the three-dimensional point cloud data using the output from the trained model 30. By having the inference unit 102 label and output the three-dimensional point cloud data using the three-dimensional point cloud data, for which the reflection intensity has been corrected by the reflection intensity correction unit 112, the point cloud identification device 10 can suppress a decrease in object identification accuracy compared to when the reflection intensity is not corrected.

[0052] FIG. 6 is a block diagram showing an example of the functional configuration of the learning device 20.

[0053] 6, the learning device 20 has, as its functional components, a characteristic correction unit 201 and a learning unit 202. The characteristic correction unit 201 has a reflection intensity feature grasping unit 211 and a reflection intensity correction unit 212. Each functional component is realized by the CPU 21 reading out a learning program stored in the ROM 22 or storage 24, expanding it in the RAM 23, and executing it.

[0054] Similar to the reflection intensity feature grasping unit 111, the reflection intensity feature grasping unit 211 obtains the reference three-dimensional point cloud data X ｉ The characteristic distribution of reflection intensity is extracted from the input 3D point cloud data.

[0055] Similar to the reflection intensity correction unit 112, the reflection intensity correction unit 212 corrects the reflection intensity distribution of the three-dimensional point cloud data to be corrected by approximating it to the reflection intensity characteristic distribution of the reference three-dimensional point cloud data, based on the reflection intensity characteristic distribution of the reference three-dimensional point cloud data obtained by the reflection intensity characteristic understanding unit 211.

[0056] The learning unit 202 performs machine learning using the three-dimensional point cloud data whose reflection intensities have been corrected by the reflection intensity correction unit 212, and generates a trained model 30. By having the learning unit 202 generate the trained model 30 using the three-dimensional point cloud data whose reflection intensities have been corrected by the reflection intensity correction unit 212, the learning device 20 can generate a trained model 30 that suppresses a decrease in object identification accuracy compared to when the reflection intensities are not corrected.

[0057] Next, the operation of the point cloud identification device 10 will be described.

[0058] 7 is a flowchart showing the flow of the point cloud identification process by the point cloud identification device 10. The CPU 11 reads out a point cloud identification program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing the point cloud identification process.

[0059] In step S111, the CPU 11 functions as the reflection intensity feature grasping unit 111 to extract a reflection intensity feature distribution of the reference three-dimensional point cloud data from the input three-dimensional point cloud data.

[0060] Following step S111, in step S112, the CPU 11, as the reflection intensity correction unit 112, corrects the reflection intensity distribution of the three-dimensional point cloud data to be corrected by approximating it to the reflection intensity characteristic distribution of the reference three-dimensional point cloud data, based on the reflection intensity characteristic distribution of the reference three-dimensional point cloud data obtained in step S101.

[0061] Following step S112, in step S113, the CPU 11, as the inference unit 102, inputs the three-dimensional point cloud data whose reflection intensity has been corrected in step S102 into the trained model 30, and uses the output from the trained model 30 to label and output the three-dimensional point cloud data.

[0062] Next, the operation of the learning device 20 will be described.

[0063] 8 is a flowchart showing the flow of the learning process by the learning device 20. The learning process is performed by the CPU 21 reading out a learning program from the ROM 22 or the storage 24, loading it into the RAM 23, and executing it.

[0064] In step S211, the CPU 21 functions as the reflection intensity feature grasping unit 211 to extract a reflection intensity feature distribution of the reference three-dimensional point cloud data from the input three-dimensional point cloud data.

[0065] Following step S211, in step S212, the CPU 21, as the reflection intensity correction unit 212, corrects the reflection intensity distribution of the three-dimensional point cloud data to be corrected by approximating it to the reflection intensity characteristic distribution of the reference three-dimensional point cloud data, based on the reflection intensity characteristic distribution of the reference three-dimensional point cloud data obtained in step S201.

[0066] Following step S212, in step S213, the CPU 21, as the learning unit 202, performs machine learning using the three-dimensional point cloud data whose reflection intensity has been corrected in step S212, and generates a trained model 30.

[0067] (Second Example) As a second example, an example will be described in which, as a preprocessing step for reducing the difference in features between domains, preprocessing is performed on three-dimensional point cloud data for learning or inference, by upsampling or downsampling the data so that the point cloud density trend is uniform.

[0068] FIG. 9 is a block diagram showing an example of the functional configuration of the point cloud identification device 10.

[0069] 9 , the point cloud identification device 10 has, as functional components, a characteristic correction unit 101 and an inference unit 102. The characteristic correction unit 101 has a point cloud density correction unit 121. Each functional component is realized when the CPU 11 reads out a point cloud identification program stored in the ROM 12 or the storage 14, expands it in the RAM 13, and executes it.

[0070] The point cloud density correction unit 121 corrects the input three-dimensional point cloud data Pｉ The three-dimensional point cloud data P ｉ The point cloud density correction unit 121 performs a point cloud density correction process to correct the overall point cloud density. When performing upsampling or downsampling as the point cloud density correction process, the point cloud density correction unit 121 performs upsampling, downsampling, or a combination of upsampling and downsampling so that the density exceeds a reference value of the lower limit of the point cloud density in a space of a certain volume, or a threshold value α that is a ratio of the lower limit of the point cloud density in a space of a certain volume, as the reference. The point cloud density correction unit 121 performs upsampling, downsampling, or a combination of upsampling and downsampling so that the density falls below a reference value of the upper limit of the point cloud density in a space of a certain volume, or a threshold value β that is a ratio of the upper limit of the point cloud density in a space of a certain volume, as the reference. The point cloud density correction unit 121 performs upsampling, downsampling, or a combination of upsampling and downsampling so that the density falls below a reference value. ｉ or three-dimensional point cloud data P with corrected reflection intensity ｓ From the point cloud density correction process, the three-dimensional point cloud data P ｄ The applicable range of the point cloud density threshold, the sampling algorithm, the elements of the threshold, and the calculation method are not limited to those described above.

[0071] The inference unit 102 calculates the three-dimensional point cloud data P whose point cloud density has been corrected by the point cloud density correction unit 121. ｄ is input to the trained model 30, and the output from the trained model 30 is used to calculate the three-dimensional point cloud data P ｄ The point cloud identification device 10 uses the three-dimensional point cloud data whose point cloud density has been corrected by the point cloud density correction unit 121, and the inference unit 102 calculates the three-dimensional point cloud data P ｄ By labeling and outputting the point cloud density, it is possible to suppress the decrease in object identification accuracy compared to when the point cloud density is not corrected.

[0072] FIG. 10 is a block diagram showing an example of the functional configuration of the learning device 20.

[0073] 10 , the learning device 20 has, as functional components, a characteristic correction unit 201 and a learning unit 202. The characteristic correction unit 201 has a point cloud density correction unit 221. Each functional component is realized by the CPU 21 reading out a learning program stored in the ROM 22 or the storage 24, expanding it in the RAM 23, and executing it.

[0074] Similar to the point cloud density correction unit 121, the point cloud density correction unit 221 performs a point cloud density correction process to correct the point cloud density of the entire three-dimensional point cloud data so that the point cloud density distribution of the input three-dimensional point cloud data becomes uniform.

[0075] The learning unit 202 calculates the three-dimensional point cloud data P whose point cloud density has been corrected by the point cloud density correction unit 221. ｄ The learning device 20 performs machine learning using the three-dimensional point cloud data P whose point cloud density has been corrected by the point cloud density correction unit 221, and generates a trained model 30. ｄ By the learning unit 202 generating the trained model 30 using the above, it is possible to generate a trained model 30 that reduces the decrease in object identification accuracy compared to when the point cloud density is not corrected.

[0076] Next, the operation of the point cloud identification device 10 will be described.

[0077] 11 is a flowchart showing the flow of the point cloud identification process by the point cloud identification device 10. The CPU 11 reads out a point cloud identification program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing the point cloud identification process.

[0078] In step S121, the CPU 11 functions as the point cloud density corrector 121 to perform a point cloud density correction process for correcting the point cloud density of the entire three-dimensional point cloud data so as to make the point cloud density distribution of the input three-dimensional point cloud data uniform.

[0079] Following step S121, in step S122, the CPU 11, as the inference unit 102, inputs the three-dimensional point cloud data whose point cloud density has been corrected in step S121 into the trained model 30, and uses the output from the trained model 30 to label and output the three-dimensional point cloud data.

[0080] Next, the operation of the learning device 20 will be described.

[0081] 12 is a flowchart showing the flow of the learning process by the learning device 20. The learning process is performed by the CPU 21 reading out a learning program from the ROM 22 or the storage 24, expanding it into the RAM 23, and executing it.

[0082] In step S221, the CPU 21 functions as the point cloud density correcting unit 221 to perform a point cloud density correction process for correcting the point cloud density of the entire three-dimensional point cloud data so as to make the point cloud density distribution of the input three-dimensional point cloud data uniform.

[0083] Following step S221, in step S222, the CPU 21, as the learning unit 202, performs machine learning using the three-dimensional point cloud data whose point cloud density has been corrected in step S221, and generates a trained model 30.

[0084] Third Example As a third example, similar to the second example, an example will be described in which preprocessing is performed on three-dimensional point cloud data for learning or inference, in order to reduce differences in features between domains, by upsampling or downsampling the data so that the point cloud density distribution is uniform. In the third example, an example will be described in which preprocessing is performed on three-dimensional point cloud data to be input, in order to uniformize the point cloud density distribution.

[0085] FIG. 13 is a block diagram showing an example of the functional configuration of the point cloud identification device 10.

[0086] 13 , the point cloud identification device 10 has, as functional components, a characteristic correction unit 101 and an inference unit 102. The characteristic correction unit 101 has a point cloud density analysis unit 131 and a point cloud density correction unit 132. Each functional component is realized when the CPU 11 reads out a point cloud identification program stored in the ROM 12 or storage 14, expands it in the RAM 13, and executes it.

[0087] The point cloud density analysis unit 131 analyzes the point cloud density distribution of the input three-dimensional point cloud data and grasps the characteristics of the point cloud density distribution. The analysis result of the point cloud density distribution by the point cloud density analysis unit 131 is used in the point cloud density correction process by the point cloud density correction unit 132.

[0088] The point cloud density corrector 132 performs a point cloud density correction process to correct the point cloud density of the entire three-dimensional point cloud data to uniformize the point cloud density distribution of the input three-dimensional point cloud data, using the analysis result of the point cloud density distribution by the point cloud density analyzer 131. The point cloud density correction process may be the same as that in the second embodiment.

[0089] The inference unit 102 calculates the three-dimensional point cloud data P whose point cloud density has been corrected by the point cloud density correction unit 132. ｄ is input to the trained model 30, and the three-dimensional point cloud data is labeled and output using the output from the trained model 30. The point cloud identification device 10 has the inference unit 102 label and output the three-dimensional point cloud data using the three-dimensional point cloud data whose point cloud density has been corrected by the point cloud density correction unit 132, thereby making it possible to suppress a decrease in object identification accuracy compared to when the point cloud density is not corrected.

[0090] FIG. 14 is a block diagram showing an example of the functional configuration of the learning device 20.

[0091] 14, the learning device 20 has, as functional components, a characteristic correction unit 201 and a learning unit 202. The characteristic correction unit 201 has a point cloud density analysis unit 231 and a point cloud density correction unit 232. Each functional component is realized when the CPU 21 reads out a learning program stored in the ROM 22 or the storage 24, expands it in the RAM 23, and executes it.

[0092] The point cloud density analysis unit 231 analyzes the point cloud density distribution of the input three-dimensional point cloud data and grasps the characteristics of the point cloud density distribution, similar to the point cloud density analysis unit 131. The analysis result of the point cloud density distribution by the point cloud density analysis unit 231 is used in the point cloud density correction process by the point cloud density correction unit 232.

[0093] Similar to the point cloud density correction unit 132, the point cloud density correction unit 232 uses the analysis results of the point cloud density distribution by the point cloud density analysis unit 231 to perform a point cloud density correction process that corrects the point cloud density of the entire three-dimensional point cloud data so that the point cloud density distribution of the input three-dimensional point cloud data is uniform.

[0094] The learning unit 202 uses the three-dimensional point cloud data P whose point cloud density has been corrected by the point cloud density correction unit 232. ｄ The learning device 20 performs machine learning using the three-dimensional point cloud data, and generates a trained model 30. By having the learning unit 202 generate the trained model 30 using the three-dimensional point cloud data whose point cloud density has been corrected by the point cloud density correction unit 232, the learning device 20 can generate a trained model 30 in which the deterioration in object identification accuracy is suppressed compared to when the point cloud density is not corrected.

[0095] Next, the operation of the point cloud identification device 10 will be described.

[0096] 15 is a flowchart showing the flow of the point cloud identification process by the point cloud identification device 10. The CPU 11 reads out a point cloud identification program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing the point cloud identification process.

[0097] In step S131, the CPU 11 functions as the point cloud density analysis unit 131 to analyze the point cloud density distribution of the input three-dimensional point cloud data and grasp the characteristics of the point cloud density distribution.

[0098] Following step S131, in step S132, the CPU 11, as the point cloud density correction unit 132, uses the analysis results of the point cloud density distribution to perform a point cloud density correction process to correct the point cloud density of the entire three-dimensional point cloud data so that the point cloud density distribution of the input three-dimensional point cloud data is made uniform.

[0099] Following step S132, in step S133, the CPU 11, as the inference unit 102, inputs the three-dimensional point cloud data whose point cloud density has been corrected in step S132 into the trained model 30, and uses the output from the trained model 30 to label and output the three-dimensional point cloud data.

[0100] Next, the operation of the learning device 20 will be described.

[0101] 16 is a flowchart showing the flow of the learning process by the learning device 20. The learning process is performed by the CPU 21 reading out a learning program from the ROM 22 or the storage 24, loading it into the RAM 23, and executing it.

[0102] In step S231, the CPU 21 functions as the point cloud density analysis unit 231 to analyze the point cloud density distribution of the input three-dimensional point cloud data and grasp the characteristics of the point cloud density distribution.

[0103] Following step S231, in step S232, the CPU 21, as the point cloud density correction unit 232, uses the analysis results of the point cloud density distribution to perform a point cloud density correction process to correct the point cloud density of the entire three-dimensional point cloud data so that the point cloud density distribution of the input three-dimensional point cloud data is uniform.

[0104] Following step S232, in step S233, the CPU 21, as the learning unit 202, performs machine learning using the three-dimensional point cloud data whose point cloud density has been corrected in step S232, and generates a trained model 30.

[0105] (Fourth Example) As a fourth example, an example will be described in which, as preprocessing to reduce the difference in features between domains, a scan line is simulated using the time-varying spatial point cloud density trend of the reference three-dimensional point cloud data as a feature, and preprocessing is performed to correct the point cloud density distribution so that the input three-dimensional point cloud data is virtually measured by a measuring device.

[0106] FIG. 17 is a block diagram showing an example of the functional configuration of the point cloud identification device 10.

[0107] 17 , the point cloud identification device 10 has, as its functional components, a characteristic correction unit 101 and an inference unit 102. The characteristic correction unit 101 has a point cloud density trend understanding unit 141 and a point cloud density correction unit 142. Each functional component is realized when the CPU 11 reads out a point cloud identification program stored in the ROM 12 or storage 14, expands it in the RAM 13, and executes it.

[0108] The point cloud density trend grasping unit 141 extracts the feature of the point cloud density of the reference point cloud from the input three-dimensional point cloud data. The feature of the point cloud density is, for example, the number N of laser irradiation beams per unit time during measurement, simulating the irradiation laser beam obtained from the irradiation direction φ. For example, in the case of reproducing a rotary LiDAR mounted on an MMS (Mobile Mapping System), a virtual time-varying scan line is simulated, and for example, a feature D is a set of equations of straight lines of laser irradiation in the measurement direction over time. （Ｎ，φ） is acquired and used in the subsequent point cloud density correction unit 142. （Ｎ，φ） may be given as an input to the point cloud density correction unit 142, or may be estimated from the input point cloud. ｉ and the three-dimensional point cloud data P of the domain B to be corrected ｉ The correspondence relationship may be, for example, a correspondence relationship between a point cloud in a learning domain and a point cloud in an inference domain, a correspondence relationship between a point cloud in an inference domain and the learning domain, or a correspondence relationship between a point cloud other than the learning and inference domains and the learning or inference domain. However, in the present disclosure, the parameters and elements used in feature extraction are not limited to these.

[0109] The point cloud density correction unit 142 performs a point cloud density correction process to correct the point cloud density of the input three-dimensional point cloud data based on the features obtained by the point cloud density trend understanding unit 141 so that the point cloud density is approximated to a point cloud density distribution when the input three-dimensional point cloud data is measured by a virtual measuring device that simulates the features. （Ｎ，φ） For example, the intersection points when a laser is irradiated onto a meshed point cloud are used as a virtual point cloud, and the three-dimensional point cloud data P ｉ The three-dimensional point cloud data P obtained by correcting the point cloud density from ｄ However, the algorithm used in the point cloud density correction process for correcting the point cloud density to make it appear as if it were measured using a virtual measuring device is not limited to this.

[0110] The inference unit 102 calculates the three-dimensional point cloud data P whose point cloud density has been corrected by the point cloud density correction unit 142. ｄis input to the trained model 30, and the three-dimensional point cloud data is labeled and output using the output from the trained model 30. The point cloud identification device 10 has the inference unit 102 label and output the three-dimensional point cloud data using the three-dimensional point cloud data whose point cloud density has been corrected by the point cloud density correction unit 142, thereby making it possible to suppress a decrease in object identification accuracy compared to when the point cloud density is not corrected.

[0111] FIG. 18 is a block diagram showing an example of the functional configuration of the learning device 20.

[0112] 18 , the learning device 20 has, as its functional components, a characteristic correction unit 201 and a learning unit 202. The characteristic correction unit 201 has a point cloud density trend grasping unit 241 and a point cloud density correction unit 242. Each functional component is realized by the CPU 21 reading out a learning program stored in the ROM 22 or the storage 24, expanding it in the RAM 23, and executing it.

[0113] The point cloud density trend grasping unit 241 extracts point cloud density features of a reference point cloud from the input three-dimensional point cloud data, similar to the point cloud density trend grasping unit 141. The point cloud density feature extraction result by the point cloud density trend grasping unit 241 is used in the point cloud density correction process by the point cloud density correction unit 242.

[0114] Similar to the point cloud density correction unit 142, the point cloud density correction unit 242 uses the results of the extraction of point cloud density features by the point cloud density trend understanding unit 241 to perform point cloud density correction processing to correct the point cloud density of the input three-dimensional point cloud data so that it approximates the point cloud density distribution when the input three-dimensional point cloud data is measured using a virtual measuring device that simulates the features.

[0115] The learning unit 202 calculates the three-dimensional point cloud data P whose point cloud density has been corrected by the point cloud density correction unit 242. ｄ The learning device 20 performs machine learning using the three-dimensional point cloud data, and generates a trained model 30. By having the learning unit 202 generate the trained model 30 using the three-dimensional point cloud data whose point cloud density has been corrected by the point cloud density correction unit 242, the learning device 20 can generate a trained model 30 in which the deterioration in object identification accuracy is suppressed compared to when the point cloud density is not corrected.

[0116] Next, the operation of the point cloud identification device 10 will be described.

[0117] 19 is a flowchart showing the flow of the point cloud identification process by the point cloud identification device 10. The CPU 11 reads out a point cloud identification program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing the point cloud identification process.

[0118] In step S141, the CPU 11 functions as the point cloud density trend grasping unit 141 to extract the characteristics of the point cloud density of the reference point cloud from the input three-dimensional point cloud data.

[0119] Following step S141, in step S142, the CPU 11, as the point cloud density correction unit 142, performs a point cloud density correction process using the extracted results of the point cloud density features to correct the point cloud density of the input three-dimensional point cloud data so that it approximates the point cloud density distribution when the input three-dimensional point cloud data is measured using a virtual measuring device that simulates the features.

[0120] Following step S142, in step S143, the CPU 11, as the inference unit 102, inputs the three-dimensional point cloud data whose point cloud density has been corrected in step S142 into the trained model 30, and uses the output from the trained model 30 to label and output the three-dimensional point cloud data.

[0121] Next, the operation of the learning device 20 will be described.

[0122] 20 is a flowchart showing the flow of the learning process by the learning device 20. The learning process is performed by the CPU 21 reading out a learning program from the ROM 22 or the storage 24, expanding it into the RAM 23, and executing it.

[0123] In step S241, the CPU 11 functions as the point cloud density trend grasping unit 241 to extract the characteristics of the point cloud density of the reference point cloud from the input three-dimensional point cloud data.

[0124] Following step S241, in step S242, the CPU 11, as the point cloud density correction unit 242, performs a point cloud density correction process using the extracted results of the point cloud density features to correct the point cloud density of the input three-dimensional point cloud data so that it approximates the point cloud density distribution when the input three-dimensional point cloud data is measured using a virtual measuring device that simulates the features.

[0125] Following step S242, in step S243, the CPU 21, as the learning unit 202, performs machine learning using the three-dimensional point cloud data whose point cloud density has been corrected in step S242, and generates a trained model 30.

[0126] The above-described embodiments may be implemented independently or in combination. For example, either the reflection intensity or the point cloud density may be corrected, or both may be corrected. Furthermore, in the above-described embodiments, if the result of the point cloud identification process performed by the point cloud identification device 10 differs from that of the actual object, the learned model 30 may be retrained by the learning device 20 so that the result of the point cloud identification process performed by the point cloud identification device 10 matches that of the actual object.

[0127] As described above, according to an embodiment of the present disclosure, a learning device 20 is provided that corrects at least one of the reflection intensity or point cloud density of three-dimensional point cloud data and generates a trained model 30 by learning using a DNN with the corrected three-dimensional point cloud data. Furthermore, according to an embodiment of the present disclosure, a point cloud identification device 10 is provided that corrects at least one of the reflection intensity or point cloud density of three-dimensional point cloud data and labels the three-dimensional point cloud data by inputting the corrected three-dimensional point cloud data into the trained model 30. The correction may be performed at least during learning or inference. That is, the learning device 20 may correct at least one of the reflection intensity or point cloud density of the three-dimensional point cloud data during learning, and the point cloud identification device 10 may not perform the correction during inference. Furthermore, the learning device 20 may correct at least one of the reflection intensity or point cloud density of the three-dimensional point cloud data during learning, and the point cloud identification device 10 may also correct at least one of the reflection intensity or point cloud density of the three-dimensional point cloud data during inference.

[0128] The point cloud identification device 10 according to an embodiment of the present disclosure can reduce the decline in object identification accuracy by correcting at least one of the reflection intensity or point cloud density of three-dimensional point cloud data and performing inference using the corrected three-dimensional point cloud data and a trained model 30 generated based on the corrected three-dimensional point cloud data.

[0129] In addition, the point cloud identification processing and learning processing executed by the CPU by reading software (programs) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electrical circuits, which are processors having circuit configurations designed specifically for executing specific processes, such as programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and application-specific integrated circuits (ASICs). The point cloud identification processing and learning processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0130] In addition, in each of the above embodiments, the point cloud identification program is pre-stored (installed) in the storage 14, and the learning program is pre-stored in the storage 24. However, this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0131] The following supplementary note is further disclosed regarding the above embodiment: (Supplementary Note 1) A point cloud identification device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: input three-dimensional point cloud data of an inference target among a plurality of pieces of three-dimensional point cloud data obtained by a sensor that measures an object, to a trained model that has been machine-learned using labeled training three-dimensional point cloud data, and to infer an object corresponding to the three-dimensional point cloud data of the inference target by outputting the labeled three-dimensional point cloud data of the inference target from the trained model; and perform correction to reduce differences in characteristics of the plurality of pieces of three-dimensional point cloud data at least either before the machine learning or before the inference.

[0132] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform point cloud identification processing, wherein the point cloud identification processing comprises: inputting three-dimensional point cloud data of an inference target among a plurality of three-dimensional point cloud data obtained by a sensor that measures an object into a trained model that has been machine-learned using labeled three-dimensional point cloud data for training; and outputting the labeled three-dimensional point cloud data of the inference target from the trained model, thereby inferring an object that corresponds to the three-dimensional point cloud data of the inference target; and performing correction to reduce differences in characteristics of the plurality of three-dimensional point cloud data at least either before the machine learning or before the inference.

[0133] 10 Point cloud identification device 20 Learning device 30 Trained model

Claims

1. A point cloud identification device comprising: an inference unit that inputs three-dimensional point cloud data of an inference target among multiple three-dimensional point cloud data obtained by a sensor that measures an object into a trained model that has been machine-learned using labeled training three-dimensional point cloud data, and infers an object corresponding to the three-dimensional point cloud data of the inference target by outputting the labeled three-dimensional point cloud data of the inference target from the trained model; and a characteristic correction unit that performs correction to reduce differences in characteristics of the multiple three-dimensional point cloud data at least either before the machine learning or before inference by the inference unit.

2. The point cloud identification device according to claim 1, wherein the characteristic correction unit extracts a characteristic feature distribution of a first three-dimensional point cloud data serving as a reference from among the plurality of three-dimensional point cloud data, and corrects the characteristics of a second three-dimensional point cloud data to be corrected from among the plurality of three-dimensional point cloud data so as to approximate the extracted characteristic distribution.

3. A point cloud identification method in which a processor inputs three-dimensional point cloud data of an inference target among multiple three-dimensional point cloud data obtained by a sensor that measures an object into a trained model that has been machine-learned using labeled training three-dimensional point cloud data, and outputs the labeled three-dimensional point cloud data of the inference target from the trained model, thereby inferring an object that corresponds to the three-dimensional point cloud data of the inference target, and performs a process of making a correction to reduce differences in characteristics of the multiple three-dimensional point cloud data at least either before the machine learning or before the inference.

4. A point cloud identification program that causes a computer to execute a process of inputting three-dimensional point cloud data of an inference target among multiple three-dimensional point cloud data obtained by a sensor that measures an object into a trained model that has been machine-learned using labeled training three-dimensional point cloud data, and outputting the labeled three-dimensional point cloud data of the inference target from the trained model, thereby inferring an object that corresponds to the three-dimensional point cloud data of the inference target, and performing a correction to reduce differences in characteristics of the multiple three-dimensional point cloud data at least either before the machine learning or before the inference.

Images