Point cloud data recognition device, method, and program

Abstract

The problem this invention aims to solve is to improve the accuracy of class label recognition of point cloud data by enabling the generation of images that reflect the features of point cloud data. [Solution] One aspect of this invention involves acquiring point cloud data of a target space and generating images projected from the point cloud data according to a plurality of projection parameters set for each class of the point cloud data. Then, feature quantities are extracted from each of the images, and the class of the imaged point cloud data is recognized based on each of the feature quantities.

Description

Technical Field

[0001] One aspect of this invention relates to a point cloud data recognition device, method, and program for recognizing class labels of three-dimensional point cloud data using, for example, a machine learning model.

Background Art

[0002] Point cloud data measured using a depth camera capable of measuring depth in a three-dimensional space, a laser radar (Light Detection and Ranging: LiDAR), etc. is expected to be used for creating high-precision three-dimensional maps, urban planning, simulations, etc. used in autonomous vehicles and autonomous mobile robots. In addition, research and development on technologies for recognizing objects from the measured point cloud data are also widely carried out.

[0003] By the way, in recent years, many methods have been proposed for extracting features from point cloud data using a machine learning model to which deep learning is applied and recognizing the class label of the point cloud based on the extracted features. However, point cloud data often records specific regions or objects, and it is difficult to obtain diverse data compared to modalities such as images and texts. In addition, the work of attaching learning annotations to the obtained point cloud data also requires a great deal of cost.

[0004] Therefore, for example, in Non-Patent Documents 1-3, methods have been proposed for utilizing an image recognition model pre-trained using a large amount of data in order to recognize the class label of a point cloud.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] However, the methods described in Non-Patent Documents 1-3 involve converting point cloud data into an image via a virtual camera, then using a machine learning model to extract features from the image, and finally identifying class labels based on the extracted features. Therefore, depending on the virtual camera's orientation, position, distortion, scale, and other configuration factors, the image generated from the point cloud data may change, resulting in images unsuitable for class label recognition. In other words, images unsuitable for class label recognition may be rendered. In such cases, a decrease in the accuracy of class label recognition is unavoidable.

[0007] This invention was made in view of the above circumstances, and aims to provide a technology that enables the generation of images reflecting the features of point cloud data, thereby improving the accuracy of class label recognition of point cloud data.

[0008] To solve the above problems, one aspect of the point cloud data recognition device or method according to the present invention acquires point cloud data of a target space and generates images projected from the point cloud data according to a plurality of projection parameters set for each class of the point cloud data. Then, features are extracted from each of the images and the class of the imaged point cloud data is recognized based on each of the features.

[0009] According to one aspect of this invention, point cloud data is imaged according to projection parameters set to optimal values ​​for each class of point cloud. That is, the point cloud data is projected onto an image by a virtual camera set to an optimal position, thereby making it possible to generate an image that appropriately reflects the features of the point cloud data. Therefore, by recognizing the class based on the features extracted from the image thus generated, it becomes possible to recognize class labels with high accuracy. [Effects of the Invention]

[0010] In other words, according to one aspect of this invention, it is possible to generate an image that reflects the features of point cloud data, thereby providing a technology that improves the accuracy of recognizing class labels of point cloud data. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows an example of the configuration of a system equipped with a point cloud data recognition device according to the first embodiment of this invention. [Figure 2] Figure 2 is a block diagram showing an example of the hardware configuration of a point cloud data recognition device according to the first embodiment of this invention. [Figure 3]Figure 3 is a block diagram showing an example of the software configuration of a point cloud data recognition device according to the first embodiment of this invention. [Figure 4] Figure 4 is a block diagram showing the software configuration of the point cloud image processing unit of the point cloud data recognition device shown in Figure 3. [Figure 5] Figure 5 is a flowchart showing an example of the processing procedure and content of the projection parameter optimization process performed by the control unit of the point cloud data recognition device shown in Figure 3 during the learning phase. [Figure 6] Figure 6 is a flowchart showing an example of the processing procedure and content of the solution candidate generation process, which is part of the point cloud class label identification process shown in Figure 5. [Figure 7] Figure 7 is a flowchart showing the processing procedure and processing content of the point cloud class label identification process performed by the control unit of the point cloud data recognition device shown in Figure 3 during the estimation phase, as in the first embodiment. [Figure 8] Figure 8 is a flowchart showing an example of the processing procedure and content of the image conversion process for point cloud data, which is part of the point cloud class label identification process shown in Figure 7. [Figure 9] Figure 9 is a diagram illustrating an example of the operation of the point cloud class label identification process shown in Figure 7. [Figure 10] Figure 10 is a flowchart showing an example of the processing procedure and processing content of the point cloud class label identification process performed in the estimation phase by the control unit of the point cloud data recognition device according to the second embodiment of this invention. [Figure 11] Figure 11 is a diagram illustrating an example of the operation of the point cloud class label identification process performed by the point cloud data recognition device according to the third embodiment of this invention. [Figure 12] Figure 12 is a diagram illustrating an example of the operation of the point cloud class label identification process performed by the point cloud data recognition device according to the fourth embodiment of this invention. [Modes for carrying out the invention]

[0012] Embodiments of this invention will be described below with reference to the drawings.

[0013] [First Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the configuration of a system including a point cloud data recognition device according to the first embodiment of the present invention.

[0014] The system of the first embodiment connects a point cloud data measurement device MS and a terminal device US to a point cloud data recognition device SM via a network NW.

[0015] The point cloud data measurement device MS is composed of, for example, a depth camera or a lidar (LiDAR), and transmits three-dimensional point cloud data (hereinafter simply referred to as point cloud data) obtained by measuring a target range including an object as a target object to the point cloud data recognition device SV via the network NW.

[0016] The terminal device US is used by a system administrator or a user and is composed of, for example, a personal computer. The terminal device US is used to input learning point cloud data, verification data, and point cloud imaging conditions to the point cloud data recognition device SV, or to receive data representing class labels of point clouds identified by the point cloud data recognition device SV. Note that the terminal device US is not limited to a personal computer, and for example, a smartphone or a tablet terminal may be used.

[0017] The network NW includes, for example, a wide area network centered on the Internet and an access network for accessing this wide area network. As the access network, for example, a public data communication network using wireless or a LAN (Local Area Network) is used, but it is not limited to these.

[0018] The point cloud data recognition device SV, the point cloud data measurement device MS, and the terminal device US may be connected via a wired LAN, a wireless interface using a low-power data communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or directly via a signal cable.

[0019] (2) Point cloud data recognition device SV The point cloud data recognition device SV is installed, for example, in a server computer located on the Web or in the cloud. Alternatively, the point cloud data recognition device SV may be installed in a terminal device, such as a personal computer used by a system administrator.

[0020] Figures 2 and 3 are block diagrams showing an example of the hardware and software configuration of the point cloud data recognition device SV.

[0021] The point cloud data recognition device SV includes a control unit 1 that uses a hardware processor such as a Central Processing Unit (CPU), and a storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter referred to as I / F) unit 4 are connected to this control unit 1 via a bus 5.

[0022] The input / output interface unit 4 transmits and receives various types of data between the point cloud data measurement device MS and the terminal device US in accordance with the communication protocol defined in the network NW.

[0023] The program storage unit 2 is configured, for example, as a storage medium, by combining a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), with a non-volatile memory such as ROM (Read Only Memory). In addition to middleware such as an OS (Operating System), it stores various programs necessary to execute various control processes according to the first embodiment of this invention.

[0024] The data storage unit 3 is configured, for example, as a storage medium, by combining a non-volatile memory that can be written to and read at any time, such as an HDD or SSD, with a volatile memory such as RAM (Random Access Memory). The storage area is provided with a projection parameter storage unit 31, an imaging condition storage unit 32, and an image feature extraction model storage unit 33, which are necessary storage units for carrying out the first embodiment.

[0025] The projection parameter storage unit 31 stores projection parameters used to image point cloud data using a machine learning model, that is, attitude transformation parameters for transforming the arrangement state such as the attitude and position of a virtual camera. The projection parameters are optimized for each class of point cloud data and are represented by a matrix and position vector that represent the attitude transformation.

[0026] The image conversion condition storage unit 32 stores various conditions used for image correction processes such as point cloud coloring, pixelation, sharpening, or denoising, which are performed during the image conversion process of point cloud data. These image conversion conditions are registered in advance, for example, from the system administrator's terminal device US.

[0027] The image feature extraction model storage unit 33 stores the weight parameters of a trained machine learning model used to extract features from an image generated based on point cloud data.

[0028] The control unit 1 includes, as processing functions for carrying out the first embodiment, a point cloud data acquisition processing unit 11, a point cloud image processing unit 12, an image feature extraction processing unit 13, a point cloud class label identification processing unit 14, a class label output processing unit 15, and a learning processing unit 16.

[0029] Each of these processing units 11 to 16 is implemented by having the hardware processor of the control unit 1 execute the application program stored in the program storage unit 2. Note that some or all of the above processing units 11 to 16 may be implemented using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

[0030] In the projection parameter learning phase, the point cloud data acquisition processing unit 11 acquires multiple point cloud data for learning and corresponding validation class labels from the system administrator's terminal device US. In the estimation phase, the point cloud data acquisition processing unit 11 acquires point cloud data output from the point cloud data measurement device MS.

[0031] The point cloud image processing unit 12 generates an image by projecting the point cloud data acquired by the point cloud data acquisition processing unit 11 using projection parameters stored in the projection parameter storage unit 31, for example, using a machine learning model composed of a neural network.

[0032] Figure 4 is a block diagram showing the functional configuration of the point cloud imaging processing unit 12. Specifically, the point cloud imaging processing unit 12 includes a virtual camera placement processing unit 121, a point cloud coloring processing unit 122, a rasterization processing unit 123, and an image correction processing unit 124 as its processing functions.

[0033] The virtual camera placement processing unit 121 sets the placement of virtual cameras based on the projection parameters for each point cloud class stored in the projection parameter storage unit 31, and generates images by projecting the point cloud data using the virtual cameras in this state.

[0034] The point cloud coloring processing unit 122 colors the point cloud of the image based on the conditions that specify the illumination position in the target space, which are stored in the imaging condition storage unit 32.

[0035] The rasterization processing unit 123 pixels the image after the colorization process based on the resolution and point cloud radius specified in the image conversion condition storage unit 32.

[0036] The image correction processing unit 124 performs sharpening and denoising on the image pixelated by the rasterization processing unit 123 based on the conditions for sharpening and denoising stored in the image conversion condition storage unit 32.

[0037] The image feature extraction processing unit 13 uses a trained machine learning model, whose weight parameters are stored in the image feature extraction model storage unit 33, to extract features from each image generated by the point cloud imaging processing unit 12 for each point cloud class. The machine learning model for feature extraction is, for example, composed of a neural network.

[0038] The point cloud class label identification processing unit 14 estimates (identifies) the class label of the point cloud based on the feature quantities extracted from the images corresponding to each class by the image feature extraction processing unit 13.

[0039] The class label output processing unit 15 calculates the confidence level of the class labels identified for each class from the image features by the point cloud class label identification processing unit 14, selects the class label with the highest confidence level, and transmits the selected class label from the input / output I / F unit 4 to the requesting terminal device US.

[0040] During the learning phase, the learning processing unit 16 optimizes the projection parameters used by the point cloud imaging processing unit 12 when given multiple training point cloud data sets and validation class labels prepared for each point cloud class. The learning processing unit 16 then stores the optimized projection parameters for each point cloud class in the projection parameter storage unit 31. An example of the learning process will be explained in the operation example.

[0041] (Example of operation) Next, we will explain an example of the operation of the point cloud data recognition device SV configured as described above.

[0042] (1) Learning Phase The point cloud data recognition device SV first performs a learning process in the point cloud imaging processing unit 12 to optimize the projection parameters used to image the point cloud data for each point cloud class.

[0043] Figure 5 is a flowchart showing an example of the processing procedure and content of the projection parameter optimization process performed by the control unit 1 of the point cloud data recognition device SV during the learning phase.

[0044] In step S10, the control unit 1 of the point cloud data recognition device SV first acquires, under the control of the point cloud data acquisition processing unit 11, multiple training point cloud data sets and corresponding verification class labels from the terminal device US, which have been prepared in advance, for example by the system administrator.

[0045] Once the above-mentioned training point cloud data and validation class labels are acquired, the control unit 1 of the point cloud data recognition device SV performs the following process to optimize the projection parameters under the control of the learning processing unit 16.

[0046] For example, suppose we are given a set of data consisting of training point cloud data and validation class labels (P_1, y_1), (P_2, y_2), ..., (P_N, y_N). Then the learning processing unit 16 uses parameters representing attitude transformations, such as Euler angles or quaternions, as design variables, and performs optimization with the classification accuracy or cross-entropy error of the class labels identified by the point cloud class label identification processing unit 14 as the target variable. For optimization, gradient-based optimization algorithms such as the steepest descent method or Newton's method may be used, or heuristic optimization algorithms such as genetic algorithms, evolutionary strategy algorithms, or particle swarm optimization may be used.

[0047] The above learning process is executed in the control unit 1 of the point cloud data recognition device SV as follows. Specifically, in step S11, one of several point cloud classes is selected. Here, the multiple point cloud classes correspond to multiple virtual camera arrangement states that are considered when recognizing objects on the point cloud coordinate system. The virtual camera arrangement state is defined by the camera's orientation and position. The orientation is represented by an orientation matrix, and the position is represented by a position vector.

[0048] Next, in step S12, the process of randomly generating P sets of candidate solutions for the projection parameters is performed as follows.

[0049] Figure 6 is a flowchart illustrating an example of this solution candidate generation process. Specifically, in step S121, the point cloud imaging processing unit 12 generates an image by projecting the training point cloud data according to the solution candidate projection parameters. Next, in step S122, the image feature extraction processing unit 13 extracts features from the image using a feature extractor composed of a trained machine learning model stored in the image feature extraction model storage unit 33. For extracting image features, a feature extractor based on edge patterns or grayscale may be used, or a feature extractor pre-trained by a deep learning method such as ResNet or Vision Transformer may be used.

[0050] Next, in step S123, the point cloud class label discrimination processing unit 14 estimates the class labels of the given point cloud data based on the extracted image features. As an estimation method, for example, statistical machine learning algorithms such as the nearest neighbor method, Support Vector Machine (SVM), or Random Forest may be used, or a method that estimates the classification result using a fully connected layer of a neural network may be used.

[0051] The estimated class labels are output from the point cloud class label identification processing unit 14 to the learning processing unit 16 in step S124.

[0052] In step S13, the learning processing unit 16 obtains a class label for the point cloud data of one of the candidate solutions for the projection parameters of group P. In step S14, it calculates the classification accuracy (identification accuracy) for the candidate solution for the projection parameters using the verification class label and stores it in the working memory area of ​​the data storage unit 3.

[0053] Then, in step S15, the learning processing unit 16 determines whether the calculation and determination of classification accuracy for all solution candidates in group P has been completed. If it has not been completed, step S16 selects the next solution candidate and executes the calculation and determination of classification accuracy in steps S13 to S15. Thereafter, the learning processing unit 16 repeatedly performs the calculation and determination of classification accuracy for all solution candidates in group P in the same manner.

[0054] On the other hand, once the calculation and determination of classification accuracy for all solution candidates in group P is complete, the learning processing unit 16 determines in step S17 whether or not the selection of all classes has been completed. If there are any unselected classes remaining, the learning processing unit 16 returns to step S11 to select the next class, and for the selected class, it performs the calculation and determination of classification accuracy for each solution candidate in group P of the projection parameters as described above in steps S12 to S17. Similarly, the learning processing unit 16 repeatedly performs the calculation and determination of classification accuracy for each solution candidate in group P for each unselected class.

[0055] Once the calculation and determination of the classification accuracy for the P sets of candidate projection parameters has been completed for all classes, the learning processing unit 16 selects the projection parameter with the highest classification accuracy for each class in step S18, and stores the selected projection parameter in the projection parameter storage unit 31, associating it with the corresponding class.

[0056] (2) Estimation Phase Once the projection parameters have been optimized during the learning phase, the control unit 1 of the point cloud data recognition device SV performs the following process to recognize class labels from the measured point cloud data.

[0057] Figure 7 is a flowchart showing an example of the processing procedure and processing content of the class label recognition process performed by the control unit 1 of the point cloud data recognition device SV.

[0058] In other words, in step S20, the control unit 1 of the point cloud data recognition device SV first acquires point cloud data measured for the target space by the point cloud data measurement device MS under the control of the point cloud data acquisition processing unit 11 via the input / output I / F unit 4. The target space includes the object to be recognized.

[0059] In step S21, the control unit 1 of the point cloud data recognition device SV then performs the following process to image the point cloud data according to the projection parameters stored in the projection parameter storage unit 31, under the control of the point cloud imaging processing unit 12.

[0060] Figure 8 is a flowchart showing an example of the processing procedure and processing content of the image processing performed by the point cloud image processing unit 12 described above.

[0061] In other words, in step S211, the point cloud imaging processing unit 12 first determines the placement state of the virtual camera in the point cloud coordinate system based on the pose matrix and position vector included in the projection parameters stored in the projection parameter storage unit 31 by class. That is, it sets the placement state of the virtual camera to an optimized state corresponding to the point cloud class. Then, the point cloud imaging processing unit 12 generates point cloud images projected by the virtual camera for each point cloud class.

[0062] The attitude matrix may be given in a different form, such as Euler angles or quaternions. The position vector may be given in the form of position coordinates in the point cloud coordinate system, or it may be given indirectly by moving the given point cloud data to the origin.

[0063] Next, in step S212, the point cloud imaging processing unit 12 reads conditions specifying the position of illumination in the target space from the imaging condition storage unit 32, and performs point cloud coloring processing on the generated point cloud image based on the read conditions. Subsequently, in step S213, the point cloud imaging processing processing unit 12 reads conditions specifying the resolution and point cloud radius from the imaging condition storage unit 32, and performs pixelation processing on the image after the coloring processing based on the read conditions. Then, in step S214, the point cloud imaging processing processing unit 12 reads conditions specifying the targets for sharpening and denoising from the imaging condition storage unit 32, and performs correction processing for sharpening and denoising on the pixelated image according to the read conditions, and outputs the corrected image to the image feature extraction processing unit 13.

[0064] In step S22, the control unit 1 of the point cloud data recognition device SV, under the control of the image feature extraction processing unit 13, inputs the point cloud images output by the point cloud imaging processing unit 12 for each point cloud class into the feature extractor, which is stored in the image feature extraction model storage unit 33 and consists of a trained machine learning model. The control unit then obtains the feature quantities of the point cloud images for each point cloud class from this feature extractor.

[0065] As mentioned in the learning phase, feature extractors based on edge patterns or grayscale may be used for extracting image features, or feature extractors pre-trained using deep learning methods such as ResNet or Vision Transformer may be used.

[0066] The control unit 1 of the point cloud data recognition device SV then, in step S23, under the control of the point cloud class label identification processing unit 14, estimates (identifies) the class label of the measured point cloud data based on the feature quantities of the point cloud image obtained for each point cloud class, and calculates the confidence level of the estimated class label.

[0067] As for the estimation method, as mentioned in the learning phase, statistical machine learning algorithms such as the nearest neighbor method, Support Vector Machine (SVM), and Random Forest may be used, or a method that estimates the classification result using the fully connected layers of a neural network may be used.

[0068] In step S24, the control unit 1 of the point cloud data recognition device SV, under the control of the class label output processing unit 15, calculates, for example, the average confidence score based on the confidence scores of the class labels of each class obtained by the point cloud class label identification processing unit 14, and selects the class label of the class with the highest value. The selected class label is then transmitted from the input / output I / F unit 4 to, for example, the system administrator's or user's terminal device US.

[0069] Figure 9 is used to explain the process of estimating and outputting the class label with the highest confidence level from the point cloud data described above.

[0070] This example demonstrates the process of acquiring point cloud data of a target space containing an object called "chair" and recognizing its class label. Specifically, C classes, Class1 to ClassC, are defined as point cloud classes, and projection parameters R1 to Rc, corresponding to each of these C classes, are pre-optimized and set during the learning phase. Then, images are generated by projecting the point cloud data using these projection parameters R1 to Rc, and features are extracted from each generated image. Based on these features, the class label is identified. Finally, the average confidence score Pi (i=1 to C) of each identified class label is calculated. Pi = 1 / C(Pi R1 +Pi R2 +…+Pi Rc ) Each is calculated using the following method, and the argmaxPi that has the largest average confidence score Pi of the calculated class label is selected and output.

[0071] (effect) As described above, in the first embodiment, during the learning phase, multiple training point cloud data and corresponding validation class labels are used to set optimized projection parameters for each class so as to minimize evaluation metrics such as cross-entropy loss, and these parameters are stored in the projection parameter storage unit 31. Then, during the estimation phase, point cloud data measured for the target space is acquired, and images are generated by projecting the acquired point cloud data using the projection parameters set for each class. Features are extracted from each generated image, and class labels are estimated based on the extracted features. The class label with the highest confidence level among the estimated class labels is then output.

[0072] Therefore, the point cloud data is projected using projection parameters optimized for each class during the learning phase, resulting in the generation of images. In other words, the point cloud data is imaged using a virtual camera set to the optimal pose and position for each class. As a result, feature quantities that appropriately reflect the features of the point cloud are extracted from the generated images, and based on these extracted feature quantities, it becomes possible to accurately recognize the class labels of the point cloud data. Furthermore, as a result, for example, in autonomous mobile robots and drones, when recognizing objects in three-dimensional space while changing their line of sight, improved accuracy in object recognition can be expected.

[0073] [Second Embodiment] A second embodiment of this invention shows a modification of the first embodiment, in which, in the estimation phase, instead of using a predetermined initial orientation for the placement of a virtual camera, the initial orientation is estimated from the acquired point cloud data, and a projected image of the point cloud data is generated using the virtual camera whose initial orientation has been estimated.

[0074] Figure 10 is a flowchart showing an example of the processing procedure and processing content of the point cloud data recognition process performed by the control unit 1 of the point cloud data recognition device SV according to the second embodiment of the present invention during the estimation phase. In Figure 10, steps that have the same processing content as those in Figure 7 are denoted by the same reference numerals, and detailed explanations are omitted.

[0075] In step S20, the control unit 1 of the point cloud data recognition device SV acquires point cloud data of the target space using the point cloud data acquisition processing unit 11. First, in step S25, it estimates the initial pose of a virtual camera from the point cloud data.

[0076] For example, the control unit 1 of the point cloud data recognition device SV determines the initial posture for each point cloud data set using a method that sets the posture that maximizes the number of points corresponding to the projection plane of a virtual camera as the initial posture, or a method that determines the initial posture from the orientation orthogonal to the first principal component axis and the second principal component axis when the point cloud data is subjected to principal component analysis.

[0077] Then, in steps S21 to S24, the control unit 1 of the point cloud data recognition device SV performs the following processing for each class, similar to the first embodiment: imaging of the point cloud data, extraction of image features, estimation of class labels, and output of the class label with the highest class confidence level.

[0078] By doing this, the initial orientation of the virtual camera can be set to be more appropriate for each point cloud dataset, making it possible to recognize class labels with higher accuracy using a virtual camera that contains more feature data from the point cloud.

[0079] [Third Embodiment] In the first embodiment, as shown in Figure 9, projection parameters are applied in parallel to all classes of point cloud data to generate projected images, and the optimal class label is selected and output based on the class label confidence score estimated from the features of each generated image.

[0080] In contrast, the third embodiment of this invention selects one projection parameter for each class from the projection parameter storage unit 31 in a cyclic manner, converts the point cloud data into an image, determines whether the class label estimated from the feature quantities of the image matches the class label corresponding to the selected projection parameter, and outputs the class label if they match.

[0081] Figure 11 is a diagram illustrating the operation of the point cloud data recognition process performed by the control unit 1 of the point cloud data recognition device SV according to the third embodiment of this invention.

[0082] As shown in Figure 11, the control unit 1 of the point cloud data recognition device SV, under the control of the point cloud imaging processing unit 12, randomly selects one projection parameter at a time from the list of projection parameters R1, R2, ..., Rc stored in the projection parameter storage unit 31 in a cyclical manner, and generates an image by projecting the point cloud data using the selected projection parameter Rr. Subsequently, the control unit 1 of the point cloud data recognition device SV extracts features from the image using the image feature extraction processing unit 13, estimates class labels from the features using the point cloud class label identification processing unit 14 and calculates the average confidence level, and saves the calculated average confidence level in the work area of ​​the data storage unit 3 in step S31.

[0083] Next, in step S32, the control unit 1 of the point cloud data recognition device SV determines from the value of the counter whether all class projection parameters R1 to Rc have been selected. If there are any unselected projection parameters remaining, the counter value is incremented, and then the point cloud imaging processing unit 12 selects the unselected projection parameter Rr. The control unit 1 of the point cloud data recognition device SV then images the point cloud data using the selected projection parameters, estimates the class label from the features of this image, and saves its confidence level.

[0084] Thereafter, the control unit 1 of the point cloud data recognition device SV repeatedly performs the process of estimating a class label from the features of the projected image and storing its confidence level in the data storage unit 3 each time a projection parameter Rr is selected.

[0085] In step S32, the control unit 1 of the point cloud data recognition device SV determines that the selection of all projection parameters R1 to Rc has been completed. In step S33, the class label output processing unit 15 integrates the class confidence scores corresponding to all stored projection parameters. For example, it calculates the average of the class confidence scores corresponding to all projection parameters and selects the class label with the highest class confidence score based on this calculation result.

[0086] In step S34, the class label output processing unit 15 finally transmits the class label with the highest selected class confidence level from the input / output I / F unit 4 to the terminal device US.

[0087] According to the third embodiment, a series of processes from image processing of point cloud data using projection parameters to calculation of class confidence can be performed using a single processing system. This makes it possible to simplify and miniaturize the configuration of the processing system compared to processing class labels of point cloud data in parallel for multiple projection parameters.

[0088] [Fourth Embodiment] A fourth embodiment of this invention uses a visual language model, trained so that image features and language features match, as a machine learning model for extracting features from projected images. The feature distance is calculated between the image features extracted from the image projected from point cloud data using projection parameters and the text features representing the class corresponding to the projection parameters, and the class label with the closest calculated feature distance is output.

[0089] Figure 12 is a diagram illustrating the operation of the point cloud data recognition process performed by the control unit 1 of the point cloud data recognition device SV according to the fourth embodiment of this invention.

[0090] The control unit 1 of the point cloud data recognition device SV includes a visual language model, which is pre-trained so that image features and language features match, as a machine learning model used by the image feature extraction processing unit 13.

[0091] The control unit 1 of the point cloud data recognition device SV acquires point cloud data from the target space using the point cloud data acquisition processing unit 11, and also acquires text data representing the class of the point cloud data. As text data representing the class, for example, text data representing the name of the object to be recognized that exists in the target space is used. For example, word labels such as "chair" and "desk" are used, but modifiers may be added to the words, or the object may be described with a descriptive phrase such as "a photo of chair" may be used.

[0092] The control unit 1 of the point cloud data recognition device SV generates images by projecting the point cloud data using projection parameters R1 to Rr stored in the projection parameter storage unit 31, under the control of the point cloud image processing unit 12. Then, the generated images and the text data of the class corresponding to the projection parameters R1 to Rr are input to the visual language model of the image feature extraction processing unit 13. As a result, the visual language model outputs feature distances d1 to dc between the feature quantities of the image and the feature quantities of the text data for each class.

[0093] The control unit 1 of the point cloud data recognition device SV calculates the average of the above feature distances d1 to dc for each class in the class label output processing unit 15, and selects the feature distance argmindi (i=1 to c) with the smallest calculated average value. Then, the class label corresponding to the selected feature distance argmindi is transmitted from the input / output I / F unit 4 to the terminal device US as the class label with the highest class confidence.

[0094] According to the fourth embodiment, the optimal class label is estimated based on the feature distance di between the image features of the point cloud data and the features of the text data representing the class name. Therefore, it is possible to recognize the class label with higher accuracy compared to when the class label is recognized based solely on the image features.

[0095] [Other embodiments] In the first to fourth embodiments, the processing functions of the control unit 1 of the point cloud data recognition device SV were described as being provided on a single server computer or personal computer. However, the processing functions of the control unit 1 of the point cloud data recognition device SV may be distributed across multiple server computers or personal computers.

[0096] Furthermore, the types and configurations of machine learning models that realize each processing function of the control unit 1 of the point cloud data recognition device SV, the processing procedures and content of the image processing, image feature extraction processing, and class label identification processing, the types of objects, etc., can be modified in various ways without departing from the spirit of this invention.

[0097] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0098] In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and materialized without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. [Explanation of Symbols]

[0099] SV…Point Cloud Data Recognition Device MS... Point Cloud Data Measurement Device US…terminal device NW...Network 1…Control Unit 2…Program memory 3…Data storage unit 4…Input / Output I / F section 5... Bus 11…Point cloud data acquisition processing unit 12…Point cloud image processing unit 13…Image Feature Extraction Processing Unit 14…Point cloud class label identification processing unit 15...Class label output processing unit 16…Learning Processing Unit 31...Projection parameter storage unit 32…Image conversion condition storage unit 33…Image Feature Extraction Model Memory Unit

Claims

1. A first processing unit that acquires point cloud data of the target space, A second processing unit that maintains a plurality of projection parameters set for each class of point cloud data and generates images projected from the point cloud data according to the plurality of projection parameters, A third processing unit that extracts feature quantities from each of the aforementioned images, A fourth processing unit recognizes the class of the imaged point cloud data based on each of the aforementioned features. A point cloud data recognition device equipped with the following features.

2. The point cloud data recognition device according to claim 1, further comprising a fifth processing unit that optimizes a plurality of projection parameters based on a plurality of training point cloud data prepared in correspondence with the target space and a verification class set in correspondence with the training point cloud data.

3. A point cloud data recognition method performed by an information processing device, The process of acquiring point cloud data of the target space, A process of maintaining a plurality of projection parameters set for each class of point cloud data, and generating images by projecting the point cloud data according to the plurality of projection parameters, The process of extracting features from each of the aforementioned images, A process for recognizing the class of the imaged point cloud data based on each of the aforementioned features, A point cloud data recognition method comprising the following features.

4. A program that causes a processor in a point cloud data recognition device to perform the processing performed by each processing unit in the point cloud data recognition device described in claim 1 or 2.

Images