Point cloud classification method, device, electronic device and storage medium

Abstract

The present invention provides a point cloud classification method, device, electronic device and storage medium; the method comprises: determining the geometric mapping relationship between the target point cloud and the pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud; generating a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud; inputting the image and the point cloud feature image into a trained neural network for classification to obtain the classification result of the image; determining the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud classification, and in particular to a point cloud classification method, device, electronic equipment and storage medium. Background Art

[0002] The fusion of airborne laser point clouds and images can achieve better performance in object classification than using a single type of data. The spectral information provided by aerial remote sensing images can eliminate confusion between certain objects with similar geometric features. However, high-resolution image data increases the intra-class spectral differences and reduces the inter-class spectral similarity, which poses challenges to the extraction and classification of image information, resulting in reduced spectral statistical separability of different classes. Airborne laser point clouds lack radiation information, but they are convenient for distinguishing objects with different heights and shapes (e.g., grass, trees, roads, and tops of buildings). Therefore, the fusion of airborne laser point clouds and optical images can make up for their respective shortcomings.

[0003] Existing methods for airborne point cloud and image fusion classification usually assign image spectral information to point clouds and then classify multispectral point clouds. However, large-scale airborne point cloud data does not sample the surface of objects densely enough, often with only dozens of measurement points per square meter. Lower point density makes semantic scene interpretation more challenging because the local geometric structures of different categories may be very similar. Therefore, projecting high-density image information onto low-density point clouds loses a lot of image information and reduces the accuracy of point cloud classification. Summary of the invention

[0004] The embodiment of the present invention provides a point cloud classification method, device, electronic device and storage medium. The technical solution of the embodiment of the present invention is implemented as follows:

[0005] An embodiment of the present invention provides a point cloud classification method, including:

[0006] Determine a geometric mapping relationship between the target point cloud and pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud;

[0007] Generate a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud;

[0008] Inputting the image and the point cloud feature image into the trained neural network for classification to obtain the classification result of the image;

[0009] The classification result of the target point cloud is determined according to the classification result of the image and the geometric mapping relationship.

[0010] In the above scheme, determining the geometric mapping relationship between the target point cloud and the pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud includes:

[0011] According to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud, determining whether the point of the target point cloud is within the field of view of the image;

[0012] If the judgment result is within the field of view of the image, the geometric mapping relationship between the target point cloud and the pixels in the image is determined according to the point coordinates of the target point cloud and the internal and external orientation elements of the image.

[0013] In the above solution, generating a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud includes:

[0014] According to the geometric mapping relationship and the point cloud features of the target point cloud, a point cloud feature image that is geometrically consistent with the image is generated based on a bilinear interpolation algorithm.

[0015] In the above solution, the image includes: a multi-view image;

[0016] Determining the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship includes: determining pixels on the image at different perspectives corresponding to the target point in the target point cloud according to the mapping relationship;

[0017] When the classification results of the pixels on the image at different viewing angles corresponding to the target point are inconsistent, the classification result with the largest proportion among the classification results of the pixels on the image at different viewing angles is used as the classification result of the target point.

[0018] In the above scheme, the method further includes:

[0019] Constructing a neural network; the neural network is used for semantic segmentation of images;

[0020] Constructing a sample data set; the sample data set includes: a sample image and a sample point cloud feature image that is geometrically consistent with the sample image;

[0021] The neural network is trained using the sample data set and the loss function to obtain a trained neural network.

[0022] The embodiment of the present invention further provides a point cloud classification device, comprising:

[0023] A projection unit, used to determine a geometric mapping relationship between the target point cloud and pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud;

[0024] A generating unit, configured to generate a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud;

[0025] A classification unit, used for inputting the image and the point cloud feature image into the trained neural network for classification to obtain a classification result of the image;

[0026] A determination unit is used to determine the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship.

[0027] In the above solution, the projection unit is also used for:

[0028] According to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud, determining whether the point of the target point cloud is within the field of view of the image;

[0029] If the judgment result is within the field of view of the image, the geometric mapping relationship between the target point cloud and the pixels in the image is determined according to the point coordinates of the target point cloud and the internal and external orientation elements of the image.

[0030] In the above scheme, the generating unit is further used to generate a point cloud feature image that is geometrically consistent with the image based on a bilinear interpolation algorithm according to the geometric mapping relationship and the point cloud features of the target point cloud.

[0031] In the above scheme, the image includes: a multi-view image; the determination unit is also used to: determine the pixels on the image of different perspectives corresponding to the target point in the target point cloud according to the mapping relationship; and when the classification results of the pixels on the image of different perspectives corresponding to the target point are inconsistent, the classification result with the largest proportion in the classification results of the pixels on the image of different perspectives is used as the classification result of the target point.

[0032] In the above solution, the device further comprises: a construction unit, wherein the construction unit is used to:

[0033] Constructing a neural network; the neural network is used for semantic segmentation of images;

[0034] Constructing a sample data set; the sample data set includes: a sample image and a sample point cloud feature image that is geometrically consistent with the sample image;

[0035] The neural network is trained using the sample data set and the loss function to obtain a trained neural network.

[0036] An embodiment of the present invention further provides an electronic device, which comprises at least: a processor and a storage medium configured to store executable instructions, wherein: the processor is configured to execute the stored executable instructions, and the executable instructions are configured to execute the point cloud classification method provided in the above embodiment.

[0037] An embodiment of the present invention further provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implements the point cloud classification method provided by the embodiment of the present invention.

[0038] The embodiment of the present invention generates a point cloud feature image that is geometrically consistent with the image, and classifies the point cloud in combination with the three-dimensional geometric features of the point cloud feature image and the image features. On the one hand, by generating a point cloud feature image that is geometrically consistent with the image, the rich image features are fully utilized to improve the classification accuracy of the point cloud. On the other hand, by fusing the three-dimensional geometric features of the cloud with the image features, the classification accuracy of the point cloud is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of a point cloud classification method provided by an embodiment of the present invention;

[0040] Figure 2 It is a flow chart of a method for fusion classification of aerial multi-view images and airborne laser point clouds provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of the structure of a point cloud classification device provided by an embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0044] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0047] The point cloud classification method provided by the embodiment of the present invention is described below. Figure 1 , Figure 1 is a flow chart of a point cloud classification method provided by an embodiment of the present invention; in some embodiments, the point cloud classification method can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The point cloud classification method provided by an embodiment of the present invention includes:

[0048] Step S110: determining a geometric mapping relationship between the target point cloud and pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud;

[0049] Step S120: generating a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud;

[0050] Step S130: inputting the image and the point cloud feature image into the trained neural network for classification to obtain the classification result of the image;

[0051] Step S140: Determine the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship.

[0052] In one embodiment, the image corresponding to the target point cloud at least includes an image of an object or scene corresponding to the target point cloud.

[0053] In one embodiment, the image includes: a high-resolution aerial image. The target point cloud is point cloud data to be classified. The target point cloud includes an airborne laser point cloud. The image can be obtained by drone photogrammetry.

[0054] In one embodiment, the image and the target point cloud are data of two different imaging mechanisms of the same object or scene (ie, aerial image and drone-borne laser point cloud).

[0055] In one embodiment, the internal and external elements of the image include: internal orientation elements of the image and external orientation elements of the image, wherein the internal orientation elements include parameters of the image node of the camera lens relative to the image position, and the internal orientation elements are generally considered to be known and can be obtained by identifying the camera. The external orientation elements include parameters of the spatial position and posture of the image or the main beam of photography at the moment of photography. The external orientation elements can be calculated by methods such as the beam method.

[0056] In one embodiment, the point coordinates of the target point cloud are the position of each spatial point in the target point cloud.

[0057] In one embodiment, determining the geometric mapping relationship between the target point cloud and the pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud includes: determining the pixel coordinate values ​​of the target point cloud projected on the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud. Specifically, the pixel coordinate values ​​of the point projection of the target point cloud on the image can be calculated using the collinear equation according to the internal and external orientation elements of the image and the point coordinates of the target point cloud.

[0058] In one embodiment, the method further includes: extracting point cloud features of the target point cloud. Specifically, first determine the best area for point cloud feature extraction, and calculate the structure tensor for each point according to the best neighborhood of each point, namely, isotropy O, anisotropy A, flatness P, sphericity S, and characteristic entropy E.

[0059] In addition, the normalized height NH of the point, the intensity value I of the point cloud, and the Z coordinate of the normal are used as features to determine the tilt of the ground. Finally, the feature vector F used by the point cloud is:

[0060] F=[O,A,P,S,E,NH,I,Z]

[0061] In one embodiment, the pixel coordinate values ​​of the target point cloud projected on the image correspond to the point coordinates of the target point cloud one by one, thereby determining the pixel corresponding to the target point cloud on the image.

[0062] In one embodiment, based on the geometric mapping relationship between the target point cloud and the image pixels (including: the pixel coordinate values ​​of the determined target point cloud projected on the image), the points of the target point cloud corresponding to the image pixels can be known. After the classification result of the image is determined, the classification result of the pixels on the image is projected back to the point cloud to obtain the classification result of the point cloud.

[0063] In this embodiment, a point cloud feature image that is consistent with the image geometry is generated, and the point cloud is classified in combination with the three-dimensional geometric features of the point cloud feature image and the image features. On the one hand, by generating a point cloud feature image that is consistent with the image geometry, the rich image features are fully utilized to improve the classification accuracy of the point cloud. On the other hand, by fusing the three-dimensional geometric features and image features of the cloud, the classification accuracy of the point cloud is further improved.

[0064] In one embodiment, the method further comprises:

[0065] Constructing a neural network; the neural network is used for semantic segmentation of images;

[0066] Constructing a sample data set; the sample data set includes: a sample image and a sample point cloud feature image that is geometrically consistent with the sample image;

[0067] The neural network is trained using the sample data set and the loss function to obtain a trained neural network.

[0068] In one embodiment, the neural network includes but is not limited to a residual network. The neural network may include: an input layer, a feature extraction layer, and a classification output layer. The input layer of the neural network may be an 11-band input, of which 3 bands are 3 spectral bands of the remote sensing image and 8 bands are bands of the point cloud feature image. The neural network fuses the features of the remote sensing image and the point cloud feature image through the feature extraction layer, and uses the fused features to classify the final remote sensing image, thereby improving the classification accuracy.

[0069] In one embodiment, the neural network may use the Resnet-101 network as the network framework of the neural network of this embodiment.

[0070] In one embodiment, the sample image may be a remote sensing image, and the remote sensing image and the point cloud feature image are preprocessed before being input into the neural network, that is, the remote sensing image and the point cloud feature image are sliced ​​into fixed-size rectangles according to spatial consistency, and the slice image size is 256×256. Since zero-filling the edges during the convolution and pooling process will cause distortion of edge information, the classification accuracy of the neural network for the image edge is often lower than that of the image interior. To address this problem, when slicing the image, it is cut at an interval of 64 pixels both horizontally and vertically, that is, 1 / 4 of the width of the slice image, to ensure that the image edge can appear inside another input image, thereby improving the accuracy of the neural network semantic segmentation.

[0071] The neural network is trained using a sample data set and a loss function, specifically including: inputting 3-band remote sensing images and 8-band point cloud feature images in the sample data set into the neural network to obtain output results; calculating the loss of the model based on the output results and the real labeled images; calculating the correction value of each layer parameter of the network through the back propagation algorithm based on the model loss; and repeating the above steps until the model loss is less than a given threshold, thereby obtaining a trained neural network.

[0072] In one embodiment, the classification result of the image includes: the classification result of each pixel on the image. The classification result of the target point cloud includes: the classification result of each point in the target point cloud, that is, the category of each point in the target point cloud.

[0073] In one embodiment, the classification result of the target point cloud is determined according to the classification result of the image and the geometric mapping relationship, including: determining the pixels in the image corresponding to each point in the target point cloud based on the pixel coordinates of the target point cloud projected on the image; and obtaining the classification results of the points of the target point cloud corresponding to the pixels according to the classification results of the pixels of the image.

[0074] In some embodiments, determining the geometric mapping relationship between the target point cloud and pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud includes:

[0075] According to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud, determining whether the point of the target point cloud is within the field of view of the image;

[0076] If the judgment result is within the field of view of the image, the geometric mapping relationship between the target point cloud and the pixels in the image is determined according to the point coordinates of the target point cloud and the internal and external orientation elements of the image.

[0077] In one embodiment, if a point in the target point cloud is within the field of view of the image, the pixel coordinates of the point in the target point cloud projected on the image are determined based on the point coordinates of the target point cloud and the internal and external orientation elements of the image (determining the geometric mapping relationship between the point in the target point cloud and the image pixels).

[0078] In this embodiment, the point features of the target point cloud are mapped to the image only when the points in the target point cloud are within the field of view of the image, thereby improving the accuracy of the generated point cloud feature image.

[0079] In some embodiments, generating a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud includes:

[0080] According to the geometric mapping relationship and the point cloud features of the target point cloud, a point cloud feature image that is geometrically consistent with the image is generated based on a bilinear interpolation algorithm. When the image is a multi-view image, each image of each viewing angle will generate a point cloud feature map that is consistent with it.

[0081] In some embodiments, the image includes: a multi-view image;

[0082] Determining the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship includes: determining pixels on the image at different perspectives corresponding to the target point in the target point cloud according to the mapping relationship;

[0083] When the classification results of the pixels on the image at different viewing angles corresponding to the target point are inconsistent, the classification result with the largest proportion among the classification results of the pixels on the image at different viewing angles is used as the classification result of the target point.

[0084] The target point is any point in the target point cloud.

[0085] In this embodiment, since a point in the target point cloud may appear in images from multiple perspectives, that is, a point may obtain multiple classification results, if the classification results of a point in multiple images are not completely consistent, the category with the most classification results is selected as the final category of the point.

[0086] In one embodiment, the point cloud classification method provided in this embodiment can be applied to application scenarios such as tree species identification and building outline extraction.

[0087] A specific example is provided below in combination with the above embodiments:

[0088] like Figure 2 As shown in the figure, this example provides a classification method for the fusion of aerial multi-view images and airborne laser point clouds, including:

[0089] Step S210: Determine the optimal neighborhood for point cloud feature extraction.

[0090] The scale selection method based on feature entropy is used to select the best neighborhood of features for each point in the point cloud, that is, the optimal number of neighboring points is selected for each point according to the Euclidean distance in three-dimensional space.

[0091] To extract point cloud features, we need to first determine the range of the neighborhood for feature extraction. This example uses a scale selection method based on feature entropy to select the best neighborhood for each point in the point cloud, that is, to select the optimal number of neighboring points K for each point based on the Euclidean distance in three-dimensional space. Then, the three-dimensional structure tensor and its eigenvalues ​​are calculated based on each point and its corresponding K nearest neighboring points. The point and its corresponding K nearest neighboring points are put together, and the coordinates of each point are marked as P i , i = 0, 1, ..., K, where P0 represents the original point, and the mean of the coordinates of all points is calculated:

[0092]

[0093] Construct the covariance matrix from the 3D coordinates of all points:

[0094]

[0095] The eigenvalue of Cov is calculated and recorded as λ j ,j=0,1,2. Let the characteristic entropy be E(K), K is the number of nearest neighbors, then the optimal calculation formula for K is:

[0096]

[0097] Where N represents [K min ,K max ] is an integer in ].

[0098] Step S220: extracting point cloud features of the airborne laser point cloud.

[0099] Based on the best neighborhood of each point, we calculate the structure tensor for each point, namely isotropy O, anisotropy A, flatness P, sphericity S, and characteristic entropy E:

[0100] O=(λ1λ2λ3) 1 / 3

[0101] A=(λ1-λ3) / λ1

[0102] P=(λ2-λ3) / λ1

[0103] S=1-A

[0104] E=-(λ1logλ1+λ2logλ2+λ3logλ3)

[0105] In addition, the normalized height NH of the point, the intensity value I of the point cloud, and the Z coordinate of the normal are used as features to determine the degree of inclination of the ground. Finally, the feature vector F used by the point cloud is:

[0106] F=[O,A,P,S,E,NH,I,Z]

[0107] Step S230: Generate point cloud feature image.

[0108] Based on the internal and external orientation elements of the aerial multi-view image and the point coordinates of the point cloud, the collinear equation can be used to calculate whether the point is within the field of view of the image and the pixel coordinate value projected on the image. After projecting all point cloud features onto the multi-view image, bilinear interpolation is used to generate a point cloud feature image that is consistent with the image geometry. Each image from each perspective will generate a consistent point cloud feature map.

[0109] Step S240: Establish a training data set.

[0110] The multi-view image (in this example, the multi-view image can be a remote sensing image) and the point cloud feature image are sliced ​​into fixed-size rectangles according to spatial consistency, and the slice image size is 256×256. A training dataset is established based on rectangular slices. Since zero-filling the edges during convolution and pooling will cause edge information distortion, the classification accuracy of the semantic segmentation network for image edges is often lower than that for the interior of the image. To address this problem, when slicing the image, it is cut at an interval of 64 pixels both horizontally and vertically, that is, 1 / 4 of the width of the slice image, to ensure that the image edge can appear inside another input image, thereby improving the accuracy of semantic segmentation.

[0111] Step S250: Construct and train a convolutional neural network

[0112] This example uses the Resnet-101 network as the basic network framework for remote sensing image semantic segmentation. The difference is that the Resnet-101 network inputs 3-band visible light images and does not consider point cloud feature images. To address this problem, this example modifies the Resnet-101 network input layer to 11-band input, of which 3 bands are the 3 spectral bands of remote sensing images (multi-view images) and 8 bands are the bands of point cloud feature images.

[0113] After the network is built, the training data is used to train the network. Specifically, the following steps are used: inputting the spectral images (multi-view images) of 3 bands and the point cloud feature images of 8 bands in the training samples into the neural network to obtain the output results; calculating the model loss based on the output results and the real annotated images; calculating the correction value of each layer parameter of the network through the back propagation algorithm based on the model loss; and repeating the above steps until the model loss is less than the given threshold, the trained neural network (i.e., classification model) can be obtained.

[0114] Step S260: Generate airborne laser point cloud classification results.

[0115] First, the neural network is used to obtain the classification results of the multi-view image, and then the point cloud classification results are obtained based on the classification results of the multi-view image. In step 230, the correspondence between the points in the point cloud and the pixels of the multi-view image has been determined. The classification results of the pixels on the multi-view image are projected back to the point cloud to obtain the classification results of the point cloud. Since a point may appear in images from multiple perspectives, that is, a point may obtain multiple classification results, if the classification results of a point in multiple images are not completely consistent, the category with the most classification results is selected as the final category of the point.

[0116] Existing methods for airborne point cloud and image fusion classification usually assign the spectral information of high-resolution images to low-density airborne point clouds, and then classify the multispectral point clouds, which results in a large loss of image information. To address this problem, this example generates point cloud feature images that are consistent with the image geometry based on the geometric mapping relationship between the point cloud and the image, making full use of the rich image features to improve classification accuracy; and jointly trains the point cloud feature images with the spectral images through neural network, fusing the three-dimensional geometric features of the point cloud with the image features, further improving the point cloud classification accuracy.

[0117] The following is a description of the point cloud classification device provided by the embodiment of the present invention. In some embodiments, the point cloud classification device can be implemented in the form of a software module. Figure 3 , Figure 3 is a schematic diagram of the structure of a point cloud classification device provided in an embodiment of the present invention. The point cloud classification device 300 provided in an embodiment of the present invention includes:

[0118] The projection unit 310 is used to determine the geometric mapping relationship between the target point cloud and the pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud;

[0119] A generating unit 320, configured to generate a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud;

[0120] A classification unit 330, configured to input the image and the point cloud feature image into a trained neural network for classification to obtain a classification result of the image;

[0121] The determination unit 340 is used to determine the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship.

[0122] In some embodiments, the projection unit is further configured to:

[0123] According to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud, determining whether the point of the target point cloud is within the field of view of the image;

[0124] If the judgment result is within the field of view of the image, the geometric mapping relationship between the target point cloud and the pixels in the image is determined according to the point coordinates of the target point cloud and the internal and external orientation elements of the image.

[0125] In some embodiments, the generating unit is further used to generate a point cloud feature image that is geometrically consistent with the image based on a bilinear interpolation algorithm according to the geometric mapping relationship and the point cloud features of the target point cloud.

[0126] In some embodiments, the image includes: a multi-view image; the determination unit is also used to: determine the pixels on the image of different perspectives corresponding to the target point in the target point cloud according to the mapping relationship; and when the classification results of the pixels on the image of different perspectives corresponding to the target point are inconsistent, the classification result with the largest proportion in the classification results of the pixels on the images of different perspectives is used as the classification result of the target point.

[0127] In some embodiments, the device further comprises: a construction unit, wherein the construction unit is configured to:

[0128] Constructing a neural network; the neural network is used for semantic segmentation of images;

[0129] Constructing a sample data set; the sample data set includes: a sample image and a sample point cloud feature image that is geometrically consistent with the sample image;

[0130] The neural network is trained using the sample data set and the loss function to obtain a trained neural network.

[0131] An embodiment of the present invention further provides an electronic device, the device comprising at least: a processor and a storage medium configured to store executable instructions, wherein:

[0132] The processor is configured to execute stored executable instructions, and the executable instructions are configured to execute the point cloud classification method provided by an embodiment of the present invention.

[0133] It should be noted that Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device 400 at least includes: a processor 410, a communication interface 420 and a memory 430, wherein:

[0134] Processor 410 generally controls the overall operation of device 400 .

[0135] The communication interface 420 may enable the device to communicate with other devices over a network.

[0136] The memory 430 is configured to store instructions and applications executable by the processor 410, and can also cache data to be processed or processed by the processor 410 and various modules in the device 400 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0137] It should be noted that in the embodiments of the present application, if the above-mentioned point cloud classification method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a server to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0138] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the point cloud classification method provided in the above embodiment are implemented.

[0139] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0140] Of course, the device in the embodiment of the present application may also have other similar protocol interaction implementation cases. Without departing from the spirit and essence of the present application, technicians in this field can make various corresponding changes and deformations according to the embodiment of the present application, but these corresponding changes and deformations should all fall within the scope of protection of the claims attached to the method of the present application.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or modules can be electrical, mechanical or other forms.

[0148] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network modules; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0149] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A point cloud classification method, characterized in that: include: Determine a geometric mapping relationship between the target point cloud and pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud; Generate a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud; the image includes a multi-view image, and the point cloud feature image includes a point cloud feature map corresponding to the image of each viewing angle; Inputting the image and the point cloud feature image into the trained neural network for classification to obtain the classification result of the image; Determining the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship; The point cloud features include a point cloud feature vector for each point; the point cloud feature vector is based on the best neighborhood of the point and is composed of a structure tensor, a normalized height of the point, an intensity value of the point cloud, and a Z coordinate of a normal; The optimal neighborhood is determined based on a scale selection method of feature entropy, including: Put the point and its corresponding K nearest neighbor points together, and mark the coordinates of each point as P i , i = 0, 1, ..., K, where P0 represents the original point, and the mean of the coordinates of all points is calculated: Construct the covariance matrix from the 3D coordinates of all points: The eigenvalue of Cov is calculated and recorded as λ j , j = 0, 1, 2, let the characteristic entropy be recorded as E(K), K is the number of nearest neighbors, then the optimal calculation formula for K is: Where N represents [K min ,K max ] is an integer in ].

2. The method according to claim 1, characterized in that: Determining the geometric mapping relationship between the target point cloud and pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud includes: According to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud, determining whether the point of the target point cloud is within the field of view of the image; If the judgment result is within the field of view of the image, the geometric mapping relationship between the target point cloud and the pixels in the image is determined according to the point coordinates of the target point cloud and the internal and external orientation elements of the image.

3. The method according to claim 1, characterized in that: The step of generating a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud includes: According to the geometric mapping relationship and the point cloud features of the target point cloud, a point cloud feature image that is geometrically consistent with the image is generated based on a bilinear interpolation algorithm.

4. The method according to claim 1, characterized in that: Determining the classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship includes: determining pixels on the image at different perspectives corresponding to the target point in the target point cloud according to the mapping relationship; When the classification results of the pixels on the image at different viewing angles corresponding to the target point are inconsistent, the classification result with the largest proportion among the classification results of the pixels on the image at different viewing angles is used as the classification result of the target point.

5. The method according to claim 1, characterized in that The method further comprises: Constructing a neural network; the neural network is used for semantic segmentation of images; Constructing a sample data set; the sample data set includes: a sample image and a sample point cloud feature image that is geometrically consistent with the sample image; The neural network is trained using the sample data set and the loss function to obtain a trained neural network.

6. A point cloud classification device, characterized in that: include: A projection unit, used to determine a geometric mapping relationship between the target point cloud and pixels in the image according to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud; A generating unit, configured to generate a point cloud feature image that is geometrically consistent with the image according to the geometric mapping relationship and the point cloud features of the target point cloud; the image includes a multi-view image, and the point cloud feature image includes a point cloud feature map corresponding to the image of each viewing angle; A classification unit, used for inputting the image and the point cloud feature image into the trained neural network for classification to obtain a classification result of the image; A determination unit, configured to determine a classification result of the target point cloud according to the classification result of the image and the geometric mapping relationship; The point cloud features include a point cloud feature vector for each point; the point cloud feature vector is based on the best neighborhood of the point and is composed of a structure tensor, a normalized height of the point, an intensity value of the point cloud, and a Z coordinate of a normal; The optimal neighborhood is determined based on a scale selection method of feature entropy, including: Put the point and its corresponding K nearest neighbor points together, and mark the coordinates of each point as P i , i = 0, 1, ..., K, where P0 represents the original point, and the mean of the coordinates of all points is calculated: Construct the covariance matrix from the 3D coordinates of all points: The eigenvalue of Cov is calculated and recorded as λ j , j = 0, 1, 2, let the characteristic entropy be recorded as E(K), K is the number of nearest neighbors, then the optimal calculation formula for K is: Where N represents [K min ,K max ] is an integer in ].

7. The device according to claim 6, characterized in that The projection unit is also used for: According to the point coordinates of the target point cloud and the internal and external orientation elements of the image corresponding to the target point cloud, determining whether the point of the target point cloud is within the field of view of the image; If the judgment result is within the field of view of the image, the geometric mapping relationship between the target point cloud and the pixels in the image is determined according to the point coordinates of the target point cloud and the internal and external orientation elements of the image.

8. The device according to claim 6, characterized in that The generating unit is further configured to generate a point cloud feature image that is geometrically consistent with the image based on a bilinear interpolation algorithm according to the geometric mapping relationship and the point cloud features of the target point cloud.

9. An electronic device, characterized in that: The device comprises at least: a processor and a storage medium configured to store executable instructions, wherein: The processor is configured to execute stored executable instructions, and the executable instructions are configured to execute the point cloud classification method provided in any one of claims 1 to 5 above.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are configured to execute the point cloud classification method provided in any one of claims 1 to 5 above.

Images