A 3D point cloud data zero-sample classification method, device, equipment and storage medium

Abstract

The present invention discloses a method, device, equipment and storage medium for zero-sample classification of 3D point cloud data, the method comprising: obtaining 3D point cloud data to be classified, determining an initial projection direction; projecting the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data; obtaining at least two categories of semantic description information, combining with a trained hidden layer diffusion model, determining the minimum predicted disturbance noise error of the 2D point cloud data for each category of semantic description information; taking the category corresponding to the minimum value of each minimum predicted disturbance noise error as the zero-sample classification result. The zero-sample classification method for 3D point cloud data provided by the present invention uses a hidden layer diffusion model to predict the minimum predicted disturbance noise error of the projected 2D point cloud data for each category of semantic description information, thereby accurately finding the category corresponding to the semantic information that best matches the image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a 3D point cloud data zero-sample classification method, device, equipment and storage medium. Background Art

[0002] Traditional deep learning has achieved outstanding results in computer vision tasks, but its performance is highly dependent on the amount of training data. In order to simulate the human cognitive process of constructing new categories with only limited descriptive information in a situation without any samples, the zero-shot learning task was proposed, requiring the model to recognize samples from unseen categories by utilizing auxiliary information different from the sample modality.

[0003] In recent years, some large-scale 3D point cloud datasets, such as ModelNet, ShapeNet, and ScanNet, have been released for 3D object classification and scene understanding tasks. These datasets provide researchers with benchmarks for evaluating algorithm performance and comparing different methods.

[0004] Existing 3D zero-shot classification methods include methods that map 3D point cloud sample features to a unified target space for matching, and methods that match 2D images obtained by projecting 3D point clouds with text features of unseen categories in the same feature space. However, these methods need to draw on large pre-trained models to introduce prior knowledge of other modalities. The former has low classification accuracy on 3D point cloud datasets due to modal differences between text features and 3D sample features, even though they are mapped to a unified target space; the latter has the problem of projection angle selection. Existing methods all project at a pre-set projection angle. However, for uncalibrated point clouds, this method will cause the effective information of the projected subject for classification to be blocked in the 2D image, affecting the final classification performance. Summary of the invention

[0005] The present invention provides a 3D point cloud data zero-sample classification method, device, equipment and storage medium to achieve accurate classification of 3D point cloud data through zero-sample learning.

[0006] According to one aspect of the present invention, a zero-sample classification method for 3D point cloud data is provided, comprising:

[0007] Obtain the 3D point cloud data to be classified and determine the initial projection direction;

[0008] Projecting the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data;

[0009] Acquire semantic description information of at least two categories, and determine the minimum prediction disturbance noise error of the 2D point cloud data for each category of semantic description information in combination with the trained hidden layer diffusion model;

[0010] The category corresponding to the minimum value of each of the minimum prediction disturbance noise errors is taken as the zero sample classification result.

[0011] Furthermore, determining the initial projection direction includes:

[0012] Determine a center point in the 3D point cloud data, and establish a three-dimensional coordinate system with the center point as the origin;

[0013] Determine a projection viewpoint according to the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system;

[0014] The direction of the viewpoint is used as the projection rotation center axis, and the direction with a rotation angle of zero is used as the initial projection direction.

[0015] Further, determining the viewpoint of the projection according to the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system includes:

[0016] Analyze the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system according to the principal component analysis method to obtain the coordinate distribution variance;

[0017] The coordinate distribution variances are sorted in descending order, and the position of the viewpoint is determined in the direction corresponding to the minimum value of the coordinate distribution variances.

[0018] Furthermore, the hidden layer diffusion model includes an encoder and a diffusion model, and the minimum prediction noise error of the 2D point cloud data for each category of semantic description information is determined in combination with the trained hidden layer diffusion model, including:

[0019] Inputting the 2D point cloud data into the encoder to obtain hidden layer features corresponding to the 2D point cloud data;

[0020] Adding random noise to the hidden layer features to obtain noise features;

[0021] The noise characteristics and the semantic description information of each category are respectively input into the diffusion model at the same time to obtain the predicted disturbance noise and the predicted disturbance noise error corresponding to each category;

[0022] For the predicted disturbance noise error corresponding to each category, the partial derivative of the projection direction is determined respectively, the projection direction is adjusted and the partial derivative of the predicted disturbance noise error with respect to the projection direction is re-determined until the partial derivative is less than the set threshold or the number of iterations is reached, the projection direction corresponding to the last iteration is taken as the target projection direction, and the predicted disturbance noise error corresponding to the target projection direction is determined as the minimum predicted noise error.

[0023] Further, adjusting the projection direction includes:

[0024] The difference between the projection direction and the partial derivative in the previous iteration is determined as the projection direction of the next iteration.

[0025] Furthermore, the 2D point cloud data includes at least one of the following: a rendered image obtained by projecting and rendering the 3D point cloud data, a depth image obtained according to a distance between a camera and the 3D point cloud data, and an edge image obtained by extracting edges from the rendered image.

[0026] Furthermore, the 2D point cloud data is input into the encoder to obtain hidden layer features corresponding to the 2D point cloud data, including:

[0027] Inputting at least two of the rendered image, the depth image, and the edge image into the encoder respectively to obtain at least two single features;

[0028] The at least two single features are fused, and the fusion result is used as the hidden layer feature.

[0029] According to another aspect of the present invention, a 3D point cloud data zero-sample classification device is provided, comprising:

[0030] An initial projection direction determination module is used to obtain 3D point cloud data to be classified and determine the initial projection direction;

[0031] A projection module, used for projecting the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data;

[0032] A minimum prediction disturbance noise error determination module is used to obtain at least two categories of semantic description information, and determine the minimum prediction disturbance noise error of the 2D point cloud data for each category of semantic description information in combination with the trained hidden layer diffusion model;

[0033] The zero-sample classification result determination module is used to take the category corresponding to the minimum value of each of the minimum predicted disturbance noise errors as the zero-sample classification result.

[0034] Optionally, the initial projection direction determination module is further used for:

[0035] Determine a center point in the 3D point cloud data, and establish a three-dimensional coordinate system with the center point as the origin;

[0036] Determine a projection viewpoint according to the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system;

[0037] The direction of the viewpoint is used as the projection rotation center axis, and the direction with a rotation angle of zero is used as the initial projection direction.

[0038] Optionally, the initial projection direction determination module is further used for:

[0039] Analyze the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system according to the principal component analysis method to obtain the coordinate distribution variance;

[0040] The coordinate distribution variances are sorted in descending order, and the position of the viewpoint is determined in the direction corresponding to the minimum value of the coordinate distribution variances.

[0041] Optionally, the hidden layer diffusion model includes an encoder and a diffusion model, and the minimum prediction disturbance noise error determination module is further used for:

[0042] Inputting the 2D point cloud data into the encoder to obtain hidden layer features corresponding to the 2D point cloud data;

[0043] Adding random noise to the hidden layer features to obtain noise features;

[0044] The noise characteristics and the semantic description information of each category are respectively input into the diffusion model at the same time to obtain the predicted disturbance noise and the predicted disturbance noise error corresponding to each category;

[0045] For the predicted disturbance noise error corresponding to each category, the partial derivative of the projection direction is determined respectively, the projection direction is adjusted and the partial derivative of the predicted disturbance noise error with respect to the projection direction is re-determined until the partial derivative is less than the set threshold or the number of iterations is reached, the projection direction corresponding to the last iteration is taken as the target projection direction, and the predicted disturbance noise error corresponding to the target projection direction is determined as the minimum predicted noise error.

[0046] Optionally, the minimum predicted disturbance noise error determination module is further used for:

[0047] The difference between the projection direction and the partial derivative in the previous iteration is determined as the projection direction of the next iteration.

[0048] Optionally, the 2D point cloud data includes at least one of the following: a rendered image obtained by projecting and rendering the 3D point cloud data, a depth image obtained based on the distance between a camera and the 3D point cloud data, and an edge image obtained by extracting edges from the rendered image.

[0049] Optionally, the minimum predicted disturbance noise error determination module is further used for:

[0050] Inputting at least two of the rendered image, the depth image, and the edge image into the encoder respectively to obtain at least two single features;

[0051] The at least two single features are fused, and the fusion result is used as the hidden layer feature.

[0052] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0053] at least one processor; and

[0054] a memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the zero-sample classification method for 3D point cloud data described in any embodiment of the present invention.

[0056] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the 3D point cloud data zero-sample classification method described in any embodiment of the present invention when executed.

[0057] The zero-sample classification method for 3D point cloud data disclosed in the present invention first obtains the 3D point cloud data to be classified and determines the initial projection direction; then projects the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data; then obtains at least two categories of semantic description information, and combines the trained hidden layer diffusion model to determine the minimum predicted disturbance noise error of the 2D point cloud data for each category of semantic description information; finally, the category corresponding to the minimum value of each minimum predicted disturbance noise error is used as the zero-sample classification result. The zero-sample classification method for 3D point cloud data provided by the present invention uses a hidden layer diffusion model to predict the minimum predicted disturbance noise error of the projected 2D point cloud data for each category of semantic description information, so as to accurately find the category corresponding to the semantic information that best matches the image.

[0058] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 is a flowchart of a zero-sample classification method for 3D point cloud data provided according to Embodiment 1 of the present invention;

[0061] Figure 2 is a flowchart of a zero-sample classification method for 3D point cloud data provided according to Embodiment 2 of the present invention;

[0062] Figure 3 is a schematic diagram of a zero-sample classification process of 3D point cloud data provided according to the second embodiment of the present invention;

[0063] Figure 4 A schematic diagram of the structure of a 3D point cloud data zero-sample classification device provided according to Embodiment 3 of the present invention;

[0064] Figure 5 It is a structural schematic diagram of an electronic device for implementing the zero-sample classification method for 3D point cloud data of the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0067] Embodiment 1

[0068] Figure 1 This is a flowchart of a 3D point cloud data zero-sample classification method provided in the first embodiment of the present invention. This embodiment is applicable to the case of classifying 3D point cloud data in a zero-sample learning scenario. The method can be performed by a 3D point cloud data zero-sample classification device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0069] S110, obtaining 3D point cloud data to be classified, and determining an initial projection direction.

[0070] Among them, 3D point cloud data is one or more groups of data points in space. In order to classify the 3D point cloud data, it needs to be projected. The projection result is related to the projection direction. In order to find the optimal projection direction, the initial value of the projection direction can be determined and optimized through a set algorithm until the optimal projection direction is obtained. The initial value of the projection direction is the initial projection direction.

[0071] In this embodiment, after the 3D point cloud data to be classified is obtained, a three-dimensional coordinate system can be established, and the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system can be obtained, and the initial projection direction can be determined based on the analysis of each coordinate.

[0072] Optionally, the coordinate system may be re-established according to the discrete distribution of each coordinate in the three-dimensional coordinate system, and the direction with the largest discrete degree may be used as the initial projection direction.

[0073] S120 , projecting the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data.

[0074] In order to classify the 3D point cloud data, it needs to be projected, that is, the 3D point cloud data is converted into 2D point cloud data, and the 2D point cloud data is the result of the projection.

[0075] In this embodiment, after the initial projection direction is determined, the 3D point cloud data to be classified is projected according to the direction to obtain 2D point cloud data.

[0076] Preferably, in the process of projection, the third-party support library Pytorch3d can be used to project and render the 3D point cloud to be classified to obtain a 2D rendering image as the 2D point cloud data required in this embodiment. Among them, PyTorch3D is a modular, efficient and differentiable operator library for 3D deep learning, which includes a fast, modular, differentiable renderer for meshes and point clouds, and supports the analysis-by-synthesis method.

[0077] S130, obtaining semantic description information of at least two categories, and determining the minimum prediction disturbance noise error of the 2D point cloud data for each category of semantic description information in combination with the trained hidden layer diffusion model.

[0078] Among them, the semantic description information is the semantic information of the image used to illustrate the image content. In this embodiment, in order to classify the 3D point cloud data, it is necessary to obtain at least two categories of semantic description information. For example, the semantic description information can be "animal" and "plant". Then, based on the projected 2D point cloud data combined with the trained hidden layer diffusion model, the classification result of the 3D point cloud data to be classified can be determined, that is, it can be determined whether the image composed of this group of data points belongs to an animal or a plant.

[0079] Furthermore, the hidden layer diffusion model is a type of generative model that gradually applies noise to the image in the forward stage until the image is destroyed and becomes completely Gaussian noise, and then learns to restore the original image from the Gaussian noise in the reverse stage.

[0080] In this embodiment, the hidden layer diffusion model is used to determine the minimum prediction disturbance noise error of the 2D point cloud data for each category of semantic description information.

[0081] Specifically, the hidden layer diffusion model includes a pre-trained decoding and encoding process and a diffusion process in the hidden layer feature space. The encoder E can map each image sample p to a specific hidden layer feature space to obtain the corresponding hidden layer feature f=E(p), and the decoder D can map each hidden layer feature f back to the sample space to obtain the reconstructed corresponding image sample p'=D(f). The diffusion process is a disturbance addition and removal process for the hidden layer feature f. Among them, the forward diffusion process is a disturbance addition process, that is, the process of using the hidden layer feature as the initial feature (f0=f) and gradually increasing the disturbance noise until it conforms to the random noise distribution. The reverse diffusion process corresponds to the disturbance removal process, and its noise estimation module F approximately estimates the probability distribution q(f) of the feature of the t-1 step by inverting the feature of the t-1 step from the t step through the CUNet (Conditional UNet) architecture with the semantic description S as conditional information. t-1 |f t ,S), thereby predicting the disturbance noise estimate ε of the tth step t '=F(f t ,S,t) is used to remove disturbances.

[0082] S140, taking the category corresponding to the minimum value among the minimum prediction disturbance noise errors as the zero sample classification result.

[0083] In this embodiment, after determining the minimum predicted disturbance noise error of the 2D point cloud data for each category of semantic description information obtained, the minimum value of each minimum predicted disturbance noise error can be determined, and the category corresponding to the minimum value is the final zero-sample classification result of the 3D point cloud data.

[0084] Specifically, let the 3D point cloud data to be classified be x (n) The minimum prediction disturbance noise error of each category obtained through the above steps is expressed as MSE k (n) , where k represents the category. By comparing the minimum prediction disturbance noise errors corresponding to each category of the data sample, the category corresponding to the minimum value is obtained. The final classification result is obtained at this time. At this point, the purpose of classifying 3D point cloud samples in zero-sample scenarios is achieved.

[0085] The zero-sample classification method for 3D point cloud data disclosed in the present invention first obtains the 3D point cloud data to be classified and determines the initial projection direction; then projects the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data; then obtains at least two categories of semantic description information, and combines the trained hidden layer diffusion model to determine the minimum predicted disturbance noise error of the 2D point cloud data for each category of semantic description information; finally, the category corresponding to the minimum value of each minimum predicted disturbance noise error is used as the zero-sample classification result. The zero-sample classification method for 3D point cloud data provided by the present invention uses a hidden layer diffusion model to predict the minimum predicted disturbance noise error of the projected 2D point cloud data for each category of semantic description information, so as to accurately find the category corresponding to the semantic information that best matches the image.

[0086] Embodiment 2

[0087] Figure 2 This is a flowchart of a 3D point cloud data zero-sample classification method provided in the second embodiment of the present invention. This embodiment is a feature refinement and addition of the above embodiment. Figure 2 As shown, the method includes:

[0088] S210, obtaining 3D point cloud data to be classified, determining a center point in the 3D point cloud data, and establishing a three-dimensional coordinate system with the center point as the origin.

[0089] In this embodiment, after the 3D point cloud data to be classified is obtained, a three-dimensional coordinate system may be established first, so as to determine the initial projection direction in subsequent steps.

[0090] Specifically, the 3D point cloud data to be classified is represented as x (n) , when establishing a three-dimensional coordinate system, we can find x (n)The center point is taken as the origin, and then a three-dimensional coordinate system is established with three random mutually perpendicular directions as coordinate axes.

[0091] S220 , determining a projection viewpoint according to the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system.

[0092] Among them, projection is the transformation of geometric shapes represented by three-dimensional coordinates into graphics represented by two-dimensional coordinates, and the viewpoint is the projection center.

[0093] In this embodiment, after a three-dimensional coordinate system is established with the center point of the 3D point cloud data as the origin, an analysis can be performed based on the coordinates of each point cloud data in the three-dimensional coordinate system to determine a suitable position as a viewpoint.

[0094] Optionally, the method of determining the projection viewpoint based on the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system can be: analyzing the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system according to the principal component analysis method to obtain the coordinate distribution variance; sorting the coordinate distribution variance in order from large to small, and determining the position of the viewpoint in the direction corresponding to the minimum value in the coordinate distribution variance.

[0095] Specifically, the principal component analysis method can be used to analyze the sample x (n) The corresponding coordinates of all points in are analyzed, where principal component analysis (PCA) is a multivariate statistical method. It is one of the most commonly used dimensionality reduction methods. It converts a set of variable data that may be correlated into a set of linearly unrelated variables through orthogonal transformation. The converted variables are called principal components. Through the principal component analysis method, the covariance matrix can be calculated based on the coordinate data of each point, and then the covariance matrix can be decomposed by singular values ​​to obtain three mutually perpendicular direction vectors. At the same time, the variance corresponding to each vector can also be obtained, and the viewpoint is fixed in the direction corresponding to the vector with the smallest variance.

[0096] For example, the three mutually perpendicular directions obtained by the principal component analysis method are [0.75, -0.27, -0.60], [0.65, 0.15, 0.75], and [-0.11, -0.95, 0.28], and the corresponding variances are 0.32, 0.06, and 0.05, respectively. The direction with the smallest variance is [-0.11, -0.95, 0.28]. The viewpoint can be fixed in this direction, and the distance from the viewpoint to the origin is set to 2 to determine the position of the viewpoint.

[0097] S230: Taking the direction of the viewpoint as the projection rotation center axis, and taking the direction of zero rotation angle as the initial projection direction.

[0098] The direction of the viewpoint is the direction in which the variance of the viewpoint is the smallest in the above step.

[0099] In this embodiment, the direction of the viewpoint can be fixed as the projection rotation center axis, and the initial rotation angle θ (n) = 0, thereby obtaining a fixed initial rotation angle with this direction as the central axis, and obtaining a rotation angle θ based on the viewpoint position and the corresponding rotation angle θ (n) =0 initial projection direction.

[0100] S240 , projecting the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data.

[0101] In this embodiment, after the initial projection direction is determined, the 3D point cloud data to be classified is projected according to the direction to obtain 2D point cloud data, which is the result of the projection.

[0102] S250, inputting the 2D point cloud data into the encoder to obtain hidden layer features corresponding to the 2D point cloud data.

[0103] The hidden layer diffusion model includes an encoder and a diffusion model.

[0104] Specifically, the hidden diffusion model is a hidden diffusion model pre-trained using the LAION-2B data set based on the stable diffusion model, which includes an encoder E, a decoder D and a diffusion model CUNet. Preferably, in this embodiment, only the encoder E and the diffusion model CUNet are used, wherein the encoder E is used to extract the hidden layer features of the projection image, the input projection image is represented as a width and height channel number of 512*512*3, and the output is a hidden layer feature represented as a width and height channel number of 64*64*4; the input of the diffusion model is the hidden layer feature plus the sampled disturbance noise and the category semantic description, and the output is the predicted disturbance noise, and the input and output features are represented as a width and height channel number of 64*64*4.

[0105] Furthermore, let the 3D point cloud data to be classified be x (n) , the projected 2D point cloud data is p (n) , p (n) Input the encoder E of the pre-trained hidden layer diffusion model to obtain the corresponding hidden layer feature f (n) =E(p (n) ).

[0106] Furthermore, the 2D point cloud data includes at least one of the following: a rendered image obtained by projecting and rendering the 3D point cloud data, a depth image obtained according to a distance between a camera and the 3D point cloud data, and an edge image obtained by extracting edges from the rendered image.

[0107] Specifically, according to the obtained camera viewpoint and the corresponding projection direction, the third-party support library Pytorch3d is used to perform projection rendering on the classified 3D point cloud data to obtain the rendered image p r (n) At the same time, the distance from the camera to the object mesh surface can also be used to calculate the corresponding depth image p d (n) ; In addition, the edge image p of the rendered image can be extracted by the edge detection algorithm Candy e (n) .

[0108] Correspondingly, the 2D point cloud data is input into the encoder to obtain the hidden features corresponding to the 2D point cloud data by: inputting at least two of the rendered image, the depth image and the edge image into the encoder respectively to obtain at least two single features; fusing the at least two single features and using the fusion result as the hidden features.

[0109] Specifically, the 2D point cloud data can be a rendered image p r (n) , depth image p d (n) and edge image p e (n) When at least two of the rendered image, the depth image, and the edge image are obtained, they can be fused by inputting them into the encoder to obtain single features, and the obtained fused features are used as hidden layer features for the next step of processing to improve the accuracy of classification.

[0110] S260, obtaining semantic description information of at least two categories, adding random noise to the hidden layer features to obtain noise features, and inputting the noise features and the semantic description information of each category into the diffusion model at the same time to obtain the predicted disturbance noise and predicted disturbance noise error corresponding to each category.

[0111] The noise feature is the feature information containing noise obtained by adding random noise to the hidden layer feature, and the predicted disturbance noise is the disturbance noise predicted by the diffusion model.

[0112] In this embodiment, the random noise can be set to ε (random noise ε obeys the standard normal distribution), and the time step t in the diffusion process is selected. Then, in the hidden layer feature f (n) The noise characteristics of the tth step obtained by adding random noise ε can be expressed as Among them, α t =1-β t , β t is the pre-set disturbance coefficient. Let the semantic description information be S k, where k represents the category, and the noise feature f t (n) And the semantic description information S corresponding to each category k In the paired input hidden layer diffusion model, the predicted disturbance noise ε corresponding to category k at time step t can be predicted (k,t) (n) =F(f t (n) , S k , t). Then, the MSE of each type of prediction disturbance noise error of the sample in the initial projection direction can be calculated: (k,0) (n) =Dif(x (n) ,θ (k,0) (n) ,ε,,S k , t) = ‖ε-ε (k,t) (n) ‖ 2 , where θ (k,0) (n) is the sample x corresponding to the kth category (n) The initial projection direction.

[0113] According to the above method, a prediction disturbance noise error in the initial projection direction can be obtained for each category. For example, assuming there are 10 categories, that is, the value of k is a positive integer between 1 and 10, and the time step t = 600, the disturbance coefficient β t =0.8392, the diffusion model CUNet can store the predicted disturbance noise ε at time t=600 (1,600) (n) , ε (2,600) (n) , …, ε (10,600) (n) , and the corresponding prediction disturbance noise error MSE (1,0) (n) , MSE (2,0) (n) , …, MSE (10,0) (n) In the simulation experiment, the predicted disturbance noise error data under the initial projection direction are {125.0, 109.1, 116.0, 109.3, 104.5, 112.9, 110.4, 109.1, 113.6, 111.8}.

[0114] S270. For the predicted disturbance noise error corresponding to each category, determine the partial derivative with respect to the projection direction respectively, adjust the projection direction and re-determine the partial derivative of the predicted disturbance noise error with respect to the projection direction until the partial derivative is less than the set threshold or the number of iterations is reached, take the projection direction corresponding to the last iteration as the target projection direction, and determine the predicted disturbance noise error corresponding to the target projection direction as the minimum predicted noise error.

[0115] In this embodiment, in order to find the optimal projection direction corresponding to each category, the partial derivative of the predicted disturbance noise error with respect to the projection direction can be calculated in an iterative manner, and then the projection direction corresponding to each category is continuously adjusted with the goal of minimizing various types of noise until the partial derivative is less than the set threshold or the number of iterations is reached, thereby finding the minimum predicted noise error for each category.

[0116] The projection direction may be adjusted by determining the difference between the projection direction and the partial derivative in the previous iteration as the projection direction for the next iteration.

[0117] Specifically, for category k, calculate the k-th prediction disturbance noise error for the projection direction θ (k,i) (n) The partial derivative of can be:

[0118]

[0119] Among them, i is the number of iterations, x (n) is the 3D point cloud data to be classified, ε is random noise, S k is the semantic description information, and t represents the time step.

[0120] With the goal of minimizing all kinds of noise errors, the projection direction corresponding to each type of semantic description can be continuously adjusted by the gradient descent method. The adjustment method of the projection direction can be expressed as:

[0121]

[0122] Among them, q is the optimization coefficient.

[0123] When the number of iterations reaches the limited number of optimization steps or the calculated partial derivative is less than the fixed threshold, the final projection direction θ corresponding to each class of the sample can be obtained. k (n) And the minimum prediction noise error MSE for each class k (n) =Dif(x (n) ,θ k (n) ,ε,S k , t). Here, the initial projection direction in the iterative calculation of each category is the initial projection direction obtained in the above steps, that is, θ k,0(n) =θ (n) =0.

[0124] S280, taking the category corresponding to the minimum value among the minimum prediction disturbance noise errors as the zero sample classification result.

[0125] In this embodiment, after determining the minimum predicted disturbance noise error of the 2D point cloud data for each category of semantic description information obtained, the minimum value of each minimum predicted disturbance noise error can be determined, and the category corresponding to the minimum value is the final zero-sample classification result of the 3D point cloud data.

[0126] Figure 3 3D point cloud data zero-sample classification process is a schematic diagram provided by an embodiment of the present invention. As shown in the figure, after determining the initial projection direction of the 3D point cloud data to be classified, the 3D point cloud data is projected to obtain 2D point cloud data, which is then input into the encoder to obtain the corresponding hidden layer features, random noise ε is added to the hidden layer features to obtain noise features, and then the noise features and the semantic description information of each category are simultaneously input into the diffusion model to obtain the predicted disturbance noise ε corresponding to each category. (k,t) (n) and predicted disturbance noise error MSE (k,0) (n) , where the predicted disturbance noise error MSE (k,0) (n) =‖ε-ε (k,t) (n) ‖ 2 Finally, the prediction disturbance noise error corresponding to each category is iteratively calculated to find the optimal projection direction and obtain the minimum prediction noise error. The category corresponding to the minimum value of each minimum prediction disturbance noise error is taken as the zero-sample classification result.

[0127] Furthermore, since the 2D point cloud data obtained by projecting the 3D point cloud data to be classified is p (n) Can include rendered images p r (n) , depth image p d (n) and edge image p e (n) , these three images can be input into the encoder to obtain hidden features, and then the prediction disturbance noise and prediction disturbance noise error corresponding to each category can be calculated respectively. In this way, the minimum prediction noise error MSE can be obtained for each category of these three images k,r (n) , MSE k,d (n) and MSE k,e (n), then the MSE of the three minimum prediction noise errors of the same category can be averaged k-average (n) , and then take the category corresponding to the minimum value of the average error of each category as the zero-sample classification result.

[0128] For example, if the average values ​​of the minimum prediction noise errors of the 10 categories of a 3D point cloud sample to be classified are {84.8, 81.2, 85.3, 90.3, 78.1, 87.3, 84.9, 93.8, 89.1, 92.2}, the minimum value 78.1 corresponds to category 5, that is, Therefore, the category prediction result of this sample in the zero-sample scenario is the fifth category.

[0129] The zero-sample classification method for 3D point cloud data disclosed in the present invention uses a hidden layer diffusion model to predict the minimum prediction disturbance noise error of the projected 2D point cloud data for each category of semantic description information, thereby accurately finding the category corresponding to the semantic information that best matches the image. In addition, this method proposes a projection direction optimization method using principal component analysis to solve the problem of selecting the projection angle of uncalibrated samples of 3D point cloud data. In addition, this method combines renderings, depth maps, and edge maps in the projection of 3D point cloud data. Figure 3 This further improves the classification accuracy.

[0130] Embodiment 3

[0131] Figure 4 This is a schematic diagram of the structure of a 3D point cloud data zero-sample classification device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes: an initial projection direction determination module 310, a projection module 320, a minimum prediction disturbance noise error determination module 330 and a zero sample classification result determination module 340.

[0132] The initial projection direction determination module 310 is used to obtain the 3D point cloud data to be classified and determine the initial projection direction.

[0133] The projection module 320 is used to project the 3D point cloud data according to an initial projection direction to obtain 2D point cloud data.

[0134] The minimum prediction disturbance noise error determination module 330 is used to obtain at least two categories of semantic description information, and determine the minimum prediction disturbance noise error of the 2D point cloud data for each category of semantic description information in combination with the trained hidden layer diffusion model.

[0135] The zero-sample classification result determination module 340 is used to take the category corresponding to the minimum value of each minimum prediction disturbance noise error as the zero-sample classification result.

[0136] Optionally, the initial projection direction determining module 310 is further used for:

[0137] Determine the center point in the 3D point cloud data, and establish a three-dimensional coordinate system with the center point as the origin; determine the projection viewpoint according to the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system; use the direction of the viewpoint as the projection rotation center axis, and use the direction with a rotation angle of zero as the initial projection direction.

[0138] Optionally, the initial projection direction determining module 310 is further used for:

[0139] According to the principal component analysis method, the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system are analyzed to obtain the coordinate distribution variance; the coordinate distribution variance is sorted in order from large to small, and the position of the viewpoint is determined in the direction corresponding to the minimum value in the coordinate distribution variance.

[0140] Optionally, the hidden layer diffusion model includes an encoder and a diffusion model, and the minimum prediction disturbance noise error determination module 330 is further used to:

[0141] The 2D point cloud data is input into the encoder to obtain the hidden layer features corresponding to the 2D point cloud data; random noise is added to the hidden layer features to obtain noise features; the noise features and the semantic description information of each category are input into the diffusion model at the same time to obtain the prediction disturbance noise and prediction disturbance noise error corresponding to each category; for the prediction disturbance noise error corresponding to each category, the partial derivative with respect to the projection direction is determined respectively, the projection direction is adjusted and the partial derivative of the prediction disturbance noise error with respect to the projection direction is re-determined until the partial derivative is less than the set threshold or the number of iterations is reached, the projection direction corresponding to the last iteration is taken as the target projection direction, and the prediction disturbance noise error corresponding to the target projection direction is determined as the minimum prediction noise error.

[0142] Optionally, the minimum predicted disturbance noise error determination module 330 is further used to:

[0143] The difference between the projection direction and the partial derivative in the previous iteration is determined as the projection direction of the next iteration.

[0144] Optionally, the 2D point cloud data includes at least one of the following: a rendered image obtained by projecting and rendering the 3D point cloud data, a depth image obtained according to the distance between the camera and the 3D point cloud data, and an edge image obtained by extracting edges from the rendered image.

[0145] Optionally, the minimum predicted disturbance noise error determination module 330 is further used to:

[0146] At least two of the rendered image, the depth image and the edge image are respectively input into the encoder to obtain at least two single features; the at least two single features are fused, and the fusion result is used as the hidden layer feature.

[0147] The 3D point cloud data zero-sample classification device provided in the embodiment of the present invention can execute the 3D point cloud data zero-sample classification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0148] Embodiment 4

[0149] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0150] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0151] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0152] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a 3D point cloud data zero-sample classification method.

[0153] In some embodiments, the 3D point cloud data zero-shot classification method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the 3D point cloud data zero-shot classification described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the 3D point cloud data zero-shot classification method in any other appropriate manner (e.g., by means of firmware).

[0154] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0156] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0158] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0159] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0161] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A zero-sample classification method for 3D point cloud data, characterized in that: include: Obtain the 3D point cloud data to be classified and determine the initial projection direction; Projecting the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data; Acquire semantic description information of at least two categories, and determine the minimum prediction disturbance noise error of the 2D point cloud data for each category of semantic description information in combination with the trained hidden layer diffusion model; Taking the category corresponding to the minimum value among the minimum prediction disturbance noise errors as the zero sample classification result; The hidden layer diffusion model includes an encoder and a diffusion model. The minimum prediction noise error of the 2D point cloud data for each category of semantic description information is determined in combination with the trained hidden layer diffusion model, including: Inputting the 2D point cloud data into the encoder to obtain hidden layer features corresponding to the 2D point cloud data; Adding random noise to the hidden layer features to obtain noise features; The noise characteristics and the semantic description information of each category are respectively input into the diffusion model at the same time to obtain the predicted disturbance noise and the predicted disturbance noise error corresponding to each category; For the predicted disturbance noise error corresponding to each category, the partial derivative of the projection direction is determined respectively, the projection direction is adjusted and the partial derivative of the predicted disturbance noise error with respect to the projection direction is re-determined until the partial derivative is less than the set threshold or the number of iterations is reached, the projection direction corresponding to the last iteration is taken as the target projection direction, and the predicted disturbance noise error corresponding to the target projection direction is determined as the minimum predicted noise error.

2. The method according to claim 1, characterized in that Determine the initial projection direction, including: Determine a center point in the 3D point cloud data, and establish a three-dimensional coordinate system with the center point as the origin; Determine a projection viewpoint according to the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system; The direction of the viewpoint is used as the projection rotation center axis, and the direction with a rotation angle of zero is used as the initial projection direction.

3. The method according to claim 2, characterized in that Determining a projection viewpoint according to the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system includes: Analyze the coordinates of each point in the 3D point cloud data in the three-dimensional coordinate system according to the principal component analysis method to obtain the coordinate distribution variance; The coordinate distribution variances are sorted in descending order, and the position of the viewpoint is determined in the direction corresponding to the minimum value of the coordinate distribution variances.

4. The method according to claim 1, characterized in that: Adjusting the projection direction includes: The difference between the projection direction and the partial derivative in the previous iteration is determined as the projection direction of the next iteration.

5. The method according to claim 1, characterized in that The 2D point cloud data includes at least one of the following: a rendered image obtained by projecting and rendering the 3D point cloud data, a depth image obtained according to a distance between a camera and the 3D point cloud data, and an edge image obtained by extracting edges from the rendered image.

6. The method according to claim 5, characterized in that Inputting the 2D point cloud data into the encoder to obtain hidden layer features corresponding to the 2D point cloud data includes: Inputting at least two of the rendered image, the depth image, and the edge image into the encoder respectively to obtain at least two single features; The at least two single features are fused, and the fusion result is used as the hidden layer feature.

7. A 3D point cloud data zero-sample classification device, characterized in that: include: An initial projection direction determination module is used to obtain 3D point cloud data to be classified and determine the initial projection direction; A projection module, used for projecting the 3D point cloud data according to the initial projection direction to obtain 2D point cloud data; A minimum prediction disturbance noise error determination module is used to obtain at least two categories of semantic description information, and determine the minimum prediction disturbance noise error of the 2D point cloud data for each category of semantic description information in combination with the trained hidden layer diffusion model; A zero-sample classification result determination module, used to take the category corresponding to the minimum value of each of the minimum predicted disturbance noise errors as the zero-sample classification result; The hidden layer diffusion model includes an encoder and a diffusion model, and the minimum prediction disturbance noise error determination module is further used for: Inputting the 2D point cloud data into the encoder to obtain hidden layer features corresponding to the 2D point cloud data; Adding random noise to the hidden layer features to obtain noise features; The noise characteristics and the semantic description information of each category are respectively input into the diffusion model at the same time to obtain the predicted disturbance noise and the predicted disturbance noise error corresponding to each category; For the predicted disturbance noise error corresponding to each category, the partial derivative of the projection direction is determined respectively, the projection direction is adjusted and the partial derivative of the predicted disturbance noise error with respect to the projection direction is re-determined until the partial derivative is less than the set threshold or the number of iterations is reached, the projection direction corresponding to the last iteration is taken as the target projection direction, and the predicted disturbance noise error corresponding to the target projection direction is determined as the minimum predicted noise error.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the 3D point cloud data zero-sample classification method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the 3D point cloud data zero-sample classification method according to any one of claims 1 to 6 when executed.

Images