Method for generating point cloud normal vectors based on three-dimensional visual image, storage medium and terminal

Abstract

This invention discloses a method, storage medium, and terminal for generating point cloud normal vectors based on 3D visual images. The method includes usage steps, which include the following sub-steps: acquiring a first point cloud of the 3D visual image to be processed; upsampling the first point cloud to obtain a second point cloud; clustering the second point cloud based on the first point cloud to obtain a third point cloud; and calculating the corresponding point cloud normal vector from the third point cloud. This invention integrates the features of the first point cloud and the upsampled second point cloud, using the clustered third point cloud as the basis for calculating the point cloud normal vector. Since upsampling makes the point cloud denser, and denser point clouds have higher expressive power, this invention also has excellent normal vector estimation capabilities for sparse and difficult-to-distinguish point clouds.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional image processing, and in particular to a method for generating point cloud normal vectors based on three-dimensional visual images, a storage medium, and a terminal. Background Technology

[0002] With the advancement of technology, traditional two-dimensional visual images can no longer meet the needs of daily life, and deep learning technology based on point clouds has gradually gained public recognition and application. This has led to applications such as virtual reality, autonomous driving, metaverse, and more.

[0003] Point clouds, as a data structure describing the surface of an object, can clearly capture the shape information of the object. However, relying solely on the distribution of point clouds for judgment has a large error and is prone to various misjudgments, which can lead to security issues. Therefore, the normal vectors of the point cloud are collected.

[0004] The purpose of point cloud normal vector estimation is to enhance the expressive power of point clouds and play an auxiliary role in various applications; it also helps in 3D reconstruction and 3D object repair. In existing technologies, after acquiring point cloud data of a 3D visual image, the point cloud data is directly processed to obtain the corresponding point cloud normal vectors, thereby enabling subsequent processing. However, this method is not accurate for sparse and difficult-to-distinguish point cloud data. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for generating point cloud normal vectors based on three-dimensional visual images, a storage medium, and a terminal.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A first aspect of the present invention provides a method for generating point cloud normal vectors based on three-dimensional visual images, comprising a usage step, wherein the usage step includes the following sub-steps:

[0008] Obtain the first point cloud of the 3D visual image to be processed;

[0009] Upsample the first point cloud to obtain the second point cloud;

[0010] Using the first point cloud as a baseline, the second point cloud is clustered to obtain the third point cloud;

[0011] Calculate the third point cloud to generate the corresponding point cloud normal vector.

[0012] Furthermore, the method also includes a training step, which comprises the following sub-steps:

[0013] Obtain the first point cloud and reference normal vector of the existing 3D visual image;

[0014] Upsample the first point cloud to obtain the second point cloud;

[0015] Using the first point cloud as a baseline, the second point cloud is clustered to obtain the third point cloud;

[0016] Calculate the third point cloud to generate the corresponding point cloud normal vector;

[0017] The second point cloud is reduced using a distance loss function, and / or the point cloud normal vectors are reduced using a 2-norm loss function based on the reference normal vector.

[0018] Further, the upsampling of the first point cloud to obtain the second point cloud includes:

[0019] Perform the furthest-distance sampling on the first point cloud to obtain a temporary point cloud p including a preset number of points;

[0020] The first feature f of the temporary point cloud p is extracted using 1*1 convolution;

[0021] The first feature f is sampled at the furthest distance to obtain the sampled second feature f';

[0022] Using the second feature f' as the benchmark, cluster the first feature f to obtain the third feature f” after clustering;

[0023] The difference is calculated for each of the first cluster features M in the third feature f”, and then a 1*1 convolution is used to calculate the fourth feature f”' as the local feature F;

[0024] Max pooling is used to calculate local features F, resulting in global features F';

[0025] The overall feature F' is calculated by a fully connected layer to obtain the upsampled second point cloud.

[0026] Furthermore, during the calculation process, the fourth feature f”’ is used as the first feature f, and the calculation of the first feature f to the fourth feature f”’ is repeated multiple times. The fourth feature f”’ obtained from the last calculation is used as the calculated local feature F.

[0027] Further, the step of calculating the third point cloud to generate the corresponding point cloud normal vector includes:

[0028] For each second clustering feature M' of the third point cloud, a 1*1 convolution is applied to calculate the query feature, key feature, and value feature;

[0029] The absolute value of the difference between the query feature (query) and the key feature (key) is calculated to obtain the relative feature offset.

[0030] The attention feature is obtained by calculating the matrix dot product between the relative feature offset and the value feature;

[0031] The residual feature F is obtained by summing the second clustering feature and the attention feature of the third point cloud. res ;

[0032] For residual characteristics F res The third clustering feature M” is obtained by using a fully connected layer for computation;

[0033] The third cluster feature M is calculated sequentially using max pooling and fully connected layers to generate the corresponding point cloud normal vector.

[0034] Furthermore, during the calculation process, the third cluster feature M” is used as the second cluster feature M’, and the calculation of the second cluster feature M’ to the third cluster feature M” is repeated multiple times. The third cluster feature M” obtained from the last calculation is used as the fourth cluster feature M”’.

[0035] The fourth cluster feature M”' is calculated by sequentially using max pooling and fully connected layers, generating the corresponding point cloud normal vector.

[0036] Furthermore, the reduction of the second point cloud using the distance loss function includes:

[0037] Calculate the distance loss L using the first and second point clouds. distance The calculation formula is as follows:

[0038]

[0039] In the formula, p0 represents the first point cloud, point represents a point within the first point cloud; p' represents the second point cloud, and point' represents a point within the second point cloud.

[0040] Furthermore, the point cloud normal vector is reduced based on the reference normal vector using a 2-norm loss function, and the calculation formula is as follows:

[0041]

[0042] In the formula, N represents the point cloud normal vector, and N' represents the reference normal vector.

[0043] In a second aspect, the present invention provides a storage medium having computer instructions stored thereon, wherein the computer instructions, when executed, perform the steps of the point cloud normal vector generation method based on a three-dimensional visual image.

[0044] A third aspect of the present invention provides a terminal, including a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the steps of the point cloud normal vector generation method based on a three-dimensional visual image when executing the computer instructions.

[0045] The beneficial effects of this invention are:

[0046] (1) In an exemplary embodiment of the present invention, compared with the prior art of directly calculating the first point cloud p0 of the three-dimensional visual image to generate the corresponding point cloud normal vector N, in this exemplary embodiment, the features of the first point cloud p0 and the second point cloud p' obtained by upsampling are combined to obtain the third point cloud p0 obtained by clustering. j As the basis for calculating the point cloud normal vector N, upsampling makes the point cloud denser, and the denser the point cloud, the better its expressive power. Therefore, it also has excellent normal vector estimation capabilities for sparse and indistinguishable point clouds.

[0047] (2) In another exemplary embodiment of the present invention, the second point cloud p' is reduced using a distance loss function, and / or the point cloud normal vector N is reduced using a 2-norm loss function based on the reference normal vector N', thereby training the model used in the whole method to make the model extraction effect better.

[0048] (3) In another exemplary embodiment of the present invention, a specific implementation method for upsampling the first point cloud to obtain the second point cloud is disclosed. Compared with the traditional method, this method has the advantages of high generation accuracy and more stable performance when facing interference.

[0049] (4) In another exemplary embodiment of the present invention, the local feature F is calculated repeatedly in a loop, which enhances the feature and improves the fitting performance of the model. Through multiple iterations, the model can more easily learn complete and expressive local features.

[0050] (5) In another exemplary embodiment of the present invention, the specific implementation of calculating the third point cloud and generating the corresponding point cloud normal vector can effectively capture the local features of the point cloud, fit various complex point cloud surfaces, and thus generate accurate point cloud normal vectors.

[0051] (6) In another exemplary embodiment of the present invention, the calculation of the point cloud normal vector is achieved by using a loop calculation method. This not only effectively captures the local features of the point cloud, but also, due to the high degree of freedom of the function, it can better fit various complex point cloud surfaces, thereby generating more accurate point cloud normal vectors. Attached Figure Description

[0052] Figure 1This is a flowchart of a point cloud normal vector generation method based on a three-dimensional visual image provided in an exemplary embodiment of the present invention;

[0053] Figure 2 This is a flowchart of a point cloud normal vector generation method based on a three-dimensional visual image, provided in another exemplary embodiment of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0056] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0057] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0058] The following content will explain the terms:

[0059] 3D visual images utilize the difference in visual perception between the two eyes and the principle of optical refraction to allow viewers to directly perceive a three-dimensional image on a plane. Objects in the image can either protrude from the picture or be deeply embedded within it, appearing vivid and lifelike, creating a strong visual impact. It is fundamentally different from planar images, which reflect the two-dimensional relationships of objects (vertical and horizontal) and also convey a sense of depth. Current conventional photography uses 2D planar imaging, losing the third dimension of the physical world (geometric data such as size and distance). 3D imaging technology, however, enables images to contain the three-dimensional coordinates of every point in space. This facilitates 3D image recognition and analysis by computers.

[0060] Point cloud: After acquiring the spatial coordinates of each sampling point on the surface of an object, the resulting set of points is called a "point cloud." In reverse engineering, the set of point data on the surface of a product obtained through measuring instruments is also called a point cloud. Point clouds obtained using a 3D coordinate measuring machine typically have a smaller number of points and larger spacing between them, called sparse point clouds; while point clouds obtained using a 3D laser scanner or photogrammetric scanner have a larger number of points and are denser, called dense point clouds. Point clouds obtained based on laser measurement principles include 3D coordinates (XYZ) and laser reflection intensity (Intensity); point clouds obtained based on photogrammetry principles include 3D coordinates (XYZ) and color information (RGB); point clouds obtained by combining laser measurement and photogrammetry principles include 3D coordinates (XYZ), laser reflection intensity (Intensity), and color information (RGB).

[0061] Point cloud normal vectors: Normal vectors are one of the important attributes of each point in a point cloud. As can be seen from spatial transformations, the angle and curvature values ​​of the normal vectors at each point in a point cloud do not change with the motion of the object, exhibiting rigid body invariance. Point cloud surface normal vectors are an important geometric surface property with wide applications in computer graphics, such as determining a reasonable light source position when performing lighting rendering and other visualization effects. Furthermore, many point cloud algorithms are based on accurate normal vector estimation, such as many surface reconstruction algorithms, point cloud segmentation algorithms, point cloud denoising algorithms, and feature description algorithms.

[0062] Clustering: Dividing a dataset into different classes or clusters according to a specific criterion (e.g., distance), maximizing the similarity of data objects within the same cluster and maximizing the differences between data objects in different clusters; after clustering, data of the same class are grouped together as much as possible, while data of different classes are separated as much as possible. Since clustering refers to grouping similar data together, the specific classification does not concern itself with the class label; the goal is simply to aggregate similar data together. Clustering is an unsupervised learning method.

[0063] In existing technologies, after acquiring point cloud data of a 3D visual image, the point cloud data is directly processed to obtain the corresponding point cloud normal vectors, thereby enabling subsequent processing. However, this method is not accurate for sparse and difficult-to-distinguish point cloud data.

[0064] See Figure 1 , Figure 1 The present invention illustrates a point cloud normal vector generation method based on a 3D visual image, which includes a usage step, the usage step comprising the following sub-steps:

[0065] Obtain the first point cloud p0 of the 3D visual image to be processed;

[0066] Upsample the first point cloud p0 to obtain the second point cloud p';

[0067] Using the first point cloud p0 as the baseline, cluster the second point cloud p' to obtain the third point cloud p. j ;

[0068] For the third point cloud p j Perform calculations to generate the corresponding point cloud normal vector N.

[0069] Specifically, compared to the prior art which directly calculates the first point cloud p0 of the 3D visual image to generate the corresponding point cloud normal vector N, in this exemplary embodiment, the features of the first point cloud p0 and the second point cloud p' obtained by upsampling are combined to obtain the third point cloud p' obtained by clustering. j As the basis for calculating the point cloud normal vector N, upsampling makes the point cloud denser, and denser point clouds have a higher expressive power. Therefore, it also has excellent normal vector estimation capabilities for sparse and indistinguishable point clouds. This method is applicable to neural networks, exhibiting excellent generalization ability and capable of predicting normal vectors for various point cloud structures. Furthermore, because it relies on neural networks, this method can generate normal vectors in real time, saving time.

[0070] It should be noted that the first point cloud p0 of the 3D visual image can be acquired using radar. Meanwhile, the final generated point cloud normal vector N can be used for subsequent 3D reconstruction and 3D object repair.

[0071] More preferably, in an exemplary embodiment, such as Figure 2 As shown, the method further includes a training step, which comprises the following sub-steps:

[0072] Obtain the first point cloud p0 and the reference normal vector N' of the existing 3D visual image;

[0073] Upsample the first point cloud p0 to obtain the second point cloud p';

[0074] Using the first point cloud p0 as the baseline, cluster the second point cloud p' to obtain the third point cloud p. j ;

[0075] For the third point cloud p j Perform calculations to generate the corresponding point cloud normal vector N;

[0076] Reduce the second point cloud p' using the distance loss function, and / or reduce the point cloud normal vector N based on the reference normal vector N' using the 2-norm loss function.

[0077] Specifically, in this exemplary embodiment, a training step can be performed before the usage step, or before performing the usage step again. The training step is basically the same as the usage step, except that: (1) the input data includes the first point cloud p0 of the 3D visual image and the reference normal vector N'; (2) finally, the second point cloud p' is reduced using a distance loss function, and / or the point cloud normal vector N is reduced using a 2-norm loss function based on the reference normal vector N', thereby training the model used by the entire method to improve the model extraction effect.

[0078] More preferably, in an exemplary embodiment, the upsampling of the first point cloud p0 to obtain the second point cloud p' includes:

[0079] S21: Perform the farthest distance sampling on the first point cloud p0 to obtain a temporary point cloud p including a preset number of points;

[0080] S22: Extract the first feature f of the temporary point cloud p using 1*1 convolution, f = conv(p);

[0081] S23: Perform the farthest distance sampling on the first feature f to obtain the sampled second feature f';

[0082] S24: Using the second feature f' as the benchmark, cluster the first feature f to obtain the third feature f” after clustering;

[0083] S25: Subtract each first cluster feature M from the third feature f”, and then use a 1*1 convolution to calculate the fourth feature f”’ as the local feature F, f″′=conv(f″-f i ), where (f i ∈M);

[0084] S26: Use max pooling to calculate the local feature F and obtain the global feature F', F' = max pooling(F);

[0085] S27: The overall feature F' is calculated through a fully connected layer to obtain the upsampled second point cloud, p′=MLP(F′).

[0086] Specifically, in this exemplary embodiment, this step can be applied to both the training step and the usage step.

[0087] In existing technologies, the locations of generated points are typically rigidly defined (e.g., N points are generated within a certain radius of each point in the first point cloud p0 to achieve an N-fold upsampling effect). However, in the upsampling process of this exemplary embodiment, the present invention first extracts local (S22-S25) and global features of the point cloud (S26), then discards redundant information and noise using a symmetric function (S26 max pooling), retaining the remaining features; and uses these features to generate the upsampled second point cloud p' (S27). Compared to traditional methods, this method has the advantages of high generation accuracy and more stable performance when facing interference.

[0088] More specifically, step S22 enhances the point cloud features (or iteratively enhances them if multiple calculations are performed subsequently); step S23 downsamples the point cloud to reduce its density and thus computational cost; steps S24 and S25 calculate local features of the point cloud. Through these local features, the model learns more refined information (relative to global features), thereby improving its performance. Furthermore, step S25 uses feature subtraction to pass the features between each point in the clustering of S24, thus obtaining relative local relationships.

[0089] The model achieves high accuracy because it comprehensively utilizes both local and global information, and the neural network possesses stronger fitting capabilities, enabling it to fit various complex situations. Its greater stability in the face of interference primarily depends on steps S23 and S26. External interference manifests as noise in the input point cloud. Step S23 downsamples the point cloud, eliminating interference to some extent. Simultaneously, step S26, global maximum pooling, discards most features (redundancy and noise). From a data volume perspective, if the input is 1024 points and the pooling layer outputs only 1 point, it's equivalent to discarding features from 1023 points, retaining only those important to the model; thus ensuring stability when facing interference.

[0090] More preferably, in an exemplary embodiment, during the calculation process, the fourth feature f”’ is used as the first feature f, and the calculation of the first feature f to the fourth feature f”’ is repeated multiple times, and the fourth feature f”’ obtained from the last calculation is used as the calculated local feature F.

[0091] Specifically, in this exemplary embodiment, the calculation is repeated multiple times (preferably three times) in steps S22 to S25 of the above exemplary embodiment. Its function is to hierarchically downsample the point cloud and extract local features, extracting features at multiple scales to improve feature representation ability and model performance.

[0092] For example, during the first execution of steps S22-S25, local features of layer A are extracted, and each point in the output of step S25 of layer A already possesses local features. During the second execution of steps S22-S25, layer B further extracts features based on the local features of layer A. Layer B integrates the local features of layer A to calculate a "larger-scale local feature" (a local feature with a larger local scope).

[0093] More preferably, in an exemplary embodiment, the third point cloud p j Perform calculations to generate the corresponding point cloud normal vector N, including:

[0094] S41: For the third point cloud p j Each second cluster feature M' is convolved using a 1*1 convolution to calculate the query feature, key feature, and value feature, as shown in the following formula:

[0095] query = conv1(M′)

[0096] key = conv2(M′)

[0097] value = conv3(M′)

[0098] S42: Calculate the absolute value of the difference between the query feature (query) and the key feature (key) to obtain the relative feature offset. The calculation formula is as follows:

[0099] offset = |query-key|

[0100] S43: Calculate the matrix dot product (dot) between the relative feature offset and the value feature to obtain the attention feature. The calculation formula is as follows:

[0101] attention = offset(dot) value

[0102] S44: For the third point cloud p j The second clustering feature M' and the attention feature are added together, and the residual is calculated to obtain the residual feature F. res ;

[0103] S45: For residual characteristics F res The third cluster feature M” is obtained by using a fully connected layer, and the calculation formula is as follows:

[0104] M” = Relu(MLP(F) res ))

[0105] In the formula, ReLU represents the ReLU activation function. The purpose is to increase the nonlinear expressive power of the neural network and further enhance the overall expressive power of the neural network.

[0106] S46: Calculate the third clustering feature M using max pooling and a fully connected layer sequentially, generating the corresponding point cloud normal vector N. The calculation formula is as follows:

[0107] N = MLP(maxpooling(M″′))

[0108] Specifically, in this exemplary embodiment, this step can be applied to both the training step and the usage step.

[0109] Among them, the existing technology adopts a deep learning-based method: using a network based on PointNet or PointNet++ (a mature technology based on symmetric functions) to extract the features of each point in the point cloud, and then using a convolutional neural network to generate point cloud normal vectors for each point. There are various derivatives of this type of method, but the core idea is the same. However, the existing methods have the following problems: (1) Normal vector estimation requires fine local features because the normal vector is directly related to the object surface reflected by the point cloud. The existing methods (PointNet, or PointNet++) use symmetric functions to synthesize and extract point cloud features. The granularity of symmetric function calculation is coarse (low precision), and the features extracted by symmetric functions tend to be consistent locally (all points calculated by symmetric functions have the same features, resulting in a lack of differences in features between points), which cannot reflect fine local features. This feature is not suitable for calculating normal vectors. (2) At the same time, because the existing methods use symmetric functions, the number of function forms that can be fitted is too small, so the function freedom is not high.

[0110] In this exemplary embodiment, local features of the point cloud can be effectively captured, various complex point cloud surfaces can be fitted, and thus accurate point cloud normal vectors can be generated. Specifically:

[0111] For fitting complex surfaces: A complex point cloud surface can be differentiated into simple surfaces within its local structure. Therefore, fitting a complex surface means being able to capture local surface structure information. This exemplary embodiment describes local surface structure information as follows: First, the point cloud is divided into multiple clusters. Then, during the cluster calculation in step S42, the relative relationships between each point in the cluster are calculated (e.g., assuming a cluster contains points A and B and their corresponding features, the difference between the features of points A and B is calculated, and the resulting feature difference describes the relationship between A and B). These relative relationships are used to describe the local surface structure information.

[0112] More preferably, in an exemplary embodiment, during the calculation process, the third cluster feature M” is used as the second cluster feature M’, and the calculation of the second cluster feature M’ to the third cluster feature M” is repeated multiple times, and the third cluster feature M” obtained from the last calculation is used as the fourth cluster feature M”’.

[0113] The fourth cluster feature M”' is calculated by sequentially using max pooling and fully connected layers, generating the corresponding point cloud normal vector.

[0114] Specifically, in this exemplary embodiment, the calculation is repeated multiple times (preferably four times) in steps S41 to S45 of the above exemplary embodiment. This not only effectively captures local features of the point cloud, but also, due to the high degree of freedom of the function, it can better fit various complex point cloud surfaces, thereby generating more accurate point cloud normal vectors.

[0115] Specifically, regarding degrees of freedom: Degrees of freedom are defined as a measure of the richness of functional forms that a neural network can fit. First, calculating the point cloud normal vectors requires a fine local structure, therefore degrees of freedom in a global sense are not considered. Second, in this exemplary embodiment, the local structure is described by clustering. In the local structure (clustering), due to the presence of two or more fully connected neural network layers (multiple calculations in S45 and S46), it is theoretically possible to fit any function. Correspondingly, being able to fit any function is reflected here as being able to fit various local structures.

[0116] More preferably, in an exemplary embodiment, the reduction of the second point cloud p' using the distance loss function includes:

[0117] Using the first point cloud p0 and the second point cloud p', calculate the distance loss L. distance The calculation formula is as follows:

[0118]

[0119] In the formula, p0 represents the first point cloud, point represents a point within the first point cloud; p' represents the second point cloud, and point' represents a point within the second point cloud.

[0120] Specifically, in this exemplary embodiment, this step can be applied to the training step.

[0121] More preferably, in an exemplary embodiment, the reduction of the point cloud normal vector N based on the reference normal vector N' using a 2-norm loss function is calculated using the following formula:

[0122]

[0123] In the formula, N represents the point cloud normal vector, and N' represents the reference normal vector.

[0124] Specifically, in this exemplary embodiment, this step can be applied to the training step.

[0125] Having the same inventive concept as the above exemplary embodiments, another exemplary embodiment of the present invention provides a point cloud normal vector generation system based on three-dimensional visual images, including a normal vector generation module, the normal vector generation module comprising:

[0126] Point cloud acquisition submodule: used to acquire the first point cloud of the 3D visual image to be processed;

[0127] Point cloud upsampling submodule: used to upsample the first point cloud to obtain the second point cloud;

[0128] Point cloud clustering module: used to cluster the second point cloud based on the first point cloud to obtain the third point cloud;

[0129] Point cloud normal vector generation submodule: used to calculate the third point cloud and generate the corresponding point cloud normal vector.

[0130] Correspondingly, the system also includes a training module, which includes the following sub-steps:

[0131] Training point cloud acquisition submodule: used to acquire the first point cloud and reference normal vector of an existing 3D vision image;

[0132] Training point cloud upsampling submodule: used to upsample the first point cloud to obtain the second point cloud;

[0133] Training point cloud clustering module: used to cluster the second point cloud based on the first point cloud to obtain the third point cloud;

[0134] The training point cloud normal vector generation submodule is used to calculate the corresponding point cloud normal vectors for the third point cloud.

[0135] Normal Vector and Upsampling Reduction Submodule: Used to reduce the second point cloud using a distance loss function, and / or reduce the point cloud normal vectors based on the reference normal vector using a 2-norm loss function.

[0136] Correspondingly, the upsampling of the first point cloud to obtain the second point cloud includes:

[0137] Perform the furthest-distance sampling on the first point cloud to obtain a temporary point cloud p including a preset number of points;

[0138] The first feature f of the temporary point cloud p is extracted using 1*1 convolution;

[0139] The first feature f is sampled at the furthest distance to obtain the sampled second feature f';

[0140] Using the second feature f' as the benchmark, cluster the first feature f to obtain the third feature f” after clustering;

[0141] The difference is calculated for each of the first cluster features M in the third feature f”, and then a 1*1 convolution is used to calculate the fourth feature f”' as the local feature F;

[0142] Max pooling is used to calculate local features F, resulting in global features F';

[0143] The overall feature F' is calculated by a fully connected layer to obtain the upsampled second point cloud.

[0144] Correspondingly, during the calculation process, the fourth feature f”’ is used as the first feature f, and the calculation of the first feature f to the fourth feature f”’ is repeated multiple times. The fourth feature f”’ obtained from the last calculation is used as the calculated local feature F.

[0145] Correspondingly, the step of calculating the third point cloud to generate the corresponding point cloud normal vector includes:

[0146] For each second clustering feature M' of the third point cloud, a 1*1 convolution is applied to calculate the query feature, key feature, and value feature;

[0147] The absolute value of the difference between the query feature (query) and the key feature (key) is calculated to obtain the relative feature offset.

[0148] The attention feature is obtained by calculating the matrix dot product between the relative feature offset and the value feature;

[0149] The residual feature F is obtained by summing the second clustering feature and the attention feature of the third point cloud. res ;

[0150] For residual characteristics F res The third clustering feature M” is obtained by using a fully connected layer for computation;

[0151] The third cluster feature M is calculated sequentially using max pooling and fully connected layers to generate the corresponding point cloud normal vector.

[0152] Correspondingly, during the calculation process, the third cluster feature M” is used as the second cluster feature M’, and the calculation of the second cluster feature M’ to the third cluster feature M” is repeated multiple times, and the third cluster feature M” obtained from the last calculation is used as the fourth cluster feature M”’.

[0153] The fourth cluster feature M”' is calculated by sequentially using max pooling and fully connected layers, generating the corresponding point cloud normal vector.

[0154] Correspondingly, the reduction of the second point cloud using the distance loss function includes:

[0155] Calculate the distance loss L using the first and second point clouds. distance The calculation formula is as follows:

[0156]

[0157] In the formula, p0 represents the first point cloud, point represents a point within the first point cloud; p' represents the second point cloud, and point' represents a point within the second point cloud.

[0158] Correspondingly, the calculation formula for reducing the point cloud normal vector based on the reference normal vector and using the 2-norm loss function is as follows:

[0159]

[0160] In the formula, N represents the point cloud normal vector, and N' represents the reference normal vector.

[0161] Having the same inventive concept as the above exemplary embodiments, another exemplary embodiment of the present invention provides a storage medium storing computer instructions, which, when executed, perform the steps of the point cloud normal vector generation method based on a three-dimensional visual image.

[0162] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product (program product). The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0163] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0164] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0165] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0166] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] Having the same inventive concept as the exemplary embodiments described above, another exemplary embodiment of the present invention provides a terminal, including a memory and a processor. The memory stores computer instructions executable by the processor. When the processor executes the computer instructions, it performs the steps of the point cloud normal vector generation method based on a three-dimensional visual image.

[0168] The components of the terminal may include, but are not limited to: at least one processing unit, at least one storage unit, and a bus connecting different system components (including storage units and processing units).

[0169] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Embodiments" section of this specification, based on various exemplary embodiments of the present invention. For example, the processing unit can perform actions such as... Figure 1 The method shown in the figure.

[0170] The storage unit may include readable media in the form of volatile storage units, such as random access memory (RAM) and / or cache storage units, and may further include read-only memory (ROM).

[0171] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0172] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0173] The terminal can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable user interaction with the terminal, and / or any device that enables the terminal to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the terminal can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the terminal via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the terminal, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0174] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to this exemplary embodiment.

[0175] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for generating point cloud normal vectors based on three-dimensional visual images, characterized in that: The method comprises a use step, which comprises the following sub-steps: Obtaining a first point cloud of a three-dimensional visual image to be processed; Upsampling the first point cloud to obtain a second point cloud; Clustering the second point cloud based on the first point cloud to obtain a third point cloud; Calculating the third point cloud to generate a corresponding point cloud normal vector; The up-sampling of the first point cloud to obtain the second point cloud comprises: Performing farthest distance sampling on the first point cloud to obtain a temporary point cloud p comprising a preset number of points; Extracting a first feature f of the temporary point cloud p using 1*1 convolution; Performing farthest distance sampling on the first feature f to obtain a second feature f' after sampling; Clustering the first feature f based on the second feature f' to obtain a third feature f'' after clustering; Subtracting each first clustered feature M in the third feature f'' and then using 1*1 convolution to calculate a fourth feature f''' as a local feature F; Using maximum value pooling to calculate the local feature F to obtain an overall feature F'; Calculating the overall feature F' through a fully connected layer to obtain the up-sampled second point cloud.

2. The method of claim 1, wherein: The method further comprises a training step, which comprises the following sub-steps: Obtaining a first point cloud and a reference normal vector of an existing three-dimensional visual image; Upsampling the first point cloud to obtain a second point cloud; Clustering the second point cloud based on the first point cloud to obtain a third point cloud; Calculating the third point cloud to generate a corresponding point cloud normal vector; Using a distance loss function to reduce the second point cloud, and / or using a 2-norm loss function to reduce the point cloud normal vector based on the reference normal vector.

3. The method of claim 1, wherein: In the calculation process, the fourth feature f ’’’ is calculated as the first feature f , the first feature f to the fourth feature f ’’’ are repeatedly calculated, and the fourth feature f ’’’ obtained in the last calculation is taken as the calculated local feature F .

4. The method of claim 1 or 2, wherein: The calculation of the third point cloud to generate the corresponding point cloud normal vector comprises: each second cluster of the third point cloud M query features query , key features key , and value features value ; Query features query and key features key Calculate the absolute value after the difference, get the relative feature offset offset ; relative feature offset offset and value features value computing a matrix dot product to obtain attention features attention ; second clustering features of the third point cloud and the attention features attention after addition, residual calculation is performed to obtain residual features F res ; On residual features F res Using fully connected layers computation, third clustering features are obtained M ’’; calculating third clustering features using max-pooling and fully connected layers in sequence M generate corresponding point cloud normal vectors.

5. The method of claim 4, wherein: In the calculation process, the third cluster feature M ’’ is calculated as the second cluster feature M ’ is repeated several times to complete the calculation of the second cluster feature M ’ to the third cluster feature M ’’ and the third cluster feature M ’’ obtained in the last calculation is taken as the fourth cluster feature M ’’’; The fourth clustering feature is calculated using max-pooling and fully connected layers in sequence M corresponding point cloud normal vectors are generated.

6. A storage medium having stored thereon computer instructions, characterized in that, The computer instructions are used to perform the steps of the point cloud normal vector generation method based on a three-dimensional visual image according to any one of claims 1-5.

7. A terminal comprising a memory and a processor, the memory having stored thereon computer instructions executable on the processor, wherein, The processor is used to perform the steps of the point cloud normal vector generation method based on a three-dimensional visual image according to any one of claims 1-5.

Images