Point cloud completion method based on cross-modal and deep repair and related equipment

Abstract

The invention relates to the technical field of point cloud completion, and provides a point cloud completion method based on cross-modal and deep repair and related equipment. The method comprises the following steps: acquiring target point cloud data and auxiliary data corresponding to the target point cloud data; encoding the target point cloud data and the auxiliary data by using an encoder to obtain point cloud encoding features; decoding the point cloud coding features by using a decoder to obtain point cloud coordinate features under a plurality of density scales, and performing deep repair on each point cloud coordinate feature according to the auxiliary data to obtain a complemented point cloud under each density scale; constructing a comprehensive loss function according to all the complemented point clouds, and optimizing an encoder and a decoder by using the comprehensive loss function to obtain an optimized encoder and an optimized decoder; and complementing the to-be-complemented point cloud data by using the optimized encoder and the optimized decoder to obtain a final complemented point cloud. According to the method, the accuracy of point cloud repair can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of point cloud completion, and in particular to a point cloud completion method based on cross-modality and deep patching and related equipment. Background Art

[0002] With the rapid development and progress in fields such as autonomous driving and robot navigation, 3D point cloud technology has been widely used. As an efficient way to express 3D space, point cloud data provides an important foundation for environmental perception, object recognition and 3D reconstruction. However, in real applications, due to the hardware limitations of the equipment (such as limited resolution of lidar scanning) and the complexity of the acquisition environment (such as occlusion, lighting conditions, etc.), the directly acquired point cloud data is often sparse, incomplete and noisy. This defect seriously affects the application of point clouds in high-precision tasks, such as high-precision map construction, 3D object detection, instance segmentation, object surface detection and refined 3D reconstruction, making it difficult for these downstream tasks to achieve good results.

[0003] Currently, existing point cloud completion methods are mainly based on deep learning technology. These methods directly infer the missing geometric information from sparse point clouds to generate dense point clouds. However, relying solely on sparse point clouds for completion still has the following limitations: (1) Insufficient ability to process cross-modal information: The local geometric information contained in sparse point clouds is limited, making it difficult to accurately infer the details of complex shapes. However, existing point cloud fusion and completion methods are mostly result-level fusion, or choose to directly complete multimodal data. This approach is relatively rough and it is difficult to effectively combine point clouds and multimodal information for collaborative completion.

[0004] (2) Insufficient understanding of geometric details: How to learn local and global information is a very critical task in point cloud completion, and it is still a big challenge for existing methods. For example, when the shape structure of the object is relatively complex or the data has an extremely sparse and dense distribution, it is difficult for existing point cloud completion methods to efficiently extract useful features.

[0005] (3) Multimodal information is not fully utilized: Most completion methods do not fully exploit multimodal depth information and fail to effectively use depth information to assist point cloud completion.

[0006] From the above, it can be seen that the current point cloud completion method has the problem of poor accuracy of point cloud completion. Summary of the invention

[0007] The present application provides a point cloud completion method and related equipment based on cross-modality and deep patching, which can solve the problem of poor accuracy of point cloud completion.

[0008] In a first aspect, an embodiment of the present application provides a point cloud completion method based on cross-modality and depth patching, the point cloud completion method comprising: Obtain target point cloud data and auxiliary data corresponding to the target point cloud data; The target point cloud data and auxiliary data are encoded by an encoder to obtain point cloud encoding features; the point cloud encoding features are used to describe the local structure information and global skeleton information of the target point cloud data; The decoder is used to decode the point cloud coding features to obtain the point cloud coordinate features at multiple density scales, and each point cloud coordinate feature is deeply repaired according to the auxiliary data to obtain the completed point cloud at each density scale; the point cloud coordinate features are used to describe the coordinates of multiple data points at the corresponding density scale; A comprehensive loss function is constructed based on all completed point clouds, and the encoder and decoder are optimized using the comprehensive loss function to obtain an optimized encoder and an optimized decoder; the comprehensive loss function is used to describe the accuracy of all completed point clouds; The optimized encoder and the optimized decoder are used to complete the point cloud data to obtain the final completed point cloud.

[0009] Optionally, the encoder includes a multi-layer feature extraction model and a multi-layer feature fusion model; The input data of the first input terminal of the multi-layer feature extraction model is auxiliary data, the input data of the second input terminal of the multi-layer feature extraction model is target point cloud data, and the output terminal of the multi-layer feature extraction model is connected to the first input terminal of the multi-layer feature fusion model; The input data of the second input end of the multi-layer feature fusion model is the target point cloud data, and the output data of the output end of the multi-layer feature fusion model is the point cloud coding feature.

[0010] Optionally, the multi-layer feature extraction model includes: a first residual neural module, a second residual neural module, a third residual neural module, a first upsampling module, a second upsampling module, a first enhanced continuous convolution module, a second enhanced continuous convolution module and a third enhanced continuous convolution module; The input end of the first residual neural module is the first input end of the multi-layer feature extraction model, the input end of the first enhanced continuous convolution module is the second input end of the multi-layer feature extraction model, and the output end of the first enhanced continuous convolution module, the output end of the second enhanced continuous convolution module and the output end of the third enhanced continuous convolution module are all output ends of the multi-layer feature extraction model; The output end of the first residual neural module is respectively connected to the input end of the second residual neural module and the input end of the first enhanced continuous convolution module, the output end of the second residual neural module is respectively connected to the input end of the third residual neural module and the input end of the first upsampling module, the output end of the first upsampling module is connected to the input end of the second enhanced continuous convolution module, the output end of the third residual neural module is connected to the input end of the second upsampling module, the output end of the second upsampling module is connected to the input end of the third enhanced continuous convolution module, the output end of the first enhanced continuous convolution module is connected to the input end of the second enhanced continuous convolution module, and the output end of the second enhanced continuous convolution module is connected to the input end of the third enhanced continuous convolution module.

[0011] Optionally, the multi-layer feature fusion model includes: a first multi-core edge convolution module, a second multi-core edge convolution module, a third multi-core edge convolution module, a fourth multi-core edge convolution module, a first cascade module, a second cascade module, a third cascade module, a first global structure perception module, a second global structure perception module, a clustering module, a maximum pooling module, a first expansion module, and a second expansion module; The input end of the first multi-core edge convolution module is the second input end of the multi-layer feature fusion model, the first input end of the first cascade module is the first input end of the multi-layer feature fusion model, and the output end of the third cascade module is the output end of the multi-layer feature fusion module; The output end of the first multi-core edge convolution module is respectively connected to the input end of the second multi-core edge convolution module and the input end of the second cascade module, the output end of the second multi-core edge convolution module is respectively connected to the input end of the third multi-core edge convolution module and the input end of the second cascade module, the output end of the third multi-core edge convolution module is respectively connected to the second input end of the first cascade module and the input end of the second cascade module, the output end of the first cascade module is respectively connected to the input end of the fourth multi-core edge convolution module, the input end of the first global structure perception module, and the input end of the clustering module, the output end of the fourth multi-core edge convolution module is respectively connected to the third The input end of the cascade module is connected, the output end of the clustering module is connected to the input end of the first global structure perception module, the output end of the first global structure perception module is connected to the input end of the second global structure perception module, the output end of the second global structure perception module is connected to the input end of the first expansion module, the output end of the first expansion module is connected to the input end of the third cascade module, the output end of the second cascade module is connected to the input end of the maximum pooling module, the output end of the maximum pooling module is connected to the input end of the second expansion module, and the output end of the second expansion module is connected to the input end of the third cascade module.

[0012] Optionally, the first global structure perception module and the second global structure perception module are both global structure perception modules; The global structure perception module includes a convolutional layer, a first nonlinear transformation layer, a second nonlinear transformation layer, a third nonlinear transformation layer, a multi-head interactive attention layer, a first residual normalization layer, a feature adaptive selection layer, and a second residual normalization layer; The input end of the convolutional layer is the input end of the global structure perception module, and the output end of the second residual normalization layer is the output end of the global structure perception module; The output end of the convolution layer is respectively connected to the input end of the first nonlinear transformation layer, the input end of the second nonlinear transformation layer, the input end of the third nonlinear transformation layer, and the input end of the first residual normalization layer; the input end of the multi-head interactive attention layer is respectively connected to the output end of the first nonlinear transformation layer, the output end of the second nonlinear transformation layer, and the output end of the third nonlinear transformation layer; the output end of the multi-head interactive attention layer is connected to the input end of the first residual normalization layer; the output end of the first residual normalization layer is respectively connected to the input end of the second residual normalization layer and the input end of the feature adaptive selection layer; and the output end of the feature adaptive selection layer is connected to the input end of the second residual normalization layer.

[0013] Optionally, the decoder includes a multi-layer feature decoding model, a registration model, and a depth completion model; The output end of the multi-layer feature decoding model and the output end of the registration model are both connected to the input end of the deep completion model. The input data of the input end of the multi-layer feature decoding model is the point cloud coding feature, the output data of the output end of the multi-layer feature decoding model is the point cloud coordinate feature at multiple density scales, the input data of the input end of the registration model is auxiliary data, and the output data of the output end of the deep completion model is the completed point cloud at multiple density scales.

[0014] Optionally, the multi-layer feature decoding model includes a first mapping module, a second mapping module, a third mapping module, a first addition module, a second addition module, a third addition module, and a multi-layer perception module; The input end of the first mapping module, the input end of the second mapping module, the input end of the third mapping module, the input end of the first addition module, the input end of the second addition module, and the input end of the third addition module are all input ends of the multi-layer feature decoding model, and the output end of the multi-layer perception module is the output end of the multi-layer feature decoding model; The output end of the first mapping module is connected to the input end of the first adding module, the output end of the second mapping module is connected to the input end of the second adding module, and the output end of the third mapping module is connected to the input end of the third adding module.

[0015] Optionally, the comprehensive loss function is: in, represents the value of the comprehensive loss function, and represents the hyperparameter, , represents the multi-scale completion loss, Represents the overall loss: in, represents the weighting parameter, Represents the probability of the discriminator's judgment on the generated completed point cloud, represents the completion of the point cloud, Represents the probability of the discriminator's judgment on the real point cloud, Indicates the real point cloud corresponding to the completed point cloud, represents the completion loss at the first density scale, represents the completion loss at the second density scale, Represents the completion loss at the third density scale: in, , Indicates The sampling point set of the completed point cloud at the density scale, express The number of sampling points in Indicates The sampling point set of the real point cloud at the density scale, express The number of sampling points in .

[0016] In a second aspect, an embodiment of the present application provides a point cloud completion device based on cross-modality and depth patching, comprising: An acquisition module, used for acquiring target point cloud data and auxiliary data corresponding to the target point cloud data; The encoding module is used to encode the target point cloud data and the auxiliary data using an encoder to obtain point cloud encoding features; the point cloud encoding features are used to describe the local structure information and global skeleton information of the target point cloud data; The decoding module is used to decode the point cloud coding features using a decoder to obtain the point cloud coordinate features at multiple density scales, and to perform deep patching on each point cloud coordinate feature according to the auxiliary data to obtain the completed point cloud at each density scale; the point cloud coordinate features are used to describe the coordinates of multiple data points at the corresponding density scale; The optimization module is used to construct a comprehensive loss function based on all completed point clouds, and use the comprehensive loss function to optimize the encoder and decoder to obtain an optimized encoder and an optimized decoder; the comprehensive loss function is used to describe the accuracy of all completed point clouds; The completion module is used to complete the point cloud data to be completed using the optimized encoder and the optimized decoder to obtain the final completed point cloud.

[0017] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned point cloud completion method based on cross-modality and depth patching when executing the above-mentioned computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned point cloud completion method based on cross-modality and depth patching.

[0019] The above solution of the present application has the following beneficial effects: In an embodiment of the present application, by acquiring target point cloud data and auxiliary data corresponding to the target point cloud data, an encoder is used to encode the target point cloud data and the auxiliary data to obtain point cloud coding features, and then a decoder is used to decode the point cloud coding features to obtain point cloud coordinate features at multiple density scales, and each point cloud coordinate feature is deeply repaired according to the auxiliary data to obtain a completed point cloud at each density scale, and then a comprehensive loss function is constructed according to all completed point clouds, and the encoder and decoder are optimized using the comprehensive loss function to obtain an optimized encoder and an optimized decoder, and finally the optimized encoder and the optimized decoder are used to complete the point cloud data to obtain the final completed point cloud. Among them, the point cloud coding features are obtained based on the target point cloud data and the auxiliary data, which can improve the information richness in the point cloud coding features, and then improve the accuracy of the point cloud coordinate features obtained by decoding the point cloud coding features. The accuracy of the completed point cloud obtained according to the accurate point cloud coordinate features is improved, and the comprehensive loss function is constructed to optimize the encoder and decoder, which can improve the performance of the encoder and decoder, and further improve the accuracy of point cloud completion.

[0020] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1A flowchart of a point cloud completion method based on cross-modality and depth patching provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an encoder provided in one embodiment of the present application; Figure 3 A schematic diagram of the structure of a global structure perception module provided in one embodiment of the present application; Figure 4 A schematic diagram of the structure of a decoder provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a discriminator provided in one embodiment of the present application; Figure 6 A schematic diagram of a specific process of a point cloud completion method based on cross-modality and depth patching provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a point cloud completion device based on cross-modality and depth repair provided by an embodiment of the present application; Figure 8 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0024] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0025] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0027] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] In response to the problem of poor accuracy of existing point cloud completion, an embodiment of the present application provides a point cloud completion method based on cross-modality and deep patching. The point cloud completion method obtains target point cloud data and auxiliary data corresponding to the target point cloud data, and then uses an encoder to encode the target point cloud data and the auxiliary data to obtain point cloud coding features, and then uses a decoder to decode the point cloud coding features to obtain point cloud coordinate features at multiple density scales, and deeply patches each point cloud coordinate feature according to the auxiliary data to obtain a completed point cloud at each density scale, and then constructs a comprehensive loss function based on all completed point clouds, and uses the comprehensive loss function to optimize the encoder and decoder to obtain an optimized encoder and an optimized decoder, and finally uses the optimized encoder and optimized decoder to complete the point cloud data to obtain the final completed point cloud. Among them, obtaining point cloud coding features based on target point cloud data and auxiliary data can improve the information richness in the point cloud coding features, and then improve the accuracy of the point cloud coordinate features obtained by decoding the point cloud coding features. The accuracy of the completed point cloud obtained according to the precise point cloud coordinate features is improved, and a comprehensive loss function is constructed to optimize the encoder and decoder, which can improve the performance of the encoder and decoder and further improve the accuracy of point cloud completion.

[0030] Next, an exemplary description is given of the point cloud completion method based on cross-modality and deep patching provided in this application.

[0031] like Figure 1 As shown, the point cloud completion method based on cross-modality and deep patching provided by the present application includes the following steps: Step 11, obtaining target point cloud data and auxiliary data corresponding to the target point cloud data.

[0032] The above-mentioned target point cloud data can be point cloud data of the surrounding environment or objects of the autonomous driving vehicle, and the above-mentioned auxiliary data is used to provide relevant information corresponding to the target point cloud data (such as color, texture information, etc.), such as the auxiliary data is an image of the surrounding environment of the autonomous driving vehicle, three-dimensional modeling, etc.

[0033] In some embodiments of the present application, target point cloud data and corresponding auxiliary data may be acquired using devices such as laser scanners and depth cameras.

[0034] For example, after acquiring the target point cloud data, data preprocessing is an indispensable step, the main purpose of which is to clean the data, including removing noise points and outliers, performing coordinate transformation, unifying scales, etc., so as to obtain reliable, enhanced, and standardized point cloud data. The preprocessing steps include: 1. Noise reduction The purpose of denoising is to remove noise points in the point cloud caused by sensor errors, environmental interference or the acquisition process, so as to improve the quality of the point cloud and the effect of subsequent algorithms. Typical algorithms include statistical filtering, radius filtering and smoothing filtering. This paper adopts statistical filtering to denoise point cloud data.

[0035] (1) Finding the neighborhood: For each point , determine its The ball neighborhood inside is the corresponding number of neighborhood points.

[0036] in, is the data point set in the target point cloud data, the radius The value of is estimated by the average distance between points in the point cloud.

[0037] in, Usually a constant between 1 and 2.

[0038] (2) Calculate the mean neighborhood distance: Calculate the average distance of the local neighborhood of each point .

[0039] ; (3) Global statistical analysis: Calculate the global mean of the neighborhood distances of all points and standard deviation : ; (4) Remove outliers: set the outlier threshold , if the mean distance of the neighborhood of a point is If the following conditions are met, it will be removed. The point cloud that is finally retained is recorded as .

[0040] ( Usually choose 1 or 2) 2. Normalization The purpose of normalization is to normalize the point cloud data to a unified coordinate system and scale, which helps the model converge quickly and stabilize. This module includes two steps: centroid normalization and scale normalization. The specific process is as follows: (1) Centroid Normalization Calculate the centroid: ; Center of mass translation: ; (2) Scale normalization Compute the maximum distance to the centroid: ; Zoom operation: .

[0041] It is worth mentioning that high-quality preprocessed data is an important prerequisite and guarantee for subsequent effective analysis and processing in the model to obtain high-precision output.

[0042] Step 12: Encode the target point cloud data and auxiliary data using an encoder to obtain point cloud coding features.

[0043] The above point cloud encoding features are used to describe the local structure information and global skeleton information of the target point cloud data.

[0044] like Figure 2 As shown, the encoder includes a multi-layer feature extraction model and a multi-layer feature fusion model; The input data of the first input terminal of the multi-layer feature extraction model is auxiliary data, the input data of the second input terminal of the multi-layer feature extraction model is target point cloud data, and the output terminal of the multi-layer feature extraction model is connected to the first input terminal of the multi-layer feature fusion model; The input data of the second input end of the multi-layer feature fusion model is the target point cloud data, and the output data of the output end of the multi-layer feature fusion model is the point cloud coding feature.

[0045] The multi-layer feature extraction model includes: a first residual neural module, a second residual neural module, a third residual neural module, a first upsampling module, a second upsampling module, a first enhanced continuous convolution module, a second enhanced continuous convolution module and a third enhanced continuous convolution module; The input end of the first residual neural module is the first input end of the multi-layer feature extraction model, the input end of the first enhanced continuous convolution module is the second input end of the multi-layer feature extraction model, and the output end of the first enhanced continuous convolution module, the output end of the second enhanced continuous convolution module and the output end of the third enhanced continuous convolution module are all output ends of the multi-layer feature extraction model; The output end of the first residual neural module is respectively connected to the input end of the second residual neural module and the input end of the first enhanced continuous convolution module, the output end of the second residual neural module is respectively connected to the input end of the third residual neural module and the input end of the first upsampling module, the output end of the first upsampling module is connected to the input end of the second enhanced continuous convolution module, the output end of the third residual neural module is connected to the input end of the second upsampling module, the output end of the second upsampling module is connected to the input end of the third enhanced continuous convolution module, the output end of the first enhanced continuous convolution module is connected to the input end of the second enhanced continuous convolution module, and the output end of the second enhanced continuous convolution module is connected to the input end of the third enhanced continuous convolution module.

[0046] The multi-layer feature fusion model includes: a first multi-core edge convolution module, a second multi-core edge convolution module, a third multi-core edge convolution module, a fourth multi-core edge convolution module, a first cascade module, a second cascade module, a third cascade module, a first global structure perception module, a second global structure perception module, a clustering module, a maximum pooling module, a first expansion module, and a second expansion module; The input end of the first multi-core edge convolution module is the second input end of the multi-layer feature fusion model, the first input end of the first cascade module is the first input end of the multi-layer feature fusion model, and the output end of the third cascade module is the output end of the multi-layer feature fusion module; The output end of the first multi-core edge convolution module is respectively connected to the input end of the second multi-core edge convolution module and the input end of the second cascade module, the output end of the second multi-core edge convolution module is respectively connected to the input end of the third multi-core edge convolution module and the input end of the second cascade module, the output end of the third multi-core edge convolution module is respectively connected to the second input end of the first cascade module and the input end of the second cascade module, the output end of the first cascade module is respectively connected to the input end of the fourth multi-core edge convolution module, the input end of the first global structure perception module, and the input end of the clustering module, the output end of the fourth multi-core edge convolution module is respectively connected to the third The input end of the cascade module is connected, the output end of the clustering module is connected to the input end of the first global structure perception module, the output end of the first global structure perception module is connected to the input end of the second global structure perception module, the output end of the second global structure perception module is connected to the input end of the first expansion module, the output end of the first expansion module is connected to the input end of the third cascade module, the output end of the second cascade module is connected to the input end of the maximum pooling module, the output end of the maximum pooling module is connected to the input end of the second expansion module, and the output end of the second expansion module is connected to the input end of the third cascade module.

[0047] It should be noted that the branch composed of the first global structure perception module, the second global structure perception module, the clustering module and the first expansion module is used to extract global skeleton information, the output data of the output end of the first expansion module is data describing the global skeleton information, and the fourth multi-core edge convolution module is used to extract local structure information, and the output data of its output end is data describing the local structure information.

[0048] The first global structure perception module and the second global structure perception module are both global structure perception modules; like Figure 3 As shown, the global structure perception module includes a convolution layer, a first nonlinear transformation layer, a second nonlinear transformation layer, a third nonlinear transformation layer, a multi-head interactive attention layer, a first residual normalization layer, a feature adaptive selection layer, and a second residual normalization layer; The input end of the convolutional layer is the input end of the global structure perception module, and the output end of the second residual normalization layer is the output end of the global structure perception module; The output end of the convolutional layer is respectively connected to the input end of the first nonlinear transformation layer, the input end of the second nonlinear transformation layer, the input end of the third nonlinear transformation layer, and the input end of the first residual normalization layer; the input end of the multi-head interactive attention layer is respectively connected to the output end of the first nonlinear transformation layer, the output end of the second nonlinear transformation layer, and the output end of the third nonlinear transformation layer; the output end of the multi-head interactive attention layer is connected to the input end of the first residual normalization layer; the output end of the first residual normalization layer is respectively connected to the input end of the second residual normalization layer and the input end of the feature adaptive selection layer; and the output end of the feature adaptive selection layer is connected to the input end of the second residual normalization layer.

[0049] It should be noted that the first residual neural module, the second residual neural module, and the third residual neural module can all be residual neural network operations for feature extraction of input data, the first upsampling module and the second upsampling module are used to upsample the input data, and the upsampling scale of the second upsampling module is larger than the upsampling scale of the first upsampling module, the first enhanced continuous convolution module, the second enhanced continuous convolution module, and the third enhanced continuous convolution module are all used to perform enhanced continuous convolution operations for feature enhancement of input data, and the first multi-core edge convolution module The second multi-core edge convolution module, the third multi-core edge convolution module, and the fourth multi-core edge convolution module are all multi-core edge convolutions, which are used to capture the geometry and feature distribution of the input data. The first cascade module, the second cascade module, and the third cascade module are all used to cascade the corresponding input data. The first global structure perception module and the second global structure perception module are both used to extract global features of the input data. The clustering module is used to cluster the input data. The maximum pooling module is used to perform maximum pooling on the input data. The first expansion module and the second expansion module are both used to expand the dimension of the input data.

[0050] The above-mentioned convolution layer is used to perform convolution operations on the input data. The first nonlinear transformation layer, the second nonlinear transformation layer, and the third nonlinear transformation layer are used to calculate the key vector, query vector, and value vector in the multi-head interactive attention mechanism, respectively. The multi-head interactive attention layer is used to perform multi-head attention mechanism and interaction operations on the input data. The first residual normalization layer and the second residual normalization layer are used to perform addition and normalization operations on the input data. The feature adaptive selection layer is used to perform feature selection on the input data.

[0051] Exemplarily, the operation process of enhancing continuous convolution is: For each point in the point cloud of the input enhanced continuous convolution, called a query point, the K-nearest neighbor algorithm (KNN) is used to find the k points closest to it to form the neighborhood of the point, and then the neighborhood points are projected to the image plane through the internal and external parameter matrix of the camera. In order to enhance the expressiveness of the projection point features, a feature aggregation strategy based on a 3×3 grid is designed. Specifically, through the mean operation, the pixel features in the 3×3 grid area around the projection point are aggregated into a feature vector. If the area is less than 3×3 in size, it is filled with 0. After this operation, the image feature set of the query point is obtained. .

[0052] in Yes The pixel coordinates of are the height and width of the image.

[0053] Next, a multi-layer perceptron (MLP) is used to fuse the image features of k neighborhood points. The input of MLP consists of three parts: image features, geometric offsets between query points and neighborhood points, and semantic offsets between query points and neighborhood points.

[0054] , in is the neighborhood of i (including i), is the image feature of the neighborhood point j, is the geometric offset between point j and point i, is the semantic offset between the two, and concat(·) is the concatenation of multiple vectors.

[0055] Finally, by aggregating the features of the MLP output by summing, we can get the enhanced features of the enhanced continuous convolution output.

[0056] The operational expression of the above multi-core edge convolution is: in, Represents the output data of multi-core edge convolution, Represents a splicing operation, Indicates indivual The output data, , represents the number of MLPs in multi-core edge convolution, Indicates the midpoint of the input data The density of and represents the weight coefficient, represents the eigenvector of point i, represents the eigenvector of point j, Represents the distance offset between two points. Indicate point Neighborhood of represents the number of points in the neighborhood, Indicate point and Point The number of common neighborhood points between Indicate point The number of neighboring points.

[0057] The operation process in the above global structure perception module is as follows: the global structure perception module GCP receives input data X, which is a matrix of size N×C, corresponding to the feature vector of N points. First, the dimension of the point is reduced by 1×1 convolution to obtain , to remove redundant information and reduce computational complexity. In order to capture complex relationships, MLP is used instead of linear projection to perform nonlinear transformations to obtain Q, K, and V matrices. Then, multi-head interactive attention in the form of residuals is used to learn features.

[0058] in, represents the normalized data, Use vector attention to adjust different feature channels and adopt self-attention to capture the interaction between heads. The operation is: in It is a standard self-attention operation, through which the model can learn the relationship between attention heads and improve the expressiveness of features. represents the output of the i-th point after the vector attention mechanism, represents the query vector corresponding to the i-th point under the current attention head, represents the key vector corresponding to the jth point under the current attention head, represents the value vector corresponding to the jth point under the current attention head, represents the output of the first attention head, represents the output of the second attention head, represents the output of the 8th attention head, Represents the output data after multi-head self-attention operation.

[0059] In order to enhance the model's fitting and selection capabilities, a feature adaptive selection layer FAS is added to adaptively adjust the contribution of the channel to improve the overall effect. Finally, the output of GCP can be expressed as: in represents global average pooling, is a 1×1 convolution, is the sigmoid function, Indicates the data after the result FAS layer processing.

[0060] Step 13, use a decoder to decode the point cloud coding features to obtain point cloud coordinate features at multiple density scales, and perform deep patching on each point cloud coordinate feature according to the auxiliary data to obtain a completed point cloud at each density scale.

[0061] The above-mentioned point cloud coordinate features are used to describe the coordinates of multiple data points at the corresponding density scale. The data points are points in the point cloud data. In related technologies, point cloud completion refers to the point cloud at a certain density scale (such as 10 points in a unit volume). The points in the point cloud are added through a certain algorithm or method to complete the missing parts. In this step, point cloud completion is performed at multiple density scales to meet the needs of completing point clouds of different density scales in actual tasks.

[0062] like Figure 4 As shown, the decoder includes a multi-layer feature decoding model, a registration model, and a depth completion model; The output end of the multi-layer feature decoding model and the output end of the registration model are both connected to the input end of the deep completion model. The input data of the input end of the multi-layer feature decoding model is the point cloud coding feature, the output data of the output end of the multi-layer feature decoding model is the point cloud coordinate feature at multiple density scales, the input data of the input end of the registration model is auxiliary data, and the output data of the output end of the deep completion model is the completed point cloud at multiple density scales.

[0063] The multi-layer feature decoding model includes a first mapping module, a second mapping module, a third mapping module, a first addition module, a second addition module, a third addition module, and a multi-layer perception module; The input end of the first mapping module, the input end of the second mapping module, the input end of the third mapping module, the input end of the first addition module, the input end of the second addition module, and the input end of the third addition module are all input ends of the multi-layer feature decoding model, and the output end of the multi-layer perception module is the output end of the multi-layer feature decoding model; The output end of the first mapping module is connected to the input end of the first adding module, the output end of the second mapping module is connected to the input end of the second adding module, and the output end of the third mapping module is connected to the input end of the third adding module.

[0064] It should be noted that the registration model is used to generate and register the auxiliary data for point cloud. The registration model adopts the iterative closest point (ICP) algorithm. The deep completion model is used to implement the deep patching of each point cloud coordinate feature according to the auxiliary data to obtain the step of completing the point cloud at each density scale. The multi-layer feature decoding model is used to implement the decoding of the point cloud encoding features to obtain the step of point cloud coordinate features at multiple density scales. The above-mentioned first mapping module, second mapping module, and third mapping module are all used to map the input data to the corresponding dimensions, and the dimensions corresponding to the first mapping module, the second mapping module, and the third mapping module are all different. Multiple dimensions correspond to multiple density scales one by one. The first addition module, the second addition module, and the third addition module are used to perform addition operations on the input data. The multi-layer perception module is used to perform multi-layer perceptron operations on the input data.

[0065] Exemplarily, the operation process in the above multi-layer feature decoding model is: First, the features of the three branches are expanded. For branch 1 (the first mapping module and the first addition module), the input feature After MLP processing, the output features of the same size are compared with the original features. Add point by point in the point dimension, and after reshape operation, output .

[0066] For branches 2 (the second mapping module and the second addition module) and 3 (the third mapping module and the third addition module), the input features First, map it to dimension through MLP and , and also the original features To expand, adopt the direct copy method (replicate r times and Then, the expanded mapping features and the original features are added point by point, and the added features are reshaped and finally output respectively. and .

[0067] The calculation process of the above branch can be expressed as: in, Represents the input data of the input branch, .

[0068] Then, a shared MLP is used to regress the three-dimensional coordinates of the point cloud. The MLP is designed to have 5 layers. Except for the input layer, the dimensions of each layer are [1024-512-64-3], and a Relu activation function is added after each layer.

[0069] Through this MLP structure, the input high-dimensional features will be mapped into three-dimensional coordinates. For each branch, point clouds of different densities will be generated, providing multi-scale results for the sparse point cloud completion task.

[0070] The above-mentioned depth completion model can be a multi-layer perceptron MLP. For each point cloud coordinate feature, the depth completion model and the registration model simultaneously realize the point cloud completion process as follows: (1) Generate deep point cloud by aligning the model The depth map (i.e. auxiliary data) is converted into three-dimensional space according to the camera's intrinsic parameter matrix to generate a point cloud in the camera coordinate system. For each pixel in the depth map and the corresponding depth value , its coordinates in three-dimensional space can be calculated as follows .

[0071] in, are the pixel coordinates in the depth map, is the principal point coordinate of the camera, and the parameters It uses pixels to describe the length of the focal length in the x-axis direction. It uses pixels to describe the length of the focal length in the y-axis direction. is the depth value of the pixel in the depth map.

[0072] Then, with the help of the external parameter matrix, the points in the camera coordinate system are transformed into the original three-dimensional point cloud space. Through the camera's rotation matrix R and translation vector t, the point set can be rotated and translated in the following way to map the points in the camera coordinate system to the global coordinate system: in, is the coordinate of the pixel in the camera coordinate system, R is the rotation matrix of the camera, t is the translation vector, It is the coordinate converted to the point cloud coordinate system.

[0073] (2) Two point cloud registration in the registration model Use the ICP algorithm to align the point cloud generated by the depth map and the point cloud regressed from the three-dimensional coordinates (i.e., the point cloud generated based on the point cloud coordinate features) to prepare for subsequent detail correction. The ICP algorithm is used to find the optimal rigid transformation (rotation and translation) between the two point clouds to minimize the point-to-point error of the two point clouds. The detailed operation steps are as follows: ① Select the initial point cloud: determine the point cloud regressed from the three-dimensional coordinates as the target point cloud , the point cloud generated by the depth map is the source point cloud .

[0074] ②Match the nearest neighbor point: for each source point , calculate its difference with each point in the target point cloud The distance between the two points is calculated, and the point with the smallest distance is selected to form a pairing point set. .

[0075] ; in, Represents the Euclidean distance between two points.

[0076] ③Calculate the transformation matrix: Assume the source point Transformed into , then the optimal R and t are obtained by minimizing the following objective function: in, represents the Euclidean distance.

[0077] ④ Apply transformation: transform each point in the source point cloud Applying this transformation results in a new aligned source point cloud, which makes the source point cloud as close to the target point cloud as possible.

[0078] ⑤ Repeated iteration: Continuously calculate the new transformation matrix and update the source point cloud position. When the loss function of each iteration, that is, the change in the mean square error function based on the Euclidean distance mentioned above, is lower than the set threshold or reaches the maximum number of iterations, the algorithm terminates.

[0079] (3) 3D coordinate adjustment in the depth completion model After obtaining the depth point cloud, the point cloud after 3D coordinate regression can be corrected in detail based on the depth point cloud to make it more accurate. Here, the MLP structure can be used to adjust the coordinates of the point cloud. The specific steps are as follows: ① Finding the neighborhood: point cloud after three-dimensional coordinate regression Each point in , according to K nearest neighbors, and depth point cloud Select k nearest neighbor points to form two neighborhoods and .

[0080] ②Coordinate correction: Use MLP to fuse the information from these K nearest points to “correct” Each point The input of MLP consists of two parts: point cloud coordinates and and The coordinate offset of the corresponding neighboring points. In general, for each point in the point cloud after 3D coordinate regression, MLP outputs the final corrected coordinate by averaging the MLP outputs of all its neighbors. : Among them, is the coordinate offset of the corresponding points in the two neighborhoods, It is a cascade operation of multiple vectors.

[0081] It is worth mentioning that obtaining point cloud coding features based on target point cloud data and auxiliary data can improve the information richness in the point cloud coding features, thereby improving the accuracy of the point cloud coordinate features obtained by decoding the point cloud coding features, and improving the accuracy of the completed point cloud obtained based on the precise point cloud coordinate features.

[0082] Step 14: construct a comprehensive loss function based on all completed point clouds, and use the comprehensive loss function to optimize the encoder and decoder to obtain an optimized encoder and an optimized decoder.

[0083] The above comprehensive loss function is used to describe the accuracy of all completed point clouds.

[0084] Specifically, the comprehensive loss function is: in, represents the value of the comprehensive loss function, and represents the hyperparameter, , represents the multi-scale completion loss, Represents the overall loss: in, represents the weighting parameter, Represents the probability of the discriminator's judgment on the generated completed point cloud, represents the completion of the point cloud, Represents the probability of the discriminator's judgment on the real point cloud, Indicates the real point cloud corresponding to the completed point cloud, represents the completion loss at the first density scale, represents the completion loss at the second density scale, Represents the completion loss at the third density scale: in, , Indicates The sampling point set of the completed point cloud at the density scale, express The number of sampling points in Indicates The sampling point set of the real point cloud at the density scale, express The number of sampling points in .

[0085] It should be noted that after constructing the comprehensive loss function, the completed point clouds of multiple target point cloud data used for training obtained through the above steps are substituted into the comprehensive loss function respectively to obtain the comprehensive loss function value corresponding to each target point cloud data, and then the average of these comprehensive loss function values ​​is calculated to determine whether the average value is less than or equal to the preset loss value. If so, the encoder is used as the optimized encoder and the decoder is used as the optimized decoder. Otherwise, the parameters in the encoder and decoder are adjusted, and the step of using the encoder to encode the target point cloud data and auxiliary data to obtain the point cloud coding features is returned.

[0086] In some embodiments of the present application, a discriminator can be used to analyze the difference between each generated completed point cloud and the real point cloud corresponding to the completed point cloud (with the same density scale as the completed point cloud) to improve the accuracy of the completed point cloud (the discriminator can give a judgment probability that the completed point cloud is a real point cloud, and the encoder and decoder can be trained according to the judgment probability, and the completed point cloud can be regenerated). Figure 5 As shown in Figure 1, the discriminator includes a serial mapping layer, a connection layer, a mapping layer, a maximum pooling layer, and a fully connected layer connected in sequence. The input data of the input end of the serial mapping layer and the connection layer is the completed point cloud, and the output data of the output end of the fully connected layer is the probability that the completed point cloud is a real point cloud. The calculation expression of the discriminator is: ; ; ; in, Represents the output data of the mapping layer, represents the completion of the point cloud, represents the operation of the serial mapping layer, represents the operation of the connection layer, represents the operation of the mapping layer, represents the maximum value under the jth channel, represents the global vector after maximum pooling, represents the operation of the fully connected layer, represents the activation function, Indicates the probability that the completed point cloud is a real point cloud.

[0087] Step 15, using the optimized encoder and the optimized decoder to complete the point cloud data to obtain the final completed point cloud.

[0088] The above-mentioned point cloud data to be completed is the point cloud data that needs to be completed.

[0089] Specifically, the optimized encoder is used to encode the point cloud data to be completed and the auxiliary data corresponding to the point cloud data to be completed to obtain the point cloud coding features, and then the optimized decoder is used to decode the point cloud coding features to obtain the point cloud coordinate features at multiple density scales, and each point cloud coordinate feature is deeply patched according to the auxiliary data to obtain the final completed point cloud at each density scale.

[0090] The method of the present application is illustrated below with reference to a specific example.

[0091] The process of the method of this application is as follows Figure 6 As shown in the figure, the lidar collects point cloud data, while the camera collects image data (i.e., auxiliary data), and then performs image enhancement feature learning (i.e., the processing process of the multi-scale feature extraction model in the encoder): the image data is enhanced and continuously convolved to obtain a feature map, and then the point cloud data is subjected to point cloud structure information learning (i.e., the processing process of the multi-scale feature fusion model in the encoder): on the one hand, global skeleton extraction is performed, and multi-head interactive attention processing is performed on the multimodal data; on the other hand, local structure extraction is performed, and multi-kernel edge convolution processing is performed on the multimodal data, and then point cloud coordinates are reconstructed (i.e., the processing process of the multi-layer feature decoding model in the decoder): feature expansion and three-dimensional coordinate regression are performed to obtain point cloud coordinate features, and then depth detail correction is performed (i.e., the processing process of the registration model and the depth completion model in the decoder): the depth map is corrected according to the point cloud after projection and ICP registration to obtain the completed point cloud, and finally the difference between the completed point cloud and the real point cloud is obtained through the discriminator module.

[0092] It is worth mentioning that obtaining point cloud coding features based on target point cloud data and auxiliary data can improve the information richness in the point cloud coding features, and then improve the accuracy of the point cloud coordinate features obtained by decoding the point cloud coding features. The accuracy of the completed point cloud obtained based on the precise point cloud coordinate features is improved, and constructing a comprehensive loss function to optimize the encoder and decoder can improve the performance of the encoder and decoder, and further improve the accuracy of point cloud completion.

[0093] In addition, the method of the present application extracts the information of auxiliary data by enhancing continuous convolution, captures the point cloud structure by multi-core edge convolution, and then fuses the features of the two modalities at an early stage. Then, with the help of two independent branch networks, the skeleton information and local structure of the point cloud are extracted and fused respectively, thereby enhancing the expressive ability of the features and achieving local and global consistency. At the same time, with the help of the point cloud generated by the auxiliary data, the multi-scale point cloud is repaired and adjusted in detail, thereby improving the accuracy of the completion, providing technical support for high-precision three-dimensional perception in the fields of unmanned driving, intelligent manufacturing, etc.

[0094] The following is an exemplary description of the point cloud completion device based on cross-modality and depth patching provided by the present application.

[0095] like Figure 7 As shown, an embodiment of the present application provides a point cloud completion device based on cross-modality and depth patching, and the point cloud completion device based on cross-modality and depth patching 700 includes: An acquisition module 701 is used to acquire target point cloud data and auxiliary data corresponding to the target point cloud data; The encoding module 702 is used to encode the target point cloud data and the auxiliary data using an encoder to obtain point cloud encoding features; the point cloud encoding features are used to describe the local structure information and global skeleton information of the target point cloud data; The decoding module 703 is used to decode the point cloud coding features using a decoder to obtain point cloud coordinate features at multiple density scales, and perform deep patching on each point cloud coordinate feature according to the auxiliary data to obtain a completed point cloud at each density scale; the point cloud coordinate feature is used to describe the coordinates of multiple data points at the corresponding density scale; The optimization module 704 is used to construct a comprehensive loss function according to all the completed point clouds, and use the comprehensive loss function to optimize the encoder and the decoder to obtain an optimized encoder and an optimized decoder; the comprehensive loss function is used to describe the accuracy of all the completed point clouds; The completion module 705 is used to complete the point cloud data to be completed using the optimized encoder and the optimized decoder to obtain the final completed point cloud.

[0096] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0097] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0098] like Figure 8 As shown, an embodiment of the present application provides a terminal device. The terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 8 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0099] Specifically, when the processor D100 executes the computer program D102, by obtaining point cloud coding features based on the target point cloud data and auxiliary data, it is possible to improve the information richness in the point cloud coding features, thereby improving the accuracy of the point cloud coordinate features obtained by decoding the point cloud coding features. The accuracy of the completed point cloud obtained based on the precise point cloud coordinate features is improved, and a comprehensive loss function is constructed to optimize the encoder and decoder, which can improve the performance of the encoder and decoder and further improve the accuracy of point cloud completion.

[0100] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0101] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0102] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0103] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0104] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium that can carry the computer program code to the point cloud completion method device / terminal device based on cross-modality and deep patching. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0105] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0107] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A point cloud completion method based on cross-modality and deep inpainting, characterized in that: include: Acquire target point cloud data and auxiliary data corresponding to the target point cloud data; Encoding the target point cloud data and the auxiliary data using an encoder to obtain point cloud coding features; the point cloud coding features are used to describe local structural information and global skeleton information of the target point cloud data; Decoding the point cloud coding features using a decoder to obtain point cloud coordinate features at multiple density scales, and performing deep patching on each of the point cloud coordinate features according to the auxiliary data to obtain a completed point cloud at each density scale; the point cloud coordinate features are used to describe the coordinates of multiple data points at the corresponding density scale; Constructing a comprehensive loss function according to all completed point clouds, and optimizing the encoder and the decoder using the comprehensive loss function to obtain an optimized encoder and an optimized decoder; the comprehensive loss function is used to describe the accuracy of all completed point clouds; The optimized encoder and the optimized decoder are used to complete the point cloud data to obtain the final completed point cloud.

2. The point cloud completion method according to claim 1, characterized in that: The encoder includes a multi-layer feature extraction model and a multi-layer feature fusion model; The input data of the first input end of the multi-layer feature extraction model is the auxiliary data, the input data of the second input end of the multi-layer feature extraction model is the target point cloud data, and the output end of the multi-layer feature extraction model is connected to the first input end of the multi-layer feature fusion model; The input data of the second input end of the multi-layer feature fusion model is the target point cloud data, and the output data of the output end of the multi-layer feature fusion model is the point cloud coding feature.

3. The point cloud completion method according to claim 2, characterized in that: The multi-layer feature extraction model includes: a first residual neural module, a second residual neural module, a third residual neural module, a first upsampling module, a second upsampling module, a first enhanced continuous convolution module, a second enhanced continuous convolution module and a third enhanced continuous convolution module; The input end of the first residual neural module is the first input end of the multi-layer feature extraction model, the input end of the first enhanced continuous convolution module is the second input end of the multi-layer feature extraction model, and the output end of the first enhanced continuous convolution module, the output end of the second enhanced continuous convolution module and the output end of the third enhanced continuous convolution module are all output ends of the multi-layer feature extraction model; The output end of the first residual neural module is respectively connected to the input end of the second residual neural module and the input end of the first enhanced continuous convolution module, the output end of the second residual neural module is respectively connected to the input end of the third residual neural module and the input end of the first upsampling module, the output end of the first upsampling module is connected to the input end of the second enhanced continuous convolution module, the output end of the third residual neural module is connected to the input end of the second upsampling module, the output end of the second upsampling module is connected to the input end of the third enhanced continuous convolution module, the output end of the first enhanced continuous convolution module is connected to the input end of the second enhanced continuous convolution module, and the output end of the second enhanced continuous convolution module is connected to the input end of the third enhanced continuous convolution module.

4. The point cloud completion method according to claim 2, characterized in that: The multi-layer feature fusion model includes: a first multi-core edge convolution module, a second multi-core edge convolution module, a third multi-core edge convolution module, a fourth multi-core edge convolution module, a first cascade module, a second cascade module, a third cascade module, a first global structure perception module, a second global structure perception module, a clustering module, a maximum pooling module, a first expansion module, and a second expansion module; The input end of the first multi-core edge convolution module is the second input end of the multi-layer feature fusion model, the first input end of the first cascade module is the first input end of the multi-layer feature fusion model, and the output end of the third cascade module is the output end of the multi-layer feature fusion module; The output end of the first multi-core edge convolution module is respectively connected to the input end of the second multi-core edge convolution module and the input end of the second cascade module, the output end of the second multi-core edge convolution module is respectively connected to the input end of the third multi-core edge convolution module and the input end of the second cascade module, the output end of the third multi-core edge convolution module is respectively connected to the second input end of the first cascade module and the input end of the second cascade module, the output end of the first cascade module is respectively connected to the input end of the fourth multi-core edge convolution module, the input end of the first global structure perception module, and the input end of the clustering module, the output end of the fourth multi-core edge convolution module is respectively connected to the output end of the third cascade module. The input end of the cascade module is connected to the input end of the first global structure perception module, the output end of the first global structure perception module is connected to the input end of the second global structure perception module, the output end of the second global structure perception module is connected to the input end of the first expansion module, the output end of the first expansion module is connected to the input end of the third cascade module, the output end of the second cascade module is connected to the input end of the maximum pooling module, the output end of the maximum pooling module is connected to the input end of the second expansion module, and the output end of the second expansion module is connected to the input end of the third cascade module.

5. The point cloud completion method according to claim 4, characterized in that: The first global structure perception module and the second global structure perception module are both global structure perception modules; The global structure perception module includes a convolution layer, a first nonlinear transformation layer, a second nonlinear transformation layer, a third nonlinear transformation layer, a multi-head interactive attention layer, a first residual normalization layer, a feature adaptive selection layer, and a second residual normalization layer; The input end of the convolutional layer is the input end of the global structure perception module, and the output end of the second residual normalization layer is the output end of the global structure perception module; The output end of the convolutional layer is respectively connected to the input end of the first nonlinear transformation layer, the input end of the second nonlinear transformation layer, the input end of the third nonlinear transformation layer, and the input end of the first residual normalization layer; the input end of the multi-head interactive attention layer is respectively connected to the output end of the first nonlinear transformation layer, the output end of the second nonlinear transformation layer, and the output end of the third nonlinear transformation layer; the output end of the multi-head interactive attention layer is connected to the input end of the first residual normalization layer; the output end of the first residual normalization layer is respectively connected to the input end of the second residual normalization layer and the input end of the feature adaptive selection layer; the output end of the feature adaptive selection layer is connected to the input end of the second residual normalization layer.

6. The point cloud completion method according to claim 1, characterized in that: The decoder includes a multi-layer feature decoding model, a registration model, and a depth completion model; The output end of the multi-layer feature decoding model and the output end of the registration model are both connected to the input end of the depth completion model, the input data of the input end of the multi-layer feature decoding model is the point cloud coding feature, the output data of the output end of the multi-layer feature decoding model is the point cloud coordinate feature under multiple density scales, the input data of the input end of the registration model is the auxiliary data, and the output data of the output end of the depth completion model is the completed point cloud under multiple density scales.

7. The point cloud completion method according to claim 6, characterized in that: The multi-layer feature decoding model includes a first mapping module, a second mapping module, a third mapping module, a first addition module, a second addition module, a third addition module, and a multi-layer perception module; The input end of the first mapping module, the input end of the second mapping module, the input end of the third mapping module, the input end of the first adding module, the input end of the second adding module, and the input end of the third adding module are all input ends of the multi-layer feature decoding model, and the output end of the multi-layer perception module is the output end of the multi-layer feature decoding model; The output end of the first mapping module is connected to the input end of the first adding module, the output end of the second mapping module is connected to the input end of the second adding module, and the output end of the third mapping module is connected to the input end of the third adding module.

8. The point cloud completion method according to claim 1, characterized in that: The comprehensive loss function is: in, represents the value of the comprehensive loss function, and represents the hyperparameter, , represents the multi-scale completion loss, Represents the overall loss: in, represents the weighting parameter, Represents the probability of the discriminator's judgment on the generated completed point cloud, represents the completion of the point cloud, Represents the probability of the discriminator's judgment on the real point cloud, Indicates the real point cloud corresponding to the completed point cloud, represents the completion loss at the first density scale, represents the completion loss at the second density scale, Represents the completion loss at the third density scale: in, , Indicates The sampling point set of the completed point cloud at the density scale, express The number of sampling points in Indicates The sampling point set of the real point cloud at the density scale, express The number of sampling points in .

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the point cloud completion method based on cross-modality and depth patching as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the point cloud completion method based on cross-modality and depth patching as described in any one of claims 1 to 8 is implemented.

Images