Multi-stage Mama point cloud completion method and device based on semantic and geometric guidance

Abstract

The invention discloses a multi-stage Mama point cloud completion method and device based on semantic and geometric guidance, and relates to the field of point cloud processing, and the method comprises the steps: constructing and training a multi-stage Mama point cloud completion model based on semantic and geometric guidance, and obtaining a trained multi-stage Mama point cloud completion model; the multi-sorting strategy Mama decoder unit comprises a plurality of stages of multi-sorting strategy Mama decoders which are connected in sequence; obtaining an incomplete point cloud to be complemented and inputting the incomplete point cloud to the trained multi-stage Mama point cloud complementation model, enabling the incomplete point cloud to pass through a Transform-Mama combined point cloud local feature coding unit to obtain coding features, and inputting the coding features into a sparse point cloud generation unit to obtain sparse point cloud; the sparse point clouds are input into the multi-sorting strategy Mama decoder unit to obtain decoding features, the decoding features pass through the point cloud up-sampling unit to obtain predicted complete point clouds, and the problems of quadratic complexity and local detail loss of an existing Transform encoder-decoder structure are solved.

Description

Technical Field

[0001] The present invention relates to the field of point cloud processing, and in particular to a multi-stage Mamba point cloud completion method and device based on semantics and geometry guidance. Background Art

[0002] Point cloud completion is one of the key tasks in computer vision and point cloud processing, which aims to generate complete and high-quality point clouds from incomplete and low-quality point clouds. Since Convolutional Neural Network (CNN) has demonstrated its powerful feature representation, end-to-end trainable paradigm, and excellent performance in two-dimensional tasks, early point cloud completion work attempted to migrate mature methods from two-dimensional completion tasks to three-dimensional point clouds through voxelization and three-dimensional convolution. However, these methods face huge computational costs and grow cubically with increasing spatial resolution. With the emergence of PointNet and PointNet++, directly processing three-dimensional point cloud coordinates has become the mainstream of point cloud analysis tasks.

[0003] In recent years, Transformer has become the mainstream framework for point cloud completion tasks because it can better learn the structural features and long-range correlations between local parts of point clouds. PoinTr regards point cloud completion as a set-to-set conversion problem and proposes a Transformer encoder-decoder structure for point cloud completion. By representing the point cloud as a set of unordered points with position embeddings, the point cloud can be converted into a series of local combinations. At the same time, in order to facilitate Transformer to better utilize the inductive bias of the three-dimensional geometric structure of the point cloud, PoinTr further designs a geometric perception module that can explicitly model local geometric relationships.

[0004] However, the quadratic complexity of the attention mechanism in Transformer hinders the scalability of the completion network in long sequence tasks. At the same time, the pooling operation of the encoder will cause the loss of local details, making it difficult to obtain a complete point cloud with fine-grained local details. Summary of the invention

[0005] The purpose of this application is to propose a multi-stage Mamba point cloud completion method and device based on semantics and geometry guidance to address the above-mentioned technical problems, so as to overcome the problems of quadratic complexity and local detail loss of the existing Transformer encoder-decoder structure.

[0006] In a first aspect, the present invention provides a multi-stage Mamba point cloud completion method based on semantics and geometry guidance, comprising the following steps:

[0007] A multi-stage Mamba point cloud completion model based on semantics and geometry guidance is constructed and trained to obtain a trained multi-stage Mamba point cloud completion model; the multi-stage Mamba point cloud completion model includes a sequentially connected Transformer-Mamba joint point cloud local feature encoding unit, a sparse point cloud generation unit, a multi-sorting strategy Mamba decoder unit and a point cloud upsampling unit; the sparse point cloud generation unit includes a plurality of sequentially connected skeleton point generation calibration layers; the multi-sorting strategy Mamba decoder unit includes a plurality of sequentially connected stages of multi-sorting strategy Mamba decoders, and each stage of the multi-sorting strategy Mamba decoder includes four sequentially connected bidirectional Mamba layers;

[0008] The incomplete point cloud to be completed is obtained and input into the trained multi-stage Mamba point cloud completion model. The incomplete point cloud passes through the Transformer-Mamba joint point cloud local feature encoding unit to obtain the encoded features, which are input into the sparse point cloud generation unit to obtain the sparse point cloud; the sparse point cloud is input into the multi-sorting strategy Mamba decoder unit. The multi-sorting strategy Mamba decoder of each stage adopts 4 sorting strategies, which are randomly and non-repeatedly assigned to 4 bidirectional Mamba layers to learn the spatial information of different angles in the sparse point cloud to obtain the decoding features. The decoding features pass through the point cloud upsampling unit to obtain the predicted complete point cloud.

[0009] Preferably, the point cloud local feature encoding unit of the Transformer-Mamba joint includes a farthest point sampling layer, a K nearest neighbor query layer, an intra-block Transformer encoding unit, an importance score prediction layer and an inter-block Mamba encoding unit connected in sequence, and the intra-block Transformer encoding unit includes a sequentially connected The intra-block Transformer encoder and the inter-block Mamba encoding unit consist of sequentially connected The importance score prediction layer includes a nonlinear projection layer and a score prediction layer connected in sequence. The nonlinear projection layer is composed of the first convolution layer, the first batch of normalization layers, the first ReLU activation function layer, the second convolution layer, and the layer normalization layer. The score prediction layer is composed of the third convolution layer, the second batch of normalization layers, the second ReLU activation function layer, and the fourth convolution layer. The calculation process of the Transformer-Mamba joint point cloud local feature encoding unit is as follows:

[0010] Input the incomplete point cloud into the farthest point sampling layer, select M center points and construct the center point set , and select the K points closest to each center point through the K nearest neighbor query layer to construct a local point cloud set to obtain a local point cloud block ,in, represents the set of real numbers, represents the mth local point cloud block of the downsampling, M represents the total number of center points, represents the kth nearest neighbor point in the local point cloud block, K represents the total number of nearest neighbor points, Represents the coordinates of the kth nearest neighbor point in the mth local point cloud block, as shown in the following formula:

[0011] ;

[0012] in, Represents the function corresponding to the farthest point sampling layer, Represents the function corresponding to the K nearest neighbor query layer, Indicates the coordinates of incomplete point cloud;

[0013] The local point cloud block Input the Transformer encoding unit in the block to obtain the local point cloud block features , the calculation expression of the Transformer encoder in the i-th block is as follows:

[0014]

[0015]

[0016] Among them, C represents the feature dimension, represents the Transformer encoder in the i-th block, Represents the number of Transformer encoders in the block, Representation layer normalization operation, Represents the function corresponding to the multi-head self-attention layer, represents the function corresponding to the feedforward network, represents the output features of the multi-head self-attention unit in the Transformer encoder in the i-th block, represents the point cloud block features output by the Transformer encoder in the i-1th block, Represents the point cloud block features output by the Transformer encoder in the i-th block, hour, for Repeat the above steps until the point cloud block features output by the Transformer encoder in the last block are obtained and used as the local point cloud block features ;

[0017] Local point cloud block features Input to the importance score prediction layer to obtain the importance score of the local point cloud block, as shown in the following formula:

[0018] ;

[0019] ;

[0020] in, represents the nonlinear projection layer, Represents the importance score of the local point cloud block, represents the function corresponding to the score prediction layer, Represents the local point cloud block features after projection;

[0021] According to the importance score of the local point cloud block, the local point cloud block features are Sort and get the sorted local point cloud block features And input the inter-block Mamba encoding unit to model the dependencies between local point cloud blocks. The expression of the inter-block Mamba encoder is as follows:

[0022] ;

[0023] ;

[0024] in, represents the i'th inter-block Mamba encoder, Indicates the number of inter-block Mamba encoders, represents the output features of the bidirectional Mamba unit in the i'th inter-block Mamba encoder, Represents the function corresponding to the bidirectional Mamba layer, represents the local point cloud block features enhanced by the i'th inter-block Mamba encoder, represents the local point cloud block feature enhanced by the i'-1th inter-block Mamba encoder. When i'=1, is the sorted local point cloud block feature ;

[0025] The bidirectional Mamba unit includes a layer normalization operation, a first branch, a second branch and a first linear projection layer. The first branch includes a second linear projection layer and a first SiLU activation function layer connected in sequence. The second branch includes a third linear projection layer, a depth-separable convolution layer, a second SiLU activation function layer, and a cascade of parallel forward state space equation units and reverse state space equation units. The forward state space equation unit includes a forward state space equation and a layer normalization operation connected in sequence. The reverse state space equation unit includes a reverse state space equation and a layer normalization operation connected in sequence. The first linear projection layer, the second linear projection layer and the third linear projection layer all use multi-layer perceptrons. The calculation process of the bidirectional Mamba unit is as follows:

[0026] ;

[0027] ;

[0028] in, Indicates the hidden state, represents the function corresponding to the depthwise separable convolutional layer, represents the function corresponding to the multi-layer perceptron layer, represents the forward state space equation, represents the inverse state-space equation, represents the SiLU activation function;

[0029] The enhanced local point cloud block features output by the last inter-block Mamba encoder are used as encoding features .

[0030] Preferably, each layer of the skeleton point generation calibration layer includes a semantically guided skeleton point cloud generation module and a geometrically guided skeleton point cloud calibration module connected in sequence, and the input feature of the skeleton point generation calibration layer of the current layer is a coded feature or a feature formed by connecting the input feature of the skeleton point generation calibration layer of the previous layer and the calibrated skeleton point feature obtained by the skeleton point cloud calibration module of the previous layer, as shown in the following formula:

[0031] ;

[0032] in, Indicates a connection operation. , Indicates the total number of skeleton point generated calibration layers; when hour, is the encoding feature, Represents the input features of the j-th layer skeleton point generation calibration layer, Represents the input features of the j-1th layer skeleton point generation calibration layer, Represents the calibration skeleton point features obtained by the skeleton point cloud calibration module of the j-1th layer;

[0033] In the skeleton point generation calibration layer of the current layer, the input features of the skeleton point generation calibration layer of the current layer first pass through the semantically guided skeleton point cloud generation module of the current layer to obtain the first skeleton point prediction features of the current layer and the skeleton point coordinates of the current layer, and the skeleton point coordinates of the current layer, the input point set of the previous layer, the first skeleton point prediction features of the current layer and the input features of the skeleton point generation calibration layer of the current layer are input into the geometrically guided skeleton point cloud calibration module of the current layer to obtain the second skeleton point prediction features of the current layer and the calibrated skeleton point coordinates of the current layer; the calibrated skeleton point coordinates of the current layer are connected with the input point set of the previous layer to obtain the input point set of the current layer and input it into the skeleton point generation calibration layer of the next layer, the skeleton point prediction features of the current layer are connected with the calibrated skeleton point features of the previous layer to obtain the calibrated skeleton point features of the current layer and input it into the skeleton point generation calibration layer of the next layer, and the above steps are repeated until the skeleton point coordinates generated by the skeleton point generation calibration layer of the last layer are obtained and used as the sparse point cloud coordinates, and the calibrated skeleton point features generated by the skeleton point generation calibration layer of the last layer are obtained and used as the sparse point cloud features, and the sparse point cloud coordinates and the sparse point cloud features constitute a sparse point cloud.

[0034] Preferably, the input features of the skeleton point generation calibration layer of the current layer are first passed through the semantically guided skeleton point cloud generation module of the current layer to obtain the first skeleton point prediction features of the current layer and the skeleton point coordinates of the current layer, and the skeleton point coordinates of the current layer, the input point set of the previous layer, the first skeleton point prediction features of the current layer and the input features of the skeleton point generation calibration layer of the current layer are input into the geometrically guided skeleton point cloud calibration module of the current layer to obtain the second skeleton point prediction features of the current layer and the calibrated skeleton point coordinates of the current layer, specifically including:

[0035] In the j-th layer semantically guided skeleton point cloud generation module, the j-th layer skeleton points are firstly used to generate the input features of the calibration layer. Perform pooling operation to obtain global features , and the skeleton points of the jth layer generate the input features of the calibration layer and global features The feature difference between them is input to the fourth linear projection layer for linear projection, and we get Skeleton point features of the current layer , which is expressed as follows:

[0036] ;

[0037] ;

[0038] in, Represents the function corresponding to the pooling layer in the semantically guided skeleton point cloud generation module of the jth layer, the pooling layer in the semantically guided skeleton point cloud generation module of the first layer is an importance-aware pooling layer, and the importance-aware pooling layer is used to re-weight and sum the importance scores predicted by the importance score prediction layer, and the pooling layers in the semantically guided skeleton point cloud generation modules of the remaining layers are maximum pooling layers;

[0039] Generate the input features of the calibration layer from the skeleton points of the jth layer and the skeleton point features of the current layer After connection, the farthest point sampling and the closest point sampling of the features are sorted, and the skeleton point features are modeled in two Mamba units. and input features The semantic correlation between them is used to obtain the first skeleton point prediction feature of the jth layer with semantic interaction information. , which is expressed as follows:

[0040] ;

[0041] in, Indicates the farthest feature point sampling, Indicates the nearest feature point sampling, Indicates a connection operation. Represents two Mamba units. The calculation process of two Mamba units is as follows:

[0042] ;

[0043] ;

[0044] ;

[0045] in, is the function corresponding to the RMS normalization layer, Represents the input features of the j-th layer skeleton point generation calibration layer and the skeleton point features of the current layer Input features after sampling and sorting the farthest feature points after connection , Represents the input features of the j-th layer skeleton point generation calibration layer and the skeleton point features of the current layer Input features after connection and feature nearest point sampling and sorting , represents the state space equation;

[0046] The first skeleton point prediction feature of the jth layer Input to the fifth linear projection layer, the prediction is The coordinates of the skeleton points at the jth layer , whose expression is as follows:

[0047] ;

[0048] Among them, the fourth linear projection layer and the fifth linear projection layer adopt multi-layer perceptron;

[0049] In the j-th layer geometry-guided skeleton point cloud calibration module, the j-th layer skeleton point coordinates and the input point set of the j-1th layer After connection, the farthest point sampling and the nearest point sampling are sorted respectively, and the first skeleton point of the jth layer is predicted according to the sorting results. And the skeleton points of the jth layer generate the input features of the calibration layer The connection results are sorted and input into two Mamba units to model the geometric correlation between the two, and the second skeleton point prediction feature of the jth layer containing spatial geometric information is obtained. , whose expression is as follows:

[0050] ;

[0051] in, represents the farthest point sampling, represents the nearest point sampling, and Respectively represent the sampling sorting by the farthest point and the sampling sorting by the nearest point. Input point set The center point set ;

[0052] Secondly, the second skeleton point prediction feature of the jth layer With global features After adding, input the sixth linear projection layer to get the point offset , and the skeleton point coordinates of the jth layer obtained in the semantically guided skeleton point cloud generation unit Add together to get the calibrated skeleton point coordinates of the jth layer , whose expression is as follows:

[0053] ;

[0054] ;

[0055] Among them, the sixth linear projection layer adopts a multi-layer perceptron.

[0056] Preferably, the calculation expression of the decoding feature is:

[0057] ;

[0058] ;

[0059] in, represents random assignment, Indicates the sorting method. , , and They are Z-curve sorting strategy, reverse Z-curve sorting strategy, Hilbert sorting strategy and reverse Hilbert sorting strategy. represents the function corresponding to the bidirectional Mamba layer executed in the mth stage according to the nth sorting method, , , represents the total number of stages in the multi-order strategy Mamba decoder unit, represents sparse point cloud features, represents the decoding feature;

[0060] The decoded features are passed through the point cloud upsampling unit to obtain the predicted complete point cloud, which includes:

[0061] Using decoding features Compute the first affine parameter and the second affine parameter , and guide the two-dimensional grid with an affine function Deformation, three-dimensional offset obtained by three-dimensional deformation At the same time, each point coordinate in the sparse point cloud coordinates is copied O times and added to the three-dimensional offset to finally get the predicted complete point cloud , whose expression is as follows:

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] in, represents the maximum pooling operation, Represents the function corresponding to the multilayer perceptron, and Represents a two-dimensional grid The mean and standard deviation of Indicates a copy operation. represents the sparse point cloud coordinates, Represents the first A sparse point, Indicates The decoding features of sparse points, represents the ReLU activation function, Indicates The second affine parameter corresponding to the sparse points, Indicates The point coordinates of sparse points, Indicated by The upsampled points generated by the sparse points.

[0068] Preferably, the total loss function used in the training process of the multi-stage Mamba point cloud completion model is the sum of the importance loss and the chamfer loss. The calculation process of the importance loss is shown in the following formula:

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] in, Represents the function corresponding to the pooling layer. The pooling layer here is the importance-aware pooling layer. The importance-aware pooling layer is used to re-weight and sum the importance scores predicted by the importance score prediction layer. represents the global encoding feature, represents the global encoding feature after projection, represents the true importance score calculated by cosine similarity, represents L1 smoothing loss, Indicates loss of importance;

[0074] The calculation process of chamfer loss is as follows:

[0075] ;

[0076] in, represents the chamfer loss, Represents the true complete point cloud, represents the number of points in the predicted complete point cloud, Represents the number of points of the true complete point cloud, represents the complete point cloud of the prediction, and Respectively represent one point in the predicted complete point cloud and one point in the real complete point cloud, Represents the square of the Euclidean distance.

[0077] In a second aspect, the present invention provides a multi-stage Mamba point cloud completion device based on semantics and geometry guidance, comprising:

[0078] A model building module is configured to build and train a multi-stage Mamba point cloud completion model based on semantics and geometry guidance to obtain a trained multi-stage Mamba point cloud completion model; the multi-stage Mamba point cloud completion model includes a sequentially connected Transformer-Mamba joint point cloud local feature encoding unit, a sparse point cloud generation unit, a multi-sorting strategy Mamba decoder unit and a point cloud upsampling unit; the sparse point cloud generation unit includes a plurality of sequentially connected skeleton point generation calibration layers; the multi-sorting strategy Mamba decoder unit includes a plurality of sequentially connected stages of multi-sorting strategy Mamba decoders, and each stage of the multi-sorting strategy Mamba decoder includes four sequentially connected bidirectional Mamba layers;

[0079] The completion module is configured to obtain the incomplete point cloud to be completed and input it into the trained multi-stage Mamba point cloud completion model. The incomplete point cloud passes through the Transformer-Mamba joint point cloud local feature encoding unit to obtain the encoded features, and the encoded features are input into the sparse point cloud generation unit to obtain the sparse point cloud; the sparse point cloud is input into the multi-sorting strategy Mamba decoder unit. The multi-sorting strategy Mamba decoder in each stage adopts 4 sorting strategies, and the 4 sorting strategies are randomly and non-repeatedly assigned to 4 bidirectional Mamba layers to learn the spatial information of different angles in the sparse point cloud to obtain the decoding features. The decoding features pass through the point cloud upsampling unit to obtain the predicted complete point cloud.

[0080] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0081] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0082] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] (1) The semantically and geometrically guided multi-stage Mamba point cloud completion method proposed in this paper introduces the Mamba framework with improved global attention and linear complexity, and serializes the input point cloud through a point cloud sorting strategy jointly guided by semantics and geometry, giving full play to the Mamba framework's ability to model causal sequences.

[0085] (2) The semantically and geometrically guided multi-stage Mamba point cloud completion method proposed in this paper designs a skeleton point generation and calibration module, which converts point cloud completion into a stage-by-stage generation process, encourages the interaction between local features, and significantly improves the detail reconstruction capability of the point cloud completion method.

[0086] (3) The multi-stage Mamba point cloud completion method based on semantic and geometric guidance proposed in the present invention combines the importance loss and chamfer loss to construct a total loss function, and uses the total loss function to train the multi-stage Mamba point cloud completion model, which not only constrains the generation of importance scores, but also constrains the final point cloud generation, thereby greatly improving the prediction accuracy of the trained multi-stage Mamba point cloud completion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0088] Figure 1 A flowchart of a multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to an embodiment of the present application;

[0089] Figure 2 A schematic diagram of a multi-stage Mamba point cloud completion model of a multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to an embodiment of the present application;

[0090] Figure 3 A schematic diagram of a point cloud local feature encoding unit of a Transformer-Mamba joint method for completing a multi-stage Mamba point cloud based on semantics and geometry guidance according to an embodiment of the present application;

[0091] Figure 4 A schematic diagram of a bidirectional Mamba unit of a multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to an embodiment of the present application;

[0092] Figure 5A schematic diagram of a semantically guided skeleton point cloud generation module of a semantically and geometrically guided multi-stage Mamba point cloud completion method according to an embodiment of the present application;

[0093] Figure 6 A schematic diagram of a geometry-guided skeleton point cloud calibration module of a semantically and geometry-guided multi-stage Mamba point cloud completion method according to an embodiment of the present application;

[0094] Figure 7 A schematic diagram of a multi-sorting strategy Mamba decoder of a multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to an embodiment of the present application;

[0095] Figure 8 A schematic diagram of a point cloud upsampling unit of a multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to an embodiment of the present application;

[0096] Fig. 9 A schematic diagram of a calculation process of a three-dimensional offset of a multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to an embodiment of the present application;

[0097] Fig.10 A schematic diagram of a calculation process of importance loss of a multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to an embodiment of the present application;

[0098] Fig.11 A schematic diagram of a multi-stage Mamba point cloud completion device based on semantics and geometry guidance according to an embodiment of the present application;

[0099] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0100] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0101] Figure 1 A multi-stage Mamba point cloud completion method based on semantics and geometry guidance provided by an embodiment of the present application is shown, comprising the following steps:

[0102] S1, construct and train a multi-stage Mamba point cloud completion model based on semantics and geometry guidance to obtain a trained multi-stage Mamba point cloud completion model; the multi-stage Mamba point cloud completion model includes a Transformer-Mamba joint point cloud local feature encoding unit, a sparse point cloud generation unit, a multi-sorting strategy Mamba decoder unit and a point cloud upsampling unit connected in sequence; the sparse point cloud generation unit includes several layers of skeleton point generation calibration layers connected in sequence; the multi-sorting strategy Mamba decoder unit includes several stages of multi-sorting strategy Mamba decoders connected in sequence, and the multi-sorting strategy Mamba decoder of each stage includes 4 bidirectional Mamba layers connected in sequence.

[0103] Specifically, refer to Figure 2 An embodiment of the present application proposes a multi-stage Mamba point cloud completion model, which includes a sequentially connected Transformer-Mamba joint point cloud local feature encoding unit, a sparse point cloud generation unit, a multi-sorting strategy Mamba decoder unit and a point cloud upsampling unit.

[0104] In a specific embodiment, the point cloud local feature encoding unit of the Transformer-Mamba joint includes a farthest point sampling layer, a K nearest neighbor query layer, an intra-block Transformer encoding unit, an importance score prediction layer, and an inter-block Mamba encoding unit connected in sequence, and the intra-block Transformer encoding unit includes a sequentially connected The intra-block Transformer encoder and the inter-block Mamba encoding unit consist of sequentially connected The importance score prediction layer includes a nonlinear projection layer and a score prediction layer connected in sequence. The nonlinear projection layer is composed of the first convolution layer, the first batch of normalization layers, the first ReLU activation function layer, the second convolution layer, and the layer normalization layer. The score prediction layer is composed of the third convolution layer, the second batch of normalization layers, the second ReLU activation function layer, and the fourth convolution layer. The calculation process of the Transformer-Mamba joint point cloud local feature encoding unit is as follows:

[0105] Input the incomplete point cloud into the farthest point sampling layer, select M center points and construct the center point set , and select the K points closest to each center point through the K nearest neighbor query layer to construct a local point cloud set to obtain a local point cloud block ,in, represents the set of real numbers, represents the mth local point cloud block of the downsampling, M represents the total number of center points, represents the kth nearest neighbor point in the local point cloud block, K represents the total number of nearest neighbor points, Represents the coordinates of the kth nearest neighbor point in the mth local point cloud block, as shown in the following formula:

[0106] ;

[0107] in, Represents the function corresponding to the farthest point sampling layer, Represents the function corresponding to the K nearest neighbor query layer, Indicates the coordinates of incomplete point cloud;

[0108] The local point cloud block Input the Transformer encoding unit in the block to obtain the local point cloud block features , the calculation expression of the Transformer encoder in the i-th block is as follows:

[0109]

[0110]

[0111] Among them, C represents the feature dimension, represents the Transformer encoder in the i-th block, Represents the number of Transformer encoders in the block, Representation layer normalization operation, Represents the function corresponding to the multi-head self-attention layer, represents the function corresponding to the feedforward network, represents the output features of the multi-head self-attention unit in the Transformer encoder in the i-th block, represents the point cloud block features output by the Transformer encoder in the i-1th block, Represents the point cloud block features output by the Transformer encoder in the i-th block, hour, for Repeat the above steps until the point cloud block features output by the Transformer encoder in the last block are obtained and used as the local point cloud block features ;

[0112] Local point cloud block features Input to the importance score prediction layer to obtain the importance score of the local point cloud block, as shown in the following formula:

[0113] ;

[0114] ;

[0115] in, represents the nonlinear projection layer, Represents the importance score of the local point cloud block, represents the function corresponding to the score prediction layer, Represents the local point cloud block features after projection;

[0116] According to the importance score of the local point cloud block, the local point cloud block features are Sort and get the sorted local point cloud block features And input the inter-block Mamba encoding unit to model the dependencies between local point cloud blocks. The expression of the inter-block Mamba encoder is as follows:

[0117] ;

[0118] ;

[0119] in, represents the i'th inter-block Mamba encoder, Indicates the number of inter-block Mamba encoders, represents the output features of the bidirectional Mamba unit in the i'th inter-block Mamba encoder, Represents the function corresponding to the bidirectional Mamba layer, represents the local point cloud block features enhanced by the i'th inter-block Mamba encoder, represents the local point cloud block feature enhanced by the i'-1th inter-block Mamba encoder. When i'=1, is the sorted local point cloud block feature ;

[0120] The bidirectional Mamba unit includes a layer normalization operation, a first branch, a second branch and a first linear projection layer. The first branch includes a second linear projection layer and a first SiLU activation function layer connected in sequence. The second branch includes a third linear projection layer, a depth-separable convolution layer, a second SiLU activation function layer, and a cascade of parallel forward state space equation units and reverse state space equation units. The forward state space equation unit includes a forward state space equation and a layer normalization operation connected in sequence. The reverse state space equation unit includes a reverse state space equation and a layer normalization operation connected in sequence. The first linear projection layer, the second linear projection layer and the third linear projection layer all use multi-layer perceptrons. The calculation process of the bidirectional Mamba unit is as follows:

[0121] ;

[0122] ;

[0123] in, Indicates the hidden state, represents the function corresponding to the depthwise separable convolutional layer, represents the function corresponding to the multi-layer perceptron layer, represents the forward state space equation, represents the inverse state-space equation, represents the SiLU activation function;

[0124] The enhanced local point cloud block features output by the last inter-block Mamba encoder are used as encoding features .

[0125] Specifically, refer to Figure 3 The Transformer-Mamba joint point cloud local feature encoding unit is constructed based on the standard Mamba model by adding three key designs: (1) intra-block Transformer encoding unit; (2) importance score prediction layer; (3) inter-block Mamba encoding unit.

[0126] Specifically, by introducing the intra-block Transformer encoding unit to locally aggregate local point cloud blocks, local point cloud block features are obtained to solve the problem of insufficient local modeling ability of the Mamba model. Then, the local point cloud block features are calculated and sorted through the importance score prediction layer to reduce the adverse effects of random sorting on the Mamba model. Finally, the sorted local point cloud block features are scanned from the front and back directions through the inter-block Mamba encoding unit to achieve bidirectional interaction of the local point cloud block sequence. Therefore, the proposed intra-block Transformer encoding unit and importance score prediction layer can provide the inter-block Mamba encoding unit with an enhanced representation of the serialized local point cloud blocks, ensuring that each enhanced local point cloud block feature fully aggregates information from all other local point cloud blocks. Reference Figure 4 The bidirectional Mamba unit is improved on the basis of the Transformer encoder, replacing the multi-head self-attention layer and layer normalization in the multi-head self-attention unit in the Transformer with the bidirectional Mamba layer.

[0127] In a specific embodiment, each layer of the skeleton point generation calibration layer includes a semantically guided skeleton point cloud generation module and a geometrically guided skeleton point cloud calibration module connected in sequence, and the input feature of the skeleton point generation calibration layer of the current layer is a coded feature or a feature formed by connecting the input feature of the skeleton point generation calibration layer of the previous layer and the calibration skeleton point feature obtained by the skeleton point cloud calibration module of the previous layer, as shown in the following formula:

[0128] ;

[0129] in, Indicates a connection operation. , Indicates the total number of skeleton point generated calibration layers; when hour, is the encoding feature, Represents the input features of the j-th layer skeleton point generation calibration layer, Represents the input features of the j-1th layer skeleton point generation calibration layer, Represents the calibration skeleton point features obtained by the skeleton point cloud calibration module of the j-1th layer;

[0130] In the skeleton point generation calibration layer of the current layer, the input features of the skeleton point generation calibration layer of the current layer first pass through the semantically guided skeleton point cloud generation module of the current layer to obtain the first skeleton point prediction features of the current layer and the skeleton point coordinates of the current layer, and the skeleton point coordinates of the current layer, the input point set of the previous layer, the first skeleton point prediction features of the current layer and the input features of the skeleton point generation calibration layer of the current layer are input into the geometrically guided skeleton point cloud calibration module of the current layer to obtain the second skeleton point prediction features of the current layer and the calibrated skeleton point coordinates of the current layer; the calibrated skeleton point coordinates of the current layer are connected with the input point set of the previous layer to obtain the input point set of the current layer and input it into the skeleton point generation calibration layer of the next layer, the skeleton point prediction features of the current layer are connected with the calibrated skeleton point features of the previous layer to obtain the calibrated skeleton point features of the current layer and input it into the skeleton point generation calibration layer of the next layer, and the above steps are repeated until the skeleton point coordinates generated by the skeleton point generation calibration layer of the last layer are obtained and used as the sparse point cloud coordinates, and the calibrated skeleton point features generated by the skeleton point generation calibration layer of the last layer are obtained and used as the sparse point cloud features, and the sparse point cloud coordinates and the sparse point cloud features constitute a sparse point cloud.

[0131] In a specific embodiment, the input features of the skeleton point generation calibration layer of the current layer are first passed through the semantically guided skeleton point cloud generation module of the current layer to obtain the first skeleton point prediction features of the current layer and the skeleton point coordinates of the current layer, and the skeleton point coordinates of the current layer, the input point set of the previous layer, the first skeleton point prediction features of the current layer and the input features of the skeleton point generation calibration layer of the current layer are input into the geometrically guided skeleton point cloud calibration module of the current layer to obtain the second skeleton point prediction features of the current layer and the calibrated skeleton point coordinates of the current layer, specifically including:

[0132] In the j-th layer semantically guided skeleton point cloud generation module, the j-th layer skeleton points are firstly used to generate the input features of the calibration layer. Perform pooling operation to obtain global features , and the skeleton points of the jth layer generate the input features of the calibration layer and global features The feature difference between them is input to the fourth linear projection layer for linear projection, and we get Skeleton point features of the current layer , which is expressed as follows:

[0133] ;

[0134] ;

[0135] in, Represents the function corresponding to the pooling layer in the semantically guided skeleton point cloud generation module of the jth layer, the pooling layer in the semantically guided skeleton point cloud generation module of the first layer is an importance-aware pooling layer, and the importance-aware pooling layer is used to re-weight and sum the importance scores predicted by the importance score prediction layer, and the pooling layers in the semantically guided skeleton point cloud generation modules of the remaining layers are maximum pooling layers;

[0136] Generate the input features of the calibration layer from the skeleton points of the jth layer and the skeleton point features of the current layer After connection, the farthest point sampling and the closest point sampling of the features are sorted, and the skeleton point features are modeled in two Mamba units. and input features The semantic correlation between them is used to obtain the first skeleton point prediction feature of the jth layer with semantic interaction information. , which is expressed as follows:

[0137] ;

[0138] in, Indicates the farthest feature point sampling, Indicates the nearest feature point sampling, Indicates a connection operation. Represents two Mamba units. The calculation process of two Mamba units is as follows:

[0139] ;

[0140] ;

[0141] ;

[0142] in, is the function corresponding to the RMS normalization layer, Represents the input features of the j-th layer skeleton point generation calibration layer and the skeleton point features of the current layer Input features after sampling and sorting the farthest feature points after connection , Represents the input features of the j-th layer skeleton point generation calibration layer and the skeleton point features of the current layer Input features after connection and feature nearest point sampling and sorting , represents the state space equation;

[0143] The first skeleton point prediction feature of the jth layer Input to the fifth linear projection layer, the prediction is The coordinates of the skeleton points at the jth layer , whose expression is as follows:

[0144] ;

[0145] Among them, the fourth linear projection layer and the fifth linear projection layer adopt multi-layer perceptron;

[0146] In the j-th layer geometry-guided skeleton point cloud calibration module, the j-th layer skeleton point coordinates and the input point set of the j-1th layer After connection, the farthest point sampling and the nearest point sampling are sorted respectively, and the first skeleton point of the jth layer is predicted according to the sorting results. And the skeleton points of the jth layer generate the input features of the calibration layer The connection results are sorted and input into two Mamba units to model the geometric correlation between the two, and the second skeleton point prediction feature of the jth layer containing spatial geometric information is obtained. , whose expression is as follows:

[0147] ;

[0148] in, represents the farthest point sampling, represents the nearest point sampling, and Respectively represent the sampling sorting by the farthest point and the sampling sorting by the nearest point. Input point set The center point set ;

[0149] Secondly, the second skeleton point prediction feature of the jth layer With global features After adding, input the sixth linear projection layer to get the point offset , and the skeleton point coordinates of the jth layer obtained in the semantically guided skeleton point cloud generation unit Add together to get the calibrated skeleton point coordinates of the jth layer , whose expression is as follows:

[0150] ;

[0151] ;

[0152] Among them, the sixth linear projection layer adopts a multi-layer perceptron.

[0153] Specifically, refer to Figure 5 The semantically guided skeleton point cloud generation module and the geometrically guided skeleton point cloud calibration module constitute a total of L layers of skeleton point generation calibration layers. The semantically guided skeleton point cloud generation module in each layer corresponds to the geometrically guided skeleton point cloud calibration module one by one. The input feature of the skeleton point generation calibration layer of the current layer is Generate input features of the calibration layer from the skeleton points of the previous layer The second skeleton point prediction feature output by the skeleton point cloud calibration module guided by the geometry of the previous layer Connection composition, where Encoding feature The skeleton point coordinates of the jth layer are generated by the skeleton point calibration layer output and the first skeleton point prediction feature of the jth layer . refer to Figure 6 , further predicting the features of the first skeleton point of the jth layer And the skeleton points of the jth layer generate the input features of the calibration layer The connection result is input into the geometry-guided skeleton point cloud calibration module, combined with the skeleton point coordinates of the jth layer and the input point set of the j-1th layer Perform calibration to obtain the calibrated skeleton point coordinates and the second skeleton point feature . Coordinates of calibrated skeleton points and input coordinates Connect , and input the skeleton points of the next layer to generate the geometry-guided skeleton point cloud calibration module in the calibration layer, the second skeleton point feature and input features Connect , and input the skeleton points of the next layer to generate the calibration layer.

[0154] Finally, a complete but coarse-grained sparse point cloud is obtained through multi-stage generation and calibration. The coordinates of the sparse point cloud are , the sparse point cloud features are .

[0155] In a specific embodiment, the calculation expression of the decoding feature is:

[0156] ;

[0157] ;

[0158] in, represents random assignment, Indicates the sorting method. , , and They are Z-curve sorting strategy, reverse Z-curve sorting strategy, Hilbert sorting strategy and reverse Hilbert sorting strategy. represents the function corresponding to the bidirectional Mamba layer executed in the mth stage according to the nth sorting method, , , represents the total number of stages in the multi-order strategy Mamba decoder unit, represents sparse point cloud features, represents the decoding feature;

[0159] The decoded features are passed through the point cloud upsampling unit to obtain the predicted complete point cloud, which includes:

[0160] Using decoding features Compute the first affine parameter and the second affine parameter , and guide the two-dimensional grid with an affine function Deformation, three-dimensional offset obtained by three-dimensional deformation At the same time, each point coordinate in the sparse point cloud coordinates is copied O times and added to the three-dimensional offset to finally get the predicted complete point cloud , whose expression is as follows:

[0161] ;

[0162] ;

[0163] ;

[0164] ;

[0165] ;

[0166] in, represents the maximum pooling operation, Represents the function corresponding to the multilayer perceptron, and Represents a two-dimensional grid The mean and standard deviation of Indicates a copy operation. represents the sparse point cloud coordinates, Represents the first A sparse point, Indicates The decoding features of sparse points, represents the ReLU activation function, Indicates The second affine parameter corresponding to the sparse points, Indicates The point coordinates of sparse points, Indicated by The upsampled points generated by the sparse points.

[0167] Specifically, refer to Figure 7 , the multi-sort strategy Mamba decoder unit has a total of The multi-sorting strategy Mamba decoder in each stage consists of four bidirectional Mamba layers. The multi-sorting strategy Mamba decoder adopts four sorting strategies, namely Z-curve sorting strategy, reverse Z-curve sorting strategy, Hilbert sorting strategy and reverse Hilbert sorting strategy. The four sorting strategies are randomly and non-repeatedly assigned to the four bidirectional Mamba layers to learn the spatial information of different angles of the sparse point cloud and finally obtain the decoding features.

[0168] For further reference, Figure 8 and 9 , the decoded features are input into the point cloud upsampling unit to generate upsampling points, which are combined with the sparse point cloud coordinates to obtain a complete point cloud with complete fine-grained The seventh linear projection layer and the eighth linear projection layer in the point cloud upsampling unit both use multi-layer perceptrons.

[0169] In a specific embodiment, the total loss function used in the training process of the multi-stage Mamba point cloud completion model is the sum of the importance loss and the chamfer loss. The calculation process of the importance loss is shown in the following formula:

[0170] ;

[0171] ;

[0172] ;

[0173] ;

[0174] in, Represents the function corresponding to the pooling layer. The pooling layer here is the importance-aware pooling layer. The importance-aware pooling layer is used to re-weight and sum the importance scores predicted by the importance score prediction layer. represents the global encoding feature, represents the global encoding feature after projection, represents the true importance score calculated by cosine similarity, represents L1 smoothing loss, Indicates loss of importance;

[0175] The calculation process of chamfer loss is as follows:

[0176] ;

[0177] in, represents the chamfer loss, Represents the true complete point cloud, represents the number of points in the predicted complete point cloud, Represents the number of points of the true complete point cloud, represents the complete point cloud of the prediction, and Respectively represent one point in the predicted complete point cloud and one point in the real complete point cloud, Represents the square of the Euclidean distance.

[0178] Specifically, refer to Fig.10 First, the importance loss used to constrain the importance score prediction module is calculated. In the process of calculating the importance loss, the projected local point cloud block features and the global encoding features are first projected to the same feature space through a nonlinear projection layer, and then the cosine similarity between the projected local point cloud block features and the global encoding features is calculated. Finally, the L1 smoothing loss between the importance score and the cosine similarity is calculated to improve the prediction accuracy.

[0179] Finally, the chamfer loss between the true complete point cloud and the predicted complete point cloud is calculated and added to the importance loss as the final loss function , as shown below:

[0180] .

[0181] S2, obtain the incomplete point cloud to be completed and input it into the trained multi-stage Mamba point cloud completion model. The incomplete point cloud passes through the Transformer-Mamba joint point cloud local feature encoding unit to obtain the encoded features, and the encoded features are input into the sparse point cloud generation unit to obtain the sparse point cloud; the sparse point cloud is input into the multi-sorting strategy Mamba decoder unit. The multi-sorting strategy Mamba decoder of each stage adopts 4 sorting strategies, and the 4 sorting strategies are randomly and non-repeatedly assigned to 4 bidirectional Mamba layers to learn the spatial information of different angles in the sparse point cloud to obtain the decoding features. The decoding features pass through the point cloud upsampling unit to obtain the predicted complete point cloud.

[0182] Specifically, the trained multi-stage Mamba point cloud completion model is deployed, and the incomplete point cloud to be completed is input into the trained multi-stage Mamba point cloud completion model to obtain the predicted complete point cloud.

[0183] Further references Fig.11 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a multi-stage Mamba point cloud completion device based on semantics and geometry guidance. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0184] The embodiment of the present application provides a multi-stage Mamba point cloud completion device based on semantics and geometry guidance, including:

[0185] The model construction module 1 is configured to construct and train a multi-stage Mamba point cloud completion model based on semantics and geometry guidance to obtain a trained multi-stage Mamba point cloud completion model; the multi-stage Mamba point cloud completion model includes a sequentially connected Transformer-Mamba joint point cloud local feature encoding unit, a sparse point cloud generation unit, a multi-sorting strategy Mamba decoder unit and a point cloud upsampling unit; the sparse point cloud generation unit includes a plurality of sequentially connected skeleton point generation calibration layers; the multi-sorting strategy Mamba decoder unit includes a plurality of sequentially connected stages of multi-sorting strategy Mamba decoders, and each stage of the multi-sorting strategy Mamba decoder includes four sequentially connected bidirectional Mamba layers;

[0186] The completion module 2 is configured to obtain the incomplete point cloud to be completed and input it into the trained multi-stage Mamba point cloud completion model. The incomplete point cloud passes through the Transformer-Mamba joint point cloud local feature encoding unit to obtain the encoding features, and the encoding features are input into the sparse point cloud generation unit to obtain the sparse point cloud; the sparse point cloud is input into the multi-sorting strategy Mamba decoder unit. The multi-sorting strategy Mamba decoder of each stage adopts 4 sorting strategies, and the 4 sorting strategies are randomly and non-repeatedly assigned to 4 bidirectional Mamba layers to learn the spatial information of different angles in the sparse point cloud to obtain the decoding features. The decoding features pass through the point cloud upsampling unit to obtain the predicted complete point cloud.

[0187] Fig.12 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Fig.12As shown, the electronic device of this embodiment includes: a processor 1201 and a memory 1202; wherein the memory 1202 is used to store computer-executable instructions; the processor 1201 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0188] Optionally, the memory 1202 may be independent or integrated with the processor 1201 .

[0189] When the memory 1202 is independently provided, the electronic device further includes a bus 1203 for connecting the memory 1202 and the processor 1201 .

[0190] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 1201 executes the computer execution instructions, the above method is implemented.

[0191] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 1201, the above method is implemented.

[0192] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0193] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0194] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0195] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 1201 to perform some steps of the methods of various embodiments of the present application.

[0196] It should be understood that the processor 1201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 1201 may be any conventional processor 1201, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 1201 for execution, or may be executed by a combination of hardware and software modules in the processor 1201.

[0197] The memory 1202 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0198] The bus 1203 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1203 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 1203 in the drawings of the present application is not limited to only one bus 1203 or one type of bus 1203.

[0199] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0200] An exemplary storage medium is coupled to the processor 1201, so that the processor 1201 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 1201. The processor 1201 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 1201 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0201] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-stage Mamba point cloud completion method based on semantics and geometry guidance, characterized in that: The following steps are involved: A multi-stage Mamba point cloud completion model based on semantics and geometry guidance is constructed and trained to obtain a trained multi-stage Mamba point cloud completion model; the multi-stage Mamba point cloud completion model includes a sequentially connected Transformer-Mamba joint point cloud local feature encoding unit, a sparse point cloud generation unit, a multi-sorting strategy Mamba decoder unit and a point cloud upsampling unit; the sparse point cloud generation unit includes a plurality of sequentially connected layers of skeleton point generation calibration layers; the multi-sorting strategy Mamba decoder unit includes a plurality of sequentially connected stages of multi-sorting strategy Mamba decoders, and each stage of the multi-sorting strategy Mamba decoder includes four sequentially connected bidirectional Mamba layers; Acquire an incomplete point cloud to be completed and input it into the trained multi-stage Mamba point cloud completion model, the incomplete point cloud passes through the Transformer-Mamba joint point cloud local feature encoding unit to obtain encoding features, and the encoding features are input into the sparse point cloud generation unit to obtain a sparse point cloud; The sparse point cloud is input into the multi-sorting strategy Mamba decoder unit. The multi-sorting strategy Mamba decoder in each stage adopts 4 sorting strategies, which are randomly and non-repetitively assigned to 4 bidirectional Mamba layers to learn spatial information at different angles in the sparse point cloud and obtain decoding features. The decoding features are passed through the point cloud upsampling unit to obtain a predicted complete point cloud.

2. The multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to claim 1, characterized in that: The Transformer-Mamba joint point cloud local feature encoding unit includes a farthest point sampling layer, a K nearest neighbor query layer, an intra-block Transformer encoding unit, an importance score prediction layer and an inter-block Mamba encoding unit connected in sequence, and the intra-block Transformer encoding unit includes a The intra-block Transformer encoders, the inter-block Mamba encoding unit includes sequentially connected The inter-block Mamba encoder is used, the importance score prediction layer includes a nonlinear projection layer and a score prediction layer connected in sequence, the nonlinear projection layer is composed of a first convolution layer, a first batch of standardization layers, a first ReLU activation function layer, a second convolution layer, and a layer standardization layer cascaded, and the score prediction layer is composed of a third convolution layer, a second batch of standardization layers, a second ReLU activation function layer, and a fourth convolution layer cascaded; the calculation process of the Transformer-Mamba joint point cloud local feature encoding unit is as follows: Input the incomplete point cloud into the farthest point sampling layer to select M center points and construct a center point set , and select the K points closest to each center point through the K nearest neighbor query layer to construct a local point cloud set to obtain a local point cloud block ,in, represents the set of real numbers, represents the mth local point cloud block of the downsampling, M represents the total number of center points, represents the kth nearest neighbor point in the local point cloud block, K represents the total number of nearest neighbor points, Represents the coordinates of the kth nearest neighbor point in the mth local point cloud block, as shown in the following formula: ； in, Represents the function corresponding to the farthest point sampling layer, Represents the function corresponding to the K nearest neighbor query layer, Indicates the coordinates of incomplete point cloud; The local point cloud block Input the Transformer encoding unit in the block to obtain the local point cloud block features , the calculation expression of the Transformer encoder in the i-th block is as follows: ； ； Among them, C represents the feature dimension, represents the Transformer encoder in the i-th block, Represents the number of Transformer encoders in the block, Representation layer normalization operation, Represents the function corresponding to the multi-head self-attention layer, represents the function corresponding to the feedforward network, represents the output features of the multi-head self-attention unit in the Transformer encoder in the i-th block, represents the point cloud block features output by the Transformer encoder in the i-1th block, Represents the point cloud block features output by the Transformer encoder in the i-th block, hour, for Repeat the above steps until the point cloud block features output by the Transformer encoder in the last block are obtained and used as the local point cloud block features ; The local point cloud block feature Input to the importance score prediction layer to obtain the importance score of the local point cloud block, as shown in the following formula: ； ； in, represents the nonlinear projection layer, Represents the importance score of the local point cloud block, represents the function corresponding to the score prediction layer, Represents the local point cloud block features after projection; The local point cloud block features are sorted according to the importance score of the local point cloud block. Sort and get the sorted local point cloud block features And input the inter-block Mamba encoding unit to model the dependencies between local point cloud blocks. The expression of the inter-block Mamba encoder is as follows: ； ； in, represents the i'th inter-block Mamba encoder, Indicates the number of inter-block Mamba encoders, represents the output features of the bidirectional Mamba unit in the i'th inter-block Mamba encoder, Represents the function corresponding to the bidirectional Mamba layer, represents the local point cloud block features enhanced by the i'th inter-block Mamba encoder, represents the local point cloud block feature enhanced by the i'-1th inter-block Mamba encoder. When i'=1, is the sorted local point cloud block feature ; The bidirectional Mamba unit includes a layer normalization operation, a first branch, a second branch and a first linear projection layer, the first branch includes a second linear projection layer and a first SiLU activation function layer connected in sequence, the second branch includes a third linear projection layer, a depth-separable convolution layer, a second SiLU activation function layer and a cascade of parallel forward state space equation units and reverse state space equation units, the forward state space equation unit includes a forward state space equation and a layer normalization operation connected in sequence, the reverse state space equation unit includes a reverse state space equation and a layer normalization operation connected in sequence, the first linear projection layer, the second linear projection layer and the third linear projection layer all use multilayer perceptrons, and the calculation process of the bidirectional Mamba unit is as follows: ； ； in, Indicates the hidden state, represents the function corresponding to the depthwise separable convolutional layer, represents the function corresponding to the multi-layer perceptron layer, represents the forward state space equation, represents the inverse state-space equation, represents the SiLU activation function; The enhanced local point cloud block features output by the last inter-block Mamba encoder are used as encoding features .

3. The multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to claim 2 is characterized in that: The skeleton point generation calibration layer of each layer includes a semantically guided skeleton point cloud generation module and a geometrically guided skeleton point cloud calibration module connected in sequence. The input feature of the skeleton point generation calibration layer of the current layer is the coded feature or the input feature of the skeleton point generation calibration layer of the previous layer and the calibrated skeleton point feature obtained by the skeleton point cloud calibration module of the previous layer, as shown in the following formula: ； in, Indicates a connection operation. , Indicates the total number of skeleton point generated calibration layers; when hour, is the encoding feature, Represents the input features of the j-th layer skeleton point generation calibration layer, Represents the input features of the j-1th layer skeleton point generation calibration layer, Represents the calibration skeleton point features obtained by the skeleton point cloud calibration module of the j-1th layer; In the skeleton point generation calibration layer of the current layer, the input features of the skeleton point generation calibration layer of the current layer first pass through the semantic-guided skeleton point cloud generation module of the current layer to obtain the first skeleton point prediction features of the current layer and the skeleton point coordinates of the current layer, and the skeleton point coordinates of the current layer, the input point set of the previous layer, the first skeleton point prediction features of the current layer and the input features of the skeleton point generation calibration layer of the current layer are input into the geometric-guided skeleton point cloud calibration module of the current layer to obtain the second skeleton point prediction features of the current layer and the calibrated skeleton point coordinates of the current layer; the calibrated skeleton point coordinates of the current layer are compared with the calibrated skeleton point coordinates of the previous layer. The input point set of the current layer is connected to obtain the input point set of the current layer and input it into the skeleton point generation calibration layer of the next layer, the skeleton point prediction features of the current layer are connected with the calibration skeleton point features of the previous layer to obtain the calibration skeleton point features of the current layer and input them into the skeleton point generation calibration layer of the next layer, and the above steps are repeated until the skeleton point coordinates generated by the skeleton point generation calibration layer of the last layer are obtained and used as sparse point cloud coordinates, and the calibration skeleton point features generated by the skeleton point generation calibration layer of the last layer are obtained and used as sparse point cloud features, and the sparse point cloud coordinates and sparse point cloud features constitute the sparse point cloud.

4. The multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to claim 3 is characterized in that: The input features of the skeleton point generation calibration layer of the current layer are first passed through the semantically guided skeleton point cloud generation module of the current layer to obtain the first skeleton point prediction features of the current layer and the skeleton point coordinates of the current layer, and the skeleton point coordinates of the current layer, the input point set of the previous layer, the first skeleton point prediction features of the current layer and the input features of the skeleton point generation calibration layer of the current layer are input into the geometrically guided skeleton point cloud calibration module of the current layer to obtain the second skeleton point prediction features of the current layer and the calibrated skeleton point coordinates of the current layer, specifically including: In the semantically guided skeleton point cloud generation module of the jth layer, the input features of the calibration layer are first generated for the skeleton points of the jth layer. Perform pooling operation to obtain global features , and the skeleton points of the jth layer generate the input features of the calibration layer and global features The feature difference between them is input to the fourth linear projection layer for linear projection, and we get Skeleton point features of the current layer , which is expressed as follows: ； ； in, Represents the function corresponding to the pooling layer in the semantically guided skeleton point cloud generation module of the jth layer, the pooling layer in the semantically guided skeleton point cloud generation module of the first layer is an importance-aware pooling layer, the importance-aware pooling layer is used to re-weighted sum according to the importance score predicted by the importance score prediction layer, and the pooling layers in the semantically guided skeleton point cloud generation modules of the remaining layers are maximum pooling layers; Generate the input features of the calibration layer from the skeleton points of the jth layer and the skeleton point features of the current layer After connection, the farthest point sampling and the closest point sampling of the features are sorted, and the skeleton point features are modeled in two Mamba units. and input features The semantic correlation between them is used to obtain the first skeleton point prediction feature of the jth layer with semantic interaction information. , which is expressed as follows: ； in, Indicates the farthest feature point sampling, Indicates the nearest feature point sampling, Indicates a connection operation. Represents two Mamba units. The calculation process of two Mamba units is as follows: ； ； ； in, is the function corresponding to the RMS normalization layer, Represents the input features of the j-th layer skeleton point generation calibration layer and the skeleton point features of the current layer Input features after sampling and sorting the farthest feature points after connection , Represents the input features of the j-th layer skeleton point generation calibration layer and the skeleton point features of the current layer Input features after connection and feature nearest point sampling and sorting , represents the state space equation; The first skeleton point prediction feature of the jth layer Input to the fifth linear projection layer, the prediction is The coordinates of the skeleton points at the jth layer , whose expression is as follows: ； Wherein, the fourth linear projection layer and the fifth linear projection layer adopt a multi-layer perceptron; In the j-th layer geometry-guided skeleton point cloud calibration module, the j-th layer skeleton point coordinates and the input point set of the j-1th layer After connection, the farthest point sampling and the nearest point sampling are sorted respectively, and the first skeleton point of the jth layer is predicted according to the sorting results. And the skeleton points of the jth layer generate the input features of the calibration layer The connection results are sorted and input into two Mamba units to model the geometric correlation between the two, and the second skeleton point prediction feature of the jth layer containing spatial geometric information is obtained. , whose expression is as follows: ； in, represents the farthest point sampling, represents the nearest point sampling, and Respectively represent the sampling sorting by the farthest point and the sampling sorting by the nearest point. Input point set The center point set ; Secondly, the second skeleton point prediction feature of the jth layer With global features After adding, input the sixth linear projection layer to get the point offset The skeleton point coordinates of the jth layer obtained in the semantically guided skeleton point cloud generation unit are Add together to get the calibrated skeleton point coordinates of the jth layer , whose expression is as follows: ； ； Wherein, the sixth linear projection layer adopts a multi-layer perceptron.

5. The multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to claim 3, characterized in that: The calculation expression of the decoding feature is: ； ； in, represents random assignment, Indicates the sorting method. , , and They are Z-curve sorting strategy, reverse Z-curve sorting strategy, Hilbert sorting strategy and reverse Hilbert sorting strategy. represents the function corresponding to the bidirectional Mamba layer executed in the mth stage according to the nth sorting method, , , represents the total number of stages in the multi-order strategy Mamba decoder unit, represents sparse point cloud features, represents the decoding feature; The decoded features are passed through the point cloud upsampling unit to obtain a predicted complete point cloud, which specifically includes: Using decoding features Compute the first affine parameter and the second affine parameter , and guide the two-dimensional grid with an affine function Deformation, three-dimensional offset obtained by three-dimensional deformation At the same time, each point coordinate in the sparse point cloud coordinates is copied O times and added to the three-dimensional offset to finally get the predicted complete point cloud , whose expression is as follows: ； ； ； ； ； in, represents the maximum pooling operation, Represents the function corresponding to the multilayer perceptron, and Represents a two-dimensional grid The mean and standard deviation of Indicates a copy operation. represents the sparse point cloud coordinates, Represents the first A sparse point, Indicates The decoding features of sparse points, represents the ReLU activation function, Indicates The second affine parameter corresponding to the sparse points, Indicates The point coordinates of sparse points, Indicated by The upsampled points generated by the sparse points.

6. The multi-stage Mamba point cloud completion method based on semantics and geometry guidance according to claim 2, characterized in that: The total loss function used in the training process of the multi-stage Mamba point cloud completion model is the sum of the importance loss and the chamfer loss. The calculation process of the importance loss is shown in the following formula: ； ； ； ； in, Represents the function corresponding to the pooling layer. The pooling layer here is the importance-aware pooling layer. The importance-aware pooling layer is used to re-weight and sum the importance scores predicted by the importance score prediction layer. represents the global encoding feature, represents the global encoding feature after projection, represents the true importance score calculated by cosine similarity, represents L1 smoothing loss, Indicates loss of importance; The calculation process of the chamfer loss is shown in the following formula: ； in, represents the chamfer loss, Represents the true complete point cloud, represents the number of points in the predicted complete point cloud, Represents the number of points of the true complete point cloud, represents the complete point cloud of the prediction, and Respectively represent one point in the predicted complete point cloud and one point in the real complete point cloud, Represents the square of the Euclidean distance.

7. A multi-stage Mamba point cloud completion device based on semantics and geometry guidance, characterized in that: include: A model construction module is configured to construct and train a multi-stage Mamba point cloud completion model based on semantics and geometry guidance to obtain a trained multi-stage Mamba point cloud completion model; the multi-stage Mamba point cloud completion model includes a sequentially connected Transformer-Mamba joint point cloud local feature encoding unit, a sparse point cloud generation unit, a multi-sorting strategy Mamba decoder unit and a point cloud upsampling unit; the sparse point cloud generation unit includes a plurality of sequentially connected layers of skeleton point generation calibration layers; the multi-sorting strategy Mamba decoder unit includes a plurality of sequentially connected stages of multi-sorting strategy Mamba decoders, and each stage of the multi-sorting strategy Mamba decoder includes four sequentially connected bidirectional Mamba layers; A completion module is configured to obtain an incomplete point cloud to be completed and input it into the trained multi-stage Mamba point cloud completion model, the incomplete point cloud passes through the Transformer-Mamba joint point cloud local feature encoding unit to obtain encoding features, and the encoding features are input into the sparse point cloud generation unit to obtain a sparse point cloud; The sparse point cloud is input into the multi-sorting strategy Mamba decoder unit. The multi-sorting strategy Mamba decoder in each stage adopts 4 sorting strategies, which are randomly and non-repetitively assigned to 4 bidirectional Mamba layers to learn spatial information at different angles in the sparse point cloud and obtain decoding features. The decoding features are passed through the point cloud upsampling unit to obtain a predicted complete point cloud.

8. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images