A point cloud up-sampling method and system based on a Mamba model

Abstract

The application belongs to the technical field of image processing, and particularly relates to a point cloud up-sampling method and system based on a Mamba model, which comprises the following steps: obtaining a sparse point cloud, dividing the point cloud into at least two voxels, sequencing the at least two voxels according to a space-filling curve to obtain a one-dimensional sequence, performing feature extraction on the one-dimensional sequence to obtain multi-scale voxel features, inputting the multi-scale voxel features into a Mamba module for time series modeling, and outputting an enhanced time series feature and a hidden state corresponding to each voxel, predicting an interpolation weight matrix of each voxel according to the hidden state, generating a predicted point in each voxel, predicting a coordinate offset of the predicted point relative to a voxel center, and finally fusing the new point with the point cloud to output a dense point cloud, so that the point cloud up-sampling scheme can effectively improve the data density and geometric accuracy of the sparse point cloud and also consider the processing efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a point cloud up-sampling method and system based on a Mamba model. BACKGROUND

[0002] Three-dimensional point cloud, as one of the most basic three-dimensional data expression forms, plays a key role in 3D immersive remote interaction, autonomous driving, augmented / virtual reality and other frontier fields. In actual engineering applications, the value of high-precision point cloud data is particularly significant - for example, the point cloud map required for city-level three-dimensional modeling not only requires complete preservation of original geometric features such as building facades and road details, but also usually has a data scale of millions to billions of point cloud units, and the single-point coordinate precision needs to be controlled at the centimeter level. Large-scale point cloud units not only produce huge bandwidth pressure, but also are difficult to achieve high-density sampling and high-precision restoration in high-speed dynamic scenes. For the above reasons, sparse sampling strategies (such as reducing laser beam density) are often forced to be used.

[0003] In order to solve the problems of local missing and uneven distribution of point cloud data caused by sparse sampling strategies, two technical paths are proposed in the prior art:

[0004] 1. Traditional method based on prior conditions: rely on artificially defined geometric assumptions to construct an up-sampling model: the typical approach is to use local geometric properties of point cloud (such as normal vector, curvature, neighborhood distribution, etc.) as prior constraints, and to achieve point cloud density improvement through mathematical interpolation or surface fitting, which has fast solving ability in simple scenes, but is limited in generalization.

[0005] 2. Neural network-based methods: This method gets rid of the restriction of explicit geometric prior, and automatically learns the mapping relationship from sparse input to dense output through large-scale data set driven end-to-end training. This kind of method constructs diversified network architecture: Convolutional Neural Networks (CNNs) usually voxelize or project point clouds into distance images, extract spatial features with 3D convolution or 2D convolution, and are suitable for point cloud data with regularized representation; Graph Convolutional Networks (GCNs) model point clouds as graph structures, capture local geometric correlation through neighborhood feature aggregation, and effectively handle the unordered and unstructured characteristics of point clouds; Generative Adversarial Networks (GANs) introduce an adversarial training mechanism, constrain the geometric rationality of generated point clouds through discriminators, and significantly improve the distribution uniformity and structural authenticity of generated points. The core advantage of this kind of method is that it can automatically mine the implicit geometric feature generation rule from massive data - for example, through multilayer perceptron (MLP) to abstract the multi-scale structural features of point clouds layer by layer, or use Transformer to model global geometric dependency - so that the geometric feature recovery capability in complex scenes significantly surpasses traditional methods, and it is especially good at handling the detail reconstruction of regions with sudden curvature changes (such as edges and corners).

[0006] Although the above neural network method breaks through the bottleneck of prior assumptions through computing power, it still has problems such as weak model interpretability and high dependence on training data quality.

[0007] Therefore, there is an urgent need for a sparse point cloud processing scheme with higher restoration accuracy. SUMMARY

[0008] The purpose of the present application is to provide a point cloud up-sampling method and system based on Mamba model, to partially alleviate or solve the above problems, and to improve the restoration accuracy of sparse point cloud to some extent.

[0009] In order to solve the above-mentioned technical problems, the present application specifically adopts the following technical solutions:

[0010] The first aspect of the present application is to provide a point cloud up-sampling method based on Mamba model, comprising:

[0011] S100, acquiring a sparse point cloud ; wherein, is the number of points, and 3 is the coordinate dimension;

[0012] S200, dividing the point cloud into at least two voxels;

[0013] S300, sequencing the at least two voxels according to the space-filling curve to obtain a one-dimensional sequence ; wherein, is the sequence length, is the feature dimension;

[0014] S400, feature extraction is performed on the one-dimensional sequence to obtain multi-scale voxel features ; wherein, is the i-th feature dimension, is the i-th sequence length, is the i+1-th sequence length;

[0015] S500, inputting the multi-scale voxel features into a Mamba module based on a selective state space model for time series modeling, and outputting enhanced time series features and hidden states corresponding to each voxel ;

[0016] S600, predicting an interpolation weight matrix of each voxel according to the hidden state ; wherein, is the up-sampling multiple of the voxel;

[0017] S700, decoding the hidden state into a coordinate residual of a predicted point based on the interpolation weight matrix and the hidden state, generating predicted points in each voxel according to the coordinate residual, and predicting a coordinate offset of the predicted point relative to the center of the voxel;

[0018] S800, summing the coordinate offset and the center coordinates of the corresponding voxel to generate new point coordinates, and fusing the new points with the point cloud to output a dense point cloud is the total number of points of the point cloud after fusing new points.

[0019] As an improvement, in step S400, the specific steps of feature extraction on the one-dimensional sequence to obtain multi-scale voxel features include:

[0020] S401, calculating the statistical features of the point cloud in each non-empty voxel through feature encoding;

[0021] S402, inputting the statistical features into a voxel feature matrix , wherein is the number of non-empty voxels, represents the initial feature dimension, and ;

[0022] S403, capturing multi-scale features through at least two levels of sparse convolution down-sampling, each level of sparse convolution including a sparse convolution layer and a voxel pooling; the sparse convolution specifically includes:

[0023] calculating non-empty voxels through a sparse convolution layer, using a hash table to accelerate neighborhood query and outputting corresponding feature maps ;

[0024]

[0025] wherein, is a neighborhood of voxel , is the i-th feature dimension, is an index of the neighborhood voxel, is an index of the current voxel, is a bias term, is an input feature vector of voxel u, is a sparse convolution kernel weight matrix whose index is determined by the relative offset ;

[0026] The voxel grid resolution is reduced by voxel pooling, and the feature aggregation is performed on the feature map to output the down-sampled voxel features .

[0027] As an improvement, in S600, the specific steps of predicting the interpolation weight matrix of each voxel according to the hidden state include:

[0028] S601, input the hidden state, and map the hidden state to a high-dimensional space to generate an initial weight matrix ;

[0029]

[0030] wherein, is a learnable projection matrix, is a bias term;

[0031] S602, L2-normalize each row of the initial weight matrix to constrain the numerical range of the weight matrix, to obtain the interpolation weight matrix:

[0032]

[0033] wherein, is the initial interpolation weight matrix corresponding to the i-th voxel, is the j-th row vector of the matrix , is the L2 norm of the j-th row vector, is the normalized interpolation weight vector.

[0034] As an improvement, the statistical features include the mean coordinate (x, , the number of in-voxel points), the density (p ) and the normal vector.

[0035] As an improvement, before S400, it further includes the steps of:

[0036] S350, judge whether the distance of the point cloud and the center of the voxel meets a preset distance range, if yes, keep the current point cloud, otherwise, correct the position of the point cloud until the point cloud meets the distance range; Wherein, the method of correction includes: moving the point cloud along a first direction by a preset distance, the first direction is the extension direction of the line connecting the point cloud and the center of the voxel.

[0037] Another aspect of the application provides a point cloud upsampling system based on Mamba model, comprising:

[0038] Data acquisition module: configured to acquire sparse point cloud ; wherein, is the number of points, and 3 is the coordinate dimension;

[0039] Point cloud voxelization and serialization module: configured to divide the point cloud into at least two voxels, and serialize the at least two voxels according to space filling curve to obtain one-dimensional sequence ; wherein, is the sequence length, is the feature dimension;

[0040] Multi-scale feature extraction module: configured to extract features from the one-dimensional sequence to obtain multi-scale voxel features ; wherein, is the i th feature dimension, is the i th sequence length, is the i+1 th sequence length;

[0041] Timing modeling module: configured to input the multi-scale voxel features into the Mamba module based on the selective state space model for timing modeling, and output the enhanced timing features and hidden state corresponding to each voxel ;

[0042] Interpolation weight generation module: configured to predict the interpolation weight matrix of each voxel according to the hidden state ; wherein, is the upsampling multiple of the voxel;

[0043] Coordinate residual prediction module: configured to decode the hidden state into the coordinate residual of the predicted point based on the interpolation weight matrix and the hidden state, generate predicted points in each voxel according to the coordinate residual, and predict the coordinate offset of the predicted point relative to the center of the voxel.

[0044] The data fusion module is configured to sum the coordinate offset and the center coordinate of the corresponding voxel, generate a new point coordinate, fuse the new point with the point cloud, and output a dense point cloud. The total number of points in the point cloud after fusion of the new points.

[0045] As an improvement, the multi-scale feature extraction module is specifically configured to:

[0046] Calculate the statistical features of the point cloud in each non-empty voxel through feature encoding;

[0047] Input the statistical features into the voxel feature matrix , wherein is the number of non-empty voxels, represents the initial feature dimension, and ;

[0048] Capture multi-scale features through at least two levels of sparse convolution downsampling, and each level of sparse convolution includes a sparse convolution layer and a voxel pooling; the sparse convolution specifically includes:

[0049] Calculate the non-empty voxels through the sparse convolution layer, use a hash table to accelerate neighborhood query, and output the corresponding feature map ;

[0050]

[0051] wherein, is the neighborhood of the voxel , is the i-th feature dimension, is the index of the neighborhood voxel, is the index of the current voxel, is the bias term, is the input feature vector of the voxel u, is the sparse convolution kernel weight matrix, the index of which is determined by the relative offset ;

[0052] Reduce the voxel grid resolution through voxel pooling, and perform feature aggregation on the feature map to output the downsampled voxel features .

[0053] As an improvement, the coordinate residual prediction module is specifically configured to:

[0054] Input the hidden state and map the hidden state to a high-dimensional space to generate an initial weight matrix ;

[0055]

[0056] wherein,​ is a learnable projection matrix, is a bias term;

[0057] performing L2 normalization on each row of the initial weight matrix to obtain an interpolation weight matrix:

[0058]

[0059] wherein, is the initial interpolation weight matrix corresponding to the i th voxel, is the j th row vector of the matrix is the L2 norm of the j th row vector, is the normalized interpolation weight vector.

[0060] As an improvement, the statistical features include mean coordinates (x, y, z) of the point cloud, , the number of in-voxel points, the density (ρ) and the normal vector.

[0061] As an improvement, it further comprises:

[0062] a data correction module configured to judge whether the distance between the point cloud and the voxel center conforms to a preset distance range, if yes, the current point cloud is retained, otherwise the position of the point cloud is corrected until the point cloud conforms to the distance range; wherein the correction method comprises moving the point cloud by a preset distance along a first direction, the first direction being the extension direction of the line connecting the point cloud and the voxel center.

[0063] The principles and beneficial technical effects of the present application are that: for scenarios such as automatic driving, surveying and mapping, scene reconstruction and face recognition, which have high requirements for precision and image restoration speed, the present application provides a point cloud up-sampling scheme that can effectively improve the density and geometric precision of sparse point cloud data, and also takes into account the processing efficiency. By performing multi-layer analysis on various features (statistical features) of the limited sparse point cloud, and introducing an offset to correct the newly added points, at the same time, using the Mamba model and lightweight weight projection to simultaneously perform lightweight processing on the data, the restoration precision is improved, and at the same time, the rapid response demand of high-speed restoration scenarios (such as automatic driving scenarios) can also be met.

[0064] ​​Firstly, the application analyzes the multi-dimensional data (including mean coordinates, density and normal vector) of the sparse point cloud, and constructs a hierarchical feature pyramid through hierarchical downsampling and feature fusion, and then analyzes the hierarchical "global data" (especially the initial point cloud data) in real time, in blocks and in parallel through the Mamba module, and introduces the offset to correct the position of the new point through the interpolation weight generation and coordinate residual prediction; the "global data" of the new point is obtained, and then a higher precision image result can be quickly obtained.

[0065] Further, the present scheme also provides a data source correction scheme, that is, the initial sparse point cloud is corrected, so that the new point always remains valid, that is, its position is always located within the voxel, thereby ensuring the validity of the "global data" (especially the new point data), ultimately breaking through the dual bottleneck of poor global consistency and low computational efficiency of deep learning, while considering the continuity of three-dimensional structure and real-time processing capability, providing a lightweight solution for high-precision point cloud reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. In all the drawings, similar elements or parts are generally identified by similar reference signs. In the drawings, each element or part is not necessarily drawn according to the actual proportion. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0067] Figure 1 The flowchart of the point cloud upsampling method based on the Mamba model in the embodiment of the present application is shown.

[0068] Figure 2 The modular structure diagram of the point cloud upsampling system based on the Mamba model in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme of the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0070] Herein, the suffixes such as "module", "part", or "unit" used for an element are merely intended for facilitating explanation of the present application, and have no specific meaning by themselves. Therefore, "module", "part", or "unit" can be mixedly used herein. Herein, the terms "upper", "lower", "inner", "outer", "front", "rear", "one side", "the other side", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are merely used for convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", are used only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0071] Herein, unless otherwise explicitly specified and limited, the terms "mount", "provided with", "connected", and the like, should be understood broadly, for example, "connected" can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, can be directly connected, or indirectly connected through an intermediate medium, can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances. "Multiple" herein means two or more, that is, it includes two, three, four, five, etc.

[0072] Embodiment one

[0073] The present application provides a point cloud up-sampling method based on Mamba model, comprising:

[0074] S100, acquiring a sparse point cloud ; wherein, is the number of points, 3 is the coordinate dimension, is a real set.

[0075] S200, dividing the point cloud into at least two voxels; wherein each voxel includes at least one point cloud, and the point cloud is represented by the voxel center coordinates and features (such as density and normal vector, etc.) of the voxel.

[0076] S300, sequencing the at least two voxels according to a space-filling curve, to obtain a one-dimensional sequence ; wherein, is the sequence length, is the feature dimension; preferably, the space-filling curve is a Z-order curve.

[0077] S400, feature extraction is performed on the one-dimensional sequence to obtain multi-scale voxel features ; wherein, is the i-th feature dimension, is the i-th sequence length, is the i+1-th sequence length.

[0078] S500, input the multi-scale voxel features into the Mamba module based on the selective state space model for time series modeling, and output the enhanced time series features and hidden states corresponding to each voxel .

[0079] S600, predict the interpolation weight matrix of each voxel according to the hidden state ; wherein, is the up-sampling multiple of the voxel (such as =2 / 4 / 8).

[0080] S700, based on the interpolation weight matrix and the hidden state, decode the hidden state into the coordinate residual of the prediction point, generate prediction points in each voxel according to the coordinate residual, and predict the coordinate offset of the prediction point relative to the voxel center ;

[0081] S800, sum the coordinate offset and the center coordinates of the corresponding voxel to generate the new point coordinates, and fuse the new points with the point cloud to output the dense point cloud is the total number of points after fusing the new points.

[0082] The scheme analyzes the multi-dimensional data (including mean coordinates, density and normal vector) of the sparse point cloud, constructs a hierarchical feature pyramid through hierarchical down-sampling and feature fusion, then analyzes the hierarchical "global data" (especially the initial point cloud data) through the Mamba module in real time, block by block and in parallel, generates the interpolation weight and predicts the coordinate residual to introduce the offset to correct the position of the new points; obtains the "global data" of the new points, and can quickly obtain a higher precision image result.

[0083] In some embodiments, in step S400, the specific steps of feature extraction on the one-dimensional sequence to obtain multi-scale voxel features include:

[0084] S401, calculate the statistical features of the point cloud in each non-empty voxel through feature encoding.

[0085] In some embodiments, the statistical features include mean coordinates (x, y, z), , is the number of points in the voxel, is the density (ρ) and normal vector.

[0086] ​S402, input the statistical features into the voxel feature matrix ,in It is a non-empty prime number. Denotes the initial feature dimension, and (Including three-dimensional coordinate mean, one-dimensional density, and three-dimensional normal vector).

[0087] S403 captures multi-scale features through at least two levels of sparse convolution downsampling, each level of sparse convolution comprising a sparse convolutional layer and voxel pooling; the sparse convolution specifically includes:

[0088] Non-empty voxels are computed using sparse convolutional layers, and a hash table is used to accelerate neighborhood lookup and output the corresponding feature map. ;

[0089]

[0090] in, voxels The neighborhood, This represents the number of output channels (i.e., the i-th feature dimension). For indexing neighborhood voxels, For the index of the current voxel, For bias terms, Let be the input feature vector of voxel u. The sparse convolution kernel weight matrix has its indices determined by relative offsets. Confirmed. In other words, this scheme captures multi-scale features through stepwise downsampling, which reduces computation and expands the receptive field.

[0091] Voxel pooling is used to reduce the voxel grid resolution, and feature aggregation is performed on the feature map to output the downsampled voxel features. .

[0092] The feature encoding rules and the voxel feature matrix are both pre-input by the user or automatically generated by the computer.

[0093] In some embodiments, the specific steps of predicting the interpolation weight matrix for each voxel based on the hidden state in S600 include:

[0094] S601, Input the hidden state and map the hidden state to a high-dimensional space to generate an initial weight matrix. ;

[0095]

[0096] in, For learnable projection matrices, For bias terms; is the up-sampling factor;

[0097] S602, L2 normalization is performed on each row of the initial weight matrix to constrain the numerical range of the weight matrix, and an interpolation weight matrix is obtained:

[0098]

[0099] wherein, is the initial interpolation weight matrix corresponding to the third voxel, is the jth row vector of the matrix . is the L2 norm of the jth row vector, is the normalized interpolation weight vector.

[0100] In some embodiments, before S400, the method further comprises the step of:

[0101] S350, determining whether the distance between the point cloud and the voxel center conforms to a preset distance range, if so, the current point cloud is retained, otherwise the position of the point cloud is corrected until the point cloud conforms to the distance range; wherein the correction method comprises: moving the point cloud by a preset distance along a first direction, the first direction being the extension direction of the line connecting the point cloud and the voxel center, that is, the direction close to the voxel center.

[0102] wherein, the preset distance range can be obtained according to historical test data or input according to user experience, when the distance between the point cloud and the voxel center does not conform to the preset distance range (for example, the distance between the point cloud and the voxel center is greater than the preset distance, at this time it is indicated that the point cloud is located at the edge position of the voxel center), if the point cloud at this position is corrected, the result is "invalid" new points, it should be noted that "invalid" here means that the new point is located outside the voxel, if an image is generated according to the point, it will lead to the destruction of the integrity of the local structure of the point cloud, and may lead to geometric distortion or holes in the finally reconstructed three-dimensional model.

[0103] In summary, the present application provides a scheme for improving the global data and data validity of sparse point clouds for scenarios such as automatic driving, surveying and mapping, scene reconstruction, and face recognition, which have very high precision requirements.

[0104] Embodiment two

[0105] The present application also provides a point cloud up-sampling system based on a Mamba model, comprising:

[0106] a data acquisition module configured to acquire a sparse point cloud ; wherein, is the number of points, and 3 is the coordinate dimension;

[0107] a point cloud voxelization and serialization module configured to divide the point cloud into at least two voxels and serialize the at least two voxels according to a space-filling curve to obtain a one-dimensional sequence ; wherein, is a sequence length, is a feature dimension;

[0108] a multi-scale feature extraction module configured to extract features from the one-dimensional sequence to obtain multi-scale voxel features ; wherein, is an i-th feature dimension, is an i-th sequence length, is an i+1-th sequence length;

[0109] a time series modeling module configured to input the multi-scale voxel features into a Mamba module based on a selective state space model for time series modeling, and output an enhanced time series feature and a hidden state corresponding to each voxel ;

[0110] an interpolation weight generation module configured to predict an interpolation weight matrix of each voxel according to the hidden state ; wherein, is an up-sampling multiple of the voxel;

[0111] a coordinate residual prediction module configured to decode the hidden state into a coordinate residual of a predicted point based on the interpolation weight matrix and the hidden state, generate a predicted point in each voxel according to the coordinate residual, and predict a coordinate offset of the predicted point relative to a voxel center.

[0112] a data fusion module configured to sum the coordinate offset and a center coordinate of a corresponding voxel to generate a new point coordinate, fuse the new point with the point cloud, and output a dense point cloud is a total number of points of the point cloud after fusing the new point.

[0113] In some embodiments, the multi-scale feature extraction module is specifically configured to:

[0114] calculate a statistical feature of the point cloud in each non-empty voxel by feature encoding;

[0115] input the statistical feature into a voxel feature matrix , wherein is a number of non-empty voxels, represents an initial feature dimension, and ;

[0116] Capture multi-scale features through at least two levels of sparse convolution downsampling, each level of sparse convolution comprising a sparse convolution layer and a voxel pooling respectively; the sparse convolution specifically comprises:

[0117] Calculate non-empty voxels through the sparse convolution layer, accelerate neighborhood query using a hash table, and output the corresponding feature map ;

[0118]

[0119] wherein, is a neighborhood of voxel , is the number of output channels, is the index of the neighborhood voxel, is the index of the current voxel, is a bias term, is the input feature vector of voxel u, is a sparse convolution kernel weight matrix, the index of which is determined by the relative offset ;

[0120] Reduce the voxel grid resolution through voxel pooling, and perform feature aggregation on the feature map to output the downsampled voxel features .

[0121] In some embodiments, the coordinate residual prediction module is specifically configured to:

[0122] Input the hidden state and map the hidden state to a high-dimensional space to generate an initial weight matrix ;

[0123]

[0124] wherein, is a learnable projection matrix, is a bias term;

[0125] L2-normalize each row of the initial weight matrix to constrain the numerical range of the weight matrix, to obtain an interpolation weight matrix:

[0126]

[0127] wherein, is the initial interpolation weight matrix corresponding to the iii voxel, is the jth row vector of the matrix , is the L2 norm of the jth row vector, is the normalized interpolation weight vector.

[0128] In some embodiments, the statistical features include mean coordinates (x, y, z) , number of points within a voxel, density (p) ), and normal vector.

[0129] In some embodiments, further comprising:

[0130] a data correction module configured to determine whether the distance between the point cloud and the voxel center is within a preset distance range, and if so, retain the current point cloud, otherwise correct the position of the point cloud until the point cloud is within the distance range; wherein the correction method includes moving the point cloud by a preset distance along a first direction, the first direction being the extension direction of the line connecting the point cloud and the voxel center.

[0131] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the element.

[0132] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the specific embodiments described above, which are only illustrative and not restrictive. Those skilled in the art can make many modifications to the present application without departing from the spirit and scope of the present application and the claims, which are all within the protection scope of the present application.

Claims

1. A method for upsampling a point cloud based on a Mamba model, the method comprising: Comprising: S100, obtaining a sparse point cloud ; wherein, is the number of points, and 3 is the coordinate dimension; S200, dividing the point cloud into at least two voxels; wherein each voxel includes at least one point cloud, which is represented by the voxel center coordinate and the feature of the voxel, and the feature includes density and normal vector; S300, serializing the at least two voxels according to the space-filling curve to obtain a one-dimensional sequence; S400, feature extraction is performed on the one-dimensional sequence to obtain multi-scale voxel features; wherein S400 specifically comprises: S401, calculate the statistical features of the point cloud in each non-empty voxel by feature encoding; the statistical features include mean coordinates , density and normal vector; wherein, is the number of points in the voxel, are the coordinates of the X-axis, Y-axis and Z-axis, respectively; S402, input the statistical feature into a voxel feature matrix wherein is a non-empty voxel number, denotes an initial feature dimension, and ; S403, capture multi-scale features by at least two levels of sparse convolution downsampling, and each level of sparse convolution includes a sparse convolution layer and a voxel pooling; the sparse convolution specifically comprises: calculating non-empty voxels through the sparse convolution layer, using a hash table to accelerate neighborhood query and outputting corresponding feature maps; reducing voxel grid resolution through voxel pooling, and performing feature aggregation on the feature maps to output down-sampled voxel features; S500, input the multi-scale voxel features into a Mamba module based on a selective state space model for time series modeling, and output enhanced time series features and hidden states corresponding to each voxel , is the feature dimension S600, predicting an interpolation weight matrix of each voxel according to the hidden state ; wherein, is an up-sampling multiple of the voxel; S700, based on the interpolation weight matrix and the hidden state, decoding the hidden state into a coordinate residual of a prediction point, generating a prediction point in each voxel according to the coordinate residual and predicting a coordinate offset of the prediction point relative to a voxel center; a prediction point, and predicting a coordinate offset of the prediction point relative to a voxel center; S800, summing the coordinate offsets and the center coordinates of the corresponding voxels to generate new point coordinates, fusing the new points with the point cloud, and outputting a dense point cloud is the total number of points in the point cloud after fusing the new points.

2. The method of claim 1, wherein, In S600, the specific steps of predicting the interpolation weight matrix of each voxel according to the hidden state comprise: S601, input the hidden state, and map the hidden state to a high-dimensional space to generate an initial weight matrix ; wherein, is a learnable projection matrix, is a bias term; S602, L2 normalization is performed on each row of the initial weight matrix to constrain the numerical range of the weight matrix, and an interpolation weight matrix is obtained: wherein, is the initial interpolation weight matrix corresponding to the i-th voxel, is the matrix is the i-th row vector of the matrix j is the L2 norm of the i-th row vector, is the i-th row vector of the matrix j is the L2 norm of the i-th row vector, is the normalized interpolation weight vector.

3. The Mamba model-based point cloud upsampling method according to claim 1 or 2, characterized in that, Before S400, it further comprises the following steps: S350, judging whether the distance between the point cloud and the voxel center conforms to the preset distance range, if yes, retaining the current point cloud, otherwise correcting the position of the point cloud until the point cloud conforms to the distance range; wherein the correction method comprises moving the point cloud by a preset distance along a first direction, and the first direction is the extension direction of the line connecting the point cloud and the voxel center.

4. A point cloud up-sampling system based on Mamba model, characterized in that, Comprising: data acquisition module: configured to acquire a sparse point cloud ; wherein, is the number of points, and 3 is the coordinate dimension; Point cloud voxelization and serialization module: configured to divide the point cloud into at least two voxels, and serialize the at least two voxels according to the space-filling curve to obtain a one-dimensional sequence; Multi-scale feature extraction module: configured to perform feature extraction on the one-dimensional sequence to obtain multi-scale voxel features; wherein the multi-scale feature extraction module is specifically configured to: calculating statistical features of the point cloud in each non-empty voxel by feature coding; the statistical features include mean coordinates , density and normal vector; wherein, is the number of points in the voxel, are the coordinates of the X-axis, Y-axis and Z-axis, respectively; inputting the statistical features into a voxel feature matrix wherein is a number of non-empty voxels, denotes an initial feature dimension, and ; capture multi-scale features by at least two levels of sparse convolution downsampling, and each level of sparse convolution includes a sparse convolution layer and a voxel pooling; the sparse convolution specifically comprises: calculating non-empty voxels through the sparse convolution layer, using a hash table to accelerate neighborhood query and outputting corresponding feature maps; reducing voxel grid resolution through voxel pooling, and performing feature aggregation on the feature maps to output down-sampled voxel features; a time series modeling module configured to input the multi-scale voxel features into a Mamba module based on a selective state space model for time series modeling, and output enhanced time series features and hidden states corresponding to each voxel , is a feature dimension an interpolation weight generation module configured to predict an interpolation weight matrix for each voxel according to the hidden state ; wherein is an up-sampling factor for the voxel The coordinate residual prediction module is configured to decode the hidden state into coordinate residuals of the prediction points based on the interpolation weight matrix and the hidden state, and generate a prediction function within each voxel based on the coordinate residuals. A prediction point is generated, and the coordinate offset of the prediction point relative to the voxel center is predicted. The data fusion module is configured to sum the coordinate offset and the center coordinate of the corresponding voxel, generate new point coordinates, fuse the new points with the point cloud, and output a dense point cloud. The total point number of the point cloud after fusion of the new points.

5. The Mamba model-based point cloud upsampling system of claim 4, wherein, The coordinate residual prediction module is specifically configured to: inputting the hidden states and mapping the hidden states to a high-dimensional space to generate an initial weight matrix ; wherein, is a learnable projection matrix, is a bias term; perform L2 normalization on each row of the initial weight matrix to constrain the numerical range of the weight matrix, and obtain an interpolation weight matrix: in, Let be the initial interpolation weight matrix corresponding to the i-th voxel. For matrix The j row vectors For the first j The L2 norm of a row vector This is the normalized interpolation weight vector.

6. The Mamba model-based point cloud upsampling system of claim 4 or 5, wherein, Further comprising: Data correction module: configured to judge whether the distance between the point cloud and the voxel center conforms to the preset distance range, if yes, retaining the current point cloud, otherwise correcting the position of the point cloud until the point cloud conforms to the distance range; wherein the correction method comprises moving the point cloud by a preset distance along a first direction, and the first direction is the extension direction of the line connecting the point cloud and the voxel center.

Images