A single-photon point cloud 3D imaging method based on cross-modal feature fusion

Abstract

The application belongs to the technical field of three-dimensional imaging, and particularly relates to a single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion, which specifically comprises a point cloud data preprocessing algorithm, which is used for converting single-photon laser radar echo data into a three-dimensional point cloud form, realizing main target extraction in a complex scene point cloud and completing calculation denoising processing; a cross-modal feature fusion point cloud reconstruction network is designed, two-dimensional features are introduced multiple times in the reconstruction process and interacted with three-dimensional point cloud features to assist in establishing a mapping relationship between a defective point cloud and a real complete point cloud; a KNN segmentation coordinate filling algorithm is designed, and the missing coordinates of the point cloud are calculated and completed until the resolution matches a two-dimensional infrared image, so that two-dimensional image and three-dimensional point cloud pixel-level fusion imaging is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional imaging, and particularly relates to a single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion. BACKGROUND

[0002] Due to the performance advantages of good directivity, good monochromaticity and good coherence, photoelectric ranging and detection technologies based on laser technology have attracted widespread attention. Single-photon laser radar technology can meet the requirements of long-distance rapid imaging due to its characteristics of quickly acquiring scene depth, and has become a representative of a new generation of detection technology. Due to the demand for further learning and application of single-photon laser radar data into the information processing system, a large amount of imaging data is generated in the application of single-photon laser radar technology in different fields with rich scenes. Although single-photon laser radar has significant advantages in sensitivity and imaging speed, it is limited by its own principle characteristics and is easily affected by ambient light when collecting data. The signal-to-noise ratio of the received echo signal is relatively low, which leads to the problem that the point cloud data extracted from the signal contains a large number of noise points. These factors bring the following problems to the processing and application of single-photon laser radar data.

[0003] Firstly, the number of signal data points received by single-photon laser radar each time is determined by the number of detectors. However, the number of pixels of the array detector in single-photon laser radar is small due to the high average cost and power consumption of each pixel, and the arrangement is relatively sparse, which leads to low resolution of the generated point cloud data. At this time, the point cloud can represent the geometric structure of the target object to a certain extent, but the low point cloud density will lead to a large number of missing surface texture features and local features. Neither traditional point cloud analysis algorithms based on normal vectors nor single-modal neural network algorithms based on the local spatial correlation between points can effectively process single-photon point clouds. Although the point cloud resolution can be optimized by improving the scanning system structure and enhancing the sampling rate, these operations will lead to an increase in imaging time and data volume of the laser radar system, which will weaken the advantages of the single-photon imaging system. Secondly, due to the high sensitivity of single-photon laser radar to light signals, a large amount of noise signals will be mixed in the echo signal. After denoising, the noise points and discrete points can be suppressed, but part of the target object echo signal will be removed during this process. These unpredictable signal losses will lead to the loss of some spatial structure and geometric edge features after the echo data is converted into point clouds, so that the single-photon point cloud is difficult to match with other modal information, and cannot meet the demand of information fusion in downstream tasks, which is particularly serious in long-distance imaging scenarios. SUMMARY

[0004] In order to effectively improve the three-dimensional imaging resolution of single-photon lidar and enhance the characterization effect of external texture features, the patent takes single-photon three-dimensional point cloud and high-resolution infrared image as the data basis, effectively improves the signal-to-noise ratio and concentration degree of sparse point cloud by establishing the feature interaction between two-dimensional image and three-dimensional image, improves the point cloud resolution by using the complementarity of double modal features on the premise of maintaining the geometric structure of point cloud, and solves the problem of missing surface texture detail features of point cloud by coloring and fusing the point cloud with enhanced resolution and infrared image. The technical scheme is:

[0005] A single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion, comprising the following steps:

[0006] S1. Single-photon target point cloud extraction and denoising processing, obtaining main target point cloud data;

[0007] S2. Constructing a point cloud completion neural network based on cross-modal feature fusion, completing the generated point cloud by mapping the feature correlation between two-dimensional image and three-dimensional point cloud;

[0008] S3. Coordinate completion and image fusion.

[0009] Preferably, the single-photon target point cloud extraction step in step S1 is as follows:

[0010] Each pixel point on the Canny edge in the two-dimensional infrared image is traversed, the number of feature points similar to the depth map within a certain range around each pixel point is counted, and the sum of the continuous edge parts above the set threshold is recorded as the two-dimensional and three-dimensional image associated edge; In order to reduce the influence of unclear edge contour in two-dimensional infrared image on feature matching effect, the original two-dimensional image is down-sampled in gray level number, and the multi-gray level number image is matched with the point cloud depth map to count the number of two-dimensional and three-dimensional feature matching points around the contour, and the mean value of the total number of points is set as the discrimination threshold of the two-dimensional and three-dimensional image associated edge. The point number in the contour neighborhood is higher than the threshold, and the point cloud in the range is reserved to realize single-photon point cloud target extraction in complex environment.

[0011] Preferably, the single-photon target point cloud denoising processing step in step S1 is as follows:

[0012] By calculating the dispersion degree of each point in the point cloud and comparing it with the overall dispersion degree of the point cloud, it is judged whether each point in the point cloud is a target point;

[0013] The average density of the point cloud based on distance is calculated by calculating the average value of the distance of each point in the point cloud to estimate the density of the point cloud distribution, and the calculation expression is:

[0014] ;

[0015] ;

[0016] wherein, is the distance between any two points in the point cloud, is the distance between a point in the point cloud and its nearest point, N is the number of points in the point cloud, is the average distance density of the point cloud, since the nearest point distance value of the discrete point in the point cloud is larger than that of the main point of the point cloud, that is, the discrete point is greater than the average distance density of the point cloud , by setting as the screening threshold, the discrete points in the point cloud can be removed.

[0017] Preferably, the cross-modal feature fusion point cloud completion neural network comprises a core point generation module, and the core point generation module has the following processing steps:

[0018] The point cloud is input into the SALayer to learn hierarchical features, and the local feature is extracted through point cloud sampling, grouping, and processing, so that the core point group coordinates representing the local structure of the point cloud are extracted from the original point cloud and features ;

[0019] Then, the is input into the up-sampling Transformer to further generate new core point features by aggregating neighborhood features ; After the core point features and the original point cloud features are connected, a weight-shared multi-layer perception is applied to obtain a rough core point cloud, and the process expression is as follows:

[0020] ;

[0021] ;

[0022] wherein is the up-sampling Transformer, is the connection operation, is the core point cloud coordinate group after processing.

[0023] Preferably, the cross-modal feature fusion point cloud completion neural network comprises a two-dimensional feature interaction module, and the two-dimensional feature interaction module has the following processing steps:

[0024] A 2D Transformer is used to interact with the two-dimensional image features of both sides, so as to enhance the effect of the output features after the fusion with the subsequent three-dimensional point cloud features;

[0025] The cross-attention mechanism of the 2D Transformer module is used to supplement the distance position information missing in the gray image, and the view angle feature is also integrated to help feature interaction. The 2D Transformer processing process expression is as follows:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] wherein is a depth map feature, is a gray single-view image feature, is a fusion feature generated by initially splicing two two-dimensional features, is a position encoding operation, represents a projection view point, is a linear layer, is layer normalization, is a feedforward network, is a two-dimensional enhanced output feature, is an intermediate generated fusion feature.

[0031] Preferably, the point cloud completion neural network for cross-modal feature fusion comprises a fusion module, and the processing steps of the fusion module are as follows:

[0032] After the point cloud is input, three cross-attention operations and one self-attention operation are performed, and then an intermediate output point cloud is generated by a decoder. In the cross-attention operation, the input point cloud feature is transformed to generate a query vector Q, and the external point cloud feature and the two-dimensional image feature generate a key vector K and a value vector V. The mutual correlation between the features of the two modalities is generated through the attention mechanism, so as to aggregate the feature information from different sources. The expression is as follows:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] wherein , , is a corresponding weight of a vector, is a fused point cloud feature, Point cloud input features for the fusion module, , image key vector, is the image value vector, d is the Q and K matrix dimensions;

[0038] A self-attention layer is added between the cross-attention layers to perform permutation-invariant transformation on features using the global receptive field to correct for incorrectly integrated information in the image.

[0039] In the fusion module, the input point cloud features are first combined with the original point cloud features A cross-attention operation is performed to maintain its basic geometric structure, and then it is fused with the two-dimensional image features across attention to generate a fused point cloud feature. After the self-attention operation is performed on the feature, the resulting feature output module is used as the external input feature of the next-level fusion module. Finally, a cross-attention operation is performed on the result feature of the self-attention layer of the upper-level fusion module. After passing through a feature decoder composed of a weight-sharing multi-layer perceptron, the intermediate morphological point cloud generated by the fusion module is obtained as the subsequent input.

[0040] Preferably, the cross-modal feature fusion point cloud completion neural network includes an upsampling module, and the upsampling module processing steps are as follows:

[0041] Input point cloud After passing through the multi-layer perceptron, the feature vector is obtained and compared with the original point cloud features. Connect to get the initial connection feature vector ,Will( , ) is input into the cross attention module; in order to obtain multiple sets of point clouds and features with different resolutions, the input data is used as the query point cloud for downsampling by sampling the farthest point to obtain point clouds corresponding to different scales 、 and its intermediate connection feature vector 、 ; At the same time, the previous features As Generate support point cloud , repeat FPS sampling with the same sampling rate to obtain another set of multi-scale point clouds 、 and features 、 ;

[0042] The smallest point cloud is used as the initial data input into the group vector attention block, the channels of each vector in the Value matrix are evenly grouped, and the same weight parameter is shared within each group;

[0043] The grouped vector attention block applies point cloud geometric features when assigning weights to different channels, and uses the KNN algorithm to analyze the query point cloud. A point in the corresponding support point cloud Perform neighborhood point search, interpolate the features of the k neighborhood points found with the feature of the point, and obtain the feature correspondence. At this time, the local correspondence between the points can be obtained by adding the feature correspondence and the coordinate correspondence. The expression is:

[0044] ;

[0045] In the formula is a nonlinear function implemented by a multilayer perceptron, Representing point clouds The features of any point in In support of point cloud A point in the neighborhood, Represents the position encoding information between these two corresponding points;

[0046] The results obtained from all local points are normalized using the softmax function to obtain the weight corresponding to each local point;

[0047] The weights are used to calculate the features corresponding to the minimum-sized point cloud, and the feature information is transferred to the larger-sized point cloud through interpolation. Finally, the interpolated point cloud is spliced ​​with the input point cloud to generate a high-resolution point cloud with complete structural features.

[0048] Preferably, the coordinate completion and image fusion steps in step S3 are as follows:

[0049] S31. Point cloud input;

[0050] S32. Contour screening: Select a surface point cloud with continuous neighborhood points on one side and vacancies on the other side in the X or Y direction;

[0051] S33. Round the coordinates and perform surface point screening;

[0052] S34. Determine whether the XY plane pixel completion is complete; if "yes", end; if "no" proceed to the next step;

[0053] S35. Set parameters Segmentation of point cloud by KNN;

[0054] S36. Generate a more subtle local neighborhood, and use the variational differentiation algorithm again in the neighborhood to complete the calculation of the coordinate values ​​of the vacant positions in the point cloud after multiple iterations, before filling the surface holes in the point cloud;

[0055] S37. Determine the parity of i; if it is an even number, then let = , return S32; if it is "odd", then let = / 2, return S32.

[0056] S38. After the completion of the vacancy in the point cloud, a high-resolution complete point cloud is generated, which can be used to match the two-dimensional infrared image with the same resolution and pixel-level color assignment for point data. The pixel color of the infrared image is assigned to the corresponding coordinate point cloud to generate a three-dimensional fusion image containing surface texture information.

[0057] Preferably, before completing the surface cavity in the point cloud, the points generated by the feature fusion completion network that exceed the contour of the target main part in the two-dimensional infrared image need to be detected and removed. The pointPolygonTest function is used to calculate the x and y coordinates of each point in the point cloud to determine the positive and negative distance between the nearest points on the contour, so as to remove the points outside the contour to avoid affecting the subsequent coordinate completion operation; after removing the points outside the contour, the x and y coordinates of each point in the point cloud are rounded according to the pixel value to facilitate subsequent point cloud fusion and color assignment, and the z coordinate of the nearest point is selected as the minimum point at the corresponding position of each pixel to obtain the surface part of the reconstructed point cloud.

[0058] Preferably, the differential coordinate completion algorithm is used to generate reasonable vacancy position point cloud coordinates. When the point cloud surface is a smooth surface, the second-order Laplacian is equal to 0, which is represented as:

[0059] ;

[0060] is the second-order Laplacian, which is transformed into a discrete form as:

[0061] ;

[0062] ;

[0063] The matrix expression is:

[0064] ;

[0065] Solving the corresponding matrix equation can obtain the point cloud completion coordinate value; f(*) is the coordinate calculation value of the cavity position.

[0066] Compared with the prior art, the application has the following advantages:

[0067] Compared with the extraction algorithm using point cloud as the only information source, the target point cloud positioning extraction module improves the integrity and accuracy of the target point cloud extraction processing in a complex scene; compared with the neighborhood search denoising method, the target point cloud denoising module has faster processing speed and lower point cloud dispersion.

[0068] Compared with the single-modal completion network using only point cloud data as input, the cross-modal feature fusion point cloud reconstruction network maps the feature correlation between two-dimensional images and three-dimensional point clouds, and the generated point cloud shows better results in chamfer distance and F-Score indicators. The SG module reduces the complexity of the network while ensuring the integrity of the structure compared to the processing scheme of directly using the original point cloud as input. The fusion module can perform cross-modal feature correlation multiple times to generate a more complete point cloud compared to a single feature interaction algorithm. Compared with other attention upsampling modules, the upsampling module uses a grouping vector attention mechanism to reduce the complexity of network parameters and the generation rate of noise points.

[0069] The KNN segmentation coordinate filling algorithm can realize multi-scale point cloud vacancy completion, and can generate high-density point clouds more completely and uniformly compared with single-scale coordinate calculation completion scheme. The variable differential coordinate calculation algorithm can generate smoother and more reasonable point cloud surfaces compared with linear coordinate calculation, and is more suitable for point cloud completion calculation in real scene. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The overall algorithm flowchart for single-photon point cloud processing and three-dimensional imaging;

[0071] Figure 2 The point cloud data preprocessing algorithm flowchart;

[0072] Figure 3 The cross-modal feature fusion point cloud reconstruction network structure diagram;

[0073] Figure 4 The three-dimensional point cloud feature encoder;

[0074] Figure 5 The core point generator module;

[0075] Figure 6 The two / three-dimensional feature fusion module;

[0076] Figure 7 The grouping vector attention upsampling module; wherein (a) the overall architecture of the upsampling module, (b) the internal architecture of the GVA layer;

[0077] Figure 8 The KNN local variable differential completion flowchart. DETAILED DESCRIPTION

[0078] The following detailed description is exemplary in nature and provided to introduce the application. All technical and scientific terms used herein have the same meaning as commonly understood to one of ordinary skill in the art to which this application belongs unless otherwise specifically defined. It should be noted that the use of particular exemplary embodiments herein does not impose a limitation on the application.

[0079] A single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion, the algorithm overall flow chart as shown in Figure 1 First, the single-photon target point cloud extraction and denoising algorithm processes the original point cloud data, constructs the point cloud depth map and two-dimensional infrared image feature point association, and judges the image region association degree according to the number of feature points in the edge pixel neighborhood of the two-dimensional image. In the high matching degree contour, point cloud extraction is carried out, so as to reduce the dispersion degree of point cloud; then through the point cloud completion network of cross-modal feature fusion, the low resolution point cloud after denoising is processed by the core point generator, multi-level fusion module and multi-scale feature interpolation upsampling module. The features are associated with the original point cloud feature, the projected depth map feature, the infrared image feature and the enhanced image feature in turn, the two-dimensional feature is used to guide the structure optimization and completion of the point cloud image feature, and the missing parts in the point cloud are completed, which provides rich semantic information for the point cloud feature; finally, through the KNN local partial differential completion algorithm, the numerical change is realized by the variation calculus calculation in the neighborhood of the hollow position to complete the smooth coordinate completion, and the multi-scale local point cloud is obtained as the data neighborhood through the data classification characteristics of the parameter iterative KNN algorithm, and the coordinate completion processing of various precision point cloud hollow positions is completed in multiple cycles to generate a high-resolution fusion image containing distance information and surface texture information.

[0080] A preprocessing algorithm for single-photon laser radar point cloud data, containing a target point cloud positioning extraction module and a target point cloud denoising module, the algorithm flow chart as shown in Figure 1

[0081] ​The target point cloud location extraction module calculates the similarity between images of different modalities and extracts similar feature points from the real-world 2D infrared image and the depth map of the single-photon point cloud mapping. To remove background point clouds from complex point clouds and locate the main target point cloud area, the pixels on each Canny edge in the 2D infrared image are traversed, and the number of feature points similar to the depth map within a certain range around each pixel is counted. Continuous edges whose sum exceeds a set threshold are recorded as 2D and 3D image-related edges. To reduce the impact of unclear edge contours in the 2D infrared image on feature matching, the original 2D image is grayscale downsampled, and feature point matching is performed between the multi-grayscale image and the point cloud depth map. The number of 2D and 3D feature matching points around the contour of the multi-grayscale image is counted, and the mean of the sum of these points is set as the threshold for identifying 2D and 3D image-related edges. If the number of points within the contour neighborhood exceeds the threshold, the point cloud within that range is retained, thus achieving single-photon point cloud target extraction in complex environments.

[0082] The target point cloud denoising module determines whether each point in the point cloud is a target point by calculating the discrete degree of each point in the point cloud and comparing it with the overall discrete degree of the point cloud. The distance-based point cloud average density estimates the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The calculation expression is:

[0083] ;

[0084] ;

[0085] in, For any two points in the point cloud distance between is the distance between a point in the point cloud and its nearest point, N is the number of points in the point cloud, is the average distance density of the point cloud. Since the distance value of the nearest point of the discrete point in the point cloud is larger than that of the main point of the point cloud, the discrete point Greater than the average distance density of the point cloud , by setting To filter the threshold, discrete points in the point cloud can be removed.

[0086] A cross-modal feature fusion point cloud reconstruction network, the 2D encoder, 3D encoder, core point generation module (Seed Generator, SG) and 2D feature interaction module 2D Trans constitute the network feature extraction part, the fusion module (Fusion Block) and upsampling module (Upsample Block) constitute the point cloud completion part, the overall structure diagram is shown in the figure Figure 2 shown.

[0087] The two-dimensional encoder selects ResNet-18 as the backbone network; the three-dimensional encoder is as shown in Figure 3 The features in the point cloud are extracted by stacking and interleaving multiple layers of edge convolution (EC) and self-attention graph pooling (SAGP) layers, and the cardinality of the point cloud is reduced. Compared with aggregating the global features of the point cloud by using only a single global embedding, the local information loss caused by single global feature learning is avoided. The self-attention pooling layer is used to calculate the weights, which determine which parts should be retained during the pooling process. The size of the point cloud is reduced while the important structural features of the point cloud are retained.

[0088] The core point generation module is as shown in Figure 4 The point cloud is input into the SALayer to learn hierarchical features. After point cloud sampling, grouping, and local feature extraction, the core point group coordinates representing the local structure of the point cloud are extracted from the original point cloud and the features , which can obtain accurate geometric and semantic information while being robust to changes in point cloud density at different positions. Then is input into the up-sampling Transformer to further generate new core point features by aggregating neighborhood features, further enhancing the ability of the features to represent the structure information of the point cloud. In order to optimize the generation effect of the core point cloud, the is connected with the original point cloud feature , and then a weight-shared multi-layer perceptron is applied to obtain a rough core point cloud. The process expression is as follows:

[0089] ;

[0090] ;

[0091] wherein is the up-sampling Transformer, is the connection operation, is the processed core point cloud coordinate group.

[0092] The two-dimensional feature interaction module uses a 2D Transformer to interact with the two-dimensional image features of both sides, thereby enhancing the effect of the output features after fusion with the subsequent three-dimensional point cloud features, and further optimizing the overall network effect. In order to establish the semantic association between the features of the depth map and the grayscale image, the cross-attention mechanism of the 2D Transformer module is used to supplement the distance position information missing in the grayscale image, and the perspective feature is also integrated to help feature interaction. The processing process expression of the 2D Transformer is as follows:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] wherein is a depth map feature, is a gray-scale monocular image feature, is a preliminary fused feature generated by fusing two two-dimensional features, is a positional encoding operation, denotes a projection view point, is a linear layer, Layer Normalization is a layer normalization, is a Feed-Forward Network, is a two-dimensional enhanced output feature, is an intermediate fused feature.

[0098] The fusion module structure is shown in Figure 5 , which generates an intermediate output point cloud after decoding the input point cloud after three cross-attention operations and one self-attention operation. In the cross-attention operation, the input point cloud feature is transformed to generate a query vector Q, and the external point cloud feature and the two-dimensional image feature are used to generate a key vector K and a value vector V. The weight of the mutual correlation between the features of the two modalities is generated through the attention mechanism, so as to aggregate the feature information from different sources. The expression is:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] wherein , , is a vector corresponding weight, is a fused point cloud feature, is a point cloud input feature of the fusion module, , an image key vector, is an image value vector, and d is the dimension of the Q and K matrices.

[0104] A self-attention layer is added between the cross-attention layers to perform permutation-invariant transformation on the features using the global receptive field to correct the incorrectly integrated information in the image. In the fusion module, the input point cloud features are first combined with the original point cloud features. A cross-attention operation is performed to maintain its basic geometric structure, and then it is fused with the two-dimensional image features across attention to generate a fused point cloud feature. After the self-attention operation is performed on the feature, the resulting feature output module is used as the external input feature of the next-level fusion module. Finally, a cross-attention operation is performed on the result feature of the self-attention layer of the upper-level fusion module. After passing through a feature decoder composed of a weight-sharing multi-layer perceptron, the intermediate morphological point cloud generated by the fusion module is obtained as the subsequent input.

[0105] The upsampling module structure is as follows Figure 6 As shown, the input point cloud After passing through the multi-layer perceptron, the feature vector is obtained and compared with the original point cloud features. Connect ,Will( , ) is input into the cross attention module. In order to obtain multiple sets of point clouds and features with different resolutions, the input data is used as the query point cloud (Query Point Cloud) through the farthest point sampling (FPS) to obtain point clouds corresponding to different scales. 、 and its characteristics 、 ; At the same time, the previous features As Generate support point cloud (Support Point Cloud), repeat FPS sampling at the same sampling rate to obtain another set of multi-scale point clouds and features 、 、 The smallest point cloud is used as the initial data input into the grouped vector attention block, and the channels of each vector in the Value matrix are evenly grouped, and the same weight parameter is shared within each group. Compared with the traditional vector attention module, the grouped vector attention module optimizes parameter efficiency by designing attention grouping, which reduces the number of parameters used while solving the model performance problem, improves the efficiency of the model and retains the advantages of vector attention. At the same time, the grouped vector attention block applies point cloud geometric features when assigning weights to different channels, and uses the KNN algorithm to search for the points in the query point cloud. A point in the corresponding support point cloud Perform neighborhood point search, interpolate the features of the K neighborhood points found with the feature of the point, and obtain the feature correspondence. At this time, the local correspondence between the points can be obtained by adding the feature correspondence and the coordinate correspondence. The expression is:

[0106] ;

[0107] wherein is a nonlinear function implemented by a multi-layer perceptron, represents the feature of any point in the point cloud , represents a point in the neighboring points in the support point cloud , represents the positional encoding information between the two corresponding points. The result obtained for all local points is normalized using the softmax function to obtain the weight size corresponding to each local point. Then, the minimum size point cloud corresponding feature is calculated using the weight, and the feature information is transmitted to the larger size point cloud through the cubic interpolation method in PointNet++, and finally the interpolation processed point cloud and the input point cloud are spliced to generate a high-resolution point cloud with complete structure characteristics.

[0108] The coordinate completion and image fusion steps are as follows, and the flow chart is shown in Figure 7 .

[0109] S31. Point cloud input;

[0110] S32. Contour screening: select a surface point cloud with continuous neighborhood points on one side and gaps on the other side in X or Y orientation;

[0111] S33. Coordinate rounding and surface point screening;

[0112] S34. Determine whether the XY plane pixel is completed; if “yes”, end; if “no”, proceed to the next step;

[0113] S35. Set parameters by KNN to segment the point cloud;

[0114] S36. Generate a more subtle local neighborhood, and use the variational differential algorithm again in the neighborhood. After multiple iterations, the coordinates of the missing positions in the point cloud are calculated. Before completing the surface hole completion in the point cloud,

[0115] S37. Determine the parity of i; if it is “even”, let = , return to S32; if it is “odd”, let = / 2, return to S32.

[0116] S38. After the completion of the vacancy in the point cloud, a high-resolution complete point cloud is generated, which can be used to match the two-dimensional infrared image with the same resolution and pixel-level color assignment for point data. The pixel color of the infrared image is assigned to the corresponding coordinate point cloud to generate a three-dimensional fusion image containing surface texture information.

[0117] The differential coordinate completion algorithm is used to generate reasonable vacancy position point cloud coordinates. When the point cloud surface is a smooth surface, the second-order Laplacian is equal to 0, which is represented as:

[0118] ;

[0119] Transforming it into a discrete form gives:

[0120] ;

[0121] ;

[0122] The matrix expression is:

[0123] ;

[0124] Solving the corresponding matrix equation can obtain the point cloud completion coordinate value.

[0125] The KNN local differential completion algorithm divides the point cloud by the multi-scale KNN algorithm, and provides multiple scales of point cloud neighborhoods for the algorithm during the differential coordinate filling process. When determining the initial point as the starting point of the KNN algorithm, a surface point cloud with continuous neighborhood points on one side and gaps on the other side in the X or Y direction is selected, rather than randomly selecting the starting point. Then, the point cloud is segmented according to the initial K value and the differential operation is performed, and the processed point cloud is further segmented by the KNN algorithm with the modified k parameter value, thereby generating more subtle local neighborhoods. Finally, the differential algorithm is used again in the neighborhood, and after multiple iterations, the coordinates of the missing positions in the point cloud are calculated. Before completing the surface holes in the point cloud, the points generated by the feature fusion completion network that exceed the outline of the target main part in the two-dimensional infrared image are detected and removed. The pointPolygonTest function is used to calculate the x and y coordinates of each point in the point cloud, and determine the positive and negative distance between the nearest points on the outline, so as to remove the points outside the outline to avoid affecting the subsequent coordinate completion operation. After removing the points outside the outline, the x and y coordinates of each point in the point cloud are rounded according to the pixel value, so as to facilitate the subsequent point cloud fusion coloring, and the minimum point corresponding to each pixel position z coordinate, i.e. the nearest point, is selected according to the pixel coordinate, to obtain the surface part of the reconstructed point cloud. After completing the missing part in the point cloud, a high-resolution complete point cloud is generated, which can be used to match the two-dimensional infrared image with the same resolution and perform pixel-level coloring for the point data. The pixel color of the infrared image is assigned to the corresponding coordinate point cloud, and a three-dimensional fusion image containing surface texture information is generated.

[0126] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A single-photon point cloud 3D imaging method based on cross-modal feature fusion, characterized in that: The following steps are involved: S1. Single-photon target point cloud extraction and denoising to obtain the main target point cloud data; S2. Construct a point cloud completion neural network with cross-modal feature fusion to complete the generated point cloud by mapping the feature associations between the 2D image and the 3D point cloud. The cross-modal feature fusion point cloud completion neural network includes a two-dimensional feature interaction module. The processing steps of the two-dimensional feature interaction module are as follows: The two-dimensional image features of both parties are interacted through a 2D Transformer to enhance the effect of fusion of the output features with the subsequent 3D point cloud features; The cross-attention mechanism of the 2D Transformer module and the depth map features are used to supplement the distance position information missing in the grayscale image, and the perspective features are incorporated to facilitate feature interaction. The expression of the 2D Transformer processing process is: ； ； ； ； in is the depth map feature, is the grayscale single-view image feature, To initially combine two 2D features to generate fusion features, is the position encoding operation, represents the projection viewpoint, is a linear layer, is layer normalization, is a feedforward network, is the two-dimensional enhanced output feature, Generate fusion features for the intermediate; The cross-modal feature fusion point cloud completion neural network includes a fusion module, and the processing steps of the fusion module are as follows: After the point cloud is input, three cross-attention operations and one self-attention operation are performed through the decoder to generate an intermediate output point cloud. In the cross-attention operation, the input point cloud features are transformed to generate a query vector. Q , external point cloud features and two-dimensional image features generate key vectors K Sum vector V , the attention mechanism is used to generate the weights of the mutual correlation between the modal features of both parties, thereby aggregating feature information from different sources. Its expression is: in 、 、 is the vector corresponding to the weight, is the fused point cloud feature, for The point cloud input features of the fusion module, , image key vector, is the image value vector, d is the dimension of Q and K matrices; A self-attention layer is added between the cross-attention layers to perform permutation-invariant transformation on features using the global receptive field to correct for incorrectly integrated information in the image. In the fusion module, the input point cloud features are first combined with the original point cloud features A cross-attention operation is performed to maintain its basic geometric structure, and then it is fused with the two-dimensional image features through cross-attention to generate a fused point cloud feature. After the self-attention operation is performed on the feature, the resulting feature output module is used as the external input feature of the next-level fusion module. Finally, a cross-attention operation is performed on the result feature of the self-attention layer of the upper-level fusion module. After passing through a feature decoder composed of a weight-sharing multi-layer perceptron, the intermediate morphological point cloud generated by the fusion module is obtained as the subsequent input; S3. Coordinate completion and image fusion.

2. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1 is characterized in that: The steps for extracting single-photon target point cloud in step S1 are as follows: The pixel points on each Canny edge in the two-dimensional infrared image are traversed, and the number of feature points similar to the depth map within a certain range around each pixel point is counted. The continuous edge part whose sum is above the set threshold is recorded as the associated edge of the two-dimensional and three-dimensional images; in order to reduce the influence of the unclear edge contour in the two-dimensional infrared image on the feature matching effect, the grayscale level is downsampled on the basis of the original two-dimensional image, and the multi-grayscale image is matched with the point cloud depth map for feature points. The number of two-dimensional and three-dimensional feature matching points around the contour of the multi-grayscale image is counted, and the mean of the sum of the points is set as the discrimination threshold of the associated edge of the two-dimensional and three-dimensional images. If the number of points in the contour neighborhood is higher than the threshold, the point cloud within the range is retained, thereby realizing single-photon point cloud target extraction in complex environments.

3. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1 is characterized in that: The single-photon target point cloud denoising process in step S1 is as follows: By calculating the discrete degree of each point in the point cloud and comparing it with the overall discrete degree of the point cloud, it is determined whether each point in the point cloud is a target point; The average density of the point cloud based on distance is used to estimate the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The calculation expression is: in, For any two points in the point cloud distance between is the distance between a point in the point cloud and its nearest point, N is the number of points in the point cloud, is the average distance density of the point cloud. Since the distance value of the nearest point of the discrete point in the point cloud is larger than that of the main point of the point cloud, the discrete point Greater than the average distance density of the point cloud , by setting To filter the threshold, discrete points in the point cloud can be removed.

4. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The cross-modal feature fusion point cloud completion neural network includes a core point generation module. The processing steps of the core point generation module are as follows: The point cloud is input into SALayer to perform hierarchical feature learning on the data. After point cloud sampling, grouping, and local feature extraction, the core point group coordinates representing the local structure of the point cloud are extracted from the original point cloud. and features ; Then Input upsampling Transformer, further generate new core point features by aggregating neighborhood features ; The core point features With the original point cloud features After the connection, a multi-layer perceptron with weight sharing is applied to obtain the rough core point cloud. The process expression is: ； ； in For upsampling Transformer, For connection operation; It is the core point cloud coordinate group after processing.

5. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The cross-modal feature fusion point cloud completion neural network includes an upsampling module. The upsampling module processes the following steps: Input point cloud After passing through the multi-layer perceptron, the feature vector is obtained and compared with the original point cloud features. Connect to get the initial connection feature vector ,Will( ， ) is input into the cross attention module; in order to obtain multiple sets of point clouds and features with different resolutions, the input data is used as the query point cloud for downsampling by sampling the farthest point to obtain point clouds corresponding to different scales 、 and its intermediate connection feature vector 、 ; At the same time, the previous features As Generate support point cloud , repeat FPS sampling with the same sampling rate to obtain another set of multi-scale point clouds 、 and features 、 ; The smallest point cloud is used as the initial data input into the group vector attention block, the channels of each vector in the Value matrix are evenly grouped, and the same weight parameter is shared within each group; The grouped vector attention block applies point cloud geometric features when assigning weights to different channels, and uses the KNN algorithm to analyze the query point cloud. A point in the corresponding support point cloud Search for neighboring points and k Interpolate the features of the neighboring points with the features of the point to obtain the feature correspondence. At this time, the local correspondence between the points can be obtained by adding the feature correspondence and the coordinate correspondence. The expression is: ; In the formula is a nonlinear function implemented by a multilayer perceptron, Representing point clouds The features of any point in In support of point cloud A point among the neighboring points, Represents the position encoding information between these two corresponding points; The results obtained from all local points are normalized using the softmax function to obtain the weight corresponding to each local point; The weights are used to calculate the features corresponding to the minimum-sized point cloud, and the feature information is transferred to the larger-sized point cloud through interpolation. Finally, the interpolated point cloud is spliced ​​with the input point cloud to generate a high-resolution point cloud with complete structural features.

6. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: Step S3: coordinate completion and image fusion steps are as follows: S31. Point cloud input; S32. Contour filter: Select one X Direction or Y A surface point cloud with continuous neighborhood points on one side and gaps on the other side in terms of orientation; S33. Round the coordinates and perform surface point screening; S34. Determine whether the XY surface pixel completion is complete; if "yes", end; if "no" proceed to the next step; S35. Set parameters Segmentation of point cloud by KNN; S36. Generate a more subtle local neighborhood, and use the variational differentiation algorithm again in the neighborhood to complete the calculation of the coordinate values ​​of the vacant positions in the point cloud after multiple iterations, before filling the surface holes in the point cloud; S37. Determine the parity of i; if it is an even number, then let = , return to S32; if it is an odd number, then let = / 2 , return to S32. 7.S38. After filling in the gaps in the point cloud, a high-resolution complete point cloud is generated. This can be used to match a 2D infrared image of the same resolution and perform pixel-level coloring on the point data. The pixel colors of the infrared image are assigned to the corresponding coordinate point cloud, generating a 3D fused image containing surface texture information.

8. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 6, characterized in that: Before completing the surface holes in the point cloud, it is necessary to detect and remove the points in the high-resolution point cloud generated by the feature fusion completion network that are beyond the outline of the target main body in the two-dimensional infrared image. The pointPolygonTest function is used to calculate the x and y coordinates of each point in the point cloud to determine the positive and negative distance between the point and the nearest point on the outline, so as to remove the point cloud outside the outline to avoid affecting the subsequent coordinate completion operation. After removing the points outside the outline, the point cloud points are x, y The coordinates are rounded according to the pixel values ​​to facilitate the subsequent point cloud fusion and coloring, and the point with the minimum z coordinate, i.e. the nearest point, of each pixel corresponding to the pixel is filtered according to the pixel coordinates to obtain the surface part of the reconstructed point cloud.

9. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The variable differential coordinate completion algorithm is used to generate reasonable point cloud coordinates for missing locations. When the point cloud surface is a smooth surface, its second-order Laplace operator is equal to 0, which is expressed as: is the second-order Laplace operator, Transform it into discrete form: Its matrix expression is: Solving the corresponding matrix equation can obtain the point cloud completion coordinate values; f(*) Calculate the coordinates of the hole location.

Images