Single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion

Abstract

The invention belongs to the technical field of three-dimensional imaging, and particularly relates to a single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion, and the method specifically comprises a point cloud data preprocessing algorithm which is used for converting single-photon laser radar echo data into a three-dimensional point cloud form, main body target extraction in the complex scene point cloud is realized, and calculation denoising processing is completed; a cross-modal feature fusion point cloud reconstruction network is designed, two-dimensional features are introduced multiple times in the reconstruction process, interaction with three-dimensional point cloud features is established, and incomplete point clouds and real complete point clouds are assisted to construct a mapping relation; a KNN segmentation coordinate filling algorithm is designed, point cloud vacant coordinates are calculated and complemented until the resolution of the point cloud vacant coordinates is matched with a two-dimensional infrared image, and two-dimensional image and three-dimensional point cloud pixel-level fusion imaging is achieved.

Description

Technical Field

[0001] The present application belongs to the field of three-dimensional imaging technology, and specifically relates to a single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion. Background Art

[0002] Due to the advantages of lasers such as good directivity, monochromaticity, and coherence, photoelectric ranging and detection technologies based on laser technology have received widespread attention. Single-photon lidar technology, due to its ability to quickly acquire scene depth, can meet the requirements of long-distance and fast imaging, and has become the representative of the new generation of detection technology. Due to the large amount of imaging data generated in its applications in various fields with rich scenes, the demand for incorporating single-photon lidar data into information processing systems for further learning and application is becoming increasingly strong. Although single-photon lidar has significant advantages in sensitivity and imaging speed, it is easily affected by ambient light when collecting data due to its own principle characteristics. The signal-to-noise ratio of the received echo signal is relatively low, resulting in the point cloud data extracted from the signal containing a large number of noise points. These factors have brought the following problems to the processing and application of single-photon lidar data.

[0003] First, the number of signal data points received by a single-photon lidar is determined by the number of detectors. However, due to the high average cost and power consumption per pixel in a single-photon lidar array, the number of pixels is small and the arrangement is relatively sparse, resulting in low resolution of the generated point cloud data. While the point cloud can represent the target object's geometry to a certain extent, the low point cloud density results in a significant loss of surface texture and local features. Traditional point cloud analysis algorithms based on normal vector calculations, as well as single-modal neural network algorithms based on local spatial correlations between points, are unable to effectively process single-photon point clouds. While improvements to the scanning system structure and increased sampling rates can improve point cloud resolution, these operations increase the imaging time and data volume of the lidar system, diminishing the advantages of single-photon imaging systems. Second, due to the high sensitivity of single-photon lidar to optical signals, the return signal is often contaminated by noise. Denoising can suppress these noise and discrete points, but this process inevitably removes some of the target's return signal. These unpredictable signal losses will cause some spatial structures and geometric edge features to be missing after the echo data is converted into a point cloud, making it difficult to match the single-photon point cloud with other modal information and unable to meet the information fusion requirements of downstream tasks. This situation is particularly serious in long-distance imaging scenarios. Summary of the Invention

[0004] In order to effectively improve the resolution of single-photon lidar 3D imaging and enhance the characterization of appearance texture features, this patent uses single-photon 3D point clouds and high-resolution infrared images as data foundations. By establishing feature interaction between 2D and 3D images, it effectively improves the signal-to-noise ratio and concentration of sparse point clouds. It also utilizes the complementarity of bimodal features to improve the point cloud resolution while maintaining the geometric structure of the point cloud. At the same time, it fuses the resolution-enhanced point cloud with the infrared image for color, thereby solving the problem of missing surface texture detail features in the point cloud. Its technical solution is as follows: A single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion includes the following steps: S1. Single-photon target point cloud extraction and denoising to obtain the main target point cloud data; S2. Construct a point cloud completion neural network with cross-modal feature fusion to complete the generated point cloud by mapping the feature associations between the 2D image and the 3D point cloud. S3. Coordinate completion and image fusion.

[0005] Preferably, the single-photon target point cloud extraction steps in step S1 are as follows: The pixel points on each Canny edge in the two-dimensional infrared image are traversed, and the number of feature points similar to the depth map within a certain range around each pixel point is counted. The continuous edge part whose sum is above the set threshold is recorded as the associated edge of the two-dimensional and three-dimensional images; in order to reduce the influence of the unclear edge contour in the two-dimensional infrared image on the feature matching effect, the grayscale level is downsampled on the basis of the original two-dimensional image, and the multi-grayscale image is matched with the point cloud depth map for feature points. The number of two-dimensional and three-dimensional feature matching points around the contour of the multi-grayscale image is counted, and the mean of the sum of the points is set as the discrimination threshold of the associated edge of the two-dimensional and three-dimensional images. If the number of points in the contour neighborhood is higher than the threshold, the point cloud within the range is retained, thereby realizing single-photon point cloud target extraction in complex environments.

[0006] Preferably, the single-photon target point cloud denoising process in step S1 is as follows: By calculating the discrete degree of each point in the point cloud and comparing it with the overall discrete degree of the point cloud, it is determined whether each point in the point cloud is a target point; The average density of the point cloud based on distance is used to estimate the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The calculation expression is: ; ; in, For any two points in the point cloud distance between is the distance between a point in the point cloud and its nearest point, N is the number of points in the point cloud, is the average distance density of the point cloud. Since the distance value of the nearest point of the discrete point in the point cloud is larger than that of the main point of the point cloud, the discrete point Greater than the average distance density of the point cloud , by setting To filter the threshold, discrete points in the point cloud can be removed.

[0007] Preferably, the cross-modal feature fusion point cloud completion neural network includes a core point generation module, and the processing steps of the core point generation module are as follows: The point cloud is input into SALayer to perform hierarchical feature learning on the data. After point cloud sampling, grouping, and local feature extraction, the core point group coordinates representing the local structure of the point cloud are extracted from the original point cloud. and features ; Then Input upsampling Transformer, further generate new core point features by aggregating neighborhood features ; The core point features With the original point cloud features After the connection, a multi-layer perceptron with weight sharing is applied to obtain the rough core point cloud. The process expression is: ; ; in For upsampling Transformer, For connection operation; It is the core point cloud coordinate group after processing.

[0008] Preferably, the cross-modal feature fusion point cloud completion neural network includes a two-dimensional feature interaction module, and the processing steps of the two-dimensional feature interaction module are as follows: The two-dimensional image features of both parties are interacted through a 2D Transformer to enhance the effect of fusion of the output features with the subsequent 3D point cloud features; The cross-attention mechanism of the 2D Transformer module and the depth map features are used to supplement the distance position information missing in the grayscale image, and the perspective features are incorporated to facilitate feature interaction. The expression of the 2D Transformer processing process is: ; ; ; ; in is the depth map feature, is the grayscale single-view image feature, To initially combine two 2D features to generate fusion features, is the position encoding operation, represents the projection viewpoint, is a linear layer, is layer normalization, is a feedforward network, is the two-dimensional enhanced output feature, Generate fusion features for the intermediate.

[0009] Preferably, the cross-modal feature fusion point cloud completion neural network includes a fusion module, and the fusion module processing steps are as follows: After the point cloud is input, three cross-attention operations and one self-attention operation are performed through the decoder to generate an intermediate output point cloud. In the cross-attention operation, the input point cloud features are transformed to generate a query vector Q. The external point cloud features and the two-dimensional image features are combined to generate a key vector K and a value vector V. The attention mechanism is used to generate the weight of the mutual correlation between the features of both modalities, thereby aggregating feature information from different sources. The expression is: ; ; ; ; in 、 、 is the vector corresponding to the weight, is the fused point cloud feature, Point cloud input features for the fusion module, , image key vector, is the image value vector, d is the Q and K matrix dimensions; A self-attention layer is added between the cross-attention layers to perform permutation-invariant transformation on features using the global receptive field to correct for incorrectly integrated information in the image. In the fusion module, the input point cloud features are first combined with the original point cloud features A cross-attention operation is performed to maintain its basic geometric structure, and then it is fused with the two-dimensional image features across attention to generate a fused point cloud feature. After the self-attention operation is performed on the feature, the resulting feature output module is used as the external input feature of the next-level fusion module. Finally, a cross-attention operation is performed on the result feature of the self-attention layer of the upper-level fusion module. After passing through a feature decoder composed of a weight-sharing multi-layer perceptron, the intermediate morphological point cloud generated by the fusion module is obtained as the subsequent input.

[0010] Preferably, the cross-modal feature fusion point cloud completion neural network includes an upsampling module, and the upsampling module processing steps are as follows: Input point cloud After passing through the multi-layer perceptron, the feature vector is obtained and compared with the original point cloud features. Connect to get the initial connection feature vector ,Will( , ) is input into the cross attention module; in order to obtain multiple sets of point clouds and features with different resolutions, the input data is used as the query point cloud for downsampling by sampling the farthest point to obtain point clouds corresponding to different scales 、 and its intermediate connection feature vector 、 ; At the same time, the previous features As Generate support point cloud , repeat FPS sampling with the same sampling rate to obtain another set of multi-scale point clouds 、 and features 、 ; The smallest point cloud is used as the initial data input into the group vector attention block, the channels of each vector in the Value matrix are evenly grouped, and the same weight parameter is shared within each group; The grouped vector attention block applies point cloud geometric features when assigning weights to different channels, and uses the KNN algorithm to analyze the query point cloud. A point in the corresponding support point cloud Perform neighborhood point search, interpolate the features of the k neighborhood points found with the feature of the point, and obtain the feature correspondence. At this time, the local correspondence between the points can be obtained by adding the feature correspondence and the coordinate correspondence. The expression is: ; In the formula is a nonlinear function implemented by a multilayer perceptron, Representing point clouds The features of any point in In support of point cloud A point in the neighborhood, Represents the position encoding information between these two corresponding points; The results obtained from all local points are normalized using the softmax function to obtain the weight corresponding to each local point; The weights are used to calculate the features corresponding to the minimum-sized point cloud, and the feature information is transferred to the larger-sized point cloud through interpolation. Finally, the interpolated point cloud is spliced ​​with the input point cloud to generate a high-resolution point cloud with complete structural features.

[0011] Preferably, the coordinate completion and image fusion steps in step S3 are as follows: S31. Point cloud input; S32. Contour screening: Select a surface point cloud with continuous neighborhood points on one side and vacancies on the other side in the X or Y direction; S33. Round the coordinates and perform surface point screening; S34. Determine whether the XY plane pixel completion is complete; if "yes", end; if "no" proceed to the next step; S35. Set parameters Segmentation of point cloud by KNN; S36. Generate a more subtle local neighborhood, and use the variational differentiation algorithm again in the neighborhood to complete the calculation of the coordinate values ​​of the vacant positions in the point cloud after multiple iterations, before filling the surface holes in the point cloud; S37. Determine the parity of i; if it is an even number, then let = , return to S32; if it is an odd number, then let = / 2, return to S32.

[0012] S38. After filling in the gaps in the point cloud, a high-resolution complete point cloud is generated, which can be used to match with a two-dimensional infrared image of the same resolution and perform pixel-level coloring on the point data. The pixel colors of the infrared image are assigned to the corresponding coordinate point cloud to generate a three-dimensional fused image containing surface texture information.

[0013] Preferably, before completing the surface holes in the point cloud, it is necessary to detect and remove the points in the high-resolution point cloud generated by the feature fusion completion network that exceed the outline of the target main part in the two-dimensional infrared image, and use the pointPolygonTest function to calculate the x and y coordinates of each point in the point cloud, and judge the positive or negative distance between it and the nearest point on the outline, so as to remove the point cloud outside the outline to avoid affecting the subsequent coordinate completion operation; after removing the points outside the outline, the x and y coordinates of each point in the point cloud are rounded according to the pixel value to facilitate the subsequent point cloud fusion and coloring, and the point with the minimum z coordinate corresponding to each pixel position, that is, the nearest point, is filtered according to the pixel coordinates to obtain the surface part of the reconstructed point cloud.

[0014] Preferably, the variable differential coordinate completion algorithm is used to generate reasonable point cloud coordinates of the missing position. When the point cloud surface is a smooth surface, its second-order Laplace operator is equal to 0, which is expressed as: ; is the second-order Laplace operator, which is transformed into discrete form: ; ; Its matrix expression is: ; Solving the corresponding matrix equation can obtain the point cloud completion coordinate value; f(*) is the calculated value of the hole position coordinate.

[0015] Compared with the prior art, this application has the following beneficial effects: Compared with extraction algorithms that use point clouds as the only source of information, the target point cloud positioning and extraction module improves the integrity and accuracy of target point cloud extraction processing in complex scenes; the target point cloud denoising module has a faster processing speed and lower point cloud discreteness compared to the neighborhood search denoising method.

[0016] Compared with the unimodal completion network that only uses point cloud data as input, the cross-modal feature fusion point cloud reconstruction network maps the feature associations between two-dimensional images and three-dimensional point clouds, and the completed point cloud shows better results in both chamfer distance and F-Score indicators; compared with the processing solution that directly uses the original point cloud as input, the SG module reduces the network complexity while ensuring structural integrity; compared with the single feature interaction algorithm, the fusion module can perform cross-modal feature associations multiple times to generate a point cloud with a more complete structure; compared with other attention upsampling modules, the upsampling module uses a grouped vector attention mechanism to reduce the noise generation rate while reducing the complexity of network parameters.

[0017] The KNN segmentation coordinate filling algorithm can realize the completion of multi-scale point cloud gaps. Compared with the single-scale coordinate calculation completion scheme, it can generate high-density point clouds more completely and evenly. Compared with the linear coordinate calculation, the variable differential coordinate calculation algorithm can generate smoother and more reasonable point cloud surfaces, and is more suitable for completing point clouds in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the overall algorithm flow chart for single-photon point cloud processing and three-dimensional imaging; Figure 2 This is the flowchart of the point cloud data preprocessing algorithm; Figure 3 Reconstruct the network structure diagram for cross-modal feature fusion point cloud; Figure 4 It is a 3D point cloud feature encoder; Figure 5 It is the core point generator module; Figure 6 It is a two-dimensional and three-dimensional feature fusion module; Figure 7 For the group vector attention upsampling module; (a) the overall architecture of the upsampling module, (b) the internal architecture of the GVA layer; Figure 8Complete the flowchart for KNN local variational differentiation. DETAILED DESCRIPTION

[0019] The following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which this application belongs. It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to this application.

[0020] A single-photon point cloud 3D imaging method based on cross-modal feature fusion. The overall algorithm flow chart is as follows: Figure 1 As shown. First, the original point cloud data is processed by the single-photon target point cloud extraction and denoising algorithm, and the point cloud depth map is associated with the feature points of the two-dimensional infrared image. The degree of association of the image area is judged according to the number of feature points in the neighborhood of the edge pixels of the two-dimensional image, and the point cloud is extracted within the contour with a high degree of matching, thereby reducing the degree of discreteness of the point cloud; then, through the point cloud completion network with cross-modal feature fusion, the denoised low-resolution point cloud is processed by the core point generator, multi-level fusion module and multi-scale feature interpolation upsampling module, and is successively combined with the original point cloud features, projected depth map features, infrared image features and enhanced image. The proposed method establishes feature associations based on image features, uses two-dimensional features to guide the structural optimization of point cloud image features and completes the missing parts in the point cloud, providing rich semantic information for the point cloud features. Finally, the KNN local variational differential completion algorithm is used to perform variational calculus calculations in the neighborhood of the hole position to obtain numerical changes for smooth coordinate completion. The data classification characteristics of the parameter iteration KNN algorithm are used to obtain multi-scale local point clouds as data neighborhoods. The coordinate completion processing of the hole positions of point clouds with various precisions is completed in multiple cycles, generating a high-resolution fused image that contains both degree distance information and surface texture information.

[0021] A preprocessing algorithm for single-photon lidar point cloud data, including a target point cloud positioning extraction module and a target point cloud denoising module. The algorithm flow chart is as follows: Figure 1 shown.

[0022] The target point cloud location extraction module calculates the similarity between images of different modalities and extracts similar feature points from the real-world 2D infrared image and the depth map of the single-photon point cloud mapping. To remove background point clouds from complex point clouds and locate the main target point cloud area, the pixels on each Canny edge in the 2D infrared image are traversed, and the number of feature points similar to the depth map within a certain range around each pixel is counted. Continuous edges whose sum exceeds a set threshold are recorded as 2D and 3D image-related edges. To reduce the impact of unclear edge contours in the 2D infrared image on feature matching, the original 2D image is grayscale downsampled, and feature point matching is performed between the multi-grayscale image and the point cloud depth map. The number of 2D and 3D feature matching points around the contour of the multi-grayscale image is counted, and the mean of the sum of these points is set as the threshold for identifying 2D and 3D image-related edges. If the number of points within the contour neighborhood exceeds the threshold, the point cloud within that range is retained, thus achieving single-photon point cloud target extraction in complex environments.

[0023] The target point cloud denoising module determines whether each point in the point cloud is a target point by calculating the discrete degree of each point in the point cloud and comparing it with the overall discrete degree of the point cloud. The distance-based point cloud average density estimates the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The calculation expression is: ; ; in, For any two points in the point cloud distance between is the distance between a point in the point cloud and its nearest point, N is the number of points in the point cloud, is the average distance density of the point cloud. Since the distance value of the nearest point of the discrete point in the point cloud is larger than that of the main point of the point cloud, the discrete point Greater than the average distance density of the point cloud , by setting To filter the threshold, discrete points in the point cloud can be removed.

[0024] A cross-modal feature fusion point cloud reconstruction network, the 2D encoder, 3D encoder, core point generation module (Seed Generator, SG) and 2D feature interaction module 2D Trans constitute the network feature extraction part, the fusion module (Fusion Block) and upsampling module (Upsample Block) constitute the point cloud completion part, the overall structure diagram is shown in the figure Figure 2 shown.

[0025] The 2D encoder uses ResNet-18 as the backbone network; the 3D encoder uses Figure 3As shown in the figure, features in the point cloud are extracted and the cardinality of the point cloud is reduced by stacking and interleaving multiple layers of edge convolution layers (EC) and self-attention graph pooling layers (SAGP). Compared with aggregating the global features of the point cloud with only a single global embedding, the loss of local information caused by learning a single global feature is avoided. The self-attention pooling layer is used to calculate weights, which determine which parts should be retained in the pooling process, reducing the size of the point cloud while retaining the important structural feature cloud.

[0026] Core point generation module such as Figure 4 As shown, the point cloud is input into SALayer to perform hierarchical feature learning on the data. After point cloud sampling, grouping, and local feature extraction, the core point group coordinates representing the local structure of the point cloud are extracted from the original point cloud. and features , can obtain accurate geometric and semantic information while showing robustness to changes in point cloud density at different locations. Input upsampling Transformer, further generate new core point features by aggregating neighborhood features , further enhancing the feature’s ability to express point cloud structure information. In order to optimize the core point cloud generation effect, With the original point cloud features After the connection, a multi-layer perceptron with weight sharing is applied to obtain the rough core point cloud. The process expression is: ; ; in For upsampling Transformer, For the connection operation, It is the core point cloud coordinate group after processing.

[0027] The 2D feature interaction module uses a 2D Transformer to interact with the 2D image features of both parties, thereby enhancing the effect of the output features after fusion with the subsequent 3D point cloud features, further optimizing the overall network effect. In order to establish the semantic association of feature information between the depth map and the grayscale image, the cross-attention mechanism of the 2D Transformer module is used with the depth map features to supplement the distance and position information missing in the grayscale image. At the same time, the perspective feature is incorporated to facilitate feature interaction. The expression of the 2D Transformer processing process is: ; ; ; ; in is the depth map feature, is the grayscale single-view image feature, To initially combine two 2D features to generate fusion features, is the position encoding operation, represents the projection viewpoint, is a linear layer, (LayerNormalization) is layer normalization, is a feed-forward network. is the two-dimensional enhanced output feature, Generate fusion features for the intermediate.

[0028] The fusion module structure is as follows Figure 5 As shown in the figure, after the point cloud is input, three cross-attention operations and one self-attention operation are performed, and the decoder generates an intermediate output point cloud. In the cross-attention operation, the input point cloud features are transformed to generate a query vector Q, and the external point cloud features and the two-dimensional image features are combined to generate a key vector K and a value vector V. The attention mechanism generates the weight of the correlation between the features of both modalities, thereby aggregating feature information from different sources. Its expression is: ; ; ; ; in 、 、 is the vector corresponding to the weight, is the fused point cloud feature, Point cloud input features for the fusion module, , image key vector, is the image value vector, and d is the dimension of the Q and K matrices.

[0029] A self-attention layer is added between the cross-attention layers to perform permutation-invariant transformation on the features using the global receptive field to correct the incorrectly integrated information in the image. In the fusion module, the input point cloud features are first combined with the original point cloud features. A cross-attention operation is performed to maintain its basic geometric structure, and then it is fused with the two-dimensional image features across attention to generate a fused point cloud feature. After the self-attention operation is performed on the feature, the resulting feature output module is used as the external input feature of the next-level fusion module. Finally, a cross-attention operation is performed on the result feature of the self-attention layer of the upper-level fusion module. After passing through a feature decoder composed of a weight-sharing multi-layer perceptron, the intermediate morphological point cloud generated by the fusion module is obtained as the subsequent input.

[0030] The upsampling module structure is as follows Figure 6 As shown, the input point cloud After passing through the multi-layer perceptron, the feature vector is obtained and compared with the original point cloud features. Connect ,Will( , ) is input into the cross attention module. In order to obtain multiple sets of point clouds and features with different resolutions, the input data is used as the query point cloud (Query Point Cloud) through the farthest point sampling (FPS) to obtain point clouds corresponding to different scales. 、 and its characteristics 、 ; At the same time, the previous features As Generate support point cloud (Support Point Cloud), repeat FPS sampling at the same sampling rate to obtain another set of multi-scale point clouds and features 、 、 The smallest point cloud is used as the initial data input into the grouped vector attention block, and the channels of each vector in the Value matrix are evenly grouped, and the same weight parameter is shared within each group. Compared with the traditional vector attention module, the grouped vector attention module optimizes parameter efficiency by designing attention grouping, which reduces the number of parameters used while solving the model performance problem, improves the efficiency of the model and retains the advantages of vector attention. At the same time, the grouped vector attention block applies point cloud geometric features when assigning weights to different channels, and uses the KNN algorithm to search for the points in the query point cloud. A point in the corresponding support point cloud Perform neighborhood point search, interpolate the features of the K neighborhood points found with the feature of the point, and obtain the feature correspondence. At this time, the local correspondence between the points can be obtained by adding the feature correspondence and the coordinate correspondence. The expression is: ; In the formula is a nonlinear function implemented by a multilayer perceptron, Representing point clouds The features of any point in In support of point cloud A point in the neighborhood, The positional encoding information between these two corresponding points is represented. The results obtained for all local points are normalized using the softmax function to obtain the weight corresponding to each local point. The weights are then used to calculate the features corresponding to the smallest point cloud. This feature information is then transferred to a larger point cloud using the cubic interpolation method in PointNet++. Finally, the interpolated point cloud is concatenated with the input point cloud to generate a high-resolution point cloud with complete structural features.

[0031] The steps of coordinate completion and image fusion are as follows, and the flow chart is as follows Figure 7 As shown: S31. Point cloud input; S32. Contour screening: Select a surface point cloud with continuous neighborhood points on one side and vacancies on the other side in the X or Y direction; S33. Round the coordinates and perform surface point screening; S34. Determine whether the XY plane pixel completion is complete; if "yes", end; if "no" proceed to the next step; S35. Set parameters Segmentation of point cloud by KNN; S36. Generate a more subtle local neighborhood, and use the variational differentiation algorithm again in the neighborhood to complete the calculation of the coordinate values ​​of the vacant positions in the point cloud after multiple iterations, before filling the surface holes in the point cloud; S37. Determine the parity of i; if it is an even number, then let = , return to S32; if it is an odd number, then let = / 2, return to S32.

[0032] S38. After filling in the gaps in the point cloud, a high-resolution complete point cloud is generated, which can be used to match with a two-dimensional infrared image of the same resolution and perform pixel-level coloring on the point data. The pixel colors of the infrared image are assigned to the corresponding coordinate point cloud to generate a three-dimensional fused image containing surface texture information.

[0033] The variable differential coordinate completion algorithm is used to generate reasonable point cloud coordinates for missing locations. When the point cloud surface is a smooth surface, its second-order Laplace operator is equal to 0, which is expressed as: ; Transform it into discrete form: ; ; Its matrix expression is: ; Solving the corresponding matrix equation can obtain the point cloud completion coordinate values.

[0034] The KNN local variational differential completion algorithm segments the point cloud using a multi-scale KNN algorithm, providing the algorithm with point cloud neighborhoods of various scales during the variational differential coordinate filling process. When determining the initial point to start the KNN algorithm, a surface point cloud with continuous neighborhood points on one side and gaps on the other side in either the X or Y direction is selected, rather than a random starting point. After segmenting the point cloud and performing a variational differential operation based on the initial K value, the processed point cloud is further segmented using the KNN algorithm with a modified k parameter value, generating a more refined local neighborhood. Finally, the variational differential algorithm is applied again within this neighborhood, and after multiple iterations, the coordinate values ​​of the gaps in the point cloud are calculated. Before completing surface holes in the point cloud, it is necessary to detect and remove points in the high-resolution point cloud generated by the feature fusion completion network that extend beyond the outline of the target's main body in the 2D infrared image. The pointPolygonTest function is used to calculate the x and y coordinates of each point in the point cloud and determine the sign of the distance to the nearest point on the outline. Points outside the outline are then removed to prevent them from affecting subsequent coordinate completion operations. After removing points outside the outline, the x and y coordinates of each point in the point cloud are rounded to the pixel value to facilitate subsequent point cloud fusion and coloring. The point with the minimum z coordinate, or the nearest point, corresponding to each pixel is selected based on the pixel coordinates to obtain the surface portion of the reconstructed point cloud. After completing the gaps in the point cloud, a complete high-resolution point cloud is generated, which can be used to match the 2D infrared image of the same resolution and perform pixel-level coloring on the point data. The infrared image pixel colors are assigned to the corresponding coordinate point cloud, generating a 3D fused image containing surface texture information.

[0035] The foregoing description is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A single-photon point cloud 3D imaging method based on cross-modal feature fusion, characterized in that: The following steps are involved: S1. Single-photon target point cloud extraction and denoising to obtain the main target point cloud data; S2. Construct a point cloud completion neural network with cross-modal feature fusion to complete the generated point cloud by mapping the feature associations between the 2D image and the 3D point cloud. S3. Coordinate completion and image fusion.

2. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1 is characterized in that: The steps for extracting single-photon target point cloud in step S1 are as follows: The pixel points on each Canny edge in the two-dimensional infrared image are traversed, and the number of feature points similar to the depth map within a certain range around each pixel point is counted. The continuous edge part whose sum is above the set threshold is recorded as the associated edge of the two-dimensional and three-dimensional images; in order to reduce the influence of the unclear edge contour in the two-dimensional infrared image on the feature matching effect, the grayscale level is downsampled on the basis of the original two-dimensional image, and the multi-grayscale image is matched with the point cloud depth map for feature points. The number of two-dimensional and three-dimensional feature matching points around the contour of the multi-grayscale image is counted, and the mean of the sum of the points is set as the discrimination threshold of the associated edge of the two-dimensional and three-dimensional images. If the number of points in the contour neighborhood is higher than the threshold, the point cloud within the range is retained, thereby realizing single-photon point cloud target extraction in complex environments.

3. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1 is characterized in that: The single-photon target point cloud denoising process in step S1 is as follows: By calculating the discrete degree of each point in the point cloud and comparing it with the overall discrete degree of the point cloud, it is determined whether each point in the point cloud is a target point; The average density of the point cloud based on distance is used to estimate the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The calculation expression is: ； ； in, For any two points in the point cloud distance between is the distance between a point in the point cloud and its nearest point, N is the number of points in the point cloud, is the average distance density of the point cloud. Since the distance value of the nearest point of the discrete point in the point cloud is larger than that of the main point of the point cloud, the discrete point Greater than the average distance density of the point cloud , by setting To filter the threshold, discrete points in the point cloud can be removed.

4. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The cross-modal feature fusion point cloud completion neural network includes a core point generation module. The processing steps of the core point generation module are as follows: The point cloud is input into SALayer to perform hierarchical feature learning on the data. After point cloud sampling, grouping, and local feature extraction, the core point group coordinates representing the local structure of the point cloud are extracted from the original point cloud. and features ; Then Input upsampling Transformer, further generate new core point features by aggregating neighborhood features ; The core point features With the original point cloud features After the connection, a weight-sharing multilayer perceptron is applied to obtain the rough core point cloud. The process expression is: ； ； in For upsampling Transformer, For connection operation; It is the core point cloud coordinate group after processing.

5. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The cross-modal feature fusion point cloud completion neural network includes a two-dimensional feature interaction module. The processing steps of the two-dimensional feature interaction module are as follows: The two-dimensional image features of both parties are interacted through a 2D Transformer to enhance the effect of fusion of the output features with the subsequent 3D point cloud features; The cross-attention mechanism of the 2D Transformer module and the depth map features are used to supplement the distance position information missing in the grayscale image, and the perspective features are incorporated to facilitate feature interaction. The expression of the 2D Transformer processing process is: ； ； ； ； in is the depth map feature, is the grayscale single-view image feature, To initially combine two 2D features to generate fusion features, is the position encoding operation, represents the projection viewpoint, is a linear layer, is layer normalization, is a feedforward network, is the two-dimensional enhanced output feature, Generate fusion features for the intermediate.

6. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The cross-modal feature fusion point cloud completion neural network includes a fusion module, and the processing steps of the fusion module are as follows: After the point cloud is input, three cross-attention operations and one self-attention operation are performed through the decoder to generate an intermediate output point cloud. In the cross-attention operation, the input point cloud features are transformed to generate a query vector Q. The external point cloud features and the two-dimensional image features are combined to generate a key vector K and a value vector V. The attention mechanism is used to generate the weight of the mutual correlation between the features of both modalities, thereby aggregating feature information from different sources. The expression is: ； ； ； ； in 、 、 is the vector corresponding to the weight, is the fused point cloud feature, Point cloud input features for the fusion module, , image key vector, is the image value vector, d is the Q and K matrix dimensions; A self-attention layer is added between the cross-attention layers to perform permutation-invariant transformation on features using the global receptive field to correct for incorrectly integrated information in the image. In the fusion module, the input point cloud features are first combined with the original point cloud features A cross-attention operation is performed to maintain its basic geometric structure, and then it is fused with the two-dimensional image features across attention to generate a fused point cloud feature. After the self-attention operation is performed on the feature, the resulting feature output module is used as the external input feature of the next-level fusion module. Finally, a cross-attention operation is performed on the result feature of the self-attention layer of the upper-level fusion module. After passing through a feature decoder composed of a weight-sharing multi-layer perceptron, the intermediate morphological point cloud generated by the fusion module is obtained as the subsequent input.

7. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The cross-modal feature fusion point cloud completion neural network includes an upsampling module. The upsampling module processes the following steps: Input point cloud After passing through the multi-layer perceptron, the feature vector is obtained and compared with the original point cloud features. Connect to get the initial connection feature vector ,Will( , ) is input into the cross attention module; in order to obtain multiple sets of point clouds and features with different resolutions, the input data is used as the query point cloud for downsampling by sampling the farthest point to obtain point clouds corresponding to different scales 、 and its intermediate connection feature vector 、 ; At the same time, the previous features As Generate support point cloud , repeat FPS sampling with the same sampling rate to obtain another set of multi-scale point clouds 、 and features 、 ; The smallest point cloud is used as the initial data input into the group vector attention block, the channels of each vector in the Value matrix are evenly grouped, and the same weight parameter is shared within each group; The grouped vector attention block applies point cloud geometric features when assigning weights to different channels, and uses the KNN algorithm to analyze the query point cloud. A point in the corresponding support point cloud Perform neighborhood point search, interpolate the features of the k neighborhood points found with the feature of the point, and obtain the feature correspondence. At this time, the local correspondence between the points can be obtained by adding the feature correspondence and the coordinate correspondence. The expression is: ; In the formula is a nonlinear function implemented by a multilayer perceptron, Representing point clouds The features of any point in In support of point cloud A point in the neighborhood, Represents the position encoding information between these two corresponding points; The results obtained from all local points are normalized using the softmax function to obtain the weight corresponding to each local point; The weights are used to calculate the features corresponding to the minimum-sized point cloud, and the feature information is transferred to the larger-sized point cloud through interpolation. Finally, the interpolated point cloud is spliced ​​with the input point cloud to generate a high-resolution point cloud with complete structural features.

8. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: Step S3: coordinate completion and image fusion steps are as follows: S31. Point cloud input; S32. Contour screening: Select a surface point cloud with continuous neighborhood points on one side and vacancies on the other side in the X or Y direction; S33. Round the coordinates and perform surface point screening; S34. Determine whether the XY surface pixel completion is complete; if "yes", end; if "no" proceed to the next step; S35. Set parameters Segmentation of point cloud by KNN; S36. Generate a more subtle local neighborhood, and use the variational differentiation algorithm again in the neighborhood to complete the calculation of the coordinate values ​​of the vacant positions in the point cloud after multiple iterations, before filling the surface holes in the point cloud; S37. Determine the parity of i; if it is an even number, then let = , return to S32; if it is an odd number, then let = / 2, return to S32; S38. After filling in the gaps in the point cloud, a high-resolution complete point cloud is generated, which can be used to match with a two-dimensional infrared image of the same resolution and perform pixel-level coloring on the point data. The pixel colors of the infrared image are assigned to the corresponding coordinate point cloud to generate a three-dimensional fused image containing surface texture information.

9. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 8, characterized in that: Before completing the surface holes in the point cloud, it is necessary to detect and remove the points in the high-resolution point cloud generated by the feature fusion completion network that exceed the outline of the target main part in the two-dimensional infrared image. The pointPolygonTest function is used to calculate the x and y coordinates of each point in the point cloud to determine the positive or negative distance between it and the nearest point on the outline, so as to remove the point cloud outside the outline to avoid affecting the subsequent coordinate completion operation. After removing the points outside the outline, the x and y coordinates of each point in the point cloud are rounded according to the pixel value to facilitate the subsequent point cloud fusion and coloring, and the point with the minimum z coordinate corresponding to each pixel position is filtered according to the pixel coordinates, that is, the nearest point, to obtain the surface part of the reconstructed point cloud.

10. The single-photon point cloud three-dimensional imaging method based on cross-modal feature fusion according to claim 1, characterized in that: The variable differential coordinate completion algorithm is used to generate reasonable point cloud coordinates for missing locations. When the point cloud surface is a smooth surface, its second-order Laplace operator is equal to 0, which is expressed as: ； is the second-order Laplace operator, which is transformed into discrete form: ； Its matrix expression is: ； Solving the corresponding matrix equation can obtain the point cloud completion coordinate value; f(*) is the calculated value of the hole position coordinate.

Images