A missing information supplement modeling method based on multimodal fusion

Abstract

The present application relates to a missing information supplement modeling method based on multimodal fusion, which comprises the following steps: encoding an RGB image by using color components to obtain a first feature; encoding a point cloud image by using spatial position coding to obtain a second feature; aligning the feature spaces of the first feature and the second feature, and performing feature fusion to obtain a joint feature; reconstructing an image based on the joint feature to obtain a reconstructed RGB image; performing frequency domain transformation on the point cloud image and the reconstructed RGB image, respectively, and decomposing color high-frequency features, color low-frequency features, point cloud high-frequency features, and point cloud low-frequency features; using the color high-frequency features and point cloud high-frequency features as source domains, and the color low-frequency features and point cloud low-frequency features as target domains, aligning and fusing the source domains with the target domains to obtain point cloud supplementary features; performing local geometry and color feature extraction and three-dimensional surface reconstruction on the point cloud supplementary features, and combining the extracted features and the reconstructed three-dimensional surface to obtain a three-dimensional model.

Description

Technical Field

[0001] The present application relates to the field of drone information processing technology, and in particular to a missing information supplement modeling method based on multimodal fusion. Background Art

[0002] Traditional multimodal information fusion techniques have several limitations. When processing data from different modalities, these techniques often fail to fully utilize and integrate the inherent connections and complementarities between them, resulting in the loss of some useful information. Furthermore, due to the relative independence of each modality, correlation between them is poor. This is particularly evident when performing cross-modal tasks, as they lack in-depth consideration of the inertia between different modal data. Furthermore, when applied to unmanned equipment, these limitations of traditional fusion methods prevent the accurate and effective execution of subsequent tasks (such as modeling), hindering the improvement of the perception and decision-making capabilities of unmanned equipment in complex environments. Summary of the Invention

[0003] Based on this, it is necessary to provide a missing information supplement modeling method based on multimodal fusion, which includes:

[0004] S1: Acquire RGB images and point cloud images of unmanned equipment;

[0005] S2: Encode the RGB image using color components to obtain the first feature; encode the point cloud image using spatial position encoding to obtain the second feature;

[0006] S3: Align the feature spaces of the first feature and the second feature, perform feature fusion, and obtain a joint feature; reconstruct the image based on the joint feature to obtain a reconstructed RGB image;

[0007] S4: Perform frequency domain transformation on the point cloud image and the reconstructed RGB image, and decompose the color high-frequency features, color low-frequency features, point cloud high-frequency features, and point cloud low-frequency features;

[0008] S5: Use color high-frequency features and point cloud high-frequency features as source domains, and color low-frequency features and point cloud low-frequency features as target domains. Align and fuse the source and target domains to obtain point cloud supplementary features.

[0009] S6: Extract local geometric and color features and reconstruct the three-dimensional surface of the point cloud supplementary features, combine the extracted features and the reconstructed three-dimensional surface to obtain a three-dimensional model.

[0010] Preferably, encoding the RGB image using color components includes:

[0011] Decompose the RGB image into three two-dimensional matrices according to color channels;

[0012] For each two-dimensional matrix, encode it according to the distribution characteristics of its corresponding pixel values ​​to obtain the corresponding encoding matrix;

[0013] Fuse the three encoding matrices to obtain a high-dimensional feature matrix;

[0014] Perform global average pooling on the high-dimensional feature matrix to obtain the first feature.

[0015] Preferably, encoding the point cloud image using spatial position coding includes:

[0016] Encode the spatial position of the point cloud image to obtain a high-dimensional representation of each point;

[0017] Input the high-dimensional representation of each point into the multi-layer perceptron and output the feature representation of each point;

[0018] For any point, a neighborhood set is obtained by using the K-nearest neighbor algorithm or sphere radius search, and the feature representations of each point in the neighborhood set are aggregated to obtain the local feature representation of the corresponding point;

[0019] Aggregate the local feature representations of all points in the point cloud image to obtain the second feature.

[0020] Preferably, the feature space of aligning the first feature and the second feature includes:

[0021] Project the first feature into the shared space through a convolutional neural network;

[0022] Project the second feature into the shared space through a multi-layer perceptron;

[0023] The projected first feature is upsampled to the dimension of the second feature using the nearest neighbor interpolation method to align the dimensions of the first feature and the second feature.

[0024] Preferably, the process of obtaining the joint features is:

[0025] The upsampled first feature is linearly fused with the projected second feature to obtain the joint feature.

[0026] Preferably, the image reconstruction based on the joint features includes:

[0027] Calculate the adjacency matrix between the RGB image and the point cloud image through spatial neighbors;

[0028] Input the adjacency matrix and joint features into the multi-layer perceptron to obtain preliminary reconstruction features;

[0029] The preliminary reconstruction features are passed through the Linear function to obtain the reconstruction weights;

[0030] The reconstruction weights are passed through the softmax activation function and multiplied element-by-element with the preliminary reconstruction features to obtain the reconstructed features;

[0031] The reconstructed features are converted into a reconstructed RGB image through the decoder.

[0032] Preferably, S3 further includes: constructing a total loss function based on a weighted combination of a modality alignment loss, a distribution alignment loss, and an image reconstruction loss between the first feature and the second feature, and minimizing the total loss function to optimize the reconstruction process;

[0033] The modality alignment loss is based on the first and second features and is calculated using the NT-Xent loss function;

[0034] The distribution alignment loss is based on the first and second features and is calculated using the maximum mean difference loss function;

[0035] The image reconstruction loss is based on the first feature and the second feature and is calculated using the mean square error.

[0036] Preferably, S4 includes:

[0037] The 2D-BTSA function is used to perform wavelet transform on the reconstructed RGB image and decompose it into the first vector matrix of different frequency bands;

[0038] The point cloud image is subjected to wavelet transform using 3D-TDMF-FE function and decomposed into second vector matrices of different frequency bands;

[0039] The first vector matrix of the high frequency band is used as the color high frequency feature, and the first vector matrix of the low frequency band is used as the color low frequency feature;

[0040] The second vector matrix of the high-frequency band is used as the high-frequency feature of the point cloud, and the second vector matrix of the low-frequency band is used as the low-frequency feature of the point cloud.

[0041] Preferably, aligning and fusing the source domain and the target domain to obtain point cloud supplementary features includes:

[0042] Take any feature in the source domain as the anchor feature, the feature in the target domain from the same image as the anchor feature as the positive sample feature, and the features in the target domain other than the positive sample feature as the negative sample feature; calculate the triplet loss based on the anchor feature, positive sample feature and negative sample feature;

[0043] Align the source domain with the target domain by minimizing the triplet loss to obtain the aligned features;

[0044] The aligned features are input into the multi-layer perceptron to obtain the point cloud supplementary features.

[0045] Preferably, S6 includes:

[0046] The supplementary features of the point cloud include: the three-dimensional coordinates of the point in the point cloud and the color features. The color features include: red channel component, green channel component, and blue channel component.

[0047] For any point in the point cloud image, use the sphere radius to search for its neighborhood point set;

[0048] Based on a point and its neighborhood point set, the local normal vector of the point is calculated by principal component analysis; the curvature of the point is calculated based on the ratio of the eigenvalues ​​of the covariance matrix of the neighborhood points;

[0049] Calculate the color gradient based on the color features of the points and the three-dimensional coordinates of the points;

[0050] Integrate the point's three-dimensional coordinates, local normal vector, curvature, red channel component, green channel component, blue channel component, and color gradient to obtain a high-dimensional feature vector;

[0051] Input the high-dimensional feature vector into the multi-layer perceptron to generate high-dimensional features of the point;

[0052] Based on the neighborhood points of the point and the distance from the neighborhood point to the neighborhood center point, the second joint feature is calculated using point cloud neighborhood interpolation;

[0053] Input the second joint feature and the local normal vector into the Poisson surface reconstruction function to generate a local three-dimensional surface corresponding to the point;

[0054] Traverse all points in the point cloud image, obtain the high-dimensional features of all points, and generate the three-dimensional surface of the point cloud;

[0055] The high-dimensional features of all points and the three-dimensional surface of the point cloud are combined to obtain a three-dimensional model.

[0056] Beneficial Effects: By integrating multimodal information, the potential connections between different modal data can be better explored and utilized, thereby enhancing the perception and decision-making capabilities of unmanned vehicles in complex environments. This new approach is expected to promote the advancement of information processing technology for unmanned equipment and provide support for achieving higher levels of autonomy and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1This is a flowchart of the missing information supplement modeling method based on multimodal fusion in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0061] like Figure 1 As shown, this embodiment provides a missing information supplement modeling method based on multimodal fusion, the method comprising:

[0062] S1: Obtain RGB images and point cloud images of the unmanned equipment.

[0063] S2: Encode the RGB image using color components to obtain the first feature; encode the point cloud image using spatial position encoding to obtain the second feature.

[0064] Furthermore, encoding the RGB image using color components includes:

[0065] Decompose the RGB image into three two-dimensional matrices according to color channels;

[0066] For each two-dimensional matrix, encode it according to the distribution characteristics of its corresponding pixel values ​​to obtain the corresponding encoding matrix;

[0067] Linearly fuse the three encoding matrices to obtain a high-dimensional feature matrix;

[0068] Perform global average pooling on the high-dimensional feature matrix to obtain the first feature.

[0069] The encoding method based on pixel features of color components can effectively extract significant information of the image and enhance the robustness and accuracy of task processing.

[0070] Furthermore, encoding the point cloud image using spatial position coding includes:

[0071] Encode the spatial position of the point cloud image to obtain a high-dimensional representation of each point;

[0072] Input the high-dimensional representation of each point into the multi-layer perceptron and output the feature representation of each point;

[0073] For any point, a neighborhood set is obtained by using the K-nearest neighbor algorithm or sphere radius search, and the feature representations of each point in the neighborhood set are aggregated (maximum pooling or weighted summation can be used for aggregation) to obtain the local feature representation of the corresponding point;

[0074] Aggregate (global pooling or attention mechanism can be used for aggregation) the local feature representation of all points in the point cloud image to obtain the second feature.

[0075] S3: Align the feature spaces of the first feature and the second feature, perform feature fusion, and obtain a joint feature; reconstruct the image based on the joint feature to obtain a reconstructed RGB image.

[0076] Furthermore, the feature space of aligning the first feature and the second feature includes:

[0077] Project the first feature into the shared space through a convolutional neural network;

[0078] Project the second feature into the shared space through a multi-layer perceptron;

[0079] The projected first feature is upsampled to the dimension of the second feature using the nearest neighbor interpolation method to align the dimensions of the first feature and the second feature.

[0080] Furthermore, the process of obtaining the joint features is:

[0081] The upsampled first feature is linearly fused with the projected second feature to obtain the joint feature.

[0082] Furthermore, the image reconstruction based on the joint features includes:

[0083] Calculate the adjacency matrix between the RGB image and the point cloud image through spatial neighbors;

[0084] Input the adjacency matrix and joint features into the multi-layer perceptron to obtain preliminary reconstruction features;

[0085] The preliminary reconstruction features are passed through the Linear function to obtain the reconstruction weights;

[0086] The reconstruction weights are passed through the softmax activation function and multiplied element-by-element with the preliminary reconstruction features to obtain the reconstructed features;

[0087] The reconstructed features are converted into a reconstructed RGB image through the decoder.

[0088] The decoder is used to perform deconvolution operations and the context features are integrated to enhance the details.

[0089] In this embodiment, step S3 further includes: constructing a total loss function based on a weighted combination of a modality alignment loss, a distribution alignment loss, and an image reconstruction loss between the first feature and the second feature, and minimizing the total loss function to optimize the reconstruction process;

[0090] The modality alignment loss is based on the first and second features and is calculated using the NT-Xent loss function;

[0091] The distribution alignment loss is based on the first and second features and is calculated using the maximum mean difference loss function;

[0092] The image reconstruction loss is based on the first feature and the second feature and is calculated using the mean square error.

[0093] S4: Perform frequency domain transformation on the point cloud image and the reconstructed RGB image respectively, and decompose the color high-frequency features, color low-frequency features, point cloud high-frequency features, and point cloud low-frequency features.

[0094] Specifically, this step includes:

[0095] The 2D-BTSA function (two-dimensional texture spectrum analyzer, emphasizing the extraction of multi-dimensional texture information from two-dimensional images) is used to perform wavelet transform on the reconstructed RGB image and decompose it into the first vector matrix of different frequency bands. Specifically, features are extracted from the following three aspects:

[0096] Horizontal detail function: captures horizontal edge and texture information in the image;

[0097] Vertical detail function: captures vertical edge and texture information in the image;

[0098] Diagonal detail function: captures diagonal edge and texture information in the image.

[0099] The 3D-TDMF-FE function (a three-dimensional multi-frequency feature extractor that highlights the ability to capture multi-frequency features of three-dimensional point clouds, including reflection intensity) is used to perform wavelet transform on the point cloud image and decompose it into second vector matrices of different frequency bands. Specifically, features are extracted from the following three aspects:

[0100] Horizontal detail function: captures the horizontal edge and texture information of the point cloud;

[0101] Vertical detail function: captures the vertical edge and texture information of the point cloud;

[0102] Reflection intensity function: captures the reflection intensity information of the point cloud.

[0103] The first vector matrix of the high-frequency band is used as the color high-frequency feature to represent local detail information (such as edges, textures, and details); the first vector matrix of the low-frequency band is used as the color low-frequency feature to represent global characteristics (such as shape and lighting distribution);

[0104] The second vector matrix in the high-frequency band is used as the high-frequency feature of the point cloud to represent local geometric information (such as surface and point cloud density changes); the second vector matrix in the low-frequency band is used as the low-frequency feature of the point cloud to represent the overall shape characteristics and light reflection intensity.

[0105] The vector matrix in the high-frequency band usually corresponds to the detail coefficients of the wavelet transform, which reflect the rapid changes of the signal in the local area (such as edge, texture and detail information);

[0106] The vector matrix in the low-frequency band usually corresponds to the approximate coefficients, which reflect the overall trend and global structure of the signal (such as overall shape, illumination distribution and reflection intensity).

[0107] Through frequency domain analysis, point clouds and RGB images are divided into source domain and target domain. Two-dimensional and three-dimensional wavelet transforms are used to extract local and global information from multiple angles. Through feature alignment and optimization, efficient modal fusion is achieved to provide high-quality feature input for subsequent tasks.

[0108] S5: Use color high-frequency features and point cloud high-frequency features as source domains, and color low-frequency features and point cloud low-frequency features as target domains. Align and fuse the source domains with the target domains to obtain point cloud supplementary features.

[0109] Specifically, the source domain and the target domain are aligned and fused to obtain the point cloud supplementary features, including:

[0110] Take any feature in the source domain as the anchor feature, the feature in the target domain from the same image as the anchor feature as the positive sample feature, and the features in the target domain other than the positive sample feature as the negative sample feature; calculate the triplet loss based on the anchor feature, positive sample feature and negative sample feature;

[0111] Align the source domain with the target domain by minimizing the triplet loss to obtain the aligned features;

[0112] The aligned features are input into the multi-layer perceptron to obtain the point cloud supplementary features.

[0113] Furthermore, the triplet loss is calculated as:

[0114] ;

[0115] ;

[0116] ;

[0117] in, represents triplet loss; Represents the Euclidean distance from the anchor feature to the positive sample feature; Represents the Euclidean distance from the anchor feature to the negative sample feature; Represents the boundary, ensuring that the distance from the anchor feature to the negative sample feature is at least greater than the distance to the positive sample feature ;

[0118] when When , the loss is 0; when When , the loss is positive, indicating that the alignment target is not achieved and further optimization is needed.

[0119] Training process:

[0120] In each batch, triples are constructed in the following way: 1. Anchor samples: randomly sampled from source domain features ,2. Positive sample selection: find the sample that matches the anchor feature from the target domain ,3. Negative sample selection: randomly sampled from the target domain, which does not match the anchor feature ;

[0121] Calculate the loss for each triple , and average it within the current batch:

[0122] ;

[0123] in, represents the triplet loss of the i-th triplet, is the number of triplets in the current batch;

[0124] Backpropagation is used to optimize the network weights to minimize the overall loss function.

[0125] After training is completed, evaluate the alignment effect of source and target domain features. 1. Visualize the alignment results: Use t-SNE to map the features into a two-dimensional space and observe whether the distribution of source and target domain features tends to be consistent. 2. Feature similarity measurement: Calculate the average distance from the anchor feature to the positive sample feature. and the average distance to negative samples ,verify .

[0126] Through the triplet loss function, features from the source and target domains can be efficiently aligned in a shared space. By dynamically adjusting the distance differences between anchor points, positive samples, and negative samples, the relevance and separation of high-frequency and low-frequency features are ensured, providing a higher-quality feature foundation for subsequent tasks.

[0127] Among the supplementary features of the fused point cloud,

[0128] RGB enriches the surface texture and color properties of the point cloud, making the model more realistic in visualization and texture mapping;

[0129] RGB information provides semantic context for each point in the point cloud, such as the distinction of object edges or materials;

[0130] Point clouds provide three-dimensional geometric structures, and RGB supplements texture details. The combination of the two can enhance the completeness and expressiveness of modeling.

[0131] After aligning the RGB and point cloud data in the source domain (local features) with the target domain (global features and low-frequency information), the fused features are further used to complete 3D modeling. The source domain provides local texture and geometric details, while the target domain provides global structure and lighting information. The combination of the two lays the foundation for accurate 3D model construction.

[0132] S6: Extract local geometric and color features and reconstruct the three-dimensional surface of the point cloud supplementary features, combine the extracted features and the reconstructed three-dimensional surface to obtain a three-dimensional model.

[0133] Specifically, this step includes:

[0134] The supplementary features of the point cloud include: the three-dimensional coordinates of the point in the point cloud and the color features. The color features include: red channel component, green channel component, and blue channel component. Each channel component is also divided into a high-frequency part and a low-frequency part. The high-frequency part provides local texture details, and the low-frequency part provides global color and lighting information.

[0135] For any point in the point cloud image, its neighborhood point set is searched using the sphere radius.

[0136] Based on the point and its neighborhood point set, the local normal vector of the point is calculated by principal component analysis. The calculation formula is:

[0137] ;

[0138] in, Represents the local normal vector of the i-th point; Represents the neighborhood point set of the i-th point; represents the i-th neighborhood point; n represents the number of neighborhood points; Represents the square of the Euclidean norm.

[0139] The curvature of a point is calculated based on the ratio of the eigenvalues ​​of the covariance matrix of the neighboring points. The calculation formula is:

[0140] ;

[0141] in, represents the curvature of the i-th point; 、 、 Represents the eigenvalues ​​of the covariance matrix of different neighborhood points, satisfying .

[0142] Based on the color features of the points and the three-dimensional coordinates of the points, the color gradient is calculated as follows:

[0143] ;

[0144] in, Represents the color gradient of the i-th point; represents partial derivative; Represents the red channel component of the i-th point; Represents the green channel component of the i-th point; Represents the blue channel component of the i-th point; Represents the spatial horizontal coordinate of the i-th point; Represents the spatial ordinate of the i-th point; Represents the spatial vertical coordinate of the i-th point.

[0145] Integrate the point's three-dimensional coordinates, local normal vector, curvature, red channel component, green channel component, blue channel component, and color gradient to obtain a high-dimensional feature vector;

[0146] The high-dimensional feature vector is input into the multi-layer perceptron to generate high-dimensional features of the point.

[0147] Based on the distances between the neighboring points of a point and the neighborhood center point, the second joint feature is calculated by point cloud neighborhood interpolation to generate a continuous surface. The calculation formula is:

[0148] ;

[0149] in, represents the second joint feature of the i-th point; Represents the interpolation weight, which is the distance from the jth neighborhood point to the neighborhood center point.

[0150] In this embodiment, the low-frequency portion provided by the target domain can be used to perform global correction on the generated surface to ensure the overall shape and lighting consistency of the model.

[0151] Specifically, the global correction process is a key step in 3D modeling, which aims to ensure that the overall shape, lighting, and structure of the model match the expected global features. The correction process is as follows:

[0152] Define the global correction objective: The goal of global correction is to make the preliminary reconstructed model match the low-frequency parts extracted from the data in terms of shape, lighting, and structure. This may involve defining one or more optimization objectives, such as minimizing the difference between the model and the low-frequency parts, or maximizing the global consistency of the model.

[0153] Applying optimization algorithms: To achieve global correction, it is usually necessary to apply optimization algorithms to adjust the surface of the model. The algorithms include: parameterizing the model surface, converting the model surface parameters into a set of adjustable parameters (such as control point positions); defining an energy function that measures the difference between the model surface and the low-frequency part. The energy function may include multiple terms (such as shape consistency terms, lighting consistency terms, and smoothness terms); optimizing the energy function, using algorithms (such as gradient descent, simulated annealing, or genetic algorithms) to minimize the energy function, thereby adjusting the parameters of the model surface.

[0154] Iterative adjustment: Global correction is usually an iterative process that requires multiple adjustments to the model surface until a preset global consistency is achieved; in each iteration, the optimization algorithm updates the parameters of the model surface based on the difference between the current model surface and the low-frequency part.

[0155] Evaluation and refinement: During the iterative adjustment process, the global consistency of the model needs to be regularly evaluated and refined as needed, including: visual evaluation, using visualization tools to check whether the shape, lighting, and structure of the model match the expected global characteristics; quantitative evaluation, using quantitative indicators (such as mean squared error or structural similarity index) to evaluate the difference between the model and the low-frequency part.

[0156] The second joint feature and the local normal vector are input into the Poisson surface reconstruction function to generate a local three-dimensional surface corresponding to the point.

[0157] In this embodiment, texture mapping and detail enhancement are performed on the model surface through the high-frequency part:

[0158] 1. Texture mapping: Project the high-frequency part onto the reconstructed 3D surface to generate a textured model.

[0159] 2. Detail enhancement: introducing high-frequency loss function Optimize the details:

[0160] ;

[0161] in, Represents the high frequency part of the i-th point; M represents the number of points; Represents the predicted high-frequency part of the i-th point; Represents the square of the Euclidean norm.

[0162] Traverse all points in the point cloud image, obtain the high-dimensional features of all points, and generate the three-dimensional surface of the point cloud;

[0163] The high-dimensional features of all points and the three-dimensional surface of the point cloud are combined to obtain a three-dimensional model.

[0164] The final output 3D model includes:

[0165] 1. Accurate geometric information: spatial shape constructed from point cloud geometric data;

[0166] 2. High-fidelity texture information: color and texture supplemented by RGB data.

[0167] This model can be used for tasks such as scene reconstruction, target recognition, and visualization.

[0168] The missing information supplementation modeling method based on multimodal fusion provided in this embodiment has the following beneficial effects: This method can use RGB-supplemented point cloud (3D) information for modeling, enhancing the data's expressiveness. Point cloud image information complements spatial depth and three-dimensional structure. It integrates data from different modalities in a more refined manner to improve the accuracy and efficiency of information processing. By fusing multimodal information, the potential connections between different modal data can be better explored and utilized, thereby enhancing the perception and decision-making support capabilities of unmanned equipment in complex environments. This new method is expected to promote the advancement of information processing technology for unmanned equipment and provide support for achieving higher levels of autonomy and intelligence.

[0169] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A missing information supplement modeling method based on multimodal fusion, characterized in that: include: S1: Acquire RGB images and point cloud images of unmanned equipment; S2: Encode the RGB image using the color components to obtain the first feature; Encode the point cloud image using spatial position coding to obtain the second feature; S3: Align the feature spaces of the first feature and the second feature, perform feature fusion, and obtain a joint feature; reconstruct the image based on the joint feature to obtain a reconstructed RGB image; S4: Perform frequency domain transformation on the point cloud image and the reconstructed RGB image, and decompose the color high-frequency features, color low-frequency features, point cloud high-frequency features, and point cloud low-frequency features; S5: Use color high-frequency features and point cloud high-frequency features as source domains, and color low-frequency features and point cloud low-frequency features as target domains. Align and fuse the source and target domains to obtain point cloud supplementary features. S6: Extract local geometric and color features and reconstruct the three-dimensional surface of the point cloud supplementary features, combine the extracted features and the reconstructed three-dimensional surface to obtain a three-dimensional model.

2. The missing information supplement modeling method based on multimodal fusion according to claim 1 is characterized in that: The encoding of the RGB image by using color components comprises: Decompose the RGB image into three two-dimensional matrices according to color channels; For each two-dimensional matrix, encode it according to the distribution characteristics of its corresponding pixel values ​​to obtain the corresponding encoding matrix; Fuse the three encoding matrices to obtain a high-dimensional feature matrix; Perform global average pooling on the high-dimensional feature matrix to obtain the first feature.

3. The missing information supplement modeling method based on multimodal fusion according to claim 1 is characterized in that: The encoding of the point cloud image by using spatial position coding includes: Encode the spatial position of the point cloud image to obtain a high-dimensional representation of each point; Input the high-dimensional representation of each point into the multi-layer perceptron and output the feature representation of each point; For any point, a neighborhood set is obtained by using the K-nearest neighbor algorithm or sphere radius search, and the feature representations of each point in the neighborhood set are aggregated to obtain the local feature representation of the corresponding point; Aggregate the local feature representations of all points in the point cloud image to obtain the second feature.

4. The missing information supplement modeling method based on multimodal fusion according to claim 1 is characterized in that: The feature space of aligning the first feature and the second feature includes: Project the first feature into the shared space through a convolutional neural network; Project the second feature into the shared space through a multi-layer perceptron; The projected first feature is upsampled to the dimension of the second feature using the nearest neighbor interpolation method to align the dimensions of the first feature and the second feature.

5. The missing information supplement modeling method based on multimodal fusion according to claim 4 is characterized in that: The process of obtaining joint features is: The upsampled first feature is linearly fused with the projected second feature to obtain the joint feature.

6. The missing information supplement modeling method based on multimodal fusion according to claim 1 is characterized in that: The image reconstruction based on the joint features includes: Calculate the adjacency matrix between the RGB image and the point cloud image through spatial neighbors; Input the adjacency matrix and joint features into the multi-layer perceptron to obtain preliminary reconstruction features; The preliminary reconstruction features are passed through the Linear function to obtain the reconstruction weights; The reconstruction weights are passed through the softmax activation function and multiplied element-by-element with the preliminary reconstruction features to obtain the reconstructed features; The reconstructed features are converted into a reconstructed RGB image through the decoder.

7. The missing information supplement modeling method based on multimodal fusion according to claim 1 is characterized in that: S3 further includes: constructing a total loss function based on a weighted combination of a modality alignment loss, a distribution alignment loss, and an image reconstruction loss between the first feature and the second feature, and minimizing the total loss function to optimize the reconstruction process; The modality alignment loss is based on the first and second features and is calculated using the NT-Xent loss function; The distribution alignment loss is based on the first and second features and is calculated using the maximum mean difference loss function; The image reconstruction loss is based on the first feature and the second feature and is calculated using the mean square error.

8. The missing information supplement modeling method based on multimodal fusion according to claim 1, characterized in that S4 include: The 2D-BTSA function is used to perform wavelet transform on the reconstructed RGB image and decompose it into the first vector matrix of different frequency bands; The point cloud image is subjected to wavelet transform using 3D-TDMF-FE function and decomposed into second vector matrices of different frequency bands; The first vector matrix of the high frequency band is used as the color high frequency feature, and the first vector matrix of the low frequency band is used as the color low frequency feature; The second vector matrix of the high-frequency band is used as the high-frequency feature of the point cloud, and the second vector matrix of the low-frequency band is used as the low-frequency feature of the point cloud.

9. The missing information supplement modeling method based on multimodal fusion according to claim 1, characterized in that: The source domain and the target domain are aligned and fused to obtain the point cloud supplementary features, including: Take any feature in the source domain as the anchor feature, the feature in the target domain from the same image as the anchor feature as the positive sample feature, and the features in the target domain other than the positive sample feature as the negative sample feature; calculate the triplet loss based on the anchor feature, positive sample feature and negative sample feature; Align the source domain with the target domain by minimizing the triplet loss to obtain the aligned features; The aligned features are input into the multi-layer perceptron to obtain the point cloud supplementary features.

10. The missing information supplement modeling method based on multimodal fusion according to claim 1, characterized in that: S6 includes: The supplementary features of the point cloud include: the three-dimensional coordinates of the point in the point cloud and the color features. The color features include: red channel component, green channel component, and blue channel component. For any point in the point cloud image, use the sphere radius to search for its neighborhood point set; Based on a point and its neighborhood point set, the local normal vector of the point is calculated by principal component analysis; the curvature of the point is calculated based on the ratio of the eigenvalues ​​of the covariance matrix of the neighborhood points; Calculate the color gradient based on the color features of the points and the three-dimensional coordinates of the points; Integrate the point's three-dimensional coordinates, local normal vector, curvature, red channel component, green channel component, blue channel component, and color gradient to obtain a high-dimensional feature vector; Input the high-dimensional feature vector into the multi-layer perceptron to generate high-dimensional features of the point; Based on the neighborhood points of the point and the distance from the neighborhood point to the neighborhood center point, the second joint feature is calculated using point cloud neighborhood interpolation; The second joint feature and the local normal vector are input into the Poisson surface reconstruction function to generate a local three-dimensional surface corresponding to the point; Traverse all points in the point cloud image, obtain the high-dimensional features of all points, and generate the three-dimensional surface of the point cloud; The high-dimensional features of all points and the three-dimensional surface of the point cloud are combined to obtain a three-dimensional model.

Images