A cross-modal feature fusion method and system

Abstract

This invention relates to the fields of autonomous driving technology and robotics technology, and discloses a cross-modal feature fusion method and system. The method includes: acquiring original images and original point cloud data collected by an onboard camera and a LiDAR, respectively; performing dimensionality upscaling on the original images to obtain a first feature point cloud, and performing compression mapping on the first feature point cloud to obtain camera modal two-dimensional features; acquiring a second feature point cloud corresponding to the original point cloud data, and obtaining LiDAR modal two-dimensional features based on the second feature point cloud; generating a third feature point cloud based on the first and second feature point clouds, and performing compression mapping on the third feature point cloud to obtain hybrid modal two-dimensional features; and generating cross-modal fused features based on the camera modal two-dimensional features, LiDAR modal two-dimensional features, and hybrid modal two-dimensional features. The solution of this invention is beneficial for providing more accurate and reliable environmental perception capabilities for intelligent driving.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving technology and robotics, and in particular to a cross-modal feature fusion method and system. Background Technology

[0002] With the rapid development of autonomous driving technology and robotics, environmental perception, as a core module, directly determines the safety and reliability of intelligent vehicles and robots. Intelligent vehicles and robots need to perceive and understand complex and changing environments with high precision, including tasks such as object detection, obstacle recognition, path planning, and semantic segmentation. These tasks place extremely high demands on environmental perception, requiring not only the capture of rich semantic information but also accurate geometric and depth information. Single data sources often have performance limitations. For example, cameras can capture rich texture and color information but are weak in depth perception, while LiDAR has strong depth perception capabilities but is insufficient in texture information capture and resolution. Therefore, the fusion of multimodal sensors has become an important research direction in current environmental perception technology. However, due to the significant differences in characteristics between different modalities, data fusion needs to overcome consistency issues in resolution, dimensionality, and temporal sequence between modalities, making cross-modal feature fusion between different modal sensors a key focus and challenge in technical research.

[0003] Currently, most cross-modal feature fusion methods adopt fixed feature fusion strategies, that is, preset fixed fusion weights or feature selection rules in the network design stage. These methods cannot dynamically adapt to changes in modal characteristics under different scenarios, and have poor perception accuracy, robustness and reliability.

[0004] Therefore, there is still room for improvement in existing cross-modal feature fusion techniques. Summary of the Invention

[0005] In view of this, the present invention proposes a cross-modal feature fusion method and system, which solves the problems of poor adaptability, low perception accuracy, poor robustness and reliability in the cross-modal feature fusion process, and provides more accurate and reliable environmental perception capabilities for intelligent driving.

[0006] To achieve the above objectives, one aspect of the present invention provides a cross-modal feature fusion method, specifically including the following steps:

[0007] Acquire raw images and raw point cloud data collected by the vehicle-mounted camera and LiDAR respectively;

[0008] The original image is subjected to dimensionality upscaling to obtain a first feature point cloud, and the first feature point cloud is subjected to compression mapping to obtain two-dimensional features of the camera modality.

[0009] Obtain the second feature point cloud corresponding to the original point cloud data, and obtain the two-dimensional features of the lidar mode based on the second feature point cloud;

[0010] A third feature point cloud is generated based on the first feature point cloud and the second feature point cloud, and a compression mapping process is performed based on the third feature point cloud to obtain a hybrid modal two-dimensional feature.

[0011] Based on the camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features, cross-modal fusion features are generated.

[0012] In some implementations, the step of generating cross-modal fusion features based on the camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features includes:

[0013] The camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features are stitched together to generate an initial stitched feature;

[0014] Based on the first channel weights corresponding to each feature channel in the splicing features, the initial splicing features are optimized to obtain the target splicing features;

[0015] The target splicing features are input into a preset cross-modal fusion network to obtain the output results, which are then used as the cross-modal fusion features.

[0016] In some implementations, the step of inputting the target splicing features into a preset cross-modal fusion network to obtain the output result includes:

[0017] The first feature of camera mode type in the target stitching features is obtained, and the first feature is used as a query feature parameter and input into the preset cross-modal fusion network;

[0018] Obtain the second feature of the target stitching feature whose modal type is lidar mode, and input the second feature as key feature parameter and value feature parameter into the preset cross-modal fusion network;

[0019] The attention weights are obtained based on the query feature parameters and the key feature parameters;

[0020] Based on the attention weights and the value feature parameters, the initial fusion features are determined;

[0021] Obtain the third feature of the target splicing features that is a mixed modality feature, and determine the output result based on the initial fusion feature and the third feature.

[0022] In some implementations, the step of obtaining a third feature of mixed modality type from the target splicing features, and determining the output result based on the initial fusion features and the third feature, includes:

[0023] The initial fusion feature is evaluated based on a preset evaluation function to obtain a first evaluation result, and the third feature is evaluated based on the preset evaluation function to obtain a second evaluation result.

[0024] Based on the first evaluation result and the second evaluation result, the target weight coefficient is determined;

[0025] The initial fusion feature and the third feature are weighted and summed based on the target weight coefficient to obtain the output result.

[0026] In some implementations, the step of evaluating the initial fusion features based on a preset evaluation function to obtain a first evaluation result includes:

[0027] Based on a preset activation function, the second channel weights corresponding to each feature channel in the initial fusion feature are generated, and the initial fusion feature is optimized based on all the second channel weights.

[0028] The optimized initial fusion features are evaluated based on a preset evaluation function to obtain an initial evaluation result;

[0029] Determine whether the initial evaluation result meets the first preset state;

[0030] In response to the initial evaluation result not meeting the first preset state, the learning parameters are adjusted to optimize the preset activation function, and the steps of generating the second channel weights corresponding to each feature channel in the initial fusion feature based on the preset activation function and optimizing the initial fusion feature based on all the second channel weights are returned.

[0031] In response to the initial evaluation result satisfying a first preset state, the initial evaluation result is taken as the first evaluation result.

[0032] In some implementations, the step of performing dimensionality upscaling on the original image to obtain a first feature point cloud includes:

[0033] Obtain the external and internal parameters of the vehicle-mounted camera;

[0034] Obtain the depth estimation result corresponding to each pixel coordinate in the original image;

[0035] Based on the preset transformation matrix, the external parameters, the internal parameters, and the depth estimation results, the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system are obtained;

[0036] The loss result is determined based on each pixel coordinate and its corresponding three-dimensional coordinate;

[0037] Update the preset transformation matrix based on the loss result, and return to the step of obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system based on the preset transformation matrix, the external parameters, the internal parameters and the depth estimation result, until the loss result satisfies the second preset state;

[0038] The first feature point cloud is obtained based on the three-dimensional coordinates of all the pixel coordinates.

[0039] In some implementations, the step of performing compression mapping processing based on the first feature point cloud to obtain two-dimensional features of the camera modality includes:

[0040] Based on each first data point in the first feature point cloud, obtain its corresponding first feature vector and depth distribution;

[0041] Based on the corresponding first feature vector and the depth distribution, the context features of each first data point are determined;

[0042] Map each of the first data points to the bird's-eye view grid to obtain the first cell corresponding to each of the first data points in the bird's-eye view grid.

[0043] Based on each mapped first cell, the context features corresponding to all first data points are aggregated to obtain the first target feature;

[0044] Based on the first target features of all the first cells, the two-dimensional features of the camera modality are obtained.

[0045] In some implementations, the step of acquiring the second feature point cloud corresponding to the original point cloud data and obtaining the two-dimensional features of the lidar mode based on the second feature point cloud includes:

[0046] Based on a preset feature extraction model, local features of the original point cloud data are extracted, and all local features are aggregated to obtain global features to obtain the second feature point cloud.

[0047] Obtain the channel dimension feature corresponding to each feature channel in the second feature point cloud, and obtain the point dimension feature corresponding to each second data point in the second feature point cloud;

[0048] Based on preset weights, the channel dimension features corresponding to each feature channel are weighted sequentially, and the point dimension features corresponding to each second data point are weighted sequentially based on preset weights.

[0049] Based on all the weighted channel dimension features and all the weighted point dimension features, the two-dimensional features of the lidar mode are obtained.

[0050] In some embodiments, the step of generating a third feature point cloud based on the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain mixed-modality two-dimensional features includes:

[0051] Spatially align the first feature point cloud and the second feature point cloud, and query the first data point in the first feature point cloud that matches each of the second data points in the second feature point cloud.

[0052] Obtain the second feature vector corresponding to each second data point, and merge the second feature vector with the first feature vector of the first data point that matches it to obtain the third feature vector;

[0053] The third feature point cloud is generated based on all the third feature vectors.

[0054] Map each third data point in the third feature point cloud to the bird's-eye view grid to obtain the third cell corresponding to each third data point in the bird's-eye view grid.

[0055] Based on each of the mapped third cells, the third feature vectors corresponding to all the third data points are aggregated to obtain the third target feature;

[0056] The mixed-modal two-dimensional features are obtained based on the third target features of all the third cells.

[0057] Another aspect of this invention provides a cross-modal feature fusion system, comprising:

[0058] The acquisition unit is configured to acquire the raw images and raw point cloud data collected by the vehicle-mounted camera and LiDAR, respectively.

[0059] A camera modality processing unit is configured to perform dimensionality upscaling processing on the original image to obtain a first feature point cloud, and to perform compression mapping processing on the first feature point cloud to obtain two-dimensional features of the camera modality.

[0060] The lidar modal processing unit is configured to acquire a second feature point cloud corresponding to the original point cloud data, and extract lidar modal two-dimensional features based on the second feature point cloud.

[0061] A hybrid modal processing unit is configured to generate a third feature point cloud based on the first feature point cloud and the second feature point cloud, and to perform compression mapping processing based on the third feature point cloud to obtain hybrid modal two-dimensional features;

[0062] A cross-modal fusion unit is configured to generate cross-modal fusion features based on the camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features.

[0063] In another aspect of the present invention, a computer device is provided, comprising: at least one processor; and a memory storing a computer program executable on the processor, the computer program performing the steps of the method described above when executed by the processor.

[0064] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0065] This invention has at least the following beneficial technical effects: It processes the original image data of the camera modality using dimensionality-enhancing and compression mapping to obtain the 3D feature point cloud corresponding to the camera modality, and then projects it onto the bird's-eye view domain to obtain the 2D feature representation of the camera modality. Next, it extracts features from the original point cloud data collected by the LiDAR to obtain the 3D feature point cloud and 2D feature representation corresponding to the LiDAR modality. It then fuses the 3D feature point clouds corresponding to the LiDAR and camera modalities respectively to obtain the 3D feature point cloud under the hybrid modality. After converting the 3D feature point cloud corresponding to the hybrid modality into a 2D feature representation consistent with the camera modality, it utilizes a preset cross-modal fusion network. Finally, it performs channel filtering, attention weighting, and dynamic calibration on the 2D features of the camera, LiDAR, and hybrid modality to achieve selective feature enhancement and suppress redundant information, generating cross-modal fusion features containing rich semantic and geometric information. This achieves cross-modal feature fusion between the camera and LiDAR, improving perception accuracy, robustness, and reliability, and providing high-precision input for environmental perception tasks. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a block diagram of an embodiment of the cross-modal feature fusion method provided by the present invention;

[0068] Figure 2 A schematic diagram of an embodiment of the cross-modal feature fusion method framework provided by the present invention;

[0069] Figure 3A schematic diagram of an embodiment of the dynamic cross-modal feature fusion process provided by the present invention;

[0070] Figure 4 A schematic diagram of an embodiment of the dimension stretching and compression mapping process provided by the present invention;

[0071] Figure 5 This is a schematic diagram of an embodiment of the lidar modal feature extraction process provided by the present invention;

[0072] Figure 6 This is a schematic diagram of an embodiment of the cross-modal feature fusion system provided by the present invention;

[0073] Figure 7 A schematic diagram of the structure of an embodiment of the computer device provided by the present invention;

[0074] Figure 8 This is a schematic diagram of an embodiment of the computer-readable storage medium provided by the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0076] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0077] In autonomous driving and robotics systems, vehicle-mounted cameras and LiDAR are two common types of sensors. Vehicle-mounted cameras provide rich semantic information by capturing two-dimensional images, making them suitable for target classification and semantic segmentation. LiDAR provides high-precision three-dimensional geometric information through point cloud data, making it suitable for 3D scene reconstruction and obstacle detection. However, due to the significant differences in characteristics between the two modalities, data fusion requires addressing heterogeneity issues in terms of resolution, dimensionality, and temporal sequence. This makes cross-modal feature fusion of cameras and LiDAR a key focus and challenge in technical research.

[0078] Current cross-modal feature fusion techniques are mainly divided into two categories: feature-level fusion and decision-level fusion. Feature-level fusion typically involves joint modeling of multimodal data during data preprocessing or feature extraction, using deep learning networks to map data from different modalities to a unified feature space. For example, some studies utilize the geometric relationship between 2D images and 3D point clouds to project the 3D point cloud onto the 2D image plane for alignment, and then extract fused features through a joint network. Although this method can achieve preliminary alignment between modalities, it often ignores the problem of unbalanced feature representation caused by modal differences. In addition, some studies have attempted to embed image information into the point cloud space for 3D modeling, but due to the sparse and uneven distribution of point cloud data, the accuracy of feature extraction is difficult to guarantee. Decision-level fusion typically involves weighted integration of the detection results of multimodal data through independent networks after feature extraction and object detection. This method is logically simple and easy to implement, but due to the lack of deep fusion at the feature level, it is difficult to fully leverage the complementary advantages of multimodal data, and usually only plays a role in performance correction.

[0079] Furthermore, most cross-modal feature fusion methods still employ fixed feature fusion strategies, i.e., pre-setting fixed fusion weights or feature selection rules during the network design phase. However, this static strategy cannot dynamically adapt to changes in modal characteristics under different scenarios. In addition, the significant differences in resolution and dimensionality between data from vehicle cameras and LiDAR can lead to a decrease in perception accuracy if fused directly. Simultaneously, existing methods generally lack effective screening and calibration mechanisms for fused features, easily introducing irrelevant features or noise, thereby reducing the robustness and reliability of autonomous driving and robotic systems.

[0080] To address the aforementioned shortcomings, a first aspect of this invention proposes a cross-modal feature fusion method. For example... Figure 1 As shown, it includes the following steps:

[0081] Step S100: Obtain the original images and original point cloud data collected by the vehicle-mounted camera and LiDAR respectively;

[0082] Step S200: Perform dimensionality upscaling on the original image to obtain a first feature point cloud, and perform compression mapping on the first feature point cloud to obtain two-dimensional features of the camera modality.

[0083] Step S300: Obtain the second feature point cloud corresponding to the original point cloud data, and obtain the two-dimensional features of the lidar mode based on the second feature point cloud;

[0084] Step S400: Generate a third feature point cloud based on the first feature point cloud and the second feature point cloud, and perform compression mapping processing based on the third feature point cloud to obtain a mixed-modality two-dimensional feature.

[0085] Step S500: Generate cross-modal fusion features based on camera modal 2D features, lidar modal 2D features, and hybrid modal 2D features.

[0086] In some embodiments, the original images are mainly acquired in real time by the vehicle-mounted camera, and the original point cloud data are mainly acquired in real time by the LiDAR. Based on the dimensionality-enhancing compression mapping theory, this method uses scale compression mechanisms, attention mechanisms, and 3D feature extraction to obtain fused features containing rich feature information of different dimensions. Then, the fused features are processed through a preset cross-modal fusion network, forming a complete cross-modal feature fusion method.

[0087] In some embodiments, firstly, for the camera modality, multi-view images captured by different vehicle-mounted cameras are fused using coordinate fusion. Dimensional stretching and compression mapping theory is employed to stretch the fused multi-view images to form a point cloud array for the camera modality. Then, three-dimensional features of the camera modality are extracted from the point cloud array to obtain a three-dimensional point cloud, i.e., the first feature point cloud. The first feature point cloud is further projected onto a bird's-eye view domain to form a two-dimensional feature representation, i.e., the camera modality two-dimensional features. Next, for the lidar modality, local and global features of the original point cloud data are extracted using a preset feature extraction model to obtain a second feature point cloud. Based on the second feature point cloud, an attention mechanism is used to infer and predict the two-dimensional features of the lidar modality, i.e., the lidar modality two-dimensional features. Then, the second feature point cloud and the first feature point cloud are fused in three-dimensional space to obtain a third feature point cloud. Finally, a two-dimensional feature representation is extracted from the third feature point cloud using compression to obtain the hybrid modality two-dimensional features. Finally, the two-dimensional features from the camera, LiDAR, and hybrid modalities are concatenated and input into a preset cross-modal fusion network. This network performs channel filtering, attention weighting, and dynamic calibration to generate the final cross-modal fused features. The cross-modal feature fusion method described above achieves feature extraction and dynamic cross-modal fusion of both camera and LiDAR modalities.

[0088] The cross-modal feature fusion method of this invention processes the original image data of the camera modality using dimensionality enhancement and compression mapping to obtain the 3D feature point cloud corresponding to the camera modality. This 3D feature point cloud is then projected onto a bird's-eye view domain to obtain a 2D feature representation of the camera modality. Next, feature extraction is performed on the original point cloud data acquired by the LiDAR to obtain the 3D feature point cloud and 2D feature representation corresponding to the LiDAR modality. The 3D feature point clouds corresponding to the LiDAR and camera modalities are fused to obtain a 3D feature point cloud in the hybrid modality. After converting the 3D feature point cloud corresponding to the hybrid modality into a 2D feature representation consistent with the camera modality, a preset cross-modal fusion network is used. Finally, channel filtering, attention weighting, and dynamic calibration are performed on the 2D features of the camera, LiDAR, and hybrid modality to achieve selective feature enhancement and suppress redundant information. This generates cross-modal fusion features containing rich semantic and geometric information, realizing cross-modal feature fusion between the camera and LiDAR, improving perception accuracy, robustness, and reliability, and providing high-precision input for environmental perception tasks.

[0089] In some embodiments, the step of generating cross-modal fusion features based on camera modal 2D features, lidar modal 2D features, and hybrid modal 2D features includes: stitching together camera modal 2D features, lidar modal 2D features, and hybrid modal 2D features to generate initial stitched features; optimizing the initial stitched features based on the first channel weights corresponding to each feature channel in the initial stitched features to obtain target stitched features; and inputting the target stitched features into a preset cross-modal fusion network to obtain the output result and using it as the cross-modal fusion feature.

[0090] In some embodiments, current deep cross-modal feature fusion methods are essentially static fusion methods, meaning they use the same computational processing to fuse multimodal input data without considering the different computational requirements of different modalities. Once trained, this method cannot adapt to the differences in each data point during inference. To address this issue, this invention proposes a dynamic cross-modal fusion network (i.e., a pre-defined cross-modal fusion network), such as... Figure 2 and Figure 3 As shown, by dynamically adjusting the fusion method of camera modality and lidar modality features, the best option can be selected from the best features at any time. Figure 2 Conv 3 "3" means 3 3. Convolutional layer operations, "Avg pool" indicates average pooling, "Conv 1" indicates average pooling. "1" represents 1 1. Convolutional layer operations, where "sigmoid" represents the activation function. Figure 3In this code, "Conv" represents convolution operation, "BN" represents normalization operation, "ReLU" represents activation function, "Cross attention" represents cross attention mechanism, "Add&Norm" represents residual connection, "SelfAttention" represents self-attention mechanism, and "Matmul" represents matrix operation.

[0091] In some embodiments, camera modal 2D features contain 2D geometric semantic information (such as color and texture), LiDAR modal 2D features contain geometric structural information (such as depth and distance), and hybrid modal 2D features fuse preliminary information from both modalities. The 2D features from the camera modality and LiDAR modality are preprocessed separately. To facilitate cross-modal fusion, the camera modal features, LiDAR modal features, and their hybrid features are concatenated along the channel dimension to generate initial stitched features. By concatenating along the channel dimension, the feature information of each modality can be preserved without changing the consistency of information in the spatial dimension; the features of each modality remain independent after stitching.

[0092] In some embodiments, such as Figure 2 As shown, the initial concatenated features are processed using a channel attention module. This module aims to dynamically adjust the importance of feature channels in the initial concatenated features, highlighting key channels and suppressing redundant ones. Specifically, the initial concatenated features are first subjected to global average pooling and max pooling in the spatial dimension. The average pooling and max pooling results for each feature channel are used as channel-level global information, representing the global distribution of the response on the feature channel. This also ensures that the input layer near the pre-defined cross-modal fusion network can also obtain the global receptive field. Then, a multilayer perceptron is used to encapsulate the global information of each channel, generating the first channel weight corresponding to each feature channel. The process of the multilayer perceptron generating the first channel weight is shown in Formula 1.

[0093] ;

[0094] in, As the weight of the first channel, and These are the weights of two fully connected layers in a multilayer perceptron. For the average pooling result, The result is the max pooling result, and sigmoid is the activation function.

[0095] In some embodiments, the initial splicing features are filtered using the first channel weights. This involves multiplying the first channel weights channel-by-channel to the corresponding feature channels, thus filtering the initial splicing features along the channel dimension and optimizing them. The channel-filtered initial splicing features are then used as the target splicing features. Finally, the target splicing features are further processed using a dynamic cross-modal fusion network. The dynamic cross-modal fusion network can be summarized as follows: (Formula 2)

[0096] ;

[0097] in, The output of the dynamic cross-modal fusion network is [F0, F1, F2], which represents the target splicing features. Through 3 The static channel and spatial fusion function implemented by the three convolutional layers aims to reduce the channel dimension of the spliced ​​features. It is an adaptive feature selection function composed of multiple convolutional layers. This can be summarized as Formula 3:

[0098] ;

[0099] Where F is the input feature and W is the linear transformation matrix. f avg This is the global average pooling function. is the activation function. By highlighting important features in the target stitching feature through the adaptive feature selection function, the complementary information of the camera mode and the lidar mode can be effectively captured, which is beneficial to obtaining cross-modal fusion features with higher expressive power.

[0100] The cross-modal feature fusion method of this invention aggregates multimodal features by concatenating them along the channel dimension, providing a unified input for the subsequent dynamic cross-modal fusion network while preserving information from each modality. The dynamic cross-modal fusion network is then used to provide a bird's-eye view of different modalities. Figure 2 The dimensional features are processed to give the final cross-modal fusion features higher expressive power, providing reliable feature support for downstream tasks such as object detection or semantic segmentation.

[0101] In some embodiments, the step of inputting the target stitching features into a preset cross-modal fusion network to obtain the output result includes: obtaining a first feature of camera mode type in the target stitching features, using the first feature as a query feature parameter and inputting it into the preset cross-modal fusion network; obtaining a second feature of lidar mode type in the target stitching features, using the second feature as a key feature parameter and a value feature parameter and inputting it into the preset cross-modal fusion network; obtaining attention weights based on the query feature parameter and the key feature parameter; determining initial fusion features based on the attention weights and the value feature parameter; obtaining a third feature of mixed mode type in the target stitching features, and determining the output result based on the initial fusion feature and the third feature.

[0102] In some embodiments, such as Figure 3 As shown, the dynamic cross-modal fusion network, combined with a Query-Key-Value architecture, selectively fuses multimodal features based on their dynamic requirements. The camera modal features (i.e., the first feature) in the target stitching features are used as the Query, and the LiDAR modal features (i.e., the second feature) are used as the Key and Value. An attention mechanism is used to calculate the correlation between the camera modal features and the LiDAR modal features, as shown in Formula 4 below.

[0103] ;

[0104] Where Q is the Query representation after linear transformation of the first feature, K is the Key representation after linear transformation of the second feature, V is the Value representation after linear transformation of the second feature, and A is the attention weight, which represents the correlation between the first and second features. The Value features are weighted and summed according to the attention weights to obtain the fused feature representation, i.e., the initial fused feature. The weighting method is shown in Formula 5 below:

[0105] ;

[0106] in, These are the initial fusion features.

[0107] Furthermore, when fusing lidar modal features and camera modal features, some information may be lost. Therefore, hybrid modal features can be selectively inserted, that is, the hybrid modal features (i.e., the third feature) in the target stitching features are used as supplementary information to insert into the initial fusion features, and the final cross-modal fusion features are output.

[0108] The cross-modal feature fusion method of the present invention integrates visual and geometric information through a Query-Key-Value architecture to enhance the synergy between modalities. The dynamic cross-modal fusion network fully leverages the advantages of LiDAR modal features and camera modal features through attention mechanisms and dynamic feature selection strategies, and selectively inserts hybrid modal features as supplementary information to construct multimodal fusion features containing rich semantic and geometric information.

[0109] In some embodiments, the step of obtaining a third feature of mixed modality type in the target splicing features, and determining the output result based on the initial fusion features and the third feature, includes: evaluating the initial fusion features based on a preset evaluation function to obtain a first evaluation result; evaluating the third feature based on the preset evaluation function to obtain a second evaluation result; determining the target weight coefficient based on the first evaluation result and the second evaluation result; and performing weighted summation processing on the initial fusion features and the third feature based on the target weight coefficient to obtain the output result.

[0110] In some embodiments, an evaluation function is first preset, which can be expressed as the following formula 6:

[0111] ;

[0112] Where z represents the prediction result of the initial fused features after completing the mapping of downstream tasks, such as the class probability in a classification task. This indicates that z belongs to the i-th category label. The higher the predicted probability, the stronger the confidence of the initial fused feature in predicting the label of the i-th category. S(z) is the first evaluation result, which takes the maximum value among all category predicted probabilities, representing the overall confidence assessment of the initial fused feature.

[0113] After determining the preset evaluation function, the final cross-modal fusion feature can be expressed as the following formula 7:

[0114] ;

[0115] in, T To set the threshold, manual adjustments were made based on the experimental process. This indicates the third characteristic. It is an indicator function that satisfies Set to 1 if the condition is met, otherwise set to 0. The target weight coefficient, A normalization strategy can be used for adjustment, which can be expressed as Equation 8 below:

[0116] ;

[0117] in, The evaluation score for the third feature to independently complete the downstream task mapping, i.e. the second evaluation result, can be calculated by the aforementioned preset evaluation function.

[0118] The cross-modal feature fusion method of this invention introduces an evaluation function to selectively insert hybrid modal features, making the preset cross-modal fusion network more adaptable and achieving the effect of "selecting the best from the best" "at every moment". By weighted fusion of hybrid modal features and initial fusion features, the complementary information of lidar mode and camera mode is preserved to the greatest extent, and fusion features containing rich semantic and depth information are obtained, realizing cross-modal feature fusion of camera-lidar, which can improve perception accuracy, robustness and reliability.

[0119] In some embodiments, the step of evaluating the initial fusion features based on a preset evaluation function to obtain a first evaluation result includes: generating second channel weights corresponding to each feature channel in the initial fusion features based on a preset activation function, and optimizing the initial fusion features based on all second channel weights; evaluating the optimized initial fusion features based on the preset evaluation function to obtain an initial evaluation result; determining whether the initial evaluation result meets a first preset state; in response to the initial evaluation result not meeting the first preset state, adjusting the learning parameters to optimize the preset activation function, and returning the step of generating second channel weights corresponding to each feature channel in the initial fusion features based on the preset activation function, and optimizing the initial fusion features based on all second channel weights; in response to the initial evaluation result meeting the first preset state, using the initial evaluation result as the first evaluation result.

[0120] In some embodiments, a dynamic feature calibration strategy is introduced to improve the dynamic adaptability of feature fusion. This involves adjusting the channel importance of the initial fused features through a weighted adjustment of each feature channel. Specifically, adaptive weights are assigned to each feature channel in the initial fused features to amplify useful features while suppressing weakly correlated or redundant features. The core steps of the dynamic feature calibration strategy are spatial dimension compression and activation operations. First, a global response value is calculated for each feature channel, as shown in Equation 9 below:

[0121] ;

[0122] Among them, F attn Let represent the initial fused features, and c represent any feature channel. H represents the global response value, H represents the height of the feature map corresponding to the initial fused features, and W represents the width of the feature map corresponding to the initial fused features.

[0123] Then, the global response value is used to generate the second channel weight through a preset activation function, which is shown in Formula 10 below:

[0124] ;

[0125] in, b are the learning parameters, i.e., weights and biases, and σ is the activation function. This is the weight for the second channel.

[0126] Use weights Labeling the importance of each feature channel and the activation operation is similar to the gating mechanism in a recurrent neural network, through parameters. To generate weights for each feature channel , where parameters It is learned to explicitly model the correlation between feature channels.

[0127] In some embodiments, the second channel weights are considered as the importance of each feature channel after feature selection. Then, they are multiplied and weighted channel-by-channel to recalibrate the initial fused features along the channel dimension, thereby optimizing the initial fused features. A preset evaluation function is used to evaluate the optimized initial fused features, yielding an initial evaluation result. If the initial evaluation result is less than a preset result value, the learning parameters in the preset activation function are adjusted using a method such as gradient descent. The updated preset activation function is then used to generate new second channel weights to further optimize the initial fused features. This process is repeated until the initial evaluation result is greater than or equal to the preset result value, i.e., the initial evaluation result meets the first preset state. The final recalibration of the initial fused features along the channel dimension is then completed, and these features are used in subsequent feature fusion operations. The preset result value is set according to actual needs and is not specifically limited here.

[0128] The cross-modal feature fusion method of the present invention can explicitly construct the interdependencies between feature channels. It adopts a dynamic feature calibration strategy, which can automatically obtain the importance of each feature channel through learning. Then, based on this importance, useful features are enhanced and features that are not very useful for the current task are suppressed. The recalibrated initial fused features have higher expressive power, realizing effective screening and calibration of fused features, effectively capturing complementary information between camera modalities and lidar modalities, and providing reliable feature support for downstream tasks (such as object detection or semantic segmentation).

[0129] In some embodiments, the step of performing dimensionality upscaling on the original image to obtain a first feature point cloud includes: acquiring the extrinsic and intrinsic parameters of the vehicle-mounted camera; acquiring the depth estimation result corresponding to each pixel coordinate in the original image; obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to a preset transformation matrix, extrinsic parameters, intrinsic parameters, and depth estimation results; determining a loss result based on each pixel coordinate and its corresponding three-dimensional coordinates; updating the preset transformation matrix according to the loss result, and returning to the step of obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to the preset transformation matrix, extrinsic parameters, intrinsic parameters, and depth estimation results, until the loss result satisfies a second preset state; and obtaining the first feature point cloud based on the three-dimensional coordinates of all pixel coordinates.

[0130] In some embodiments, such as Figure 2 and Figure 4 As shown, the camera modality data processing method is based on the dimensionality-enhancing compression mapping theory. Dimensionality-enhancing compression mapping is a visual perception model used in autonomous driving and robotics, aiming to generate bird's-eye view representations from multi-view images, thereby achieving a three-dimensional understanding of the environment. The core idea of ​​the dimensionality-enhancing compression mapping model is to convert the two-dimensional image data acquired by the camera into a three-dimensional spatial representation, thus forming a bird's-eye view, providing accurate high-dimensional spatial information for subsequent tasks such as object detection and semantic segmentation.

[0131] In some embodiments, the multi-view RGB images (i.e., the raw images) captured by the vehicle-mounted camera are first preprocessed. These images cover a 360-degree field of view around the vehicle. After calibration, the camera's intrinsic and extrinsic parameters are obtained. The intrinsic parameters include focal length and principal point coordinates, and the extrinsic parameters include rotation matrix and translation vector. Figure 4 As shown, in the dimensionality upscaling stage, the input raw image is mapped into 3D space through depth estimation and camera intrinsic and extrinsic parameters. This process requires implicit or explicit learning of depth information, typically using multi-view geometric relationships or deep learning networks for depth estimation. In the compression stage, the point cloud distribution upscaled to 3D space is projected onto a unified bird's-eye view grid. This step requires handling geometric consistency across views to ensure that key information is not lost or confused during the fusion process. Finally, in the mapping stage, feature extraction is performed on the 2D representation projected onto the bird's-eye view grid to obtain camera modality 2D features. These extracted camera modality 2D features are used to complete downstream tasks, such as object detection, trajectory prediction, or semantic segmentation.

[0132] In some embodiments, the dimensionality enhancement stage is further explained. To achieve a unified geometric description, pixels in the original image coordinate system are transformed into a 3D point cloud in the vehicle coordinate system using camera intrinsic and extrinsic parameters and depth information. This mapping of images from different perspectives into a unified coordinate system lays the foundation for subsequent feature extraction. After coordinate system transformation, depth estimation techniques are used to map each pixel in the 2D image to 3D space, obtaining the 3D coordinates of each pixel. Depth estimation can be based on multi-view geometric relationships or deep learning networks to estimate the depth value of pixels. Depth estimation techniques include monocular depth estimation networks or stereo matching algorithms, and can also be optimized using corresponding loss functions. After obtaining depth information, the 3D coordinates of each pixel in the vehicle coordinate system are generated by combining the camera's intrinsic and extrinsic parameters and a preset transformation matrix, forming a cone-view point cloud. Preset features are used to extract features from the cone-view point cloud to generate a first feature point cloud. The 3D coordinates of each pixel describe its spatial position in the real world, providing corresponding geometric information for further processing.

[0133] In some embodiments, the simplified process of coordinate system transformation is shown in Equation 11 below:

[0134] ;

[0135] CAM e External parameters, for CAM i Internal parameters, where V is the first feature point cloud. Z c Depth refers to the distance of a pixel from the optical axis. P uv Let u be the pixel coordinate system, where u represents the pixel's position in the row direction (horizontal direction) and v represents the pixel's position in the column direction (vertical direction). , Indicates focal length. Represents a rotation matrix, and the matrix contains... This represents the translation vector. , Indicates the coordinates of the principal point.

[0136] The aforementioned coordinate system transformation requires augmenting the coordinates and parameters of the camera and vehicle to form a mapping relationship, involving operations such as matrix rotation, translation, parameter augmentation, and pooling. Currently, coordinate system transformation typically employs algebraic methods or simplified calculation methods, which often increase computational load or affect mapping accuracy. This invention, however, uses an adaptive coordinate transformation method. It introduces a preset transformation matrix M and uses deep learning to find mapping and scaling relationships to achieve adaptive adjustment of the coordinate system. Its loss function is shown in Equation 12 below:

[0137] ;

[0138] Where π is the projection function from the three-dimensional coordinates to the image plane. P i These are the generated three-dimensional coordinates. p i These are the corresponding pixel coordinates in the original image. It is a loss.

[0139] Furthermore, the preset transformation matrix M can be updated by determining whether the loss result meets a preset state. Specifically, when the loss result is greater than the preset loss value, the preset transformation matrix is ​​updated by adjusting its parameters, and new 3D coordinates are generated for each pixel in the original image. This process is repeated until the loss result is less than the preset loss value, meaning the loss result meets the second preset state. The preset loss value is set according to actual needs and is not specifically limited here.

[0140] The cross-modal feature fusion method of the present invention transforms multi-view images into a unified three-dimensional point cloud representation by dimensionality enhancement, thereby making the vehicle environment geometrically consistent and laying the foundation for three-dimensional spatial understanding. By optimizing the preset transformation matrix through deep learning, the computational complexity in traditional algebraic methods or simplified calculation methods is reduced, ensuring the accuracy of subsequent cross-modal feature fusion.

[0141] In some embodiments, the step of performing compression mapping processing based on the first feature point cloud to obtain two-dimensional features of the camera modality includes: obtaining the corresponding first feature vector and depth distribution for each first data point in the first feature point cloud; determining the context features of each first data point based on the corresponding first feature vector and depth distribution; mapping each first data point to a bird's-eye view grid to obtain the first cell corresponding to each first data point in the bird's-eye view grid; aggregating the context features corresponding to all first data points in each mapped first cell to obtain a first target feature; and obtaining the two-dimensional features of the camera modality based on the first target features of all first cells.

[0142] In some embodiments, for each data point (i.e., the first data point) in the first feature point cloud, a multidimensional feature vector and multiple discrete depth probability distributions are predicted. The discrete depth probability distributions are the depth probability distributions of each pixel in the frustum point cloud. The predicted feature vector and the depth distributions are multiplied by the outer product to generate the context features corresponding to the first data point. The context feature point cloud is obtained from all the generated context features. The calculation process is shown in Equation 13 below:

[0143] ;

[0144] Where c is the feature vector. αd For depth distribution, For contextual features.

[0145] In some embodiments, such as Figure 4 As shown, after the dimensionality scaling operation, the first feature point cloud and the corresponding context feature point cloud are obtained. The first feature point cloud contains the position information of each pixel in the original image in the vehicle coordinate system, while the context feature point cloud contains the context feature information of each pixel in the original image. Compression is performed based on this information, mapping the first feature point cloud from the vehicle coordinate system to the bird's-eye view grid, and filtering out points that fall outside the grid boundaries after translation, completing the compression process from a 3D point cloud to a 2D representation. After mapping, pooling is used to aggregate all features within each cell of the bird's-eye view grid. Specifically, for all mapped points in each cell, their feature representation is obtained through max pooling to ensure that the most representative features within the cell are retained, thus obtaining the first target feature. Pooling is also used to aggregate the context features in each cell. Finally, a convolutional neural network is used to extract features from the compressed bird's-eye view grid, generating a 2D feature map suitable for downstream tasks, i.e., camera modality 2D features. The convolutional neural network is designed based on the grid resolution to ensure that the feature map can capture global semantic information. To enhance the feature representation of key regions, a channel attention mechanism is further introduced, optimizing feature channels through weighted optimization. The optimized features possess higher semantic expressiveness and spatial consistency, effectively supporting subsequent cross-modal fusion with LiDAR modes.

[0146] The cross-modal feature fusion method of the present invention generates a unified bird's-eye view grid representation by compressing 3D point cloud data, which significantly reduces computational complexity and retains the key features of each grid. This helps to extract feature maps with global semantics and spatial consistency, thereby providing accurate high-dimensional spatial information for subsequent tasks such as object detection and semantic segmentation.

[0147] In some embodiments, the step of obtaining the second feature point cloud corresponding to the original point cloud data includes: preprocessing the original point cloud data; extracting local features of the preprocessed original point cloud data based on a preset feature extraction model; aggregating all local features to obtain global features; and obtaining the second feature point cloud based on the global features.

[0148] In some embodiments, the raw point cloud data acquired by the lidar is first preprocessed to remove redundant data and noise points. The raw point cloud data from the lidar is acquired by multiple laser beams, and each data point contains its three-dimensional coordinates and intensity value. The preprocessing process includes voxelization downsampling, noise filtering, and coordinate normalization. Voxelization downsampling reduces the density of the point cloud by dividing the point cloud data into a fixed-size three-dimensional grid and selecting representative points within each grid, while preserving spatial structure information. Noise filtering uses statistical filtering and radius filtering methods to remove isolated and outlier points, enhancing the quality and density of the point cloud. Then, a preset feature extraction model is used to extract features from the preprocessed raw point cloud data. Specifically, the preset feature extraction model first segments the raw point cloud data into multiple local regions using a hierarchical grouping strategy. The point cloud features of each local region are encoded using a multilayer perceptron to form a local geometric feature representation. These local features are then aggregated into a global feature through layer-by-layer abstraction and feature propagation. This global feature retains the geometric structure information of the point cloud, thus yielding the second feature point cloud. The preset feature extraction model can be PointNet++, an improved deep learning model used to process 3D point cloud data. PointNet++ introduces a local feature extraction mechanism through a hierarchical architecture, combining sampling and grouping strategies to gradually aggregate local and global information, thus improving the ability to model the local geometric structure of point clouds. PointNet++ is used as the backbone network to extract 3D features of LiDAR modes. The process of extracting local features can be described by the following formula 14:

[0149] ;

[0150] in, f ( x 1, x 2,…, x n Let y be an unordered set of points, γ and h be multilayer perceptron networks, the response of h is equivalent to a spatial encoding of a point, and Max is max pooling.

[0151] The process of aggregating global features in the PointNet++ model is shown in Equation 15 below:

[0152] ;

[0153] Where γ is the local feature encoding function, MaxPool represents the max pooling operation used to aggregate local features, and MLP represents a multilayer perceptron. This is a global feature.

[0154] The cross-modal feature fusion method of the present invention solves the problem of insufficient local feature capture of the original PointNet by layer-by-layer abstraction and feature propagation, and enhances the robustness to unevenly sampled point clouds. This allows the second feature point cloud to retain both local details and capture global spatial structure information, thereby providing accurate high-dimensional spatial information for subsequent tasks such as object detection and semantic segmentation.

[0155] In some embodiments, the step of obtaining the two-dimensional features of the lidar mode based on the second feature point cloud includes: obtaining the channel dimension features corresponding to each feature channel in the second feature point cloud, and obtaining the point dimension features corresponding to each second data point in the second feature point cloud; performing weighted processing on the channel dimension features corresponding to each feature channel according to a preset weight, and performing weighted processing on the point dimension features corresponding to each second data point according to a preset weight; and obtaining the two-dimensional features of the lidar mode based on all weighted channel dimension features and all weighted point dimension features.

[0156] In some embodiments, such as Figure 5 As shown, to further enhance the two-dimensional feature representation of the LiDAR modality, a columnar sensing attention module is introduced. This module generates attention features with global semantic information by weighting the input second feature point cloud along both the channel and point dimensions. The weighted two-dimensional feature representation, i.e., the LiDAR modality two-dimensional feature, is obtained by multiplying the attention weights point-by-point and channel-by-channel by the second feature point cloud. The columnar sensing attention module performs average pooling and max pooling on the input data along both the channel and point dimensions to obtain four feature representations. ,in and For channel dimension features, and The feature is a point dimension. Then, a multilayer perceptron is used to encapsulate the global information, and the attention weights for the channel dimension and the point dimension are calculated, as shown in Formula 16 below:

[0157] ;

[0158] in, ω 0 and ω 1 represents the weights of the two fully connected layers in the multilayer perceptron, i.e., the preset weights. Attention weights along the channel dimension. The attention weights are defined in the point dimension. In the channel dimension, the attention weights are multiplied with the second feature point cloud channel by channel. In the point dimension, the attention weights are multiplied with the second feature point cloud point by point, ultimately yielding the weighted two-dimensional feature representation, i.e., the two-dimensional feature of the LiDAR modality.

[0159] The cross-modal feature fusion method of the present invention enhances the detailed representation capability of LiDAR modal two-dimensional features by utilizing a columnar perception attention module, laying the foundation for subsequent cross-modal fusion.

[0160] In some embodiments, the step of generating a third feature point cloud based on a first feature point cloud and a second feature point cloud includes: spatially aligning the first feature point cloud and the second feature point cloud, and querying in the first feature point cloud a first data point that matches each second data point in the second feature point cloud; obtaining a second feature vector corresponding to each second data point, merging the second feature vector with the first feature vector of the first data point that matches it to obtain a third feature vector; and generating a third feature point cloud based on all third feature vectors.

[0161] In some embodiments, the step of performing compression mapping processing based on the third feature point cloud to obtain mixed-modality two-dimensional features includes: mapping each third data point in the third feature point cloud to a bird's-eye view grid to obtain the third cell corresponding to each third data point in the bird's-eye view grid; based on each mapped third cell, aggregating the third feature vectors corresponding to all third data points therein to obtain the third target feature; and based on the third target features of all third cells, obtaining the mixed-modality two-dimensional features.

[0162] In some embodiments, a first feature point cloud of visual data stream is generated for the camera modality. The second feature point cloud of the LiDAR modality is then added and merged with the first feature point cloud at the point level to generate a new three-dimensional feature point cloud, namely the third feature point cloud. The third feature point cloud is compressed into a hybrid modal two-dimensional feature with the same dimension as the two-dimensional feature of the camera modality through downsampling. The core idea of ​​point-level fusion is to merge the three-dimensional feature representations of different modalities through a weighted method. Specifically, for each point in the second feature point cloud, a corresponding point is found in the first feature point cloud. This can be done through coordinate projection or adjacent point matching to ensure that the two modal features correspond correctly. This achieves the addition of the feature vector of the LiDAR modality data point with the feature vector of the corresponding data point in the camera modality to generate a hybrid modal three-dimensional feature representation, namely the third feature vector.

[0163] In some embodiments, the third feature point cloud obtained after point-level fusion is downsampled to compress the 3D features into 2D bird's-eye view features. This involves mapping the third feature point cloud from the vehicle coordinate system to the bird's-eye view grid and filtering out points that fall outside the grid boundaries after translation, completing the compression process from 3D point cloud to 2D representation. Point features projected onto the same cell of the bird's-eye view grid are pooled and aggregated to generate a unified 2D feature representation, i.e., hybrid modal 2D features. Max pooling or weighted average pooling is used to ensure that the fused features capture the most representative information within the grid.

[0164] This invention relates to the field of environmental perception in autonomous driving technology, specifically a cross-modal feature fusion method based on vehicle-mounted cameras and LiDAR sensors. Through three aspects—dimensionality-enhancing compression mapping, multi-dimensional feature extraction, and dynamic fusion—it provides a more efficient and accurate solution for environmental perception tasks. Specifically, this invention proposes a cross-modal feature fusion method based on multi-source data enhancement and compression mapping, tailored to the data characteristics of vehicle-mounted cameras and LiDAR sensors. This method improves the perception accuracy and robustness of autonomous driving systems in complex environments and can be widely applied to tasks such as target detection, obstacle recognition, and path planning in autonomous driving, providing more accurate and reliable environmental perception capabilities for intelligent driving.

[0165] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 6 As shown, embodiments of the present invention also provide a cross-modal feature fusion system, comprising:

[0166] The acquisition unit 110 is configured to acquire the raw images and raw point cloud data collected by the vehicle-mounted camera and the lidar, respectively.

[0167] The camera modality processing unit 120 is configured to perform dimensionality upscaling processing on the original image to obtain a first feature point cloud, and to perform compression mapping processing on the first feature point cloud to obtain two-dimensional features of the camera modality.

[0168] The lidar mode processing unit 130 is configured to acquire a second feature point cloud corresponding to the original point cloud data, and extract lidar mode two-dimensional features based on the second feature point cloud.

[0169] The hybrid modality processing unit 140 is configured to generate a third feature point cloud based on the first feature point cloud and the second feature point cloud, and to perform compression mapping processing based on the third feature point cloud to obtain hybrid modality two-dimensional features.

[0170] The cross-modal fusion unit 150 is configured to generate cross-modal fusion features based on camera modal 2D features, lidar modal 2D features, and hybrid modal 2D features.

[0171] The cross-modal feature fusion system of this invention processes the original image data of the camera modality using dimensionality enhancement and compression mapping to obtain the three-dimensional feature point cloud corresponding to the camera modality. This three-dimensional feature point cloud is then projected onto a bird's-eye view domain to obtain a two-dimensional feature representation of the camera modality. Next, feature extraction is performed on the original point cloud data acquired by the LiDAR to obtain the three-dimensional feature point cloud and two-dimensional feature representation corresponding to the LiDAR modality. Then, the three-dimensional feature point clouds corresponding to the LiDAR modality and the camera modality are fused to obtain the three-dimensional feature point cloud corresponding to the hybrid modality. After converting the three-dimensional feature point cloud corresponding to the hybrid modality into a two-dimensional feature representation consistent with the camera modality, a preset cross-modal fusion network is used to perform channel filtering, attention weighting, and dynamic calibration on the two-dimensional features of the camera, LiDAR, and hybrid modality. This achieves selective feature enhancement and suppresses redundant information, generating cross-modal fusion features containing rich semantic and geometric information. This improves perception accuracy, robustness, and reliability, providing high-precision input for environmental perception tasks.

[0172] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 7 As shown, an embodiment of the present invention also provides a computer device 30, which includes a processor 310 and a memory 320. The memory 320 stores a computer program 321 that can be run on the processor. When the processor 310 executes the program, it performs the steps of the method described above.

[0173] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 8 As shown, embodiments of the present invention also provide a computer-readable storage medium 40, which stores a computer program 410 that, when executed by a processor, performs the methods described above.

[0174] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0175] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0176] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. The sequence numbers of the disclosed embodiments of this invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0177] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0178] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A cross-modal feature fusion method, characterized in that, include: Acquire raw images and raw point cloud data collected by the vehicle-mounted camera and LiDAR respectively; The original image is subjected to dimensionality upscaling to obtain a first feature point cloud, and the first feature point cloud is subjected to compression mapping to obtain two-dimensional features of the camera modality. Obtain the second feature point cloud corresponding to the original point cloud data, and obtain the two-dimensional features of the lidar mode based on the second feature point cloud; A third feature point cloud is generated based on the first feature point cloud and the second feature point cloud, and a compression mapping process is performed based on the third feature point cloud to obtain a hybrid modal two-dimensional feature. Based on the camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features, cross-modal fusion features are generated; The steps for generating cross-modal fusion features include: The initial stitched features are generated by stitching together the camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features; Based on the first channel weights corresponding to each feature channel in the initial splicing features, the initial splicing features are optimized to obtain the target splicing features; The target splicing features are input into a preset cross-modal fusion network to obtain the output results, which are then used as the cross-modal fusion features. The steps to obtain the output include: The first feature of camera mode type in the target stitching features is obtained, and the first feature is used as a query feature parameter and input into the preset cross-modal fusion network; Obtain the second feature of the target stitching feature whose modal type is lidar mode, and input the second feature as key feature parameter and value feature parameter into the preset cross-modal fusion network; The attention weights are obtained based on the query feature parameters and the key feature parameters; Based on the attention weights and the value feature parameters, the initial fusion features are determined; Obtain the third feature of the target splicing features that is a mixed modality feature, and determine the output result based on the initial fusion feature and the third feature; The steps for determining the output result include: The initial fusion feature is evaluated based on a preset evaluation function to obtain a first evaluation result, and the third feature is evaluated based on the preset evaluation function to obtain a second evaluation result. Based on the first evaluation result and the second evaluation result, the target weight coefficient is determined; The initial fusion feature and the third feature are weighted and summed based on the target weight coefficients to obtain the output result; The steps to obtain the first evaluation result include: Based on a preset activation function, the second channel weights corresponding to each feature channel in the initial fusion feature are generated, and the initial fusion feature is optimized based on all the second channel weights. The optimized initial fusion features are evaluated based on a preset evaluation function to obtain an initial evaluation result; Determine whether the initial evaluation result meets the first preset state; In response to the initial evaluation result not meeting the first preset state, the learning parameters are adjusted to optimize the preset activation function, and the steps of generating the second channel weights corresponding to each feature channel in the initial fusion feature based on the preset activation function and optimizing the initial fusion feature based on all the second channel weights are returned. In response to the initial evaluation result satisfying a first preset state, the initial evaluation result is taken as the first evaluation result.

2. The cross-modal feature fusion method according to claim 1, characterized in that, The step of performing dimensionality upscaling processing on the original image to obtain the first feature point cloud includes: Obtain the external and internal parameters of the vehicle-mounted camera; Obtain the depth estimation result corresponding to each pixel coordinate in the original image; Based on the preset transformation matrix, the external parameters, the internal parameters, and the depth estimation results, the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system are obtained; The loss result is determined based on each pixel coordinate and its corresponding three-dimensional coordinate; Update the preset transformation matrix based on the loss result, and return to the step of obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system based on the preset transformation matrix, the external parameters, the internal parameters and the depth estimation result, until the loss result satisfies the second preset state; The first feature point cloud is obtained based on the three-dimensional coordinates of all the pixel coordinates.

3. The cross-modal feature fusion method according to claim 2, characterized in that, The step of performing compression mapping processing based on the first feature point cloud to obtain two-dimensional features of the camera modality includes: Based on each first data point in the first feature point cloud, obtain its corresponding first feature vector and depth distribution; Based on the corresponding first feature vector and the depth distribution, the context features of each first data point are determined; Map each of the first data points to the bird's-eye view grid to obtain the first cell corresponding to each of the first data points in the bird's-eye view grid. Based on each mapped first cell, the context features corresponding to all first data points are aggregated to obtain the first target feature; Based on the first target features of all the first cells, the two-dimensional features of the camera modality are obtained.

4. The cross-modal feature fusion method according to claim 1, characterized in that, The step of obtaining the second feature point cloud corresponding to the original point cloud data and obtaining the two-dimensional features of the lidar mode based on the second feature point cloud includes: Based on a preset feature extraction model, local features of the original point cloud data are extracted, and all local features are aggregated to obtain global features to obtain the second feature point cloud. Obtain the channel dimension feature corresponding to each feature channel in the second feature point cloud, and obtain the point dimension feature corresponding to each second data point in the second feature point cloud; Based on preset weights, the channel dimension features corresponding to each feature channel are weighted sequentially, and the point dimension features corresponding to each second data point are weighted sequentially based on preset weights. Based on all the weighted channel dimension features and all the weighted point dimension features, the two-dimensional features of the lidar mode are obtained.

5. The cross-modal feature fusion method according to claim 1, characterized in that, The step of generating a third feature point cloud based on the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain a hybrid modality two-dimensional feature includes: In the first feature point cloud, query the first data point that matches each of the second data points in the second feature point cloud; Obtain the second feature vector corresponding to each second data point, and merge the second feature vector with the first feature vector of the first data point that matches it to obtain the third feature vector; The third feature point cloud is generated based on all the third feature vectors. Map each third data point in the third feature point cloud to the bird's-eye view grid to obtain the third cell corresponding to each third data point in the bird's-eye view grid. Based on each of the mapped third cells, the third feature vectors corresponding to all the third data points are aggregated to obtain the third target feature; The mixed-modal two-dimensional features are obtained based on the third target features of all the third cells.

6. A cross-modal feature fusion system, characterized in that, include: The acquisition unit is configured to acquire the raw images and raw point cloud data collected by the vehicle-mounted camera and LiDAR, respectively. A camera modality processing unit is configured to perform dimensionality upscaling processing on the original image to obtain a first feature point cloud, and to perform compression mapping processing on the first feature point cloud to obtain two-dimensional features of the camera modality. The lidar modal processing unit is configured to acquire a second feature point cloud corresponding to the original point cloud data, and extract lidar modal two-dimensional features based on the second feature point cloud. A hybrid modal processing unit is configured to generate a third feature point cloud based on the first feature point cloud and the second feature point cloud, and to perform compression mapping processing based on the third feature point cloud to obtain hybrid modal two-dimensional features; A cross-modal fusion unit is configured to generate cross-modal fusion features based on the camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features; The cross-modal fusion unit is further configured to: stitch together the camera modal 2D features, the lidar modal 2D features, and the hybrid modal 2D features to generate initial stitched features; optimize the initial stitched features based on the first channel weights corresponding to each feature channel in the initial stitched features to obtain target stitched features; input the target stitched features into a preset cross-modal fusion network to obtain the output result and use it as the cross-modal fusion feature; The first feature of camera mode type in the target stitching features is obtained, and the first feature is used as a query feature parameter and input into the preset cross-modal fusion network; the second feature of lidar mode type in the target stitching features is obtained, and the second feature is used as a key feature parameter and a value feature parameter and input into the preset cross-modal fusion network; attention weights are obtained according to the query feature parameters and the key feature parameters; initial fusion features are determined based on the attention weights and the value feature parameters. Obtain the third feature of the target splicing features that is a mixed modality feature, and determine the output result based on the initial fusion feature and the third feature; The initial fusion feature is evaluated based on a preset evaluation function to obtain a first evaluation result; the third feature is evaluated based on the preset evaluation function to obtain a second evaluation result; and a target weight coefficient is determined based on the first evaluation result and the second evaluation result. The initial fusion feature and the third feature are weighted and summed based on the target weight coefficients to obtain the output result; Based on a preset activation function, the second channel weights corresponding to each feature channel in the initial fusion feature are generated, and the initial fusion feature is optimized based on all the second channel weights. The optimized initial fusion features are evaluated based on a preset evaluation function to obtain an initial evaluation result; it is then determined whether the initial evaluation result satisfies a first preset state. In response to the initial evaluation result not meeting the first preset state, the learning parameters are adjusted to optimize the preset activation function, and the steps of generating the second channel weights corresponding to each feature channel in the initial fusion feature based on the preset activation function and optimizing the initial fusion feature based on all the second channel weights are returned. In response to the initial evaluation result satisfying a first preset state, the initial evaluation result is taken as the first evaluation result.

Images