Cross-modal feature fusion method and system

Abstract

The invention relates to the technical field of automatic driving and robots, and discloses a cross-modal feature fusion method and system. The method comprises the following steps: acquiring original images and original point cloud data respectively acquired by a vehicle-mounted camera and a laser radar; performing dimension pull-up processing based on the original image to obtain a first feature point cloud, and performing compression mapping processing based on the first feature point cloud to obtain a camera modal two-dimensional feature; obtaining a second feature point cloud corresponding to the original point cloud data, and obtaining a laser radar modal two-dimensional feature based on the second feature point cloud; generating a third feature point cloud according to the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain a mixed modal two-dimensional feature; and generating a cross-modal fusion feature based on the camera modal two-dimensional feature, the laser radar modal two-dimensional feature and the mixed modal two-dimensional feature. According to the scheme of the invention, more accurate and reliable environment perception capability can be provided for intelligent driving.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving technology and robotics, and in particular to a cross-modal feature fusion method and system. Background Art

[0002] With the rapid development of autonomous driving technology and robotics, environmental perception, as a core module, directly determines the safety and reliability of intelligent vehicles and robots. Intelligent vehicles and robots need to perceive and understand complex and changing environments with high precision, including tasks such as target detection, obstacle recognition, path planning, and semantic segmentation. These tasks place extremely high demands on environmental perception, requiring not only the capture of rich semantic information, but also accurate geometric and depth information. Single data sources often have limitations in performance. For example, cameras can capture rich texture and color information, but are weak in depth perception, while lidar has strong depth perception capabilities, but is insufficient in capturing and resolving texture information. Therefore, the fusion of multimodal sensors has become an important research direction in current environmental perception technology. However, due to the large differences in the characteristics of different modalities, the fusion of their data needs to overcome the consistency issues in resolution, dimension, and timing between modalities, making cross-modal feature fusion between sensors of different modalities the focus and difficulty of technical research.

[0003] At present, most cross-modal feature fusion methods adopt a fixed feature fusion strategy, that is, fixed fusion weights or feature selection rules are preset in the network design stage. They cannot dynamically adapt to the changes in modal characteristics in different scenarios, and have poor perception accuracy, robustness and reliability.

[0004] Therefore, there is still room for improvement in existing cross-modal feature fusion technology. Summary of the invention

[0005] In view of this, the present invention proposes a cross-modal feature fusion method and system, which solves the problems of poor adaptability, low perception accuracy, poor robustness and reliability that occur in the cross-modal feature fusion process, and provides more accurate and reliable environmental perception capabilities for intelligent driving.

[0006] Based on the above purpose, an embodiment of the present invention provides a cross-modal feature fusion method, which specifically includes the following steps: Obtain the original image and original point cloud data collected by the vehicle-mounted camera and lidar respectively; Performing dimensionality upscaling processing based on the original image to obtain a first feature point cloud, and performing compression mapping processing based on the first feature point cloud to obtain a camera modality two-dimensional feature; Acquire a second feature point cloud corresponding to the original point cloud data, and obtain a two-dimensional feature of a laser radar modality based on the second feature point cloud; Generating a third feature point cloud according to the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain a mixed-modal two-dimensional feature; Based on the camera modality two-dimensional features, the lidar modality two-dimensional features and the mixed modality two-dimensional features, a cross-modality fusion feature is generated.

[0007] In some embodiments, the step of generating a cross-modal fusion feature based on the camera modality two-dimensional feature, the lidar modality two-dimensional feature, and the mixed modality two-dimensional feature includes: Splicing the camera modality two-dimensional feature, the laser radar modality two-dimensional feature, and the mixed modality two-dimensional feature to generate an initial spliced ​​feature; Based on the first channel weights corresponding to all feature channels in the splicing feature, the initial splicing feature is optimized to obtain a target splicing feature; The target splicing feature is input into a preset cross-modal fusion network to obtain an output result and use it as the cross-modal fusion feature.

[0008] In some embodiments, the step of inputting the target splicing feature into a preset cross-modal fusion network to obtain an output result includes: Acquire a first feature of the target stitching feature whose modality type is camera modality, and use the first feature as a query feature parameter and input it into the preset cross-modal fusion network; Acquire a second feature of the target splicing feature whose modality type is a laser radar modality, and input the second feature as a key feature parameter and a value feature parameter into the preset cross-modal fusion network; Obtaining an attention weight according to the query feature parameter and the key feature parameter; Determining an initial fusion feature based on the attention weight and the value feature parameter; A third feature whose modality type is a mixed modality feature in the target splicing feature is obtained, and the output result is determined based on the initial fusion feature and the third feature.

[0009] In some embodiments, the step of obtaining a third feature of the target splicing feature whose modality type is a mixed modality feature, and determining the output result based on the initial fusion feature and the third feature, includes: The initial fusion feature is evaluated based on a preset evaluation function to obtain a first evaluation result, and the third feature is evaluated based on a preset evaluation function to obtain a second evaluation result; Determining a target weight coefficient based on the first evaluation result and the second evaluation result; The initial fusion feature and the third feature are weighted and summed based on the target weight coefficient to obtain the output result.

[0010] In some implementations, the step of evaluating the initial fusion feature based on a preset evaluation function to obtain a first evaluation result includes: Based on a preset excitation function, generating a second channel weight corresponding to each feature channel in the initial fusion feature, and optimizing the initial fusion feature based on all the second channel weights; Evaluate the optimized initial fusion features based on a preset evaluation function to obtain an initial evaluation result; Determining whether the initial evaluation result satisfies a first preset state; In response to the initial evaluation result not satisfying the first preset state, adjusting the learning parameters to optimize the preset excitation function, and returning to the step of generating a second channel weight corresponding to each feature channel in the initial fusion feature based on the preset excitation function, and optimizing the initial fusion feature based on all the second channel weights; In response to the initial evaluation result satisfying a first preset state, the initial evaluation result is used as the first evaluation result.

[0011] In some implementations, the step of performing dimensionality upscaling processing based on the original image to obtain a first feature point cloud includes: Obtaining external parameters and internal parameters of the vehicle-mounted camera; Obtaining a depth estimation result corresponding to each pixel coordinate in the original image; Obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to the preset transformation matrix, the external parameters, the internal parameters and the depth estimation result; Determine a loss result based on each pixel coordinate and its corresponding three-dimensional coordinate; updating the preset transformation matrix according to the loss result, and returning to the step of obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to the preset transformation matrix, the external parameters, the internal parameters and the depth estimation result, until the loss result satisfies a second preset state; Based on the three-dimensional coordinates of all the pixel coordinates, the first feature point cloud is obtained.

[0012] In some implementations, the step of performing compression mapping processing based on the first feature point cloud to obtain a camera modality two-dimensional feature includes: Based on each first data point in the first feature point cloud, obtaining a first feature vector and a depth distribution corresponding to the first data point; Determining a context feature of each of the first data points based on the corresponding first feature vector and the depth distribution; Mapping each of the first data points to a bird's-eye view grid to obtain a first cell corresponding to each of the first data points in the bird's-eye view grid; Based on each of the mapped first cells, aggregating context features corresponding to all first data points therein to obtain a first target feature; The camera modality two-dimensional feature is obtained based on the first target features of all the first cells.

[0013] In some embodiments, the step of obtaining a second feature point cloud corresponding to the original point cloud data, and obtaining a two-dimensional feature of a laser radar modality based on the second feature point cloud, includes: Based on a preset feature extraction model, extract local features of the original point cloud data, and aggregate all the local features to obtain global features to obtain the second feature point cloud; Obtain a channel dimension feature corresponding to each feature channel in the second feature point cloud, and obtain a point dimension feature corresponding to each second data point in the second feature point cloud; Based on the preset weight, weighted processing is performed on the channel dimension feature corresponding to each of the feature channels in sequence, and based on the preset weight, weighted processing is performed on the point dimension feature corresponding to each of the second data points in sequence; Based on the weighted channel dimensional features and the weighted point dimensional features, the two-dimensional features of the laser radar modality are obtained.

[0014] In some embodiments, the step of generating a third feature point cloud according to the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain a mixed-modal two-dimensional feature includes: Spatially aligning the first feature point cloud and the second feature point cloud, and searching the first feature point cloud for first data points that respectively match each second data point in the second feature point cloud; Obtaining a second eigenvector corresponding to each of the second data points, and merging the second eigenvector with the first eigenvector of the first data point matched therewith to obtain a third eigenvector; Based on all the third eigenvectors, generating the third feature point cloud; Mapping each third data point in the third feature point cloud to a bird's-eye view grid to obtain a third cell corresponding to each third data point in the bird's-eye view grid; Based on each of the mapped third cells, aggregating third feature vectors corresponding to all third data points therein to obtain a third target feature; Based on the third target features of all the third cells, the mixed modal two-dimensional features are obtained.

[0015] Another aspect of the embodiment of the present invention further provides a cross-modal feature fusion system, including: An acquisition unit configured to acquire original images and original point cloud data respectively collected by the vehicle-mounted camera and the laser radar; A camera modality processing unit, configured to perform dimensionality upscaling processing based on the original image to obtain a first feature point cloud, and perform compression mapping processing based on the first feature point cloud to obtain a camera modality two-dimensional feature; A laser radar modality processing unit configured to obtain a second feature point cloud corresponding to the original point cloud data, and extract a laser radar modality two-dimensional feature based on the second feature point cloud; A mixed mode processing unit, configured to generate a third feature point cloud according to the first feature point cloud and the second feature point cloud, and perform compression mapping processing based on the third feature point cloud to obtain a mixed mode two-dimensional feature; The cross-modal fusion unit is configured to generate a cross-modal fusion feature based on the camera modality two-dimensional feature, the lidar modality two-dimensional feature and the mixed modality two-dimensional feature.

[0016] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: at least one processor; and a memory, wherein the memory stores a computer program executable on the processor, and the computer program implements the steps of the above method when executed by the processor.

[0017] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores a computer program that implements the above method steps when executed by a processor.

[0018] The present invention has at least the following beneficial technical effects: the original image data of the camera modality is processed by dimensionality lifting and compression mapping to obtain the three-dimensional feature point cloud corresponding to the camera modality, which is then projected into the bird's-eye view domain to obtain the two-dimensional feature representation of the camera modality, and then the original point cloud data collected by the lidar is subjected to feature extraction to obtain the three-dimensional feature point cloud and the two-dimensional feature representation corresponding to the lidar modality, the three-dimensional feature point clouds corresponding to the lidar modality and the camera modality are respectively fused to obtain the three-dimensional feature point cloud under the hybrid modality, the three-dimensional feature point cloud corresponding to the hybrid modality is converted into a two-dimensional feature representation consistent with the camera modality, and then a preset cross-modal fusion network is used to finally perform channel screening, attention weighting and dynamic calibration on the two-dimensional features of the camera, lidar and hybrid modality to achieve feature selectivity enhancement and suppress redundant information, generate cross-modal fusion features containing rich semantic information and geometric information, realize cross-modal feature fusion of camera-lidar, improve perception accuracy, robustness and reliability, and provide high-precision input for environmental perception tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A block diagram of an embodiment of the cross-modal feature fusion method provided by the present invention; Figure 2 A schematic diagram of an embodiment of a cross-modal feature fusion method framework provided by the present invention; Figure 3 A schematic diagram of an embodiment of a dynamic cross-modal feature fusion process provided by the present invention; Figure 4 A schematic diagram of an embodiment of a dimensionality-enhancing compression mapping process provided by the present invention; Figure 5 A schematic diagram of an embodiment of a laser radar modal feature extraction process provided by the present invention; Figure 6 A schematic diagram of an embodiment of a cross-modal feature fusion system provided by the present invention; Figure 7 A schematic diagram of the structure of an embodiment of a computer device provided by the present invention; Figure 8 A schematic diagram of the structure of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0022] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.

[0023] In autonomous driving and robotic systems, on-board cameras and lidar are two common types of sensors. On-board cameras provide rich semantic information by capturing two-dimensional images, which is suitable for target classification and semantic segmentation. Lidar provides high-precision three-dimensional geometric information through point cloud data, which is suitable for three-dimensional scene reconstruction and obstacle detection. However, due to the large differences in the characteristics of the two modalities, their data fusion needs to solve heterogeneity problems in many aspects such as resolution, dimension and timing. This makes the cross-modal feature fusion of cameras and lidars the focus and difficulty of technical research.

[0024] Current cross-modal feature fusion technologies are mainly divided into two categories: feature-level fusion and decision-level fusion. Feature-level fusion usually performs joint modeling of multimodal data in the data preprocessing or feature extraction stage, and uses deep learning networks to map data from different modalities to a unified feature space. For example, some studies use the geometric relationship between two-dimensional images and three-dimensional point clouds to project three-dimensional point clouds onto the two-dimensional image plane for alignment, and then extract fusion features through a joint network. Although this method can achieve preliminary alignment between modalities, it often ignores the problem of unbalanced feature expression caused by modal differences. In addition, some studies have attempted to embed image information into the point cloud space for three-dimensional modeling, but due to the sparse and uneven distribution of point cloud data, it is difficult to ensure the accuracy of feature extraction. Decision-level fusion usually performs weighted integration of the detection results of multimodal data through an independent network after feature extraction and target detection. This method is simple in logic and easy to implement, but due to the lack of deep fusion at the feature level, it is difficult to give full play to the complementary advantages of multimodal data, and usually only plays a role in performance correction.

[0025] Moreover, most cross-modal feature fusion methods still use fixed feature fusion strategies, that is, fixed fusion weights or feature selection rules are preset during the network design phase. However, this static strategy cannot dynamically adapt to changes in modal characteristics in different scenarios. In addition, the data from on-board cameras and lidars differ significantly in resolution and dimension, and direct fusion may lead to decreased perception accuracy. At the same time, existing methods generally lack effective screening and calibration mechanisms for fusion features, which can easily introduce irrelevant features or noise, thereby reducing the robustness and reliability of autonomous driving and robotic systems.

[0026] In view of the above defects, the first aspect of the embodiment of the present invention proposes a cross-modal feature fusion method. Figure 1 As shown, it includes the following steps: Step S100, obtaining original images and original point cloud data collected by the vehicle-mounted camera and the laser radar respectively; Step S200, performing dimensionality upscaling processing based on the original image to obtain a first feature point cloud, and performing compression mapping processing based on the first feature point cloud to obtain a camera modality two-dimensional feature; Step S300, obtaining a second feature point cloud corresponding to the original point cloud data, and obtaining a two-dimensional feature of a laser radar modality based on the second feature point cloud; Step S400, generating a third feature point cloud according to the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain a mixed-modal two-dimensional feature; Step S500, generating a cross-modal fusion feature based on the camera modality two-dimensional feature, the lidar modality two-dimensional feature and the mixed modality two-dimensional feature.

[0027] In some embodiments, the original image mainly comes from the real-time acquisition of the vehicle-mounted camera, and the original point cloud data mainly comes from the real-time acquisition of the laser radar. With the dimension lifting compression mapping theory as the core idea, the scale compression mechanism, attention mechanism, three-dimensional feature extraction and other methods are used to obtain fusion features containing rich feature information of different dimensions, and then the fused features are processed through the preset cross-modal fusion network, forming a complete set of cross-modal feature fusion methods.

[0028] In some embodiments, first, for the camera modality, the multi-view views taken by different on-board cameras are fused by coordinate fusion. The dimensionality-lifting compression mapping theory is used to lift the fused multi-view images to form a point cloud array of the camera modality, and then extract the three-dimensional features of the camera modality based on the point cloud array to obtain a three-dimensional point cloud, i.e., the first feature point cloud. The first feature point cloud is further projected to the bird's-eye view domain to form a two-dimensional feature representation, i.e., the camera modality two-dimensional feature. Then, for the lidar modality, the local and global features of the original point cloud data are extracted by a preset feature extraction model to obtain a second feature point cloud, and the two-dimensional features of the lidar modality are predicted by inference through the attention mechanism based on the second feature point cloud, i.e., the lidar modality two-dimensional feature. Next, the second feature point cloud is fused with the first feature point cloud in three-dimensional space to obtain a third feature point cloud, and then the two-dimensional feature representation is compressed and extracted for the third feature point cloud to obtain a mixed modality two-dimensional feature. Finally, the two-dimensional features of the camera, lidar and mixed modalities are spliced ​​in series and input into a preset cross-modal fusion network, which performs channel screening, attention weighting and dynamic calibration to generate the final cross-modal fusion features. The cross-modal feature fusion method of the present invention realizes the feature extraction and dynamic cross-modal fusion of the camera modality and the lidar modality.

[0029] The cross-modal feature fusion method of the present invention processes the original image data of the camera modality by dimensionality lifting and compression mapping, obtains the three-dimensional feature point cloud corresponding to the camera modality, and then projects it to the bird's-eye view domain to obtain the two-dimensional feature representation of the camera modality, then performs feature extraction on the original point cloud data collected by the laser radar, obtains the three-dimensional feature point cloud and the two-dimensional feature representation corresponding to the laser radar modality, fuses the three-dimensional feature point clouds corresponding to the laser radar modality and the camera modality respectively to obtain the three-dimensional feature point cloud under the hybrid modality, converts the three-dimensional feature point cloud corresponding to the hybrid modality into a two-dimensional feature representation consistent with the camera modality, uses a preset cross-modal fusion network, and finally performs channel screening, attention weighting and dynamic calibration on the two-dimensional features of the camera, laser radar and hybrid modality, realizes feature selectivity enhancement and suppresses redundant information, generates cross-modal fusion features containing rich semantic information and geometric information, realizes cross-modal feature fusion of camera-laser radar, improves perception accuracy, robustness and reliability, and provides high-precision input for environmental perception tasks.

[0030] In some embodiments, the step of generating a cross-modal fusion feature based on camera modality two-dimensional features, lidar modality two-dimensional features and mixed modality two-dimensional features includes: stitching the camera modality two-dimensional features, lidar modality two-dimensional features and mixed modality two-dimensional features to generate an initial stitching feature; optimizing the initial stitching feature based on the first channel weights corresponding to all feature channels in the initial stitching feature to obtain a target stitching feature; inputting the target stitching feature into a preset cross-modal fusion network to obtain an output result and using it as a cross-modal fusion feature.

[0031] In some embodiments, the current deep cross-modal feature fusion method is essentially a static fusion method, that is, the multi-modal input data is processed and fused with the same calculation without considering the different calculation requirements of different modal data. Once the training is completed, this method cannot adapt to the differences of each data during the reasoning process. In order to solve this problem, the present invention proposes a dynamic cross-modal fusion network (i.e., a preset cross-modal fusion network), such as Figure 2 and Figure 3 As shown, by dynamically adjusting the fusion method of camera modality and lidar modality features, it is possible to "select the best from the best" among multi-modal features at all times. Figure 2 "Conv 3 3” means 3 3 convolutional layer operations, “Avg pool” means average pooling, “Conv 1 1" means 1 1 Convolutional layer operation, “sigmoid” represents the activation function. Figure 3 In it, "Conv" represents the convolution operation, "BN" represents the normalization operation, "ReLu" represents the activation function, "Cross attention" represents the cross attention mechanism, "Add&Norm" represents the residual connection, "SelfAttention" represents the self-attention mechanism, and "Matmul" represents the matrix operation.

[0032] In some embodiments, the camera modality two-dimensional features contain two-dimensional geometric semantic information (such as color, texture), the lidar modality two-dimensional features contain geometric structure information (such as depth, distance), and the mixed modality two-dimensional features fuse the preliminary information of the above two modalities. The two-dimensional features from the camera modality and the lidar modality are preprocessed separately. In order to facilitate cross-modal fusion, the camera modality features, the lidar modality features, and their mixed features are concatenated according to the channel dimension to generate initial spliced ​​features. By concatenating in the channel dimension, the feature information of each modality can be retained, the information consistency of the spatial dimension will not be changed, and the features of each modality remain independent after splicing.

[0033] In some embodiments, Figure 2As shown, the initial spliced ​​features are feature processed through the channel attention module. The channel attention module aims to dynamically adjust the importance of feature channels in the initial spliced ​​features to highlight key channels and suppress redundant channels. Specifically, the initial spliced ​​features are firstly globally average pooled and max pooled in the spatial dimension, and the average pooling result and max pooling result of each feature channel are used as channel-level global information, which characterizes the global distribution of responses on the feature channels, and enables the input layer close to the preset cross-modal fusion network to obtain the global receptive field. Then a multi-layer perceptron is used to encapsulate the global information of each channel and generate the first channel weight corresponding to each feature channel. The process of generating the first channel weight by the multi-layer perceptron is as follows: Formula 1: ; in, is the first channel weight, and are the weights of the two fully connected layers in the multilayer perceptron, is the average pooling result, is the maximum pooling result, and sigmoid is the activation function.

[0034] In some embodiments, the first channel weight is used to perform channel screening on the initial splicing feature, that is, the first channel weight is weighted to the corresponding feature channel by multiplication, and the initial splicing feature is screened in the channel dimension, thereby optimizing the initial splicing feature, and the initial splicing feature after channel screening is used as the target splicing feature. Finally, the target splicing feature is further processed by the dynamic cross-modal fusion network. The dynamic cross-modal fusion network can be summarized as the following formula 2: ; in, is the output result of the dynamic cross-modal fusion network, [F0, F1, F2] is the target splicing feature, is through 3 The static channel and spatial fusion function implemented by the 3 convolutional layers aims to reduce the channel dimension of the spliced ​​features. It is an adaptive feature selection function composed of multiple convolutional layers. It can be summarized as the following formula 3: ; Among them, F is the input feature, W is the linear transformation matrix, f avg is the global average pooling function, is the activation function. The adaptive feature selection function is used to highlight the important features in the target splicing features, which can effectively capture the complementary information of the camera modality and the lidar modality, and is conducive to obtaining cross-modal fusion features with higher expression ability.

[0035] The cross-modal feature fusion method of the present invention combines multi-modal features together through channel dimension splicing, provides a unified input for the subsequent dynamic cross-modal fusion network, and retains the information of each modality. The dynamic cross-modal fusion network is then used to obtain a bird's-eye view of different modalities. Figure 2 The dimensional features are processed so that the final cross-modal fusion features have higher expressive power and provide reliable feature support for downstream tasks (such as target detection or semantic segmentation).

[0036] In some embodiments, the step of inputting the target stitching feature into a preset cross-modal fusion network to obtain an output result includes: obtaining a first feature whose modality type is a camera modality in the target stitching feature, using the first feature as a query feature parameter and inputting it into the preset cross-modal fusion network; obtaining a second feature whose modality type is a lidar modality in the target stitching feature, and inputting the second feature as a key feature parameter and a value feature parameter into the preset cross-modal fusion network; obtaining an attention weight according to the query feature parameter and the key feature parameter; determining an initial fusion feature based on the attention weight and the value feature parameter; obtaining a third feature whose modality type is a mixed modality feature in the target stitching feature, and determining the output result based on the initial fusion feature and the third feature.

[0037] In some embodiments, Figure 3 As shown in the figure, the dynamic cross-modal fusion network combines the Query-Key-Value architecture to selectively fuse according to the dynamic requirements of the input multi-modal features. The camera modal feature (i.e., the first feature) in the target splicing feature is used as the Query (i.e., the query feature parameter), and the lidar modal feature (i.e., the second feature) in the target splicing feature is used as the Key (i.e., the key feature parameter) and the Value (i.e., the value feature parameter). The attention mechanism is used to calculate the correlation between the camera modal feature and the lidar modal feature. The calculation method is shown in the following formula 4: ; Among them, Q is the Query representation after the linear transformation of the first feature, K is the Key representation after the linear transformation of the second feature, V is the Value representation after the linear transformation of the second feature, and A is the attention weight, which represents the correlation between the first feature and the second feature. The Value feature is weighted and summed according to the attention weight to obtain the fused feature representation, that is, the initial fused feature. The weighting method is shown in the following formula 5: ; in, is the initial fusion feature.

[0038] Furthermore, some information may be lost when the lidar modal features are fused with the camera modal features. Therefore, the mixed modal features can be selectively inserted, that is, the mixed modal features (that is, the third features) in the target splicing features are inserted into the initial fusion features as supplementary information to output the final cross-modal fusion features.

[0039] The cross-modal feature fusion method of the present invention fuses visual and geometric information through the Query-Key-Value architecture to enhance the synergy between modalities. The dynamic cross-modal fusion network fully exploits the advantages of lidar modal features and camera modal features through the attention mechanism and dynamic feature selection strategy, and selectively inserts mixed modal features as supplementary information to construct multimodal fusion features containing rich semantic and geometric information.

[0040] In some embodiments, a third feature whose modality type is a mixed modality feature in the target splicing feature is obtained, and the step of determining the output result based on the initial fusion feature and the third feature includes: evaluating the initial fusion feature based on a preset evaluation function to obtain a first evaluation result, and evaluating the third feature based on a preset evaluation function to obtain a second evaluation result; determining a target weight coefficient based on the first evaluation result and the second evaluation result; and performing weighted summation processing on the initial fusion feature and the third feature based on the target weight coefficient to obtain an output result.

[0041] In some embodiments, an evaluation function is first preset, and the preset evaluation function can be expressed as the following formula 6: ; Among them, z represents the prediction result of the initial fusion feature to complete the downstream task mapping, such as the category probability in the classification task. Indicates that z belongs to the i-th category label The higher the probability, the stronger the confidence of the initial fusion feature in predicting the i-th category label. S(z) is the first evaluation result, which takes the maximum value of all category prediction probabilities, indicating the overall confidence evaluation of the initial fusion feature.

[0042] After determining the preset evaluation function, the final cross-modal fusion feature can be expressed as the following formula 7: ; in, T To set the threshold, manual adjustments were performed according to the experimental process. Indicates the third feature. is an indicator function that satisfies Set to 1 when yes, otherwise set to 0. is the target weight coefficient, The normalization strategy can be used for adjustment, which can be expressed as the following formula 8: ; in, The evaluation score for completing the downstream task mapping independently for the third feature, that is, the second evaluation result, can be calculated by the above-mentioned preset evaluation function.

[0043] The cross-modal feature fusion method of the present invention introduces an evaluation function to achieve selective insertion of mixed modal features, so that the preset cross-modal fusion network has higher adaptability and achieves the effect of "selecting the best from the best" at "all times". By weighted fusion of mixed modal features and initial fusion features, the complementary information of the lidar modality and the camera modality is retained to the greatest extent, and fusion features containing rich semantic information and depth information are obtained, thereby realizing cross-modal feature fusion of camera-lidar, and improving perception accuracy, robustness and reliability.

[0044] In some embodiments, the step of evaluating the initial fusion feature based on a preset evaluation function to obtain a first evaluation result includes: generating a second channel weight corresponding to each feature channel in the initial fusion feature based on a preset excitation function, and optimizing the initial fusion feature based on all second channel weights; evaluating the optimized initial fusion feature based on the preset evaluation function to obtain an initial evaluation result; judging whether the initial evaluation result satisfies a first preset state; in response to the initial evaluation result not satisfying the first preset state, adjusting the learning parameters to optimize the preset excitation function, and returning to the step of generating a second channel weight corresponding to each feature channel in the initial fusion feature based on the preset excitation function, and optimizing the initial fusion feature based on all second channel weights; in response to the initial evaluation result satisfying the first preset state, taking the initial evaluation result as the first evaluation result.

[0045] In some embodiments, in order to improve the dynamic adaptability of feature fusion, a dynamic feature calibration strategy is introduced. The channel importance of the initial fusion feature is adjusted by weighting each feature channel. Specifically, by assigning adaptive weights to each feature channel in the initial fusion feature, useful features are amplified, while weakly related or redundant features are suppressed. The core steps of the dynamic feature calibration strategy are compression and excitation operations in the spatial dimension. First, the global response value is calculated for each feature channel, and the calculation formula is shown in Formula 9 below: ; Among them, F attn represents the initial fusion feature, c represents any feature channel, represents the global response value, H represents the height of the feature map corresponding to the initial fusion feature, and W represents the width of the feature map corresponding to the initial fusion feature.

[0046] Then, the global response value is used to generate the second channel weight through a preset excitation function, and the preset excitation function is shown in the following formula 10: ; in, and b are learning parameters, i.e. weights and biases, σ is the activation function, is the second channel weight.

[0047] Use weight The importance of calibrating each feature channel and the excitation operation are similar to the gate mechanism in the recurrent neural network. To generate weights for each feature channel , where the parameters is learned to explicitly model the correlation between feature channels.

[0048] In some embodiments, the second channel weight is regarded as the importance of each feature channel after feature selection, and then weighted to the corresponding feature channel by channel by multiplication, completing the recalibration of the initial fusion feature in the channel dimension, thereby optimizing the initial fusion feature. The optimized initial fusion feature is evaluated using a preset evaluation function to obtain an initial evaluation result. If the initial evaluation result is less than the preset result value, the learning parameters in the preset excitation function are adjusted by, for example, a gradient descent optimization method, and the updated preset excitation function is used to generate a new second channel weight to optimize the initial fusion feature again. The above operation is repeated until the initial evaluation result is greater than or equal to the preset result value, that is, the initial evaluation result satisfies the first preset state, and the final recalibration of the initial fusion feature is completed in the channel dimension, and it is used for subsequent feature fusion operations. The preset result value is set according to actual needs and is not specifically limited here.

[0049] The cross-modal feature fusion method of the present invention can explicitly construct the interdependence between feature channels. It adopts a feature dynamic calibration strategy, and can automatically obtain the importance of each feature channel through learning. Then, according to this importance, useful features are promoted and features that are not very useful for the current task are suppressed. The initial fused features after recalibration have higher expressive power, and effective screening and calibration of fused features are achieved, which effectively captures the complementary information of the camera modality and the lidar modality, and provides reliable feature support for downstream tasks (such as target detection or semantic segmentation).

[0050] In some embodiments, the step of performing dimensionality upscaling processing based on the original image to obtain a first feature point cloud includes: obtaining external parameters and internal parameters of a vehicle-mounted camera; obtaining a depth estimation result corresponding to each pixel coordinate in the original image; obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to a preset transformation matrix, external parameters, internal parameters and depth estimation results; determining a loss result based on each pixel coordinate and its corresponding three-dimensional coordinates; updating the preset transformation matrix according to the loss result, and returning to the step of obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to the preset transformation matrix, external parameters, internal parameters and depth estimation results, until the loss result satisfies a second preset state; obtaining a first feature point cloud based on the three-dimensional coordinates of all pixel coordinates.

[0051] In some embodiments, Figure 2 and Figure 4 As shown in the figure, the data processing method of the camera modality is based on the theory of dimensionality lifting and compression mapping. Dimensionality lifting and compression mapping is a visual perception model used in the field of autonomous driving and robotics. It aims to generate a bird's-eye view representation through multi-view images to achieve a three-dimensional understanding of the environment. The core idea of ​​the dimensionality lifting and compression mapping model is to convert the two-dimensional image data collected by the camera into a three-dimensional spatial representation, thereby forming a bird's-eye view, thereby providing accurate high-dimensional spatial information for subsequent tasks such as target detection and semantic segmentation.

[0052] In some embodiments, the multi-view RGB images (i.e., original images) collected by the vehicle-mounted camera are first preprocessed. These images cover a 360-degree field of view around the vehicle, and the internal and external parameters of the camera are obtained through calibration. The internal parameters include the focal length and the principal point coordinates, and the external parameters include the rotation matrix and the translation vector. Figure 4 As shown in the figure, in the dimensionality lifting stage, the input original image is mapped to the three-dimensional space through depth estimation and camera internal and external parameters. This process requires implicit or explicit learning of depth information, usually using multi-view geometric relationships or estimating depth through a deep learning network. In the compression stage, the point cloud distribution lifted to the three-dimensional space is projected onto a unified bird's-eye view grid. This step requires processing geometric consistency across views to ensure that key information is not lost or confused during the fusion process of multi-view image information. Finally, in the mapping stage, feature extraction is performed on the two-dimensional representation projected onto the bird's-eye view grid to obtain the camera modality two-dimensional features, and the proposed camera modality two-dimensional features are used to complete downstream tasks such as target detection, trajectory prediction, or semantic segmentation.

[0053] In some embodiments, the dimension pulling stage is further described. In order to achieve a unified geometric description, the pixel points in the original image coordinate system are converted into a three-dimensional point cloud in the vehicle coordinate system through the camera internal and external parameters and depth information, so as to realize the mapping of images of different perspectives in a unified coordinate system, and lay the foundation for subsequent feature extraction. After completing the coordinate system conversion, each pixel point in the two-dimensional image is mapped to a three-dimensional space by a depth estimation technology to obtain the three-dimensional coordinates of each pixel point. Depth estimation can estimate the depth value of the pixel point based on a multi-perspective geometric relationship or a deep learning network. The depth estimation technology includes a monocular depth estimation network or a stereo matching algorithm, and can also optimize the depth estimation technology through a corresponding loss function. After obtaining the depth information, the three-dimensional coordinates of each pixel point in the vehicle coordinate system are generated by combining the internal and external parameters of the camera and the preset transformation matrix to form a cone view point cloud. The features of the cone view point cloud are extracted using the preset features to generate a first feature point cloud. The three-dimensional coordinates of each pixel point describe its spatial position in the real world, and provide corresponding geometric information for further processing.

[0054] In some embodiments, the simplified process of coordinate system conversion is shown in Formula 11 below: ; ,CAM e is an external parameter, is CAM i Internal parameters, V is the first feature point cloud, Z c is the depth, that is, the distance of the pixel from the optical axis. P uv is the pixel coordinate system, u represents the position of the pixel in the row direction (horizontal direction), and v represents the position of the pixel in the column direction (vertical direction). , represents the focal length, Represents the rotation matrix, the matrix Represents the translation vector. , Represents the principal point coordinates.

[0055] The above-mentioned coordinate system conversion requires augmenting the coordinates and parameters of the camera and vehicle and forming a mapping relationship, which involves operations such as matrix rotation, translation, parameter enhancement, and pooling. At present, the coordinate system conversion process usually adopts algebraic methods or simplified calculation methods, which often increase the amount of calculation or affect the mapping accuracy. The present invention adopts an adaptive coordinate conversion method, introduces a preset transformation matrix M, and realizes adaptive adjustment of the coordinate system by searching for mapping and scaling relationships through deep learning. Its loss function is shown in the following formula 12: ; Among them, π is the projection function from the three-dimensional coordinates to the image plane,P i are the generated three-dimensional coordinates, p i is the corresponding pixel coordinate point in the original image, It is a loss result.

[0056] Furthermore, the preset transformation matrix M can be updated by judging whether the loss result satisfies the preset state. That is, when the loss result is greater than the preset loss value, the preset transformation matrix is ​​updated by adjusting the parameters of the preset transformation matrix, and a new three-dimensional coordinate is generated for each pixel in the original image, and the above operation is repeated until the loss result is less than the preset loss value, that is, the loss result satisfies the second preset state. The preset loss value is set according to actual needs and is not specifically limited here.

[0057] The cross-modal feature fusion method of the present invention converts multi-view images into a unified three-dimensional point cloud representation through dimensionality lifting, so that the vehicle environment has geometric consistency, laying a foundation for three-dimensional space understanding, and optimizing the preset transformation matrix through deep learning, thereby reducing the computational complexity in traditional algebraic methods or simplified calculation methods, thereby ensuring the accuracy of subsequent cross-modal feature fusion.

[0058] In some embodiments, the step of performing compression mapping processing based on the first feature point cloud to obtain the camera modal two-dimensional feature includes: based on each first data point in the first feature point cloud, obtaining its corresponding first feature vector and depth distribution; based on the corresponding first feature vector and depth distribution, determining the context feature of each first data point; mapping each first data point to a bird's-eye view grid to obtain the first cell corresponding to each first data point in the bird's-eye view grid; based on each mapped first cell, aggregating the context features corresponding to all the first data points therein to obtain a first target feature; and obtaining the camera modal two-dimensional feature based on the first target features of all the first cells.

[0059] In some embodiments, for each data point on the first feature point cloud (i.e., the first data point), a multi-dimensional feature vector and a probability distribution of multiple discrete depths are predicted, and the probability distribution of discrete depths is the depth probability distribution of each pixel point in the cone view point cloud. The predicted feature vector and the depth distribution are outer-producted to generate the context feature corresponding to the first data point, and the context feature point cloud is obtained from all the generated context features. The calculation process is shown in the following formula 13: ; Among them, c is the eigenvector, α d is the depth distribution, is a context feature.

[0060] In some embodiments, Figure 4As shown in the figure, after the dimension lifting operation is completed, the first feature point cloud and the corresponding context feature point cloud are obtained respectively, wherein the first feature point cloud contains the position information of each pixel in the original image in the vehicle coordinate system, and the context feature point cloud contains the context feature information of each pixel in the original image. According to this information, the compression operation is performed to map the first feature point cloud from the vehicle coordinate system to the bird's-eye view grid, and the points that fall outside the boundary of the bird's-eye view grid after translation are filtered out, completing the compression process from the three-dimensional point cloud to the two-dimensional representation. After mapping, the pooling method is used to aggregate all the features in each cell in the bird's-eye view grid. Specifically, for all the points mapped in each cell, its feature representation is obtained by the maximum pooling operation, ensuring that the most representative feature in the cell is retained, that is, the first target feature is obtained. For example, the context features in each cell are aggregated using the pooling method. Finally, the compressed bird's-eye view grid is feature extracted using a convolutional neural network to generate a two-dimensional feature map suitable for downstream tasks, that is, the camera modality two-dimensional feature. The design of the convolutional neural network is based on the grid resolution to ensure that the feature map can capture global semantic information. In order to enhance the feature expression of key areas, a channel attention mechanism is further introduced to optimize the feature channels in a weighted manner. The optimized features have higher semantic expression ability and spatial consistency, which can effectively support the subsequent cross-modal fusion with the lidar modality.

[0061] The cross-modal feature fusion method of the present invention generates a unified bird's-eye view grid representation by compressing three-dimensional point cloud data, which significantly reduces the computational complexity and retains the key features of each grid. It helps to extract feature maps with global semantics and spatial consistency, thereby providing accurate high-dimensional spatial information for subsequent tasks such as target detection and semantic segmentation.

[0062] In some embodiments, the step of obtaining a second feature point cloud corresponding to the original point cloud data includes: preprocessing the original point cloud data; extracting local features of the preprocessed original point cloud data based on a preset feature extraction model; aggregating all local features to obtain global features, and obtaining a second feature point cloud based on the global features.

[0063] In some embodiments, first, the raw point cloud data collected by the laser radar is preprocessed to remove redundant data and noise points. The raw point cloud data of the laser radar is acquired by multiple laser beams, and each data point contains its three-dimensional coordinates and intensity values. The preprocessing process includes voxel downsampling, noise filtering and coordinate standardization. Among them, voxel downsampling divides the point cloud data into three-dimensional grids of fixed size, selects representative points in each grid, reduces the density of the point cloud, and maintains the spatial structure information. Noise filtering uses statistical filtering and radius filtering methods to remove isolated points and abnormal points, and enhance the quality and density of the point cloud. Then, the preprocessed raw point cloud data is feature extracted using a preset feature extraction model. That is, the preset feature extraction model first divides the raw point cloud data into multiple local areas through a hierarchical grouping strategy, and the point cloud features of each local area are encoded by a multi-layer perceptron to form a local geometric feature representation. These local features are aggregated into global features through layer-by-layer abstraction and feature propagation. This global feature also retains the geometric structure information of the point cloud, and the second feature point cloud can be extracted. The preset feature extraction model can be a PointNet++ model, which is an improved deep learning model for processing three-dimensional point cloud data. The PointNet++ model introduces a local feature extraction mechanism through a layered architecture, combines sampling and grouping strategies, and gradually aggregates local and global information, thereby improving the modeling capability of the local geometric structure of the point cloud. PointNet++ is used as the backbone network to extract the three-dimensional features of the lidar modality. The process of extracting local features can be described by the following formula 14: ; in, f ( x 1, x 2,…, x n ) is an unordered point set, γ and h are multi-layer perceptron networks, the response of h is equivalent to the spatial encoding of a point, and Max is the maximum pooling.

[0064] The process of aggregating global features of the PointNet++ model is as follows: ; Among them, γ is the local feature encoding function, MaxPool represents the maximum pooling operation, which is used to aggregate local features, and MLP represents the multi-layer perceptron. It is a global feature.

[0065] The cross-modal feature fusion method of the present invention solves the problem of insufficient local feature capture of the original PointNet through layer-by-layer abstraction and feature propagation, and enhances the robustness to unevenly sampled point clouds, so that the second feature point cloud not only retains local details but also captures global spatial structure information, thereby providing accurate high-dimensional spatial information for subsequent target detection, semantic segmentation and other tasks.

[0066] In some embodiments, the step of obtaining the two-dimensional features of the laser radar modal based on the second feature point cloud includes: obtaining the channel dimension features corresponding to each feature channel in the second feature point cloud, and obtaining the point dimension features corresponding to each second data point in the second feature point cloud; weighting the channel dimension features corresponding to each feature channel in turn based on preset weights, and weighting the point dimension features corresponding to each second data point in turn based on preset weights; and obtaining the two-dimensional features of the laser radar modal based on all weighted channel dimension features and all weighted point dimension features.

[0067] In some embodiments, Figure 5 As shown in the figure, in order to further enhance the two-dimensional feature expression of the lidar modality, a columnar perception attention module is introduced. This module generates attention features with global semantic information by weighting the channel dimension and point dimension of the input second feature point cloud respectively. According to the attention weight, the second feature point cloud is multiplied point by point and channel by channel to obtain the weighted two-dimensional feature representation, that is, the two-dimensional feature of the lidar modality. The columnar perception attention module performs average pooling and maximum pooling on the input data in the channel dimension and point dimension respectively to obtain four feature representations ,in and is the channel dimension feature, and is the point dimension feature. Then, a multi-layer perceptron is used to encapsulate global information and calculate the channel dimension and point dimension attention weights. The process is as follows: ; in, ω 0 and ω 1 is the weight of the two fully connected layers in the multi-layer perceptron, that is, the preset weight. is the attention weight in the channel dimension, is the attention weight in the point dimension. In the channel dimension, the attention weight is multiplied by the second feature point cloud channel by channel, and in the point dimension, the attention weight is multiplied by the second feature point cloud point by point, and finally a weighted two-dimensional feature representation is obtained, that is, the two-dimensional feature of the lidar modality.

[0068] The cross-modal feature fusion method of the present invention utilizes a columnar perception attention module to enhance the detail expression capability of the two-dimensional features of the lidar modality, laying the foundation for subsequent cross-modal fusion.

[0069] In some embodiments, the step of generating a third feature point cloud based on the first feature point cloud and the second feature point cloud includes: spatially aligning the first feature point cloud and the second feature point cloud, and querying the first data points in the first feature point cloud that respectively match the second data points in the second feature point cloud; obtaining the second feature vector corresponding to each second data point, merging the second feature vector with the first feature vector of the first data point that matches it to obtain a third feature vector; and generating a third feature point cloud based on all third feature vectors.

[0070] In some embodiments, the step of performing compression mapping processing based on the third feature point cloud to obtain a mixed modal two-dimensional feature includes: mapping each third data point in the third feature point cloud to a bird's-eye view grid to obtain a third cell corresponding to each third data point in the bird's-eye view grid; based on each mapped third cell, aggregating the third feature vectors corresponding to all the third data points therein to obtain a third target feature; and obtaining a mixed modal two-dimensional feature based on the third target features of all third cells.

[0071] In some embodiments, a first feature point cloud about the visual data stream is generated for the camera modality, and the second feature point cloud of the lidar modality is added and merged with the first feature point cloud at the point level to generate a new three-dimensional feature point cloud, i.e., the third feature point cloud. The third feature point cloud is compressed into a mixed modality two-dimensional feature of the same dimension as the two-dimensional feature of the camera modality by downsampling. The core idea of ​​point-level fusion is to merge the three-dimensional feature representations of different modalities in a weighted manner. Specifically, for each point of the second feature point cloud, a corresponding point is found in the first feature point cloud, and coordinate projection or adjacent point matching can be used to ensure that the two modal features can correspond correctly. Thereby, the feature vector of the lidar modality data point is added to the feature vector of the corresponding data point in the camera modality to generate a three-dimensional feature representation of the mixed modality, i.e., the third feature vector.

[0072] In some embodiments, the third feature point cloud obtained after point-level fusion is downsampled to compress the three-dimensional features into two-dimensional bird's-eye view features. That is, the third feature point cloud is mapped from the vehicle coordinate system to the bird's-eye view grid, and the points that fall outside the boundary of the bird's-eye view grid after translation are filtered out to complete the compression process from the three-dimensional point cloud to the two-dimensional representation. The point features projected into the same cell of the bird's-eye view grid are pooled and aggregated to generate a unified two-dimensional feature representation, that is, a mixed-modal two-dimensional feature. The pooling method uses maximum pooling or weighted average pooling to ensure that the fused features can capture the most representative information in the grid.

[0073] The present invention relates to the field of environmental perception in autonomous driving technology, and in particular to a cross-modal feature fusion method based on vehicle-mounted cameras and lidar sensors. Through dimensionality lifting and compression mapping, multi-dimensional feature extraction, and dynamic fusion, a more efficient and accurate solution is provided for environmental perception tasks. Specifically, the present invention proposes a cross-modal feature fusion method based on multi-source data lifting and compression mapping based on the data characteristics of vehicle-mounted cameras and lidar sensors, so as to improve the perception accuracy and robustness of the autonomous driving system in complex environments. It can be widely used in tasks such as target detection, obstacle recognition, and path planning in autonomous driving, providing more accurate and reliable environmental perception capabilities for intelligent driving.

[0074] Based on the same inventive concept, according to another aspect of the present invention, Figure 6 As shown, an embodiment of the present invention further provides a cross-modal feature fusion system, including: An acquisition unit 110 is configured to acquire original images and original point cloud data collected by the vehicle-mounted camera and the laser radar respectively; The camera modality processing unit 120 is configured to perform dimensionality upscaling processing based on the original image to obtain a first feature point cloud, and perform compression mapping processing based on the first feature point cloud to obtain a camera modality two-dimensional feature; The laser radar modality processing unit 130 is configured to obtain a second feature point cloud corresponding to the original point cloud data, and extract a laser radar modality two-dimensional feature based on the second feature point cloud; A mixed mode processing unit 140 is configured to generate a third feature point cloud according to the first feature point cloud and the second feature point cloud, and perform compression mapping processing based on the third feature point cloud to obtain a mixed mode two-dimensional feature; The cross-modal fusion unit 150 is configured to generate a cross-modal fusion feature based on the camera modality two-dimensional feature, the laser radar modality two-dimensional feature and the mixed modality two-dimensional feature.

[0075] The cross-modal feature fusion system of the present invention processes the original image data of the camera modality by dimensionality lifting and compression mapping, obtains the three-dimensional feature point cloud corresponding to the camera modality, and then projects it to the bird's-eye view domain to obtain the two-dimensional feature representation of the camera modality, then performs feature extraction on the original point cloud data collected by the laser radar, obtains the three-dimensional feature point cloud and the two-dimensional feature representation corresponding to the laser radar modality, then fuses the three-dimensional feature point clouds corresponding to the laser radar modality and the camera modality respectively to obtain the three-dimensional feature point cloud corresponding to the hybrid modality, and converts the three-dimensional feature point cloud corresponding to the hybrid modality into a two-dimensional feature representation consistent with the camera modality, and then uses a preset cross-modal fusion network to perform channel screening, attention weighting and dynamic calibration on the two-dimensional features of the camera, laser radar and hybrid modality, so as to achieve feature selectivity enhancement and suppress redundant information, generate cross-modal fusion features containing rich semantic information and geometric information, improve perception accuracy, robustness and reliability, and provide high-precision input for environmental perception tasks.

[0076] Based on the same inventive concept, according to another aspect of the present invention, Figure 7 As shown, an embodiment of the present invention further provides a computer device 30, which includes a processor 310 and a memory 320. The memory 320 stores a computer program 321 that can be run on the processor. When the processor 310 executes the program, the steps of the above method are performed.

[0077] Based on the same inventive concept, according to another aspect of the present invention, Figure 8 As shown, an embodiment of the present invention further provides a computer-readable storage medium 40, which stores a computer program 410 for executing the above method when executed by a processor.

[0078] Finally, it should be noted that a person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program can be a disk, an optical disk, a read-only storage memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding above-mentioned any method embodiments.

[0079] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0080] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0081] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.

[0082] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A cross-modal feature fusion method, characterized in that: include: Obtain the original image and original point cloud data collected by the vehicle-mounted camera and lidar respectively; Performing dimensionality upscaling processing based on the original image to obtain a first feature point cloud, and performing compression mapping processing based on the first feature point cloud to obtain a camera modality two-dimensional feature; Acquire a second feature point cloud corresponding to the original point cloud data, and obtain a two-dimensional feature of a laser radar modality based on the second feature point cloud; Generating a third feature point cloud according to the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain a mixed-modal two-dimensional feature; Based on the camera modality two-dimensional features, the lidar modality two-dimensional features and the mixed modality two-dimensional features, a cross-modality fusion feature is generated.

2. The cross-modal feature fusion method according to claim 1, characterized in that: The step of generating a cross-modal fusion feature based on the camera modality two-dimensional feature, the laser radar modality two-dimensional feature and the mixed modality two-dimensional feature comprises: Splicing the camera modality two-dimensional feature, the laser radar modality two-dimensional feature, and the mixed modality two-dimensional feature to generate an initial spliced ​​feature; Based on the first channel weights corresponding to all feature channels in the initial splicing feature, the initial splicing feature is optimized to obtain a target splicing feature; The target splicing feature is input into a preset cross-modal fusion network to obtain an output result and use it as the cross-modal fusion feature.

3. The cross-modal feature fusion method according to claim 2, characterized in that: The step of inputting the target splicing features into a preset cross-modal fusion network to obtain an output result includes: Acquire a first feature of the target stitching feature whose modality type is camera modality, use the first feature as a query feature parameter and input it into the preset cross-modal fusion network; Acquire a second feature of the target splicing feature whose modality type is a laser radar modality, and input the second feature as a key feature parameter and a value feature parameter into the preset cross-modal fusion network; Obtaining an attention weight according to the query feature parameter and the key feature parameter; Determining an initial fusion feature based on the attention weight and the value feature parameter; A third feature whose modality type is a mixed modality feature in the target splicing feature is obtained, and the output result is determined based on the initial fusion feature and the third feature.

4. The cross-modal feature fusion method according to claim 3, characterized in that: The step of obtaining a third feature whose modality type is a mixed modality feature in the target splicing feature, and determining the output result based on the initial fusion feature and the third feature, comprises: Evaluate the initial fusion feature based on a preset evaluation function to obtain a first evaluation result, and evaluate the third feature based on a preset evaluation function to obtain a second evaluation result; Determining a target weight coefficient based on the first evaluation result and the second evaluation result; The initial fusion feature and the third feature are weighted and summed based on the target weight coefficient to obtain the output result.

5. The cross-modal feature fusion method according to claim 4, characterized in that: The step of evaluating the initial fusion feature based on a preset evaluation function to obtain a first evaluation result includes: Based on a preset excitation function, generating a second channel weight corresponding to each feature channel in the initial fusion feature, and optimizing the initial fusion feature based on all the second channel weights; Evaluate the optimized initial fusion features based on a preset evaluation function to obtain an initial evaluation result; Determining whether the initial evaluation result satisfies a first preset state; In response to the initial evaluation result not satisfying the first preset state, adjusting the learning parameters to optimize the preset excitation function, and returning to the step of generating a second channel weight corresponding to each feature channel in the initial fusion feature based on the preset excitation function, and optimizing the initial fusion feature based on all the second channel weights; In response to the initial evaluation result satisfying a first preset state, the initial evaluation result is used as the first evaluation result.

6. The cross-modal feature fusion method according to claim 1, characterized in that: The step of performing dimensionality upscaling processing based on the original image to obtain a first feature point cloud includes: Obtaining external parameters and internal parameters of the vehicle-mounted camera; Obtaining a depth estimation result corresponding to each pixel coordinate in the original image; Obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to the preset transformation matrix, the external parameters, the internal parameters and the depth estimation result; Determine a loss result based on each pixel coordinate and its corresponding three-dimensional coordinate; updating the preset transformation matrix according to the loss result, and returning to the step of obtaining the three-dimensional coordinates of each pixel coordinate in the vehicle coordinate system according to the preset transformation matrix, the external parameters, the internal parameters and the depth estimation result, until the loss result satisfies a second preset state; Based on the three-dimensional coordinates of all the pixel coordinates, the first feature point cloud is obtained.

7. The cross-modal feature fusion method according to claim 6, characterized in that: The step of performing compression mapping processing based on the first feature point cloud to obtain the camera modality two-dimensional feature includes: Based on each first data point in the first feature point cloud, obtaining a first feature vector and a depth distribution corresponding to the first data point; Determining a context feature of each of the first data points based on the corresponding first feature vector and the depth distribution; Mapping each of the first data points to a bird's-eye view grid to obtain a first cell corresponding to each of the first data points in the bird's-eye view grid; Based on each of the mapped first cells, aggregating context features corresponding to all first data points therein to obtain a first target feature; The camera modality two-dimensional feature is obtained based on the first target features of all the first cells.

8. The cross-modal feature fusion method according to claim 1, characterized in that: The step of obtaining a second feature point cloud corresponding to the original point cloud data, and obtaining a two-dimensional feature of a laser radar modality based on the second feature point cloud, comprises: Based on a preset feature extraction model, extract local features of the original point cloud data, and aggregate all the local features to obtain global features to obtain the second feature point cloud; Obtain a channel dimension feature corresponding to each feature channel in the second feature point cloud, and obtain a point dimension feature corresponding to each second data point in the second feature point cloud; Based on the preset weight, weighted processing is performed on the channel dimension feature corresponding to each of the feature channels in sequence, and based on the preset weight, weighted processing is performed on the point dimension feature corresponding to each of the second data points in sequence; Based on the weighted channel dimensional features and the weighted point dimensional features, the two-dimensional features of the laser radar modality are obtained.

9. The cross-modal feature fusion method according to claim 1, characterized in that: The step of generating a third feature point cloud according to the first feature point cloud and the second feature point cloud, and performing compression mapping processing based on the third feature point cloud to obtain a mixed-modal two-dimensional feature includes: Searching the first feature point cloud for first data points that respectively match each second data point in the second feature point cloud; Obtaining a second eigenvector corresponding to each of the second data points, and merging the second eigenvector with the first eigenvector of the first data point matched therewith to obtain a third eigenvector; Based on all the third eigenvectors, generating the third feature point cloud; Mapping each third data point in the third feature point cloud to a bird's-eye view grid to obtain a third cell corresponding to each third data point in the bird's-eye view grid; Based on each of the mapped third cells, aggregating third feature vectors corresponding to all third data points therein to obtain a third target feature; Based on the third target features of all the third cells, the mixed modal two-dimensional features are obtained.

10. A cross-modal feature fusion system, characterized in that: include: An acquisition unit configured to acquire original images and original point cloud data respectively collected by the vehicle-mounted camera and the laser radar; A camera modality processing unit, configured to perform dimensionality upscaling processing based on the original image to obtain a first feature point cloud, and perform compression mapping processing based on the first feature point cloud to obtain a camera modality two-dimensional feature; A laser radar modality processing unit configured to obtain a second feature point cloud corresponding to the original point cloud data, and extract a laser radar modality two-dimensional feature based on the second feature point cloud; A mixed mode processing unit, configured to generate a third feature point cloud according to the first feature point cloud and the second feature point cloud, and perform compression mapping processing based on the third feature point cloud to obtain a mixed mode two-dimensional feature; The cross-modal fusion unit is configured to generate a cross-modal fusion feature based on the camera modality two-dimensional feature, the lidar modality two-dimensional feature and the mixed modality two-dimensional feature.

Images