A multimodal fusion method, device and storage medium for autonomous driving

Abstract

The present invention discloses a multimodal fusion method, device and storage medium for autonomous driving, which relates to the technical field of multimodal feature fusion and solves the technical problem that the current multimodal fusion method generates invalid missing modal features in the modality missing scenario. The method includes obtaining the alignment matrix between complete multimodal features through optimal transmission theory, and performing feature fusion on the complete multimodal features to obtain complete fusion features; obtaining missing multimodal features, completing the missing multimodal features according to the alignment matrix to obtain supplementary features; performing feature fusion on the supplementary features to obtain supplementary fusion features; respectively using the complete fusion features and the supplementary fusion features to train a neural network to obtain a multimodal fusion model; inputting an image with missing features into the multimodal fusion model to obtain multimodal fusion features of the image. The present invention can effectively generate missing modal features and increase the applicable scenarios of the model.

Description

Technical Field

[0001] The present invention relates to the field of multimodal feature fusion technology, and in particular to a multimodal fusion method, device, and storage medium for autonomous driving. Background Art

[0002] With the rapid development of autonomous driving technology, the fusion of multimodal sensors (such as lidar, cameras, and millimeter-wave radar) has become a core means of environmental perception. Multimodal data provides complementary information. For example, lidar provides precise three-dimensional spatial structure, while cameras capture rich texture and color information. By fusing these modalities, autonomous driving systems can gain a more comprehensive understanding of their surroundings, thereby improving the accuracy and robustness of perception.

[0003] However, in real-world applications, multimodal data often suffers from missing data in some modalities due to sensor failure, extreme weather conditions (such as rain, snow, and haze), occlusion, or hardware limitations. For example, cameras may fail in low-light conditions, while lidar may be interfered with by rain or snow. This missing modality can severely impact the performance of multimodal fusion models and even cause autonomous driving systems to make erroneous decisions.

[0004] Currently, solutions to the problem of missing modalities are mainly divided into two categories: one is to encode multimodal data into a unified feature space, and when a certain modality is missing, directly use the remaining modalities to complete the perception task. However, this type of method is essentially a passive response to modal missingness, and does not actively fill in the missing information. It is difficult to deal with scenarios where various modalities are missing, thus affecting the effect of subsequent semantic segmentation. The second is to directly synthesize the missing modal data through generative adversarial networks (GANs) or other generative models, and then input it into the multimodal fusion model. However, this type of method usually ignores the deep correlation between modalities, resulting in low quality of the generated missing data, affecting the ability of the multimodal fusion model to deal with missing scenarios, and thus also affecting the effect of subsequent semantic segmentation.

[0005] In the process of implementing the present invention, the inventors discovered that the prior art has at least the following problems:

[0006] In the current multimodal fusion method, the generation of missing modal features is ineffective in the modality-missing scenario, resulting in unsatisfactory multimodal fusion effects. Summary of the Invention

[0007] The purpose of the present invention is to provide a multimodal fusion method, device and storage medium for autonomous driving, so as to solve the technical problem in the prior art that the current multimodal fusion method is invalid in the modal missing scenario, resulting in unsatisfactory multimodal fusion effect.

[0008] The various technical effects that can be produced by the preferred technical solutions among the various technical solutions provided by the present invention are described in detail below.

[0009] To achieve the above objectives, the present invention provides the following technical solutions:

[0010] The present invention provides a multimodal fusion method for autonomous driving, comprising the following steps: obtaining an alignment matrix between complete multimodal features through optimal transmission theory, and performing feature fusion on the complete multimodal features to obtain complete fusion features; obtaining missing multimodal features, and completing the missing multimodal features according to the alignment matrix to obtain supplementary features; performing feature fusion on the supplementary features to obtain supplementary fusion features; training a neural network using the complete fusion features and the supplementary fusion features respectively to obtain a multimodal fusion model; and inputting an image with missing features into the multimodal fusion model to obtain multimodal fusion features of the image.

[0011] Optionally, before obtaining the alignment matrix between complete multimodal features through optimal transmission theory, the method also includes: obtaining the complete multimodal features; obtaining the complete multimodal features includes: extracting lidar voxel features and multi-view image features from lidar point clouds and multi-view images; unifying the lidar voxel features and multi-view image features through projection operations to obtain each modal point feature; obtaining each modal category semantic feature by multiplying the lidar voxel features and multi-view image features with corresponding distribution matrices respectively; wherein the complete multimodal features include each modal point feature and each modal category semantic feature.

[0012] Optionally, the obtaining of the alignment matrix between complete multimodal features through optimal transfer theory includes: aggregating the features of each modal point and the semantic features of each modal category into modal point-category semantic embedding features through an attention mechanism; calculating the true value of the inter-modal alignment matrix of the modal point-category semantic embedding features through optimal transfer theory, and obtaining the predicted value of the alignment matrix through a Transformer decoder; and fitting the true value and the predicted value through mean square error loss to obtain the alignment matrix.

[0013] Optionally, the multimodal features are subjected to feature fusion to obtain complete fused features, including: projecting the modal point features and the modal category semantic features into the same dimension; wherein, the modal point features are added and then projected; the modal point features include lidar point features and multi-view image point features; the modal category semantic features include lidar category semantic features and multi-view image category semantic features; and the projected modal point features and the modal category semantic features are respectively fused using a multi-head cross-attention mechanism and a feedforward neural network to obtain complete fused features.

[0014] Optionally, obtaining the missing multimodal features includes: randomly selecting the multi-view image features for local masking to obtain the remaining view image point features and the remaining view category semantic features; the missing multimodal features include lidar point features, lidar category semantic features, the remaining view image point features and the remaining view category semantic features.

[0015] Optionally, the missing multimodal features are supplemented according to the alignment matrix to obtain supplementary features, including: obtaining missing view image point features through a linear combination of the alignment matrix and the lidar point features, and connecting the remaining view image point features with the missing view image point features to obtain supplementary multi-view image point features; obtaining missing view image category semantic features through a linear combination of the alignment matrix and the lidar category semantic features, and connecting the remaining view category semantic features with the missing view image category semantic features to obtain supplementary multi-view image category semantic features; wherein the supplementary features include the supplemented multi-view image point features and the supplemented multi-view image category semantic features.

[0016] Optionally, the feature fusion of the supplementary features to obtain the supplementary fusion features includes: projecting the lidar point features, lidar category semantic features, the completed multi-view image point features, and the completed multi-view image category semantic features into the same dimension; wherein, the lidar point features and the completed multi-view image point features are added and then projected; and the lidar point features and the completed multi-view image point features, the lidar category semantic features, and the completed multi-view image category semantic features are respectively fused using a multi-head cross-attention mechanism and a feedforward neural network to obtain the supplementary fusion features.

[0017] Optionally, after obtaining the multimodal fusion features of the image, the method further includes: inputting the multimodal fusion features into a segmentation head based on a multi-layer perceptron to obtain a segmentation result of the image.

[0018] A terminal device includes: a complete feature fusion module, which is used to obtain an alignment matrix between complete multimodal features through optimal transmission theory, and perform feature fusion on the complete multimodal features to obtain complete fusion features; a missing feature supplement module, which is used to obtain missing multimodal features, and complete the missing multimodal features according to the alignment matrix to obtain supplementary features; a missing feature fusion module, which is used to perform feature fusion on the supplementary features to obtain supplementary fusion features; a multimodal fusion model training module, which is used to respectively use the complete fusion features and the supplementary fusion features to train a neural network to obtain a multimodal fusion model; and a multimodal fusion module, which is used to input an image with missing features into the multimodal fusion model to obtain multimodal fusion features of the image.

[0019] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the steps of the multimodal fusion method for autonomous driving described above.

[0020] Implementing one of the above technical solutions of the present invention has the following advantages or beneficial effects:

[0021] The present invention obtains the alignment matrix between complete multimodal features through optimal transmission theory, completes the image features of the missing perspective based on the alignment matrix, and finally fuses the multimodal features, thereby enhancing the perceptual performance of the multimodal fusion model in the missing modality scenario, thereby improving the robustness of the model, fully considering the deep correlation between modalities, and can effectively complete the missing modal features, thereby increasing the accuracy of the multimodal fusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work. In the drawings:

[0023] Figure 1 is a flowchart of a multimodal fusion method for autonomous driving according to an embodiment of the present invention;

[0024] Figure 2 This is a flowchart of step S1 of the multimodal fusion method for autonomous driving according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, in which various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods and devices that are consistent with some aspects of the present disclosure as detailed in the appended claims, and other embodiments may also be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and essence of the present invention.

[0026] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", etc. indicate the orientation or position relationship based on the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the elements referred to must have a specific orientation, be constructed and operate in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "plurality" means two or more. The terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0027] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below, in which only the parts related to the embodiment of the present invention are shown.

[0028] Example 1:

[0029] like Figure 1 As shown, the present invention provides a multimodal fusion method for autonomous driving, comprising the following steps: S1, obtaining an alignment matrix between complete multimodal features through optimal transmission theory, and performing feature fusion on the multimodal features to obtain complete fusion features; S2, obtaining missing multimodal features, and completing the missing multimodal features according to the alignment matrix to obtain supplementary features; S3, performing feature fusion on the supplementary features to obtain supplementary fusion features; S4, respectively using the complete fusion features and the supplementary fusion features to train a neural network to obtain a multimodal fusion model; S5, inputting an image with missing features into the multimodal fusion model to obtain multimodal fusion features of the image.

[0030] This embodiment obtains the alignment matrix between complete multimodal features through optimal transmission theory, completes the image features of the missing perspective based on the alignment matrix, and finally fuses the multimodal features, thereby enhancing the perceptual performance of the multimodal fusion model in the missing modality scenario, thereby improving the robustness of the model, fully considering the deep correlation between modalities, and can effectively complete the missing modal features, increasing the accuracy of the multimodal fusion model.

[0031] Next, combine Figure 1 and Figure 2 The specific implementation steps of the multimodal fusion method for autonomous driving provided by this embodiment are described in detail:

[0032] First, execute step S1 and use optimal transport theory to obtain the alignment matrix between the complete multimodal features. Then, perform feature fusion on the multimodal features to obtain the complete fused features. Optimal transport (OT) is a problem that involves finding an optimal solution. Its goal is to determine how to transform one probability distribution into another with minimal cost. Optimal transport theory can be used to mine the correlation information between the complete multimodal features, effectively completing the missing modal features.

[0033] Furthermore, before step S1, the method further includes: obtaining complete multimodal features; specifically, obtaining complete multimodal features includes:

[0034] Extract LiDAR voxel features and multi-view image features from LiDAR point clouds and multi-view images. LiDAR point clouds are surface point cloud data obtained using LiDAR scanning. LiDAR voxel features can be derived from LiDAR point clouds. LiDAR emits a laser beam and detects the return, determining the 3D position and posture of the target by measuring the distance and angle between the laser beam and the target. Multi-view images are images of the same object or scene captured by a camera from different perspectives.

[0035] Projection is used to unify the LiDAR voxel features and multi-view image features to obtain the point features of each modality. These features are then projected onto the midpoint of the LiDAR point cloud to obtain the point features of each modality. Projection can be performed using either linear or nonlinear methods, depending on the data characteristics and task requirements. This effectively unifies the features of different modalities, facilitating subsequent feature analysis and processing.

[0036] The semantic features of each modality category are obtained by multiplying the lidar voxel features and multi-view image features with the corresponding distribution matrices. The multi-modal features include the features of each modality point and the semantic features of each modality category. For the lidar modality, the lidar category semantic features are obtained by multiplying the lidar voxel features with the corresponding distribution matrix. The distribution matrix is ​​obtained by applying a multi-layer perceptron-based segmentation head to the lidar voxel features. For the camera modality, the multi-view image category semantic features are obtained by multiplying the multi-view image features with the corresponding distribution matrix. The distribution matrix is ​​obtained by applying a multi-layer perceptron-based segmentation head to the multi-view image features.

[0037] Further, if Figure 2 As shown, the optimal transfer theory is used to obtain a complete multimodal feature alignment matrix, including: S11, aggregating the modal point features and the modal category semantic features into a modal point-category semantic embedding feature through an attention mechanism; S12, calculating the true value of the modal point-category semantic embedding intermodal alignment matrix through optimal transfer theory, and obtaining the predicted value of the alignment matrix through a Transformer decoder. The Transformer is a deep learning model architecture used for natural language processing (NLP) and other sequence-to-sequence tasks. The Transformer architecture incorporates a self-attention mechanism, which makes it excellent at processing sequential data. In the Transformer model, the decoder is the component responsible for generating the target sequence. It receives the encoder output as context information and gradually generates the output sequence. S13, fitting the true and predicted values ​​using a mean squared error loss to obtain the alignment matrix. This effectively aligns features from different modalities, thereby obtaining correlation information between them. At the same time, by using the Transformer decoder with a self-attention mechanism, the true value and predicted value of the alignment matrix can be calculated more accurately, resulting in more accurate alignment results.

[0038] Furthermore, feature fusion is performed on the complete multimodal features to obtain a complete fused feature. This involves projecting the point features of each modality and the semantic features of each modality into the same dimension. The point features of each modality are summed and then projected to increase feature richness and information content. The point features of each modality include lidar point features and multi-view image point features. The semantic features of each modality include lidar category semantic features and multi-view image category semantic features. Lidar point features are features extracted from point cloud data that describe object shape, structure, and other information, such as point normals, curvature, and density. Lidar category semantic features are obtained through tasks such as semantic segmentation or object detection. These features can help the system identify different objects in the point cloud, such as roads, trees, buildings, and vehicles, and provide useful information for subsequent tasks. Multi-view image point features are key points or feature points in the image that are stable across different viewpoints. These feature points can be detected and described using feature extraction algorithms (such as SIFT and SURF). Similar to lidar category semantic features, multi-view image category semantic features are also features related to the category or semantics of an object. These features can be acquired through tasks such as image classification and object detection. A multi-head cross-attention mechanism and a feedforward neural network are used to fuse the projected point features of each modality and the category semantic features of each modality to obtain a complete fused feature.

[0039] Then, step S2 is performed to obtain the missing multimodal features, and the missing multimodal features are supplemented according to the alignment matrix to obtain supplementary features. Using the alignment matrix can effectively supplement the missing multimodal features and provide accurate feature information for multimodal fusion.

[0040] Furthermore, missing multimodal features are obtained by randomly selecting multi-view image features for local masking to obtain residual view image point features and residual view category semantic features. By locally masking multi-view image features, we can simulate real-world missing scenes and improve the generalization of the model. Missing multimodal features include lidar point features, lidar category semantic features, residual view image point features, and residual view category semantic features.

[0041] Furthermore, the missing multimodal features are supplemented based on the alignment matrix to obtain supplementary features. This includes: obtaining missing viewpoint image point features by linearly combining the alignment matrix with the lidar point features, and concatenating the remaining viewpoint image point features with the missing viewpoint image point features to obtain supplementary multi-view image point features; obtaining missing viewpoint image category semantic features by linearly combining the alignment matrix with the lidar category semantic features, and concatenating the remaining viewpoint category semantic features with the missing viewpoint image category semantic features to obtain supplementary multi-view image category semantic features. The supplementary features include supplementary multi-view image point features and supplementary multi-view image category semantic features. By linearly combining the alignment matrix with the lidar point features and the lidar category semantic features, respectively, the missing viewpoint image point features and image category semantic features are obtained. These features are concatenated with the remaining viewpoint image point features to complete the point features and category semantic features of the multi-data image. This supplementation method can effectively improve the completeness and accuracy of multi-view images.

[0042] Next, step S3 is executed to fuse the supplementary features to obtain supplementary fusion features. The supplementary fusion features take into account the correlation information between the modalities, and are effectively supplemented, solving the problem of insufficient features in missing scenarios.

[0043] Furthermore, step S3 includes projecting the lidar point features, lidar category semantic features, supplemented multi-view image point features, and supplemented multi-view image category semantic features into the same dimension; wherein the lidar point features and supplemented multi-view image point features are added before projection; and the lidar point features and supplemented multi-view image point features, lidar category semantic features, and supplemented multi-view image category semantic features are fused using a multi-head cross-attention mechanism and a feedforward neural network, respectively, to obtain supplemented fused features. The use of a multi-head cross-attention mechanism and a feedforward neural network enables effective interaction between multimodal features. The cross-attention mechanism allows different modal features to pay attention to and influence each other during the completion process, while the multi-head attention mechanism divides different modal features into multiple independent parts, allowing different parts to focus more on a specific modal feature during the completion process. Simultaneously, the feedforward neural network integrates and processes feature information from each modality. Through a series of neural network layers, the input feature information is converted into a more abstract and effective feature representation, thereby enhancing feature interaction. By combining the multi-head cross-attention mechanism and the feedforward neural network, the interaction and information integration between different modal features can be considered more comprehensively, so as to more accurately obtain the completed multimodal fusion features and improve the completeness and accuracy of the features.

[0044] Next, step S4 is performed, and the neural network is trained using the complete fusion features and the supplemented fusion features, respectively, to obtain a multimodal fusion model. Using the complete fusion features to train the model allows the model to obtain more features of the original data, thereby better learning the essential characteristics and patterns of the data. This helps improve the generalization and robustness of the model and reduces the risk of overfitting. Using the supplemented fusion features to train the model enables the model to obtain a more comprehensive feature representation. This helps improve the accuracy and performance of the model, especially when features are missing, so that it can better adapt to feature-missing scenarios.

[0045] Finally, step S5 is executed to input the feature-missing image into the multimodal fusion model to obtain the multimodal fusion features of the image. Through the trained multimodal fusion model, even in scenes with missing features, multimodal fusion features with complete information can be obtained.

[0046] As an optional implementation, after step S5, the method further includes: inputting the multimodal fusion features into a segmentation head based on a multilayer perceptron to obtain image segmentation results. By exploiting the correlation information between the lidar and camera modalities through optimal transmission theory, the missing modal features are effectively complemented. Furthermore, the complemented multimodal features are effectively fused using an attention mechanism. This solves the problem of severely degraded segmentation performance in multimodal fusion models when the modality is missing, significantly improving semantic segmentation performance.

[0047] This embodiment effectively supplements missing modal features by learning inter-modal alignment information. In addition, the inter-modal alignment matrix is ​​calculated by aggregating modal point features and category semantic features to achieve a dimensionality reduction effect, greatly reducing the amount of calculation of the alignment matrix. The category semantic features and point features are fused through a multi-head cross-attention mechanism. The fusion process is simple and effective, further enhancing the robustness of the multimodal fusion model in the missing modality scenario.

[0048] The embodiment is only a special example and does not represent only one way of implementing the present invention.

[0049] Example 2:

[0050] Based on the same inventive concept, the second embodiment of the present invention also provides a terminal device, including: a complete feature fusion module, which is used to obtain the alignment matrix between complete multimodal features through optimal transmission theory, and perform feature fusion on the complete multimodal features to obtain complete fusion features; a missing feature supplement module, which is used to obtain missing multimodal features, and complete the missing multimodal features according to the alignment matrix to obtain supplementary features; a missing feature fusion module, which is used to perform feature fusion on the supplementary features to obtain supplementary fusion features; a multimodal fusion model training module, which is used to train the neural network using complete fusion features and supplementary fusion features respectively to obtain a multimodal fusion model; a multimodal fusion module, which is used to input an image with missing features into the multimodal fusion model to obtain the multimodal fusion features of the image.

[0051] The device of this embodiment is used to execute the multimodal fusion method for autonomous driving provided in the embodiment, which can comprehensively consider the interaction and information integration between different modal features, thereby more accurately obtaining the missing features of the completed multimodal fusion and improving the completeness and accuracy of the multimodal fusion features.

[0052] Example 3:

[0053] Based on the same inventive concept, the third embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps of a multimodal fusion method for autonomous driving as described in any one of the first embodiments.

[0054] The foregoing is merely a preferred embodiment of the present invention. Those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the guidance of the present invention, these features and embodiments may be modified to suit specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be within the scope of the present invention.

Claims

1. A multimodal fusion method for autonomous driving, characterized in that: The following steps are involved: Obtain complete multimodal features; Obtaining an alignment matrix between the complete multimodal features through optimal transmission theory, and performing feature fusion on the complete multimodal features to obtain a complete fusion feature; Obtaining missing multimodal features, and completing the missing multimodal features according to the alignment matrix to obtain supplementary features; Performing feature fusion on the supplementary features to obtain supplementary fusion features; The complete fusion feature and the supplementary fusion feature are respectively used to train a neural network to obtain a multimodal fusion model; Inputting the feature-missing image into the multimodal fusion model to obtain the multimodal fusion features of the image; Acquiring the complete multimodal features includes: Extract lidar voxel features and multi-view image features from lidar point clouds and multi-view images; Unifying the laser radar voxel features and the multi-view image features through a projection operation to obtain features of each modal point; The semantic features of each modality category are obtained by multiplying the laser radar voxel features and the multi-view image features with the corresponding distribution matrices respectively; wherein the complete multimodal features include the features of each modality point and the semantic features of each modality category; The obtaining of the alignment matrix between the complete multimodal features by using the optimal transmission theory includes: Aggregating the modal point features and the modal category semantic features into modal point-category semantic embedding features through an attention mechanism; Calculating the true value of the modality point-category semantic embedding feature inter-modality alignment matrix using optimal transmission theory, and obtaining the predicted value of the alignment matrix through a Transformer decoder; Fitting the true value and the predicted value by using mean square error loss to obtain the alignment matrix; The obtaining of missing multimodal features includes: Randomly selecting the multi-view image features for local masking to obtain remaining view image point features and remaining view category semantic features; the missing multimodal features include lidar point features, lidar category semantic features, the remaining view image point features, and the remaining view category semantic features; The step of completing the missing multimodal features according to the alignment matrix to obtain supplementary features includes: Obtaining missing viewpoint image point features through a linear combination of the alignment matrix and the lidar point features, and concatenating the remaining viewpoint image point features with the missing viewpoint image point features to obtain completed multi-viewpoint image point features; The missing view image category semantic feature is obtained by linearly combining the alignment matrix and the lidar category semantic feature, and the remaining view category semantic feature is connected with the missing view image category semantic feature to obtain the completed multi-view image category semantic feature; wherein the supplementary feature includes the completed multi-view image point feature and the completed multi-view image category semantic feature.

2. A multimodal fusion method for autonomous driving according to claim 1, characterized in that: The performing feature fusion on the complete multimodal features to obtain complete fusion features includes: Projecting the modal point features and the modal category semantic features into the same dimension; wherein the modal point features are added and then projected; the modal point features include lidar point features and multi-view image point features; and the modal category semantic features include lidar category semantic features and multi-view image category semantic features; A multi-head cross-attention mechanism and a feedforward neural network are used to fuse the projected modal point features and the semantic features of each modal category to obtain complete fusion features.

3. The multimodal fusion method for autonomous driving according to claim 1, characterized in that: The performing feature fusion on the supplementary features to obtain supplementary fusion features includes: Projecting the lidar point features, the lidar category semantic features, the supplemented multi-view image point features, and the supplemented multi-view image category semantic features into the same dimension; wherein the lidar point features and the supplemented multi-view image point features are added together before the projection; A multi-head cross-attention mechanism and a feedforward neural network are used to fuse the lidar point features and the completed multi-view image point features, and the lidar category semantic features and the completed multi-view image category semantic features to obtain supplementary fusion features.

4. The multimodal fusion method for autonomous driving according to claim 1, characterized in that: After obtaining the multimodal fusion features of the image, the method further includes: The multimodal fusion features are input into a segmentation head based on a multi-layer perceptron to obtain a segmentation result of the image.

5. A multimodal fusion device for autonomous driving, characterized in that: include: A complete feature fusion module is used to obtain an alignment matrix between complete multimodal features through optimal transmission theory, and perform feature fusion on the complete multimodal features to obtain a complete fused feature; A missing feature supplementation module is used to obtain missing multimodal features and to supplement the missing multimodal features according to the alignment matrix to obtain supplementary features; A missing feature fusion module is used to fuse the supplementary features to obtain supplementary fusion features; A multimodal fusion model training module, configured to respectively train a neural network using the complete fusion feature and the supplementary fusion feature to obtain a multimodal fusion model; A multimodal fusion module, configured to input a feature-missing image into the multimodal fusion model to obtain multimodal fusion features of the image; Acquiring the complete multimodal features includes: Extract lidar voxel features and multi-view image features from lidar point clouds and multi-view images; Unifying the laser radar voxel features and the multi-view image features through a projection operation to obtain features of each modal point; The semantic features of each modality category are obtained by multiplying the laser radar voxel features and the multi-view image features with the corresponding distribution matrices respectively; wherein the complete multimodal features include the features of each modality point and the semantic features of each modality category; The method of obtaining the alignment matrix between the complete multimodal features through the optimal transmission theory includes: Aggregating the modal point features and the modal category semantic features into modal point-category semantic embedding features through an attention mechanism; Calculating the true value of the modality point-category semantic embedding feature inter-modality alignment matrix using optimal transmission theory, and obtaining the predicted value of the alignment matrix through a Transformer decoder; Fitting the true value and the predicted value by using mean square error loss to obtain the alignment matrix; The obtaining of missing multimodal features includes: Randomly selecting the multi-view image features for local masking to obtain remaining view image point features and remaining view category semantic features; the missing multimodal features include lidar point features, lidar category semantic features, the remaining view image point features, and the remaining view category semantic features; The step of completing the missing multimodal features according to the alignment matrix to obtain supplementary features includes: Obtaining missing viewpoint image point features through a linear combination of the alignment matrix and the lidar point features, and concatenating the remaining viewpoint image point features with the missing viewpoint image point features to obtain completed multi-viewpoint image point features; The missing view image category semantic feature is obtained by linearly combining the alignment matrix and the lidar category semantic feature, and the remaining view category semantic feature is connected with the missing view image category semantic feature to obtain the completed multi-view image category semantic feature; wherein the supplementary feature includes the completed multi-view image point feature and the completed multi-view image category semantic feature.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a multimodal fusion method for autonomous driving as described in any one of claims 1 to 4 are implemented.

Images