Multi-modal data fusion real-time semantic segmentation and target detection joint model

Abstract

The invention relates to the field of multi-modal computer vision, in particular to a real-time semantic segmentation and target detection joint model for multi-modal data fusion. The model is realized through the following technical means: designing a multi-branch network based on an RGB image, depth information and laser radar data, and respectively extracting different modal features; a dynamic modal selection module is introduced, an optimal modal combination is automatically selected according to environmental conditions, and the adaptive ability of the model in a complex scene is improved; realizing cross-modal information complementation by using a UAFM feature fusion module; multi-scale features are fused step by step through a step-by-step decoding structure so as to balance detail and global semantic information; a joint training mechanism is adopted, a semantic segmentation result and a target detection frame are synchronously output, and calculation redundancy is reduced. According to the method, the segmentation and detection precision in a complex scene is remarkably improved under the support of multi-modal data; through dynamic modal selection and feature sharing, efficient real-time processing is realized, and the reasoning speed is improved by more than 30% compared with that of an independent model.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal computer vision and real-time image processing, and specifically relates to a real-time semantic segmentation and target detection joint model based on multimodal data fusion and dynamic modality selection mechanism. Background Art

[0002] In areas such as autonomous driving and robotic navigation, real-time scene understanding relies on two core tasks: semantic segmentation and object detection. Semantic segmentation achieves refined scene analysis through pixel-level classification, while object detection locates and identifies key objects in the scene. However, existing technologies have the following limitations:

[0003] The limitations of single-modality data: Traditional methods typically process tasks based on a single modality (such as RGB images). In complex environments (such as low light, occlusion, and inclement weather), single-modality data struggles to provide sufficient robustness. For example, RGB images are prone to failure at night or in strong sunlight, while LiDAR or depth information can compensate for these shortcomings. However, existing models fail to effectively integrate the complementary advantages of multimodal data.

[0004] Redundancy of Independent Models: Existing technologies often implement semantic segmentation and object detection using independent models, resulting in redundant computing resources and difficulty meeting real-time requirements. For example, autonomous driving systems must run both segmentation and detection models simultaneously, increasing hardware load and latency.

[0005] Inefficiency of multimodal fusion: Previous studies have attempted to combine multimodal data (e.g., RGB+LiDAR), but most feature fusion methods rely on simple concatenation or weighted summation, failing to fully exploit the spatial and semantic correlations of cross-modal information. For example, the lack of coordinated optimization of multimodal feature channels and spatial consistency leads to redundant or lost fused feature information.

[0006] The conflict between real-time performance and accuracy: Existing real-time models (such as lightweight segmentation networks) improve speed but sacrifice multi-scale details or global semantic information. For example, some models suffer from poor segmentation of small objects due to excessive downsampling, or lack a hierarchical decoding structure to balance details and semantics.

[0007] Lack of joint training: Existing joint models often use a phased training strategy, resulting in insufficient inter-task parameter optimization. For example, the loss functions for semantic segmentation and object detection are not co-designed, making it difficult to achieve a balance between feature sharing and computational efficiency.

[0008] With the rapid development of deep learning technology, convolutional neural networks have been widely used in image segmentation, replacing traditional manual feature extraction methods. Since the introduction of fully convolutional networks (FCNs), a series of novel networks have been proposed, such as DeepLabv3+, a semantic segmentation model based on RGB images. Although highly accurate, it does not integrate multimodal data and lacks real-time performance. There is also Faster R-CNN, a classic object detection framework, which operates independently of the segmentation task and cannot achieve end-to-end joint reasoning. And PointPainting projects LiDAR point clouds onto RGB images for fusion, but the fusion process relies on manual design and lacks an adaptive feature interaction mechanism. It is clear that existing methods have significant shortcomings in terms of multimodal data utilization, joint task optimization, and the real-time-accuracy balance. They are unable to meet the requirements of scenarios such as autonomous driving for efficient, robust, and refined scene understanding. Summary of the Invention

[0009] In response to the above technical problems, the present invention proposes a real-time semantic segmentation and target detection joint model with multimodal data fusion, which achieves efficient and accurate scene understanding through a multi-branch structure, a dynamic modality selection mechanism, cross-modal feature fusion and step-by-step decoding technology.

[0010] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0011] A real-time semantic segmentation and object detection joint model for multimodal data fusion, including:

[0012] A feature extraction module is used to extract corresponding features of images of different modalities in a parallel manner;

[0013] Dynamic modality selection module, used to generate corresponding modality weights based on the features of different modalities;

[0014] UAFM feature fusion module, used to perform cross-modal fusion of different modal features based on modal weights to generate fused features;

[0015] The step-by-step decoding module is used to perform multi-scale feature fusion on the fused features and gradually restore detail information and global semantic information;

[0016] The joint output module is used to generate and output semantic segmentation results and object detection results based on detail information and global semantic information;

[0017] The joint optimization module is used to optimize the shared parameters of semantic segmentation and object detection tasks through a joint training mechanism.

[0018] The feature extraction module includes:

[0019] The first branch network is used to extract the semantic features of RGB images through the residual network;

[0020] The second branch network is used to extract the structural features of the depth map through a lightweight convolutional layer;

[0021] The third branch network is used to voxelize the lidar point cloud and extract its spatial geometric features through three-dimensional sparse convolution.

[0022] The dynamic mode selection module includes:

[0023] The modality importance score calculation module is used to perform global average pooling on the features of each modality to generate the modality importance score;

[0024] Normalization module, used to normalize the modality importance score to modality weight through the Softmax function;

[0025] The weighted fusion module is used to perform weighted fusion of multimodal features according to modal weights to generate cross-modal complementary features.

[0026] The UAFM feature fusion module includes:

[0027] The channel attention module is used to perform channel attention weighting on cross-modal complementary features of different modalities to generate inter-modal weights;

[0028] The spatial attention module is used to perform spatial attention weighting on the cross-modal complementary features after channel attention weighting;

[0029] The convolution compression module is used to perform channel splicing and convolution compression on the cross-modal complementary features after spatial attention weighting to generate fusion features.

[0030] The step-by-step decoding module includes:

[0031] Multi-level upsampling module, used to perform multi-level upsampling of fusion features;

[0032] The skip connection module is used to perform a skip connection between the current upsampled features and the low-level features of the corresponding modality encoding stage at each level of decoding;

[0033] The convolution operation module is used to fuse the features of the jump connection through convolution operations to generate decoding features containing multi-scale information.

[0034] The combined output module includes:

[0035] The semantic segmentation head consists of a sequentially connected convolution, activation function, and bilinear interpolation layer. It processes the decoded features of multi-scale information and outputs pixel-level classification results.

[0036] The object detection head consists of a sequentially connected region proposal network and a classification and regression module, which is used to generate candidate boxes based on the decoded features of multi-scale information and perform category prediction and bounding box correction on the candidate boxes.

[0037] The joint optimization module is used to construct a multi-task loss function and optimize the shared parameters of the semantic segmentation and object detection tasks through the loss function, wherein the multi-task loss function includes:

[0038] Segmentation loss: including pixel-level cross entropy loss;

[0039] Detection loss: including smooth L1 loss and focal loss;

[0040] Edge-assisted loss: This involves predicting boundaries through the edge detection head and calculating the binary cross entropy loss with the labels generated by the Canny operator.

[0041] A real-time semantic segmentation and target detection method based on multimodal data fusion includes the following steps:

[0042] Obtain RGB images, depth information, and lidar point clouds to build a dataset;

[0043] Build and train a real-time semantic segmentation and object detection joint model that fused multimodal data;

[0044] Save the trained model weights;

[0045] Use the trained model for semantic segmentation and object detection, and evaluate the model based on the segmentation and detection results.

[0046] The present invention has the following beneficial effects and advantages:

[0047] 1. Through multimodal data fusion and dynamic modality selection mechanism, the segmentation and detection accuracy in complex scenarios (such as low light, occlusion, and rainy days) is significantly improved;

[0048] 2. The joint model design reduces computational redundancy and increases inference speed by more than 30% compared to independent models;

[0049] 3. The hierarchical decoding structure and edge-assisted loss effectively balance details and semantic information, improving the segmentation accuracy of small objects by 15%;

[0050] 4. The dynamic mode selection mechanism enhances the model's adaptability and can automatically select the optimal mode combination based on environmental conditions, further improving robustness.

[0051] 5. Suitable for scenarios with strict real-time requirements such as autonomous driving and robot navigation, with high robustness and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The overall flow chart for constructing a joint model of real-time semantic segmentation and object detection for multimodal data fusion according to the present invention;

[0053] Figure 2 This is the overall network structure diagram of the real-time semantic segmentation and target detection joint model of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the art to which the present invention pertains. The terms used in the specification of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0056] This paper focuses on a real-time joint semantic segmentation and object detection model that integrates multimodal data. This method combines multimodal data, including RGB images, depth information, and LiDAR point clouds, to design a joint model capable of simultaneously performing semantic segmentation and object detection tasks. This model achieves efficient and accurate scene understanding through modules such as a multi-branch network, a dynamic modality selection module, a cross-modal feature fusion module, a step-by-step decoding architecture, a joint output module, and a joint training mechanism.

[0057] like Figure 1 and Figure 2 As shown in the figure, a real-time semantic segmentation and object detection joint model with multimodal data fusion achieves efficient and accurate scene understanding through a multi-branch structure, dynamic modality selection mechanism, cross-modal feature fusion and step-by-step decoding technology. The model includes:

[0058] Multimodal data input and feature extraction: Input data includes RGB images, depth information (Depth Map), and LiDAR point clouds. Features from each modality are extracted using a multi-branch network, including RGB, depth, and LiDAR branches. The RGB branch uses a residual network (ResNet) to extract image semantic features; the depth branch uses lightweight convolutional layers to extract structural features of the depth map; and the LiDAR branch voxelizes the point cloud and uses 3D sparse convolution to extract spatial geometric features.

[0059] Dynamic Modal Selection Mechanism: A dynamic modal selection module is introduced to automatically select the optimal modal combination based on environmental conditions (such as lighting, occlusion, and weather). This module performs global average pooling on the features of each modality to generate a modal importance score, which is then normalized into modal weights using the Softmax function. Multimodal features are weighted and fused based on the modal weights to generate cross-modal complementary features, significantly improving the model's adaptability in complex scenarios.

[0060] Cross-modal Feature Fusion Module: UAFM (Unified Attention-based Fusion Module) is used to achieve multimodal feature fusion. First, channel-wise attention is used to calculate the channel weights of each modal feature to suppress redundant information. Second, spatial attention is used to enhance the spatial alignment of cross-modal features. Finally, the weighted features are channel-wise concatenated and convolutionally compressed to generate cross-modal complementary features.

[0061] A step-by-step decoding architecture: The fused features are upsampled at multiple levels, gradually restoring resolution. At each decoding stage, the currently upsampled features are skip-connected to the lower-level features from the corresponding encoding stage. These skip-connected features are then fused via convolution operations to generate decoded features containing multi-scale information. This architecture effectively balances details (boundaries, textures) with global semantics (object categories, scene layout).

[0062] Joint output module: This module includes a semantic segmentation head and an object detection head. The semantic segmentation head, which consists of convolution, activation functions, and bilinear interpolation, outputs pixel-level classification results. The object detection head, based on a modified CenterNet architecture, simultaneously generates detection bounding boxes and class probabilities. Both heads share decoder features, reducing computational redundancy and improving inference efficiency.

[0063] Multi-task joint training mechanism: A multi-task loss function is designed, including semantic segmentation loss, object detection loss, and edge-assisted loss. The segmentation loss is a pixel-level cross-entropy loss, which is used to optimize semantic classification; the detection loss is a weighted sum of the smoothed L1 loss (box regression) and the focal loss (class imbalance optimization); and the edge-assisted loss uses the edge detection head to predict boundaries and calculates a binary cross-entropy loss with the labels generated by the Canny operator to enhance the extraction of detailed information.

[0064] A real-time semantic segmentation and object detection joint method based on multimodal data fusion includes the following steps:

[0065] Acquire multimodal input data, including RGB images, depth information, and lidar point clouds;

[0066] Extracting features of the RGB image, depth information, and lidar point cloud respectively through a multi-branch network;

[0067] Using the dynamic mode selection module, the optimal mode combination is automatically selected according to environmental conditions and the mode weight is generated;

[0068] Using a feature fusion module to perform cross-modal fusion on the features extracted by the multi-branch network to generate fused features;

[0069] The fusion features are subjected to multi-scale feature fusion through a step-by-step decoding structure, and detail information and global semantic information are gradually restored;

[0070] Based on the output of the step-by-step decoding structure, semantic segmentation results and target detection results are generated synchronously;

[0071] The shared parameters of semantic segmentation and object detection tasks are optimized through a joint training mechanism to reduce computational redundancy.

[0072] The dynamic mode selection module includes the following steps:

[0073] Perform global average pooling on the features of each modality to generate a modality importance score;

[0074] Normalize the modality importance score to modality weight through the Softmax function;

[0075] Multimodal features are weightedly fused according to modal weights to generate cross-modal complementary features.

[0076] The multi-branch network includes:

[0077] The first branch network is used to process RGB images and contains three convolutional layers and two residual modules;

[0078] The second branch network is used to process depth information and consists of three convolutional layers and two residual modules;

[0079] The third branch network is used to process lidar point clouds and includes a point cloud voxelization module and four 3D convolutional layers.

[0080] The feature fusion module adopts UAFM (Unified Attention-based Feature Fusion Module), which specifically includes the following steps:

[0081] Perform channel attention weighting on the features of different modalities to generate inter-modality weights;

[0082] Enhancing spatial consistency of cross-modal features through spatial attention mechanism;

[0083] The weighted features are channel-concatenated and convolutionally compressed to generate fused features.

[0084] The step-by-step decoding structure includes the following steps:

[0085] Perform multi-level upsampling on the fused features to gradually restore the feature resolution;

[0086] At each level of decoding, the current upsampled features are skipped and connected to the low-level features of the corresponding encoding stage;

[0087] The features of the skip connection are fused through convolution operations to generate decoding features containing multi-scale information.

[0088] The semantic segmentation result is outputted by a segmentation head, which includes: at least one convolutional layer, an activation function, and a bilinear interpolation layer; the target detection result is outputted by a detection head, which includes:

[0089] Region Proposal Network (RPN), used to generate candidate boxes;

[0090] The classification and regression module is used to predict the category of the candidate box and correct the bounding box.

[0091] The joint training mechanism includes the following steps:

[0092] Design a multi-task loss function, including semantic segmentation loss, object detection loss, and edge auxiliary loss;

[0093] The semantic segmentation loss is a cross entropy loss, which is used to optimize pixel-level classification;

[0094] The target detection loss is a weighted sum of smooth L1 loss and cross entropy loss, which is used to optimize the detection box position and category;

[0095] The edge-assisted loss generates an edge prediction map through the edge detection head and calculates the binary cross entropy loss with the true edge label to enhance the extraction of detail information.

[0096] The true edge labels are generated by applying the Canny operator to the semantic segmentation labels.

[0097] The processing of the lidar point cloud includes:

[0098] Convert raw point cloud data into voxelized representation;

[0099] Extract spatial features of point clouds through 3D sparse convolution;

[0100] Project the 3D features onto a 2D plane and align them with the RGB image and depth information.

[0101] The model achieves real-time performance during the inference phase by following these steps:

[0102] Parallel preprocessing of multimodal input data;

[0103] Use lightweight convolution design to reduce the computational complexity of branch networks;

[0104] Hardware acceleration technology is used to optimize the feature fusion and decoding process.

[0105] Example

[0106] A real-time semantic segmentation and object detection joint model for multimodal data fusion includes:

[0107] Dataset building blocks:

[0108] Obtain multimodal input data, including RGB images, depth information (Depth Map) and LiDAR point cloud (LiDAR), and divide the dataset into training set and validation set to construct a dataset that can be used for training and evaluation.

[0109] Real-time semantic segmentation and target detection model building modules:

[0110] Build a real-time semantic segmentation and object detection joint model that strikes a good balance between speed and accuracy. The model is implemented through the following modules:

[0111] Multimodal data input and feature extraction:

[0112] RGB branch: Use the residual network (ResNet) to extract the semantic features of RGB images.

[0113] Depth branch: extracts the structural features of the depth map through lightweight convolutional layers.

[0114] LiDAR branch: After voxelizing the lidar point cloud, three-dimensional sparse convolution is used to extract spatial geometric features.

[0115] Dynamic modal selection module:

[0116] Modal weight calculation: Perform global average pooling on the features of each modality to generate a modal importance score, which is normalized into a modal weight using the Softmax function.

[0117] Weighted fusion: Multimodal features are weightedly fused according to modal weights to generate cross-modal complementary features.

[0118] Cross-modal feature fusion module (UAFM):

[0119] Channel attention: Calculate the channel weights of each modal feature to suppress redundant information.

[0120] Spatial Attention: Enhancing Spatial Alignment of Cross-Modal Features.

[0121] Convolution compression: The weighted features are channel-concatenated and convolutionally compressed to generate cross-modal complementary features.

[0122] Level-by-level decoding structure:

[0123] Multi-level upsampling: progressively restoring feature resolution.

[0124] Skip connection: At each level of decoding, the current upsampled features are skipped and connected to the low-level features of the corresponding encoding stage.

[0125] Convolution operation: Fusion of skip connection features to generate decoding features containing multi-scale information.

[0126] Combined output module:

[0127] Semantic segmentation head: Consists of convolution, activation function and bilinear interpolation, and outputs pixel-level classification results. Object detection head: Based on the improved CenterNet architecture, it simultaneously generates detection boxes and category probabilities.

[0128] Joint training mechanism:

[0129] Multi-task loss function: includes semantic segmentation loss, object detection loss and edge auxiliary loss.

[0130] Segmentation loss: pixel-level cross entropy loss, optimized for semantic classification.

[0131] Detection loss: smooth L1 loss (box regression) and focal loss (class imbalance optimization).

[0132] Edge-assisted loss: The edge detection head predicts the boundary and calculates the binary cross entropy loss with the label generated by the Canny operator to enhance detail preservation.

[0133] Detailed steps for training and evaluating the joint model module:

[0134] S1: Build the dataset:

[0135] S2: Build a real-time semantic segmentation and target detection joint model: mainly including the construction of a multi-branch network, a dynamic modality selection module, a cross-modal feature fusion module, a step-by-step decoding structure and a joint output module:

[0136] S3: Train the constructed model:

[0137] S4: Save the weights of the trained model:

[0138] S5: Evaluate the model effect:

[0139] The real-time semantic segmentation and target detection joint model of multimodal data fusion described in the present invention constructs a real-time semantic segmentation and target detection joint model of multimodal data fusion through a dataset construction module, a real-time semantic segmentation and target detection model construction module, and a training and evaluation joint model module. It can segment complex scene information in real time and perform target detection simultaneously.

[0140] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A real-time semantic segmentation and object detection joint model for multimodal data fusion, characterized by: include: A feature extraction module is used to extract corresponding features of images of different modalities in a parallel manner; Dynamic modality selection module, used to generate corresponding modality weights based on the features of different modalities; UAFM feature fusion module, used to perform cross-modal fusion of different modal features based on modal weights to generate fused features; The step-by-step decoding module is used to perform multi-scale feature fusion on the fused features and gradually restore detail information and global semantic information; The joint output module is used to generate and output semantic segmentation results and object detection results based on detail information and global semantic information; The joint optimization module is used to optimize the shared parameters of semantic segmentation and object detection tasks through a joint training mechanism.

2. The real-time semantic segmentation and target detection joint model for multimodal data fusion according to claim 1 is characterized in that: The feature extraction module includes: The first branch network is used to extract the semantic features of RGB images through the residual network; The second branch network is used to extract the structural features of the depth map through a lightweight convolutional layer; The third branch network is used to voxelize the lidar point cloud and extract its spatial geometric features through three-dimensional sparse convolution.

3. The real-time semantic segmentation and target detection joint model for multimodal data fusion according to claim 1 is characterized in that: The dynamic mode selection module includes: The modality importance score calculation module is used to perform global average pooling on the features of each modality to generate the modality importance score; Normalization module, used to normalize the modality importance score to modality weight through the Softmax function; The weighted fusion module is used to perform weighted fusion of multimodal features according to modal weights to generate cross-modal complementary features.

4. The real-time semantic segmentation and target detection joint model for multimodal data fusion according to claim 1, characterized in that: The UAFM feature fusion module includes: The channel attention module is used to perform channel attention weighting on cross-modal complementary features of different modalities to generate inter-modal weights; The spatial attention module is used to perform spatial attention weighting on the cross-modal complementary features after channel attention weighting; The convolution compression module is used to perform channel splicing and convolution compression on the cross-modal complementary features after spatial attention weighting to generate fusion features.

5. The real-time semantic segmentation and target detection joint model for multimodal data fusion according to claim 1, characterized in that: The step-by-step decoding module includes: Multi-level upsampling module, used to perform multi-level upsampling of fusion features; The skip connection module is used to perform a skip connection between the current upsampled features and the low-level features of the corresponding modality encoding stage at each level of decoding; The convolution operation module is used to fuse the features of the jump connection through convolution operations to generate decoding features containing multi-scale information.

6. The real-time semantic segmentation and target detection joint model for multimodal data fusion according to claim 1, characterized in that: The combined output module includes: The semantic segmentation head consists of a sequentially connected convolution, activation function, and bilinear interpolation layer. It processes the decoded features of multi-scale information and outputs pixel-level classification results. The object detection head consists of a sequentially connected region proposal network and a classification and regression module, which is used to generate candidate boxes based on the decoded features of multi-scale information and perform category prediction and bounding box correction on the candidate boxes.

7. The real-time semantic segmentation and target detection joint model for multimodal data fusion according to claim 1, characterized in that: The joint optimization module is used to construct a multi-task loss function and optimize the shared parameters of the semantic segmentation and object detection tasks through the loss function, wherein the multi-task loss function includes: Segmentation loss: including pixel-level cross entropy loss; Detection loss: including smooth L1 loss and focal loss; Edge-assisted loss: This involves predicting boundaries through the edge detection head and calculating the binary cross entropy loss with the labels generated by the Canny operator.

8. A method for real-time semantic segmentation and target detection based on multimodal data fusion, applied to the joint model for real-time semantic segmentation and target detection based on multimodal data fusion according to claim 1, characterized in that: The following steps are involved: Obtain RGB images, depth information, and lidar point clouds to build a dataset; Build and train a real-time semantic segmentation and object detection joint model that fused multimodal data; Save the trained model weights; Use the trained model for semantic segmentation and object detection, and evaluate the model based on the segmentation and detection results.

Images