A multi-modal image fusion semantic segmentation method based on edge feature enhancement
By employing a multimodal image fusion strategy enhanced with edge features, the challenge of object recognition in low-light environments during RGB-T image fusion was solved, achieving more efficient semantic segmentation and improving segmentation accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENYANG INST OF AUTOMATION - CHINESE ACAD OF SCI
- Filing Date
- 2025-01-24
- Publication Date
- 2026-07-24
AI Technical Summary
Existing RGB-T image fusion methods struggle to effectively identify visually unrecognizable objects in low-light environments, and suffer from redundant information and noise interference, affecting segmentation accuracy and efficiency.
A multimodal image fusion strategy with edge feature enhancement is adopted. Through parallel RGB and thermal image feature extraction modules, combined with position and channel attention mechanisms, rich features are extracted, and a dedicated loss function is used to optimize model performance, with particular attention to the edge contours of foreground objects.
It significantly improves the accuracy and robustness of object recognition and segmentation, generates more refined and accurate semantic segmentation results, and enhances the model's segmentation performance in complex scenes.
Smart Images
Figure CN122454572A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision. It proposes a multimodal semantic segmentation method that utilizes the unique features of RGB-T data to address the multimodal image fusion problem. The aim is to improve segmentation performance and enhance the understanding and application capabilities of complex scenes. Background Technology
[0002] Semantic segmentation is a crucial component of computer vision, with extremely wide-ranging applications, particularly in practical applications such as autonomous driving and robot object detection. To improve segmentation accuracy and efficiency, it is typically necessary to collect and integrate visual data from multiple modalities for comprehensive data acquisition. Among these, RGB-T (data fusing red, green, and blue visible light with thermal infrared information) is a widely adopted data modality that has become a research hotspot in both academia and industry due to its unique advantages, attracting in-depth exploration and research.
[0003] Current research in the RGB-T domain primarily focuses on the conversion mechanisms and learning strategies between different modalities, with few studies delving into the unique properties inherent in each modality. This situation may inadvertently introduce redundant information and noise interference during the fusion of multimodal data. Through detailed analysis of RGB images and thermal images, although the application of attention mechanisms significantly improves the accuracy of RGB images in segmentation tasks, the ability to recognize visually difficult-to-discern objects in low-light environments still faces severe challenges. Summary of the Invention
[0004] To address the aforementioned technical challenges, this invention proposes a multimodal image fusion semantic segmentation method based on edge feature enhancement. Edge features from thermal images are cleverly integrated into the RGB image processing framework as crucial supplementary information. This strategy not only enriches the dimensions of image information but also significantly enhances the overall performance of image segmentation tasks, thereby achieving more accurate and robust object recognition and segmentation in complex scenes.
[0005] This invention proposes an enhanced RGB-T fusion semantic segmentation network, built upon an encoder-decoder architecture. In the encoder component, two parallel and independent feature extraction modules are used to process image data of different modalities. First, in the RGB image feature extraction module, an attention mechanism is introduced to effectively capture and integrate context-dependent structural information within the image. This mechanism dynamically fuses local feature information with global dependencies, thereby generating more accurate and detailed feature representations and improving the accuracy and richness of feature expression. Second, in the hot feature extraction module, an edge feature extraction algorithm is introduced to accurately extract foreground content edge information from the hot image and effectively integrate these key edge features into the RGB image features. Furthermore, to optimize the model's performance in processing image edge information and foreground object recognition, thereby improving the overall accuracy and efficiency of the segmentation task, this invention innovatively designs a dedicated loss function for RGB-T image characteristics and incorporates it as a constraint into the model. The loss function comprises two core components: first, a constraint on image content similarity, designed to ensure that the generated segmentation result maintains a high degree of consistency with the original image in terms of overall content; and second, a constraint focusing on the similarity of foreground object edge contours, which pays particular attention to accurately capturing and preserving the boundary details of objects. This design strategy of the present invention not only enhances the model's feature learning ability but also ensures that it exhibits excellent segmentation performance and high stability in complex and varied scenes.
[0006] The technical solution adopted by this invention to achieve the above objectives is: a multimodal image fusion semantic segmentation method based on edge feature enhancement, comprising the following steps:
[0007] Step 1: Perform data augmentation on the input multimodal visual data;
[0008] Step 2: Construct an encoder-decoder multimodal semantic segmentation model using parallel RGB image feature extraction and thermal image feature extraction modules; wherein, the encoder obtains local features of RGB images and edge features of thermal images through the RGB image feature extraction and thermal image feature extraction modules respectively, and then obtains the majority modality data fusion features by bit-by-bit addition, and then inputs them into the decoder to obtain the prediction map;
[0009] Step 3: Construct a loss function that includes image content similarity constraints and foreground object edge contour similarity constraints as constraints for training the encoder-decoder multimodal semantic segmentation model. Use the augmented data to train the encoder-decoder multimodal semantic segmentation model to obtain the model parameters.
[0010] Step 4: Input the multimodal visual data to be segmented into the trained encoder-decoder multimodal semantic segmentation model to obtain the prediction results.
[0011] A multimodal semantic segmentation model for encoders and decoders is constructed using parallel RGB image feature extraction and thermal image feature extraction modules, specifically including:
[0012] (1) The RGB image feature extraction module uses a parallel approach of positional attention and channel attention mechanisms to extract features from the augmented RGB image data E. rgb Feature extraction includes the following steps:
[0013] 1) Using a ResNet50 network to process the enhanced RGB image data E rgb Feature extraction is performed, and then convolution is applied to the extracted results to obtain a local feature map F of size C×H×W. l Where C is F l Number of channels, H is F l The length of W is F l The width;
[0014] 2) For the obtained F l The positional attention mechanism is used to further extract RGB image features, and the final output F of the positional attention mechanism is obtained. p ;
[0015] 3) For the obtained F l The channel attention mechanism is used to further extract RGB image features, and the final output of the channel attention mechanism is F. c ;
[0016] 4) The output features F of the two attention mechanisms p and F c Perform the same convolution operation on each part, and add the results bitwise to obtain F. rgb As the output of the RGB image feature extraction module;
[0017] (2) Using the thermal image feature extraction module to process the data-enhanced thermal image data E t Feature extraction is performed to obtain the final edge features F of the thermal image. t ;
[0018] (3) Obtain F rgb and F t The fused feature F is obtained by adding the positions one by one. e ;
[0019] (4) F e The input is fed into the decoder to obtain the predicted image P.
[0020] The acquisition of F lThe positional attention mechanism is used to further extract RGB image features, and the final output F of the positional attention mechanism is obtained. p This includes the following steps:
[0021] ①For F l Perform three l×l convolution operations in parallel and reshape the results into C×N to obtain F. p1 F p2 and F p3 Where N = H × W;
[0022] ② F p1 With F p2 Multiplying by the transpose of the product, and then applying the softmax function, yields the position attention weights P. ji for:
[0023]
[0024] Among them, P ji Let represent the attention intensity of position i to position j, and exp represent the exponential function;
[0025] ③Use P ji and F p3 Perform matrix multiplication to obtain positional attention; reshape the positional attention into C×H×W and combine it with the local feature map F. l Adding them together yields the final output F of the position attention mechanism. p :
[0026]
[0027] Here, reshape represents the reshaping function, and the parameter α is initially set to 0, with its value continuously updated throughout the training process.
[0028] The acquisition of F l The channel attention mechanism is used to further extract RGB image features, and the final output of the channel attention mechanism is F. c This includes the following steps:
[0029] ①For F l Perform three reshaping operations in parallel, reshaping its size from C×H×W to C×N, and obtain F respectively. c1 F c2 and F c3 Where N = H × W;
[0030] ②F c1 Multiply by the transpose F using matrix multiplication. c2 Then, the output is processed using softmax to obtain the channel attention weights C. sk :
[0031]
[0032] Among them, C sk This represents the attention intensity of channel k to channel s;
[0033] ③ Place C sk and F c3 Perform matrix multiplication to obtain channel attention; reshape the result into C×H×W and combine it with the local feature map F. l Adding them together, we obtain the final output F of the channel attention mechanism. c :
[0034] F c =reshape[β(C sk F c3 )] C×H×W +F l
[0035] The parameter β is initially set to 0, and its value is continuously updated throughout the training process.
[0036] The thermal image feature extraction module is used for E t Feature extraction is performed to obtain the final edge features F of the thermal image. t This includes the following steps:
[0037] 1) Use the Sobel operator to extract the edge features of the thermal image to obtain the edge image S. t ;
[0038] 2) The edge features of the thermal image are extracted using the Canny operator to obtain the edge image C. t ;
[0039] 3) Place S t With C t By adding the bits one by one, we obtain the final edge image D of the thermal image. t ;
[0040] 4) D t The input is fed into a ResNet50 network for feature extraction, and then an l×l convolution operation is performed on the result to obtain the final edge features F of the thermal image. t .
[0041] The loss function, which incorporates image content similarity constraints and foreground object edge contour similarity constraints, is constructed as follows:
[0042] (1) Construct content similarity constraints using the cross-entropy loss function:
[0043]
[0044] Where M represents the number of sample categories, y c p represents the sample label. c This represents the probability that a sample is predicted to be of the c-th class.
[0045] (2) Use the following loss function to construct constraints on edge contour similarity:
[0046]
[0047] Where x is the predicted image, i.e., the predicted image P obtained by the encoder-decoder multimodal semantic segmentation model; y is the label edge feature image, i.e., the edge features F obtained by the thermal image feature extraction module. t SSIM is a structural similarity index, as detailed below:
[0048]
[0049] Where, μ x and μ y σ represents the brightness of x and y respectively; x and σ y σ represents the contrast of x and y, respectively; xy Let C1 be the covariance of x and y; the constant is C1 = (K1 × L). 2 The constant C2 = (K2 × L) 2 Where [K1,K2] are positive constants, and L represents the dynamic range of pixel values.
[0050] A multimodal image fusion semantic segmentation system based on edge feature enhancement includes:
[0051] The multimodal visual data augmentation module is used to augment the input multimodal visual data.
[0052] The multimodal semantic segmentation model construction module is used to construct an encoder-decoder multimodal semantic segmentation model using parallel RGB image feature extraction modules and thermal image feature extraction modules. The encoder obtains local features of RGB images and edge features of thermal images through the RGB image feature extraction module and thermal image feature extraction module, respectively. Then, it obtains the majority modality data fusion features by bit-by-bit addition, and then inputs them into the decoder to obtain the prediction map.
[0053] The multimodal semantic segmentation model training module is used to construct a loss function that includes image content similarity constraints and foreground object edge contour similarity constraints as constraints for training the encoder-decoder multimodal semantic segmentation model. The encoder-decoder multimodal semantic segmentation model is trained using augmented data to obtain model parameters.
[0054] The multimodal visual data segmentation module is used to input the multimodal visual data to be segmented into the trained encoder-decoder multimodal semantic segmentation model to obtain the prediction results.
[0055] A multimodal image fusion semantic segmentation device based on edge feature enhancement includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the multimodal image fusion semantic segmentation method based on edge feature enhancement when the computer program is executed.
[0056] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned multimodal image fusion semantic segmentation method based on edge feature enhancement.
[0057] The present invention has the following beneficial effects and advantages:
[0058] This invention employs different feature extraction branches for RGB images and thermal images, fully utilizing the rich color and texture information of RGB images while leveraging the ability of thermal images to clearly display object outlines even under poor lighting conditions. Through this design, the invention effectively avoids noise interference and feature redundancy problems that may result from indiscriminately introducing thermal image features, thereby achieving optimized information integration during the fusion process. This invention can generate more refined and accurate semantic segmentation results, significantly improving the performance of segmentation tasks.
[0059] This invention designs a proprietary loss function as a constraint. This loss function consists of two core components: first, an image content similarity constraint, ensuring that the generated segmented image maintains a high degree of consistency with the original image in terms of content; and second, a foreground object edge contour similarity constraint, which improves the accuracy of the foreground object edge contours in the segmentation result. The synergistic effect of these two parts not only enhances the model's ability to understand complex scenes but also improves the refinement and accuracy of the segmentation results, effectively improving the performance and quality of the segmentation task. Attached Figure Description
[0060] Figure 1 The architecture diagram of the multimodal image fusion semantic segmentation method proposed in this invention.
[0061] Figure 2 The data augmentation structure diagram used in this invention.
[0062] Figure 3 The present invention presents a flowchart for multimodal data feature extraction and fusion based on edge feature enhancement. Detailed Implementation
[0063] The following example of multimodal semantic segmentation in an autonomous driving scenario will be used to describe the present invention in more detail. This embodiment is exemplary and intended to explain the present invention, and should not be construed as a limitation of the present invention.
[0064] Figure 1 This is a framework diagram of a multimodal image fusion semantic segmentation method based on edge feature enhancement according to an embodiment of the present invention.
[0065] like Figures 1-3 As shown, this multimodal image fusion semantic segmentation method based on edge feature enhancement includes the following steps:
[0066] Step 1: Augmentation of Original Road Scene Data. By performing reasonable transformations and simulations (such as rotation, cropping, etc.) on the existing data, the existing dataset can be expanded. This can effectively cover a variety of complex scenarios and special conditions, thereby helping the model learn more diverse features and improving the model's generalization ability and robustness.
[0067] like Figure 2 As shown, the specific process for data augmentation of input RGB image and thermal image data in a road scene includes:
[0068] 1. Define the input data RGB image sample set as Where n is the number of RGB images; the set of input data thermal image samples is defined as... The number of thermal image samples is the same as the number of RGB image samples.
[0069] 2. To I rgb and I t Data augmentation was performed using random flipping, random cropping, and random noise addition to obtain the augmented dataset E. rgb E t The expanded dataset has a size of m.
[0070] Step 2: Build an RGB-T semantic segmentation model based on the encoder-decoder architecture. The specific process includes:
[0071] 1. Use as follows Figure 3 The RGB image feature extraction module shown above is for E rgb Feature extraction is performed. First, preliminary feature extraction is conducted on the input RGB image data. Then, positional attention and channel attention mechanisms are used to further extract features. The obtained feature results are then summed bit-by-bit to obtain feature F. rgb This is the output of the RGB image feature extraction module. The specific process includes:
[0072] (1) Using a ResNet50 network to analyze E rgbPreliminary feature extraction is performed, followed by a 3×3 convolution operation on the extracted results to obtain a local feature map F of size C×H×W. l Where C=512 is the number of channels, and H=30 is the F l The length, W=40 is F l The width;
[0073] (2) Figure 3 As shown, for the obtained F l The positional attention mechanism is used to further extract RGB image features. The specific process is as follows:
[0074] 1) For F l Perform three 3×3 convolution operations in parallel and reshape the results into C×N to obtain F. p1 F p2 and F p3 Where N = H × W;
[0075] 2) F p1 With F p2 Multiplying by the transpose of the product, and then applying the softmax function, yields the position attention weights P. ji for:
[0076]
[0077] Where P ji This represents the attention intensity of position i to position j; the correlation between two positions is proportional to the similarity of their feature representations.
[0078] 3) Obtain P ji With F p3 Perform multiplication to obtain positional attention, reshape the result into C×H×W, and then combine it with the local feature map F. l Adding them together yields the final output F of the position attention mechanism. p :
[0079]
[0080] The parameter α is initially set to 0, and its value is continuously updated throughout the training process.
[0081] (3) Figure 3 As shown, for the obtained F l The channel attention mechanism is used to further extract RGB image features. The specific process is as follows:
[0082] 1) For F l Perform three reshaping operations in parallel, reshaping its size from C×H×W to C×N, and obtain F respectively. c1 F c2 and Fc3 Where N = H × W, C = 512 is the number of channels, and H = 30 and W = 40 are F l Length and width;
[0083] 2)F c1 Multiply by F using matrix multiplication. c2 The transpose of the expression is then used. The output is then processed using softmax to obtain the channel attention weights C. sk :
[0084]
[0085] Where C sk This represents the attention intensity of channel k to channel s;
[0086] 3) Obtain C sk and F c3 Perform multiplication to obtain channel attention, reshape the result into C×H×W, and then combine it with the local feature map F. l Adding them together, we obtain the final output F of the channel attention mechanism. c :
[0087] F c =reshape[β(C sk F c3 )] C×H×W +F l
[0088] The parameter β is initially set to 0, and its value is continuously updated throughout the training process.
[0089] (4) Figure 3 As shown, the output features F of the two attention mechanisms are... p and F c Perform the same convolution operation on each part, and add the results bitwise to obtain F. rgb As the output of the RGB image feature extraction module.
[0090] 2. For example Figure 3 As shown, the thermal image feature extraction module is used to extract features from E. t Feature extraction mainly involves the following steps:
[0091] (1) The Sobel operator is used to extract the edge features of the thermal image to obtain the edge image S. t .
[0092] (2) The edge features of the thermal image are extracted using the Canny operator to obtain the edge image C. t .
[0093] (3) S t With Ct By adding the bits one by one, we obtain the final edge image D of the thermal image. t .
[0094] (4) D t The input is fed into a ResNet50 network for feature extraction. Then, a 3×3 convolution operation is used on the results to enhance the model's non-linear expressive power, yielding the final edge features F of the thermal image. t .
[0095] 3. For example Figure 3 As shown, the obtained F rgb and F t The fused feature F is obtained by adding the positions one by one. e .
[0096] 4. For example Figure 3 As shown, F e The input is fed into the decoder to obtain the semantic segmentation prediction map P.
[0097] Step 3: Design a loss function as a constraint for model training. This loss function contains two core components: first, a constraint on image content similarity to ensure that the generated segmentation result maintains a high degree of consistency with the original image in overall content; second, a constraint focusing on the similarity of foreground object edge contours to accurately capture and preserve the boundary details of objects. The design process is as follows:
[0098] 1. Construct content similarity constraints using the cross-entropy loss function, defined as:
[0099]
[0100] Where M represents the number of classes in the sample, y c p represents the sample label. c This represents the predicted probability of a sample.
[0101] 2. Use the following loss function to construct constraints for edge contour similarity:
[0102]
[0103] Where x and y are the predicted image and the label edge feature image, respectively. y can be obtained through the edge feature extraction process in step two. SSIM is the structural similarity index, which is defined as follows:
[0104]
[0105] Where μ x and μ y σ represents the brightness of x and y respectively; x and σ y σ represents the contrast of x and y, respectively;xy Let C1 be the covariance of x and y; C1 = (K1 × L) 2 And C2=(K2×L) 2 Where the positive constants [K1,K2] = [0.01,0.03], and L = 255 represents the dynamic range of pixel values.
[0106] 3. The total loss function is defined as the weighted sum of the two loss functions mentioned above:
[0107] L total =γ1L CE +γ2L edge
[0108] Where γ1=γ2=0.5 are hyperparameters, satisfying γ1+γ2=1.
[0109] 4. For example Figure 1 As shown, the above loss function is used as a constraint to guide the training of the model, resulting in a pre-trained multimodal semantic segmentation model.
[0110] Step 4, as follows Figure 3 As shown, the paired test images, i.e., RGB images T, in the downstream task are... rgb With thermal image T t The input is fed into a pre-trained multimodal semantic segmentation model to obtain segmentation results T containing different color regions. p Specifically, these include: cars (blue), pedestrians (green), bicycles (yellow), ground edge lines (red), guardrails (blue-green), warning triangles (orange), and road obstacles (purple).
[0111] By fusing cross-modal features, this invention enables the model to significantly improve its sensitivity and recognition ability to target contours, thereby achieving higher detail preservation and accuracy in the final semantic segmentation results.
Claims
1. A multimodal image fusion semantic segmentation method based on edge feature enhancement, characterized in that, Includes the following steps: Step 1: Perform data augmentation on the input multimodal visual data; Step 2: Construct an encoder-decoder multimodal semantic segmentation model using parallel RGB image feature extraction and thermal image feature extraction modules; wherein, the encoder obtains local features of RGB images and edge features of thermal images through the RGB image feature extraction and thermal image feature extraction modules respectively, and then obtains the majority modality data fusion features by bit-by-bit addition, and then inputs them into the decoder to obtain the prediction map; Step 3: Construct a loss function that includes image content similarity constraints and foreground object edge contour similarity constraints as constraints for training the encoder-decoder multimodal semantic segmentation model. Use the augmented data to train the encoder-decoder multimodal semantic segmentation model to obtain the model parameters. Step 4: Input the multimodal visual data to be segmented into the trained encoder-decoder multimodal semantic segmentation model to obtain the prediction results.
2. The multimodal image fusion semantic segmentation method based on edge feature enhancement according to claim 1, characterized in that, A multimodal semantic segmentation model for encoders and decoders is constructed using parallel RGB image feature extraction and thermal image feature extraction modules, specifically including: (1) The RGB image feature extraction module uses a parallel approach of positional attention and channel attention mechanisms to extract features from the augmented RGB image data E. rgb Feature extraction includes the following steps: 1) Using a ResNet50 network to process the enhanced RGB image data E rgb Feature extraction is performed, and then convolution is applied to the extracted results to obtain a local feature map F of size C×H×W. l Where C is F l Number of channels, H is F l The length of W is F l The width; 2) For the obtained F l The positional attention mechanism is used to further extract RGB image features, and the final output F of the positional attention mechanism is obtained. p ; 3) For the obtained F l The channel attention mechanism is used to further extract RGB image features, and the final output of the channel attention mechanism is F. c ; 4) The output features F of the two attention mechanisms p and F c Perform the same convolution operation on each part, and add the results bitwise to obtain F. rgb As the output of the RGB image feature extraction module; (2) Using the thermal image feature extraction module to process the data-enhanced thermal image data E t Feature extraction is performed to obtain the final edge features F of the thermal image. t ; (3) Obtain F rgb and F t The fused feature F is obtained by adding the positions one by one. e ; (4) F e The input is fed into the decoder to obtain the predicted image P.
3. The multimodal image fusion semantic segmentation method based on edge feature enhancement according to claim 2, characterized in that, The acquisition of F l The positional attention mechanism is used to further extract RGB image features, and the final output F of the positional attention mechanism is obtained. p This includes the following steps: ①For F l Perform three l×l convolution operations in parallel and reshape the results into C×N to obtain F. p1 F p2 and F p3 Where N = H × W; ② F p1 With F p2 Multiplying by the transpose of the product, and then applying the softmax function, yields the position attention weights P. ji for: Among them, P ji Let represent the attention intensity of position i to position j, and exp represent the exponential function; ③Use P ji and F p3 Perform matrix multiplication to obtain positional attention; reshape the positional attention into C×H×W and combine it with the local feature map F. l Adding them together yields the final output F of the position attention mechanism. p : Here, reshape represents the reshaping function, and the parameter α is initially set to 0, with its value continuously updated throughout the training process.
4. The multimodal image fusion semantic segmentation method based on edge feature enhancement according to claim 2, characterized in that, The acquisition of F l The channel attention mechanism is used to further extract RGB image features, and the final output of the channel attention mechanism is F. c This includes the following steps: a. Regarding F l Perform three reshaping operations in parallel, reshaping its size from C×H×W to C×N, and obtain F respectively. c1 F c2 and F c3 Where N = H × W; bF c1 Multiply by the transpose F using matrix multiplication. c2 Then, the output is processed using softmax to obtain the channel attention weights C. sk : Among them, C sk This represents the attention intensity of channel k to channel s; c. Place C sk and F c3 Perform matrix multiplication to obtain channel attention; reshape the result into C×H×W and combine it with the local feature map F. l Adding them together, we obtain the final output F of the channel attention mechanism. c : F c =reshape[β(C sk F c3 )] C×H×W +F l The parameter β is initially set to 0, and its value is continuously updated throughout the training process.
5. The multimodal image fusion semantic segmentation method based on edge feature enhancement according to claim 1, characterized in that, The thermal image feature extraction module is used for E t Feature extraction is performed to obtain the final edge features F of the thermal image. t This includes the following steps: The Sobel operator is used to extract edge features from the thermal image, resulting in the edge image S. t ; The edge features of the thermal image are extracted using the Canny operator to obtain the edge image C. t ; S t With C t By adding the bits one by one, we obtain the final edge image D of the thermal image. t ; D t The input is fed into a ResNet50 network for feature extraction, and then an l×l convolution operation is performed on the result to obtain the final edge features F of the thermal image. t .
6. The multimodal image fusion semantic segmentation method based on edge feature enhancement according to claim 1, characterized in that, The loss function, which incorporates image content similarity constraints and foreground object edge contour similarity constraints, is constructed as follows: 4.1) Construct content similarity constraints using the cross-entropy loss function: Where M represents the number of sample categories, y c p represents the sample label. c This represents the probability that a sample is predicted to be of the c-th class. 4.2) Construct constraints for edge contour similarity using the following loss function: Where x is the predicted image, i.e., the predicted image P obtained through the encoder-decoder multimodal semantic segmentation model; y is the label edge feature image, i.e., the edge features F obtained through the thermal image feature extraction module. t SSIM is a structural similarity index, as detailed below: Where, μ x and μ y σ represents the brightness of x and y respectively; x and σ y σ represents the contrast of x and y, respectively; xy Let C1 be the covariance of x and y; the constant is C1 = (K1 × L). 2 The constant C2 = (K2 × L) 2 Where [K1,K2] are positive constants, and L represents the dynamic range of pixel values.
7. A multimodal image fusion semantic segmentation system based on edge feature enhancement, characterized in that, include: The multimodal visual data augmentation module is used to augment the input multimodal visual data. The multimodal semantic segmentation model construction module is used to construct an encoder-decoder multimodal semantic segmentation model using parallel RGB image feature extraction modules and thermal image feature extraction modules. The encoder obtains local features of RGB images and edge features of thermal images through the RGB image feature extraction module and thermal image feature extraction module, respectively. Then, it obtains the majority modality data fusion features by bit-by-bit addition, and then inputs them into the decoder to obtain the prediction map. The multimodal semantic segmentation model training module is used to construct a loss function that includes image content similarity constraints and foreground object edge contour similarity constraints as constraints for training the encoder-decoder multimodal semantic segmentation model. The encoder-decoder multimodal semantic segmentation model is trained using augmented data to obtain model parameters. The multimodal visual data segmentation module is used to input the multimodal visual data to be segmented into the trained encoder-decoder multimodal semantic segmentation model to obtain the prediction results.
8. A multimodal image fusion and semantic segmentation device based on edge feature enhancement, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, a multimodal image fusion semantic segmentation method based on edge feature enhancement as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a multimodal image fusion semantic segmentation method based on edge feature enhancement as described in any one of claims 1-6.