Cross-modal semantic segmentation system and method

Abstract

The present application relates to the technical field of image processing, and more particularly to a cross-modal semantic segmentation system and method, which combines the complementary information of infrared and visible light images, and designs different fusion modules for different levels: for shallow features, a multi-scale attention module expands the receptive field through multi-scale convolution layers, combines channel and spatial attention mechanisms, and enhances the ability to capture detailed texture information in cross-modal images; for deep features, a cross-modal adaptive weighted fusion module uses dense connection blocks and nonlinear feature enhancement units to realize efficient feature transmission and adaptive fusion and differential complementarity of different modal semantic information. In the method provided by the present application, edge labels and binary classification labels are introduced as auxiliary supervision, focusing on target boundary details and enhancing the semantic discriminability of features, and through a multi-task collaborative optimization mechanism, the segmentation accuracy and robustness of the network for complex scenes are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a cross-modal semantic segmentation system and method. BACKGROUND

[0002] Image semantic segmentation is to accurately label the semantic class of each pixel point in the image, deeply analyze the size ratio, geometric shape and spatial position information of the target, and decompose the complex visual scene into entity units with clear semantic attributes, so as to realize fine image understanding and analysis from pixel level to semantic level.

[0003] Traditional single-modal semantic segmentation algorithm mainly describes the visual characteristics of different regions in the image by artificially designing features, and then inputs the extracted features into a classifier to classify the pixels. This kind of algorithm excessively relies on artificially designed features, and the feature expression ability is limited, so the segmentation effect for complex scenes is often not ideal. With the development of deep learning technology and the availability of a large number of data sets, semantic segmentation networks based on deep learning have made great progress, and according to the network structure, they can be divided into full convolution network structure, encoder-decoder structure and pyramid structure. However, the single-modal semantic segmentation network still has the problems of poor segmentation effect and unstable segmentation under complex environment and light change condition.

[0004] In order to enhance the segmentation robustness under complex light and bad weather conditions such as rain and fog, in recent years, researchers have expanded the information dimension of visible light images by supplementing infrared modal data, and proposed cross-modal semantic segmentation algorithm. Infrared image is formed by perceiving the difference of thermal radiation of objects, and has the light immunity characteristic, which can stably obtain the target contour information in the environment where visible light fails such as dark night and thick fog. Combined with the high sensitivity of visible light image to texture details and color features, the two can form complementary information combination. However, the existing fusion methods all indiscriminately process different levels of features and cross-modal interaction between levels by designing specific modules, ignoring the feature differences between different levels and levels, resulting in the problems of boundary blur and poor segmentation accuracy of small targets in the distance of the network under complex light conditions. SUMMARY

[0005] Therefore, the present application aims to provide a cross-modal semantic segmentation system and method, which combines the complementary information of infrared and visible light, and designs different fusion modules for deep and shallow networks, namely multi-scale attention fusion module and cross-modal feature weighted fusion module. In the method, edge labels and binary classification labels are introduced as auxiliary supervision, focusing on target boundary details and enhancing the semantic discriminability of features.

[0006] To achieve the above purpose, the technical scheme of the present application is as follows:

[0007] The application discloses a cross-modal semantic segmentation system, which comprises a visible light coding branch, an infrared coding branch and a decoding branch.

[0008] Further, the multi-scale attention fusion module performs feature fusion on the shallow visible light features and infrared features, and then simultaneously performs multi-scale spatial attention operation and channel attention operation on the fused features, fuses the features after the two kinds of attention operations, and obtains output features of a corresponding scale; the adaptive weighted fusion module respectively performs dense convolution on the deep visible light features and infrared features, combines the features after the dense convolution with the input features of the other party respectively, and obtains first dense features and second dense features; the first dense features and the second dense features are fused, and weight extraction is performed on the fused features to obtain weights respectively for the first dense features and the second dense features; the weights are applied to the first dense features and the second dense features, and then the two features with the weights are fused to obtain output features of a corresponding scale.

[0009] Further, the multi-scale spatial attention operation in the multi-scale attention fusion module comprises the following steps: performing multi-scale convolution operation on the input features, adding the features of different scales obtained, and performing spatial attention operation on the added features to obtain output features.

[0010] Further, the dense convolution in the adaptive weighted fusion module comprises the following steps: performing multiple continuous convolution operations on the input features, and jump connecting the features after the convolution to other convolution operations.

[0011] Further, in the adaptive weighted fusion module, the process of fusing the first dense features and the second dense features and performing weight extraction on the fused features to obtain weights respectively for the first dense features and the second dense features comprises the following steps: performing feature fusion on the first dense features and the second dense features by first channel splicing and then convolution; performing sigmoid activation operation on the fused features to obtain weights, and then dividing the weights to obtain weights respectively for the first dense features and the second dense features.

[0012] Furthermore, in the adaptive weighted fusion module, the process of fusing two features with weights includes: multiplying the two features, and then adding the multiplied feature to both features simultaneously.

[0013] Furthermore, the decoding branch includes a deep decoding stage and a shallow decoding stage: In the deep decoding stage: the deep fusion feature with the smallest scale is decoded, the decoded feature is added to the deep fusion feature of the corresponding scale, and then the added feature is decoded again until the deep fusion feature with the largest scale is decoded to obtain the output feature; In the shallow decoding stage: the output feature of the deep decoding stage is continuously decoded at multiple scales to obtain the predicted semantic segmentation result; Each decoding operation includes: after multiple convolutions of the input feature, the convolutional feature is added to the input feature pixel by pixel, and then the added feature is upsampled to obtain the decoded output feature.

[0014] Furthermore, before each decoding operation, channel attention is performed on the input features of the decoding operation.

[0015] A cross-modal semantic segmentation method, comprising:

[0016] S1: Obtain a dataset including visible light images, infrared images, and corresponding real semantic segmentation results;

[0017] S2: Perform boundary labeling and binary classification labeling on the real semantic segmentation results obtained in step S1 to obtain the corresponding real boundary labels and real binary classification labels; combine the visible light image, infrared image, real semantic segmentation results, real boundary labels and real binary classification labels to obtain the training set;

[0018] S3: Construct the cross-modal semantic segmentation system as provided in this invention, and train the cross-modal semantic segmentation system using the training set obtained in step S2 to obtain a semantic segmentation model;

[0019] S4: Input the visible light image to be semantically segmented and its corresponding infrared image into the semantic segmentation model obtained in step S3 to obtain the corresponding predicted semantic segmentation result.

[0020] Furthermore, in step S3, the model is trained using the total loss function expressed in the following formula:

[0021] ;

[0022] Where L represents the total loss function, Indicates semantic loss, Indicates boundary loss, Indicates the binary boundary loss. , and Indicates the loss balance factor;

[0023] Calculate semantic loss During the process, the Lovász-softmax loss function is used to calculate the difference between the predicted semantic segmentation result and the corresponding real semantic segmentation result;

[0024] Calculate boundary loss In the process, the shallow fusion features of the largest scale are multiplied with the predicted semantic segmentation results, and then the cross-entropy loss function is calculated with the corresponding real boundary labels.

[0025] Calculate the binary boundary loss In the process, the shallow fusion features of the second scale are multiplied with the input features encoded at the last level, and then the cross-entropy loss function is calculated with the corresponding true binary classification labels.

[0026] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0027] This invention creates a cross-modal semantic segmentation system and method, aiming to provide a cross-modal semantic segmentation method that combines edge assistance and multi-level feature fusion. It integrates complementary information from infrared and visible light images and designs different fusion modules for different levels. For shallow features, a multi-scale attention fusion module (MAFM) is constructed, expanding the receptive field through multi-scale convolutional layers and combining channel and spatial attention mechanisms to enhance the ability to capture detailed texture information in cross-modal images. For deep features, a cross-modal feature weighted fusion module (CFWF) is designed, utilizing densely connected blocks and nonlinear feature enhancement units to achieve efficient feature transfer and adaptive fusion and complementary differences in semantic information from different modalities. The method introduces edge labels and binary classification labels as auxiliary supervision, focusing on target boundary details and enhancing the semantic discriminative power of features. Through a multi-task collaborative optimization mechanism, the network's segmentation accuracy and robustness in complex scenes are improved. Attached Figure Description

[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0029] Figure 1A schematic diagram of the cross-modal semantic segmentation system described in the embodiments of the present invention;

[0030] Figure 2 A schematic diagram of the multi-scale attention fusion module described in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the adaptive weighted fusion module described in an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of dense convolution as described in an embodiment of the present invention;

[0033] Figure 5 A schematic diagram illustrating the combination of decoding operation and channel attention operation as described in the embodiments of the present invention;

[0034] Figure 6 A flowchart illustrating the cross-modal semantic segmentation method described in the embodiments of the present invention;

[0035] Figure 7 The comparative result diagrams described in the embodiments of the present invention;

[0036] Figure 8 A comparison chart of the objective evaluation index data described in the embodiments of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0039] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0040] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0041] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] like Figure 1 As shown in the embodiment of the present invention, the cross-modal semantic segmentation system includes a visible light encoding branch, an infrared encoding branch, a multi-scale feature fusion branch, and a decoding branch: the visible light encoding branch extracts multi-scale features from the input visible light image to obtain multi-scale visible light features; the infrared encoding branch extracts multi-scale features from the input infrared image to obtain multi-scale infrared features; the multi-scale feature fusion branch includes a multi-scale attention fusion module (MAFM) and an adaptive weighted fusion module (CFWF). The multi-scale attention fusion module performs attention enhancement operations combining channel direction and spatial direction on the shallow visible light and infrared features to obtain corresponding shallow fused features; the adaptive weighted fusion module performs adaptive weight allocation on the deep visible light and infrared features, and then performs feature fusion according to the allocated weights to obtain deep fused features of the corresponding scale; the decoding branch decodes the shallow fused features and deep fused features step by step according to scale to obtain the predicted semantic segmentation result.

[0043] In this embodiment of the invention, the visible light coding branch and the infrared coding branch have the same structure. Both preferably adopt the ResNet152 network structure pre-trained on ImageNet. Compared with the first two layers, the last three layers of ResNet152 use more downsampling operations and residual blocks, which effectively expands the receptive field and improves the representation of semantic features. Therefore, the features extracted from the last three layers are used as deep features, and the features extracted from the first two layers are used as shallow features.

[0044] The multi-scale attention fusion module captures detailed texture information in shallow features through multi-scale convolutional layers and attention mechanisms, achieving fusion enhancement of low-level features such as image edges and structures. The adaptive weighted fusion module utilizes a densely connected network structure, stacking nonlinear feature enhancement units to strengthen the expression of abstract semantics such as target categories and scene relationships, achieving high-order semantic information feature fusion enhancement. The two modules work together to form a progressive feature fusion system from low-level details to high-level semantics, effectively improving the encoder's feature representation capability. The fused feature map is further enhanced with semantic information through channel feature enhancement operation (CFEM) in the decoder, and then the image is upsampled and reconstructed through the decoding module (DB) to obtain the semantic segmentation result.

[0045] Since convolutional neural networks can only fuse features of the same scale, it is easy to conclude that in the multi-scale feature fusion branch of this embodiment, the multi-scale attention fusion module fuses the visible light features extracted from the first and second layers of the visible light coding branch and the infrared features extracted from the first and second layers of the infrared coding branch at corresponding scales to obtain the first shallow fusion feature corresponding to the visible light features extracted from the first layer of the visible light coding branch, and the second shallow fusion feature corresponding to the visible light features extracted from the second layer of the visible light coding branch. Similarly, the adaptive weighted fusion module fuses the visible light features extracted from the third to fifth layers of the visible light coding branch and the infrared features extracted from the third to fifth layers of the infrared coding branch at corresponding scales to obtain the first deep fusion feature corresponding to the visible light features extracted from the third layer of the visible light coding branch, the second deep fusion feature corresponding to the visible light features extracted from the fourth layer of the visible light coding branch, and the third deep fusion feature corresponding to the visible light features extracted from the fifth layer of the visible light coding branch.

[0046] Shallow features contain a wealth of detailed information such as texture and edges. To enhance the representational power of these detailed features, this invention designs a multi-scale attention fusion module. The feature processing includes: fusing shallow visible light and infrared features; simultaneously performing multi-scale spatial attention and channel attention operations on the fused features; and fusing the features after the two attention operations to obtain output features at the corresponding scale. In some embodiments, performing multi-scale spatial attention operations in the multi-scale attention fusion module includes: performing convolution operations on the input features at multiple scales; adding the resulting features at different scales; and performing spatial attention operations on the added features to obtain output features. In the multi-scale attention fusion module provided by this invention, the multi-scale convolutional layer increases the receptive field, the channel and spatial attention mechanisms adaptively enhance important channel and positional information, suppress redundant background information, improve the accuracy and effectiveness of feature representation, and the additive fusion algorithm preserves detailed information and overall structure.

[0047] The feature processing procedure of the multi-scale attention fusion module provided in this embodiment of the invention is as follows: Figure 2 As shown, the feature fusion of shallow visible light features and shallow infrared features at the same scale is achieved by first channel stitching and then performing a 1×1 convolution operation, as follows:

[0048] S=conv 1×1 (cat(f i ,f r ));

[0049] Where S represents the fused features, conv 1×1 This indicates a convolution operation with a kernel size of 1×1, cat indicates a channel concatenation operation, and f i and f r These represent visible light and infrared characteristics at the same scale, respectively.

[0050] The fused features are subjected to both multi-scale spatial attention and channel attention operations. The features after the two attention operations are then fused by first adding them pixel by pixel and then performing a 1×1 convolution operation to obtain the output features at the corresponding scale, as shown in the following formula:

[0051] ;

[0052] Where G represents the output feature, P represents the channel salient feature obtained after performing channel attention operation on the fused features, and Q represents the spatial salient feature obtained after performing multi-scale spatial attention operation on the fused features. This indicates a pixel-by-pixel addition operation.

[0053] The multi-scale spatial attention operation on the fused features specifically includes: simultaneously inputting the fused features into three convolution operations with kernel sizes of 3×3, 5×5, and 7×7, and then summing their outputs pixel by pixel to obtain the multi-scale features, as shown in the following formula.

[0054] ;

[0055] Where, conv 3×3 conv 5×5 and conv 7×7 These represent three convolution operations with kernel sizes of 3×3, 5×5, and 7×7, respectively.

[0056] Global average pooling and global max pooling are performed simultaneously on multi-scale feature maps to focus on important information in the spatial location of the feature maps. The features after the two pooling operations are concatenated by channels and activated by Sigmoid, and then normalized to obtain the spatial attention weights, as shown in the following formula:

[0057] W s =σ(cat(GAP(H),GMP(H)));

[0058] Among them, W s σ represents the spatial attention weight, GAP represents the global average pooling operation, and GMP represents the global max pooling operation.

[0059] Multiplying the multi-scale features with the spatial attention weights pixel by pixel yields the multi-scale spatial salient features, thus completing the spatial attention operation, as shown in the following equation:

[0060] ;

[0061] in, This indicates pixel-by-pixel multiplication.

[0062] The channel attention operation on the fused features specifically includes: simultaneously performing global average pooling and global max pooling operations on the fused features to obtain two vectors describing the channel features; then concatenating the two vectors and normalizing them using Sigmoid activation to obtain channel attention weights normalized to the range of 0-1, as shown in the following formula.

[0063] W c =σ(cat(GAP(S),GMP(S)));

[0064] Among them, W c Indicates channel attention weights;

[0065] Multiplying the channel attention weights by the corresponding elements of the fused features yields the salient features of each channel, as shown in the following formula:

[0066] .

[0067] The multi-scale attention fusion module utilizes multi-scale convolution and attention mechanisms to preserve the detailed information in shallow features to the maximum extent, which helps to optimize the edge contours of segmented targets.

[0068] Deep features contain more semantic information. To enhance the high-order semantic information representation capability of deep feature maps, this invention designs a cross-modal adaptive weighted fusion module. Its feature processing includes: performing dense convolutions on deep visible light and infrared features respectively, and then combining the convolved features with the input features of the other to obtain first and second dense features. The first and second dense features are then fused, and weights are extracted from the fused features to obtain weights for the first and second dense features respectively. This process specifically includes: performing channel concatenation and convolution on the first and second dense features for feature fusion; performing sigmoid activation on the fused features to obtain weights, and then dividing the weights to obtain weights for the first and second dense features respectively. The weights are applied to the first and second dense features respectively, and then the two weighted features are fused to obtain output features at the corresponding scale. This process includes: multiplying the two features, and then adding the multiplied feature to both features simultaneously.

[0069] The feature processing process of the adaptive weighted fusion module provided in this embodiment of the invention is as follows: Figure 3 As shown, deep visible light features and deep infrared features at the same scale are densely convolved. Then, the densely convolved features are multiplied pixel-by-pixel with the input features of the other, and then combined by a convolution operation with a kernel size of 1×1 to obtain the first dense feature and the second dense feature, as shown in the following formula:

[0070] ;

[0071] ;

[0072] Among them, o i and o r F represents the first dense feature and the second dense feature, respectively. i and F r These represent deep visible light features and deep infrared features at the same scale, respectively, and Dense represents dense convolution.

[0073] The first and second dense features are first concatenated through channels, and then fused using two consecutive 3×3 convolutions. The fused features are normalized to the range of 0-1 using a sigmoid activation operation to obtain weights. Then, a split operation is performed on the weights to separate the weights for the first and second dense features, as shown in the following formula:

[0074] {w i ;w r}=split(σ(conv 3×3 (conv 3×3 (cat(o i ,o r )))));

[0075] Among them, w i and w r These represent the weights of the first dense feature and the second dense feature, respectively.

[0076] The weights are applied to the first and second dense features respectively to obtain features with cross-modal complementary information, as shown in the following equation:

[0077] ;

[0078] ;

[0079] in, and These represent the first and second dense features after being weighted, respectively.

[0080] The two weighted features are then multiplied to further extract salient information between modalities. To avoid information loss, the multiplied feature is then added to both features to obtain the output feature at the corresponding scale, as shown in the following formula:

[0081] ;

[0082] Where E represents the output feature.

[0083] The dense convolution process includes performing multiple consecutive convolution operations on the input features, and then skipping between the convolutional features and other convolutional operations. In this embodiment of the invention, the convolutions in the dense convolution are all convolutions with a kernel size of 1×1, such as... Figure 4 As shown.

[0084] In some embodiments, the decoding branch includes a deep decoding stage and a shallow decoding stage: In the deep decoding stage: the smallest scale deep fusion feature is decoded, the decoded feature is added to the corresponding scale deep fusion feature, and then the added feature is decoded again until the largest scale deep fusion feature is decoded, resulting in the output feature; In the shallow decoding stage: the output feature of the deep decoding stage is subjected to continuous multi-scale decoding operations to obtain the predicted semantic segmentation result. Each decoding operation includes: performing multiple convolutions on the input feature, adding the convolutional feature to the input feature pixel by pixel, and then upsampling the added feature to obtain the decoded output feature. In some embodiments, channel attention is performed on the input feature of the decoding operation before each decoding operation. In this embodiment, the combination of decoding operation and channel attention operation is as follows: Figure 5 As shown, where:

[0085] The channel attention operation specifically includes: capturing key information in each channel of the input features using global max pooling and global average pooling; then, adding the results of the two pooling operations pixel by pixel, performing dimensionality reduction and expansion using two 1×1 convolutional layers, and normalizing the convolutional features using the sigmoid activation function to obtain a channel weight between 0 and 1, as shown in the following formula:

[0086] ;

[0087] Where W represents the channel weight, F in Represents input features;

[0088] Finally, the input feature map is multiplied by the weights and then added element-wise to the input feature map to obtain the channel-enhanced features, as shown in the following formula:

[0089] ;

[0090] Among them, F c Features that indicate channel feature enhancement;

[0091] The decoding operation specifically includes: performing a combination of three consecutive 3×3 convolutions, batch normalization, and ReLU activation on the channel feature-enhanced features; then adding the result pixel-by-pixel with the channel feature-enhanced features to stabilize gradient propagation; finally, upsampling the summed features by a factor of 2 to obtain the output features, as shown in the following formula:

[0092] ;

[0093] Among them, F out Indicates the output feature, Up 2×CBN represents a 2x upsampling operation, and CBN represents a combination of 3×3 convolution, batch normalization, and ReLU activation.

[0094] This invention also provides a cross-modal semantic segmentation method, combining Figure 1 and Figure 6 ,include:

[0095] S1: Obtain a dataset including visible light images, infrared images, and corresponding real semantic segmentation results.

[0096] In this embodiment of the invention, the publicly available MFNet dataset is used. This dataset consists of visible light RGB images and infrared images captured using an InfReCR500 camera for urban street scenes. The dataset contains a total of 1569 image pairs, including 820 pairs of daytime images and 749 pairs of nighttime images. Besides the background category, there are eight target categories: color cones, vehicle barriers, pedestrians, curves, bicycles, guardrails, vehicles, and protrusions.

[0097] S2: Perform boundary labeling and binary classification labeling on the real semantic segmentation results obtained in step S1 to obtain the corresponding real boundary labels and real binary classification labels; combine the visible light image, infrared image, real semantic segmentation results, real boundary labels, and real binary classification labels to obtain the training set. In this embodiment of the invention, 784 pairs of images from the dataset are selected to construct the training set, including 410 pairs of daytime images and 374 pairs of nighttime images, while 394 pairs of images from the dataset are selected to construct the validation set, including 205 pairs of daytime images and 189 pairs of nighttime images.

[0098] S3: Construct the cross-modal semantic segmentation system as provided in this invention, and train the cross-modal semantic segmentation system using the training set obtained in step S2 to obtain a semantic segmentation model.

[0099] This invention introduces a boundary-assisted supervision strategy. In the decoder, the output is compared with the true boundary label and the true binary classification label to calculate the loss separately. This forces the network model to simultaneously focus on the overall segmentation of the target region and the extraction of boundary details, thereby assisting in training and optimizing network parameters and improving the accuracy of edge segmentation. Specifically, in some embodiments, the following total loss function is used for model training:

[0100] ;

[0101] Where L represents the total loss function, Indicates semantic loss, Indicates boundary loss, Indicates the binary boundary loss. , and This represents the loss balance factor. In this embodiment of the invention, after multiple experiments and adjustments, it was found that when the loss balance factor... , and The objective evaluation index of the experimental results was optimal when the corresponding values ​​were set to 0.4, 0.3, and 0.3.

[0102] Calculate semantic loss During the process, the Lovász-softmax loss function is used to calculate the difference between the predicted semantic segmentation result and the corresponding real semantic segmentation result;

[0103] Calculate boundary loss In the process, the shallow fusion features of the largest scale are multiplied with the predicted semantic segmentation results, and then the cross-entropy loss function is calculated with the corresponding real boundary labels.

[0104] Calculate the binary boundary loss In the process, the shallow fusion features of the second scale are multiplied with the input features encoded at the last level, and then the cross-entropy loss function is calculated with the corresponding true binary classification labels.

[0105] S4: Input the visible light image to be semantically segmented and its corresponding infrared image into the semantic segmentation model obtained in step S3 to obtain the corresponding predicted semantic segmentation result.

[0106] To verify the effectiveness of the method provided by this invention, a comparative experiment was conducted with three existing methods. These three methods are: Method 1: MFNet, a real-time semantic segmentation method for multispectral autonomous driving scenarios (Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes.” 2017 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 2017); Method 2: EGFNet, an edge-aware guidance semantic segmentation method for multispectral scenes (Zhou, Wujie et al. “Edge-awareGuidance Fusion Network for RGB Thermal Scene Parsing.” ArXiv abs / 2112.05144(2021)); Method 3: CMX, a Transformer-based RGB-X semantic segmentation cross-modal fusion network algorithm (Liu H, Zhang J, Yang K, et al. CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers[J]. 2022.DOI:10.48550 / arXiv.2203.04838.). Comparison results are as follows: Figure 7 As shown, Figure 7 Each column represents a visible light RGB image, an infrared image, the true semantic segmentation label value, comparison method one, comparison method two, comparison method three, and the method of this invention; Figure 7 In the results shown, compared with the segmentation results of Method 1, the feature information is severely lost and the segmentation effect is poor; compared with the segmentation results of Method 2, the edge detail information is not sufficiently extracted and the target contour segmentation is coarse; compared with the segmentation results of Method 3, the ability to segment small objects at a distance is insufficient and there are cases of missed segmentation; in the segmentation results of the method of this invention, the structural information of the segmented image is clear and complete, the complementary information of infrared and visible light images is well preserved, the detail information is rich, and the edge segmentation of small targets and objects at a distance is clear and accurate.

[0107] To better evaluate the semantic segmentation performance, in addition to the subjective evaluations mentioned above, objective evaluation metrics were used to objectively analyze the segmentation results of different semantic segmentation methods. Two evaluation metrics were set: mean Intersection over Union (mIoU) and mean accuracy (mAcc). A higher metric value indicates a better segmentation method, and vice versa. The objective evaluation metric data are as follows: Figure 8 As shown, from Figure 8 The objective evaluation data shows that the semantic segmentation method of the present invention is superior to the comparison method 1, comparison method 2 and comparison method 3 in all objective evaluation indicators, which is consistent with the above subjective evaluation results. This shows that the method of the present invention can stably obtain high-quality semantic segmentation results.

[0108] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A cross-modal semantic segmentation system, characterized in that, include: The visible light coding branch performs multi-scale feature extraction on the input visible light image to obtain multi-scale visible light features; The infrared coding branch performs multi-scale feature extraction on the input infrared image to obtain multi-scale infrared features; The multi-scale feature fusion branch includes a multi-scale attention fusion module and an adaptive weighted fusion module. The multi-scale attention fusion module performs feature fusion on shallow visible light and infrared features, and then performs multi-scale spatial attention operation and channel attention operation on the fused features. The features after the two attention operations are fused to obtain the output features of the corresponding scale. The adaptive weighted fusion module performs dense convolutions on deep visible light and infrared features respectively, and then combines the densely convolved features with the input features of the other to obtain the first dense feature and the second dense feature. The first dense feature and the second dense feature are fused, and weights are extracted from the fused features to obtain weights for the first dense feature and the second dense feature respectively. The weights are applied to the first dense feature and the second dense feature respectively, and then the two features with weights are fused to obtain the output feature of the corresponding scale. The decoding branch decodes the shallow and deep fusion features step by step according to their scale to obtain the predicted semantic segmentation results.

2. Multi-scale spatial attention operations in the multi-scale attention fusion module include: Perform convolution operations at multiple scales on the input features, and then sum the features obtained at different scales. The summed features are then subjected to spatial attention to obtain the output features.

3. The cross-modal semantic segmentation system according to claim 1, characterized in that, Dense convolutions in the adaptive weighted fusion module include: performing multiple consecutive convolution operations on the input features, and then skipping between the convolutional features and other convolutional operations.

4. The cross-modal semantic segmentation system according to claim 1, characterized in that, In the adaptive weighted fusion module, the first dense feature and the second dense feature are fused, and weights are extracted from the fused features to obtain weights for the first dense feature and the second dense feature, respectively. The process includes: The first and second dense features are first concatenated and then convolved to fuse the features. The fused features are then activated by sigmoid to obtain weights, and the weights are then divided to obtain weights for the first and second dense features respectively.

5. The cross-modal semantic segmentation system according to claim 1, characterized in that, In the adaptive weighted fusion module, the process of fusing two features with weights includes: multiplying the two features, and then adding the multiplied feature to both features.

6. The cross-modal semantic segmentation system according to claim 1, characterized in that, The decoding branch includes a deep decoding stage and a shallow decoding stage: In the deep decoding stage: the deep fusion feature with the smallest scale is decoded, the decoded feature is added to the deep fusion feature of the corresponding scale, and then the added feature is decoded again until the deep fusion feature with the largest scale is decoded to obtain the output feature. In the shallow decoding stage: continuous multi-scale decoding operations are performed on the output features of the deep decoding stage to obtain the predicted semantic segmentation results; Each decoding operation includes: performing multiple convolutions on the input features, adding the convolutional features to the input features pixel by pixel, and then performing an upsampling operation on the added features to obtain the decoded output features.

7. The cross-modal semantic segmentation system according to claim 6, characterized in that, Before each decoding operation, channel attention is performed on the input features of the decoding operation.

8. A cross-modal semantic segmentation method, characterized in that, include: S1: Obtain a dataset including visible light images, infrared images, and corresponding real semantic segmentation results; S2: Perform boundary labeling and binary classification labeling on the real semantic segmentation results obtained in step S1 to obtain the corresponding real boundary labels and real binary classification labels; combine the visible light image, infrared image, real semantic segmentation results, real boundary labels and real binary classification labels to obtain the training set; S3: Construct a cross-modal semantic segmentation system as described in any one of claims 1 to 7, and train the cross-modal semantic segmentation system using the training set obtained in step S2 to obtain a semantic segmentation model; S4: Input the visible light image to be semantically segmented and its corresponding infrared image into the semantic segmentation model obtained in step S3 to obtain the corresponding predicted semantic segmentation result.

9. The cross-modal semantic segmentation method according to claim 8, characterized in that, In step S3, the model is trained using the total loss function as follows: ； Where L represents the total loss function, Indicates semantic loss, Indicates boundary loss, Indicates the binary boundary loss. , and Indicates the loss balance factor; Calculate semantic loss During the process, the Lovász-softmax loss function is used to calculate the difference between the predicted semantic segmentation result and the corresponding real semantic segmentation result; Calculate boundary loss In the process, the shallow fusion features of the largest scale are multiplied with the predicted semantic segmentation results, and then the cross-entropy loss function is calculated with the corresponding real boundary labels. Calculate the binary boundary loss In the process, the shallow fusion features of the second scale are multiplied with the input features encoded at the last level, and then the cross-entropy loss function is calculated with the corresponding true binary classification labels.

Images