Visible-infrared semantic segmentation method based on multi-attribute feature progressive interaction
By using a semantic decoder guided by progressive interaction of multi-attribute features and detailed features, the shortcomings of visible light-infrared semantic segmentation algorithms in feature fusion and image restoration are addressed, thereby improving target perception and segmentation accuracy in unfriendly environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2024-01-11
- Publication Date
- 2026-07-24
AI Technical Summary
Existing visible-infrared semantic segmentation algorithms have shortcomings in feature fusion and image scale recovery, making it difficult to accurately perceive surrounding targets in unfriendly environments, and the decoder has difficulty fully recovering image information.
We employ a progressive interaction method with multi-attribute features, filtering features of different dimensions through an attention module. The branches are modal complementarity and modal commonality. Combined with a semantic decoder guided by detailed features, we achieve comprehensive feature integration and restoration of detailed information.
It achieves more comprehensive multimodal feature interaction and fusion of detailed information, improving the accuracy and robustness of semantic segmentation models, especially in target perception capabilities under low light, overexposure, and rainy conditions.
Smart Images
Figure CN117876683B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a visible light-infrared semantic segmentation method based on progressive interaction of multi-attribute features. Background Technology
[0002] Visible-infrared semantic segmentation helps unmanned platforms perceive surrounding targets in unfriendly environments (such as low light, overexposure, rain, and fog), which is crucial for autonomous driving and intelligent driving requirements. In recent years, deep learning-based visible-infrared semantic segmentation algorithms have achieved high levels of accuracy and robustness, but they still suffer from insufficient feature fusion when processing visible and infrared modal information. Furthermore, semantic segmentation requires a decoder capable of reconstructing image information from the feature scale to the image scale for pixel-level classification. However, existing semantic segmentation algorithms struggle to fully reconstruct image information at both the overall and detailed scales. Summary of the Invention
[0003] In view of this, the purpose of the present invention is to provide a semantic segmentation model that can better process visible light and infrared modal features and completely recover image information from the features.
[0004] To achieve the above objectives, the present invention provides the following technical solution:
[0005] A visible light-infrared semantic segmentation method based on progressive interaction of multi-attribute features includes the following steps:
[0006] Multi-attribute feature progressive interaction: First, an attention module is used to filter features of different dimensions; then, the filtered features are branched into two branches: modal complementarity and modal commonality; finally, through feature interaction and feature transfer between different receptive fields, a comprehensive integration of features with different attributes is achieved.
[0007] Semantic decoding guided by detailed features: Incorporating guiding information from detailed features, low-dimensional features are used to provide edge and contour information for semantic decoding, while high-dimensional features are used to provide distribution information for semantic decoding.
[0008] Furthermore, the use of an attention module to filter features across different dimensions specifically includes:
[0009] Visible light and thermal infrared features are first filtered through an attention module to process the different characteristics of features at different levels. Spatially, global max pooling is used to generate spatial attention weights for low-level features with large spatial scales and few channels. Channel-wise, global max pooling is used to obtain channel attention weights for high-level features. Then, the initial input is multiplied by the attention weights to obtain R. i 'and Ti ', is represented as:
[0010]
[0011]
[0012] in, Cat[] and Cat[] represent pixel-level multiplication and feature merging operations, respectively; GMP represents global max pooling, used for low-level and high-level features in the spatial and channel domains, respectively; Conv7×7 represents convolution operation with a kernel size of 7; MLP contains two convolutional layers with a kernel size of 1, and a rectified linear unit.
[0013] Furthermore, the filtering of feature branches into two branches—modal complementarity and modal commonality—specifically includes:
[0014] Filtered features R i 'and T i 'Addition and multiplication operations are performed respectively to represent the complementary and shared attributes between visible light and infrared features; four depthwise separable convolutions with different kernel sizes are used to enhance the features of these two attributes; through feature interaction blocks, the features of the two attributes interact to achieve interaction in different receptive fields and achieve fusion within the receptive field range from small to large, as represented by:'
[0015]
[0016] IB[a,b,c]=CBR(Cat[CBR(a),CBR(b),CBR(c)])
[0017] Where Fi represents the interaction characteristics of different receptive fields, DSConv k×k represents a depthwise separable convolution with a kernel size of 3 and an expansion rate of k, followed by batch normalization and ReLU layers; IB[] represents a feature interaction block used to fuse complementary and shared features from various modalities; CBR represents a convolution with a kernel size of 3, followed by batch normalization and ReLU layers.
[0018] Furthermore, the comprehensive integration of features with different attributes is achieved through feature interaction and feature transfer between different receptive fields, specifically including:
[0019] The features extracted from all four receptive fields are concatenated together and introduced into the initial input via residual connections. Additional CBR is used to merge the outputs from different levels, achieving multi-level interaction, as shown below:
[0020]
[0021] Ei =CBR(Cat[E i-1 ,CBR(F')])
[0022] Among them, E i This represents the output of the multi-attribute feature progressive interaction module. Through the multi-attribute feature progressive interaction module, the modal attributes of visible light and infrared light are effectively utilized, and the number of channels of all features is normalized to 64, which facilitates the subsequent decoding process.
[0023] Furthermore, the semantic decoding guided by detailed features specifically includes:
[0024] The decoder consists of three branches: boundary, binary, and semantic branches;
[0025] The boundary branch focuses on exploring edge information in low-dimensional features, the binary branch uses high-dimensional features to provide distribution information inside the contour, and the semantic branch integrates features from various levels to achieve accurate segmentation.
[0026] The boundary branch performs a multiplication operation on the features extracted from the first and second layers to obtain shared information, introducing semantic information to enhance the generation of the boundary map, expressed as:
[0027]
[0028] Where up indicates upsampling the feature map, scaling it to a uniform size; E1, E2, and S i-1 These represent low-level features and semantic output, respectively.
[0029] The binary branch combines and connects features from the third and fourth stages to capture complementary information and introduces semantic information to enhance the generation of the binary map, expressed as:
[0030] H i =CBR(Cat[(CBR(up(E3))+CBR(up(E4))),CBR(Cat[up(E3),up(E4)]),S i-1 ])
[0031] Finally, the semantic branch is at L i and H i With the assistance of [unclear], a semantic segmentation map is reconstructed, represented as:
[0032] S i =CBR(Cat[CBR(L i ),CBR(H i ),CBR(S i-1 )]).
[0033] The beneficial effects of this invention are as follows: Compared with classic feature fusion methods, the multi-attribute progressive feature fusion of this invention fully considers the similarity and difference between visible light modal and infrared modal features, achieving more comprehensive multi-modal feature interaction and providing a better foundation for subsequent semantic decoding; compared with semantic decoders that rely solely on semantic labels, the semantic decoding process guided by detailed features of this invention takes into account both detailed information and semantic information. This enables the semantic segmentation model of this invention to exhibit more accurate semantic segmentation results.
[0034] Other advantages, objectives, and features of the invention will be set forth in the following description and will be apparent to those skilled in the art in some respects, or may be learned by practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0035] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration:
[0036] Figure 1 This is a semantic segmentation algorithm model based on progressive interaction of multi-attribute features and a multi-attribute progressive feature interaction module;
[0037] Figure 2 A semantic decoder guided by detailed features. Detailed Implementation
[0038] This invention provides a semantic segmentation algorithm based on progressive interaction of multiple attribute features, comprising a progressive interaction module for multiple attribute features and a semantic decoder guided by detailed features. Its detailed implementation is as follows:
[0039] The visible light-infrared semantic segmentation algorithm based on progressive interaction of multi-attribute features includes the following parts:
[0040] I. Multi-attribute feature progressive interaction module. The multi-attribute feature progressive module consists of three stages: the initial adaptive stage of light-infrared feature weights, the stage of splitting visible light-infrared modal complementary features and common features, and the stage of multi-attribute feature progressive interaction.
[0041] Effective utilization of visible light and thermal infrared features is crucial for improving the accuracy of visible-infrared semantic segmentation. However, due to differences in modal attributes, simple feature addition and multiplication cannot achieve accurate feature fusion; instead, they may introduce significant noise. Furthermore, in urban street scenes, there are significant differences in object scale, which poses a challenge to achieving accurate segmentation of different objects using feature processing methods with fixed receptive fields. To address these challenges, this invention proposes a multi-attribute feature progressive interaction module. This module first uses an attention module to filter features across different dimensions. Then, it branches the filtered features into two branches: modal complementarity and modal commonality. Finally, through feature interaction and feature transfer between different receptive fields, it achieves comprehensive integration of features with different attributes.
[0042] The following is a detailed description of the multi-attribute feature progressive interaction module:
[0043] like Figure 1 As shown, visible light and thermal infrared features are first filtered through an attention module to highlight important information from both modalities. The attention module is designed to handle different characteristics of features at different levels. Specifically, global max pooling is used spatially to generate spatial attention weights for low-level features with a large spatial scale and few channels. Similarly, global max pooling is used channel-wise to obtain channel attention weights for high-level features. The initial input is then multiplied by the attention weights to obtain R. i 'and T i This process can be represented as:
[0044]
[0045]
[0046] in, `Cat[]` and `Cat[]` represent pixel-level multiplication and feature merging operations, respectively. `GMP` stands for Global Max Pooling, used for low-level and high-level features in the spatial and channel domains, respectively. `Conv7×7` represents a convolution operation with a kernel size of 7. `MLP` contains two convolutional layers with a kernel size of 1, and a rectified linear unit.
[0047] Filtered features R i 'and T i The features undergo addition and multiplication operations respectively to represent complementary and shared attributes between visible light and infrared features. To efficiently segment multi-scale targets in urban street scenes, four depthwise separable convolutions with different kernel sizes are used to enhance the features of these two attributes. Furthermore, through feature interaction blocks, the features of the two attributes interact, enabling interaction across different receptive fields and achieving fusion within receptive fields ranging from small to large. This can be represented as:
[0048]
[0049] IB[a,b,c]=CBR(Cat[CBR(a),CBR(b),CBR(c)])
[0050] Here, Fi represents the interaction characteristics of different receptive fields. DSConv k×k `IB[]` represents a depthwise separable convolution with a kernel size of 3 and an expansion rate of k, followed by batch normalization and a ReLU layer. `IB[]` represents a feature interaction block used to fuse complementary and shared features from various modalities. `CBR` represents a convolution with a kernel size of 3, followed by batch normalization and a ReLU layer.
[0051] To enhance feature fusion from different receptive fields, features extracted from all four receptive fields are concatenated together and introduced into the initial input via residual connections. Subsequently, additional CBR is used to merge outputs from different levels, achieving multi-level interaction. This operation can be represented as:
[0052]
[0053] E i =CBR(Cat[E i-1 ,CBR(F')])
[0054] Among them, E i This represents the output of the multi-attribute feature progressive interaction module. Through this module, the modal attributes of visible light and infrared light are effectively utilized, and the number of channels for all features is normalized to 64, facilitating the subsequent decoding process.
[0055] II. Detail-Guided Semantic Decoder. Detail-guided information is added to the traditional semantic label-supervised decoding structure. Low-dimensional features provide edge and contour information for semantic decoding, while high-dimensional features provide distribution information. The designed decoder achieves more accurate semantic segmentation.
[0056] In semantic segmentation, challenges often arise at the boundaries between different categories, leading to segmentation errors. Furthermore, misclassification of pixels within a contour can result in low-quality segmentation maps. Features at different levels contain detailed information that can aid in achieving accurate segmentation. Therefore, a detail-guided semantic decoder is proposed as a means of leveraging this detailed information to guide semantic map reconstruction. Figure 2The decoder consists of three branches: a boundary branch, a binary branch, and a semantic branch. The boundary branch focuses on exploring edge information in low-dimensional features, while the binary branch utilizes high-dimensional features to provide distribution information within the contour. With the assistance of these two branches, the semantic branch integrates features from various levels to achieve accurate segmentation.
[0057] To leverage the detailed information contained in the low-level features, the boundary branch performs multiplication on the features extracted from the first and second layers to obtain shared information. Furthermore, the introduction of semantic information further enhances the generation of the boundary map. This process can be expressed as:
[0058]
[0059] Here, `up` indicates an upsampling operation on the feature map, scaling it to a uniform size. E1, E2, and S i-1 These represent low-level features and semantic output, respectively. Furthermore, an additional binary branch is designed to enhance the segmentation of the semantic branch. Specifically, the binary branch combines and connects the features from the third and fourth stages to capture complementary information. This process improves the stability of contour construction and introduces semantic information to enhance the generation of the binary map. This can be expressed as:
[0060] H i =CBR(Cat[(CBR(up(E3))+CBR(up(E4))),CBR(Cat[up(E3),up(E4)]),S i-1 ])
[0061] Finally, the semantic branch is at L i and H i With the assistance of [unclear], the semantic segmentation map can be reconstructed as follows:
[0062] S i =CBR(Cat[CBR(L i ),CBR(H i ),CBR(S i-1 )])
[0063] By performing the above operations, the detail-guided semantic decoder integrates edge information from low-level features and utilizes distribution information from high-level features, thereby facilitating the generation of the final semantic segmentation map.
[0064] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.
Claims
1. A visible light-infrared semantic segmentation method based on progressive interaction of multi-attribute features, characterized in that: Includes the following steps: Multi-attribute feature progressive interaction: First, an attention module is used to filter features across different dimensions, including: Visible light and thermal infrared features are first filtered through an attention module to process the different characteristics of features at different levels. Spatially, global max pooling is used to generate spatial attention weights for low-level features with large spatial scales and few channels. Channel-wise, global max pooling is used to obtain channel attention weights for high-level features. Then, the initial input is multiplied by the attention weights to obtain... R i 'and T i ' , represented as: in, These represent pixel-level multiplication and feature merging operations, respectively. GMP Represents global max pooling, used for low-level and high-level features in the spatial and channel domains, respectively; Conv 7×7 represents a convolution operation with a kernel size of 7; MLP It contains two convolutional layers with a kernel size of 1, and one rectified linear unit; The filtered feature branches are then divided into two branches: modal complementarity and modal commonality, including: Filtered features R i 'and T i 'Addition and multiplication operations are performed respectively to represent the complementary and shared attributes between visible light and infrared features; four depthwise separable convolutions with different kernel sizes are used to enhance the features of these two attributes; through feature interaction blocks, the features of the two attributes interact to achieve interaction in different receptive fields and achieve fusion within the receptive field range from small to large, as represented by:' Where Fi represents the interaction characteristics of different receptive fields. DSConv k×k This represents a depthwise separable convolution with a kernel size of 3 and an expansion rate of 1 / 3. k This is followed by batch normalization and ReLU layers; IB [] represents a feature interaction block, used to fuse complementary and shared features from various modalities; CBR This represents a convolution with a kernel size of 3, followed by batch normalization and ReLU layers; Finally, through feature interaction and feature transfer between different receptive fields, a comprehensive integration of features with different attributes is achieved, including: The features extracted from all four receptive fields are concatenated together and introduced into the initial input via residual connections. Additional CBR is used to merge the outputs from different levels, achieving multi-level interaction, as shown below: in, E i This represents the output of the multi-attribute feature progressive interaction module. Through the multi-attribute feature progressive interaction module, the modal attributes of visible light and infrared light are effectively utilized, and the number of channels of all features is normalized to 64, which facilitates the subsequent decoding process. Detail-guided semantic decoding: Detail-guided information is incorporated, using low-dimensional features to provide edge and contour information for semantic decoding, and high-dimensional features to provide distribution information for semantic decoding; the decoder consists of three branches: boundary, binary, and semantic branches. The boundary branch focuses on exploring edge information in low-dimensional features, the binary branch uses high-dimensional features to provide distribution information inside the contour, and the semantic branch integrates features from various levels to achieve accurate segmentation. The boundary branch performs a multiplication operation on the features extracted from the first and second layers to obtain shared information, introducing semantic information to enhance the generation of the boundary map, expressed as: in, up This indicates that the feature map is upsampled and scaled to a uniform size. E 1. E 2 and S i-1 These represent low-level features and semantic output, respectively. The binary branch combines and connects features from the third and fourth stages to capture complementary information and introduces semantic information to enhance the generation of the binary map, expressed as: Finally, semantic branches in L i and H i With the assistance of [unclear], a semantic segmentation map is reconstructed, represented as: 。