Visible light-infrared semantic segmentation method based on hybrid experts

Abstract

The invention discloses a visible light-infrared semantic segmentation method based on hybrid experts. According to the method, the neural network model based on the hybrid experts is established, and visible light and infrared features can be effectively fused, so that a more accurate segmentation result is obtained. The invention provides a route selector with channel attention, and suitable expert networks can be adaptively selected for different visible light-infrared image pairs, so that the generalization performance of a model in coping with different scenes is improved. Besides, the feature fusion module based on the hybrid experts designed by the invention has sparsity, only a small amount of calculation overhead is increased while the expression ability of the model is improved, and better balance is achieved between the effect of the model and the reasoning speed. Experimental results prove that the method achieves advanced performance on a widely used MFNet data set.

Description

Technical Field

[0001] The invention relates to a semantic segmentation method in the technical field of computer vision, and in particular to a visible light-infrared semantic segmentation method and system based on mixed experts. Background Art

[0002] In the field of computer vision and image processing, semantic segmentation based on the fusion of visible light and infrared images is an important and active research direction. With the development of technology, more and more application fields require image processing systems that can work stably under different environmental conditions. These fields include autonomous driving, security monitoring, military reconnaissance, search and rescue, and industrial inspection. In these applications, the technology of effectively fusing visible light and infrared images will greatly improve the reliability and accuracy of the system. For example, at night or in bad weather conditions, visible light cameras may not be able to capture clear images, while thermal imagers can detect image information caused by temperature differences; conversely, visible light images can supplement the detail information and color information lacking in infrared images, thereby producing more refined semantic segmentation results.

[0003] How to utilize the complementarity of visible light and infrared images to improve the effect of semantic segmentation has always been the focus of research in the field of RGB-T semantic segmentation.

[0004] Deng et al. introduced a feature-enhanced attention module in FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation to mine and enhance multi-level features from both channel and spatial perspectives, thereby preserving spatial information, refining detail features, and shifting more attention to fused high-resolution features. Zhou et al. proposed a spatial attention mechanism module in MFFENet: Multiscale Feature Fusion and Enhancement Network For RGB–Thermal Urban Road Scene Parsing, which gives higher weights to foreground areas and introduces multi-label supervision to optimize model parameters. Zhang et al. adopted a bidirectional image translation-based method to reduce the modal difference between visible light features and infrared features in ABMDRNet: Adaptive-weighted Bi-directional Modality Difference Reduction Network for RGB-T Semantic Segmentation, and then adaptively selected discriminative multimodal features for semantic segmentation in a channel-weighted fusion manner. However, the above methods often design complex attention-based feature fusion modules, which significantly increase the computational overhead of the model.

[0005] Fan et al. proposed a spatially aware demand-guided recursive network in Spatial-aware Demand-guided Recursive Meshing for RGB-TSemantic Segmentation. The network actively compensates for insufficient contextual semantics in optically damaged areas through a demand-guided target masking algorithm, and refines the fused semantic features through recursive meshing. Shin et al. proposed a complementary masking strategy for visible and infrared images in Complementary Random Masking for RGB-ThermalSemantic Segmentation, which encourages the network to learn complementary feature representations from a single modality or complementary masking modalities, thereby improving the performance of pixel-level semantic segmentation. However, the above methods do not consider the generalization ability of the model in dealing with different scenarios. Summary of the invention

[0006] In order to solve the problems existing in the background technology, the present invention provides a method and system for visible light-infrared semantic segmentation based on hybrid experts. The present invention performs feature fusion between visible light and infrared modalities through a multimodal fusion module based on hybrid experts, and selects the most suitable fusion mode for different visible light-infrared image pairs, so as to adapt to complex and changeable application scenarios.

[0007] The technical solution adopted by the present invention is:

[0008] 1. A Visible-Infrared Semantic Segmentation Method Based on Hybrid Experts

[0009] (1) Obtaining visible light-infrared data set;

[0010] (2) Construct a visible light-infrared semantic segmentation model based on mixed experts;

[0011] (3) Inputting the visible light-infrared dataset into the visible light-infrared semantic segmentation model based on mixed experts and training the model to obtain a trained visible light-infrared semantic segmentation model;

[0012] (4) The visible light-infrared image pair to be segmented is input into the trained visible light-infrared semantic segmentation model, and the model outputs the semantic segmentation result.

[0013] The (1) further comprises:

[0014] Data enhancement processing is performed on the acquired visible light-infrared dataset to obtain the final visible light-infrared dataset.

[0015] In (2), the visible light-infrared semantic segmentation model based on hybrid experts includes a dual-branch encoder, a feature fusion module and a decoder; the aligned visible light image and infrared image pair are input into the visible light-infrared semantic segmentation model together, wherein the visible light image is passed through the visible light branch in the dual-branch image encoder to extract multi-scale visible light features, and the infrared image is passed through the infrared branch in the dual-branch image encoder to extract multi-scale infrared features; then the visible light features and infrared features of the same scale are input into a feature fusion module to obtain fused features of the same scale, and the multi-scale visible light features and the corresponding scale infrared features are traversed and fused respectively to obtain multi-scale fused features; finally, the multi-scale fused features are input into the decoder to obtain the final semantic segmentation result.

[0016] The feature fusion module includes a routing selector and a sparse hybrid expert network;

[0017] First, the visible light features and infrared features of the same scale are spliced ​​together in the channel dimension to obtain the visible light-infrared features and input them into the routing selector; then, the routing selector determines the expert network that needs to be activated in the sparse hybrid expert network and the corresponding expert weights according to the visible light-infrared features; the activated expert network in the sparse hybrid expert network takes the visible light-infrared features as input, and weights the outputs of each activated expert network according to the expert weights to obtain the confidence map of the visible light and the confidence map of the infrared features; finally, the confidence map of the visible light is multiplied and fused with the input visible light features to obtain the visible light fusion features, and the confidence map of the infrared features is multiplied and fused with the input infrared features to obtain the infrared fusion features, and the visible light fusion features and the infrared fusion features are element-by-element added and fused to obtain the fusion features of the current scale.

[0018] In the routing selector, first, a dimension reduction module composed of a linear layer and a normalization layer is used to reduce the dimension of the visible light-infrared feature to obtain the reduced dimension visible light-infrared feature;

[0019] Next, the reduced visible light-infrared features are passed through the average pooling layer and the maximum pooling layer respectively to obtain the average pooling output feature And the maximum pooling output These two features are passed through a multi-layer perceptron with shared weights respectively, and then the two outputs of the multi-layer perceptron are added and normalized by the Sigmoid function to obtain the initial expert weights of all expert networks in the sparse mixture of expert networks;

[0020] Then, the expert network corresponding to the largest k values ​​in the initial expert weights is used as the expert network that needs to be activated, and then the expert weights corresponding to the expert network that needs to be activated are generated.

[0021] The decoder is a Mask2Former decoder.

[0022] In (3), the ADAMW optimization algorithm is used for training during the model training process.

[0023] 2. A Visible-Infrared Semantic Segmentation System Based on Hybrid Experts

[0024] A visible light-infrared data set input module is used to obtain a visible light-infrared data set;

[0025] A semantic segmentation module is used to train a visible light-infrared semantic segmentation model based on a hybrid expert using the acquired visible light-infrared data set, obtain the trained visible light-infrared semantic segmentation model and store it;

[0026] The semantic segmentation result output module is used to input the visible light-infrared image pair to be segmented into the trained visible light-infrared semantic segmentation model, and output the semantic segmentation result output by the model to the terminal device.

[0027] 3. A computer device

[0028] The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the visible light-infrared semantic segmentation method based on mixed experts when executing the computer program.

[0029] 4. A computer-readable storage medium

[0030] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visible light-infrared semantic segmentation method based on mixed experts are implemented.

[0031] V. A computer program product

[0032] The product includes a computer program / instruction, which, when executed by a processor, implements the steps of the hybrid expert-based visible light-infrared semantic segmentation method.

[0033] Compared with the existing methods, the present invention has the following beneficial effects:

[0034] 1. The present invention proposes an effective visible light-infrared feature fusion method (ie), which can adapt to complex and changeable application scenarios and has stronger robustness.

[0035] 2. The present invention introduces channel attention in the routing selector, which can better select appropriate expert networks for different visible light-infrared image pairs, thereby improving the generalization performance of the model when dealing with different scenarios.

[0036] 3. The present invention uses a sparse hybrid expert network to improve the model's expressiveness while only increasing a small amount of computation; it can effectively fuse visible light and infrared features to obtain more accurate segmentation results. The hybrid expert-based feature fusion module designed by the present invention is sparse, and while improving the model's expressiveness, only increasing a small amount of computational overhead, it achieves a good balance between the model's effect and reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of an embodiment of the present invention.

[0038] Figure 2 Schematic diagram of a network model in an embodiment of the present invention.

[0039] Figure 3Schematic diagram of a feature fusion module used in a network model in an embodiment of the present invention.

[0040] Figure 4 Schematic diagram of a routing selector used in a feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0042] The embodiments and specific implementation process of the present invention are as follows:

[0043] like Figure 1 As shown, the present invention proposes a visible light-infrared semantic segmentation method based on mixed experts, comprising the following steps:

[0044] (1) Obtain a visible light-infrared dataset and perform data enhancement processing on the image, including random flipping, brightness transformation, contrast transformation, and saturation transformation. Specifically: the visible light image and infrared image are randomly flipped with a probability of 50%; the brightness, contrast, and saturation of the two images are multiplied by a uniformly sampled random number between 0.5 and 1.5.

[0045] (2) Construct a visible light-infrared semantic segmentation model based on mixed experts, such as Figure 2 As shown;

[0046] The visible light-infrared semantic segmentation model based on hybrid experts includes a dual-branch encoder, a feature fusion module and a decoder; the aligned visible light image and infrared image pair are input into the visible light-infrared semantic segmentation model together, wherein the visible light image is passed through the visible light branch in the dual-branch image encoder to extract multi-scale visible light features, and the multi-scale infrared features are extracted after passing through the infrared branch in the dual-branch image encoder; then the visible light features and infrared features of the same scale are input into a feature fusion module to obtain the fused features of the same scale, and the multi-scale visible light features and the corresponding scale infrared features are traversed and fused respectively to obtain the multi-scale fused features; finally, the multi-scale fused features are input into the decoder to obtain the final semantic segmentation result.

[0047] The role of the dual-branch image encoder is to extract multi-scale visible light and infrared image features. In general, an image encoder can be any network that outputs C×H×W image features. Since the semantic segmentation task relies on multi-scale feature representation, the present invention selects Swin Transformer as the backbone network and constructs a dual-branch image encoder. The two-branch Swin Transformer accepts visible light images and infrared images as inputs, respectively. The input image is first divided into multiple small non-overlapping patches, which serve as the basic processing units of the model. Each patch is flattened and converted into a series of tokens through a linear embedding layer, which are then sent to the Transformer model for processing. The model is processed by layer-by-layer Transformer blocks, gradually reducing the scale of the feature map while increasing its channel dimension, thereby gradually building a deeper feature representation. Finally, the model outputs visible light and infrared feature maps with N=4 different scales, represented as

[0048] The function of the feature fusion module is to use the complementary information of visible light and infrared images to fuse the features of the two. Figure 3 As shown, the feature fusion module includes a routing selector and a sparse hybrid expert network (i.e., a sparse hybrid expert network);

[0049] First, the visible light features and infrared features of the same scale are spliced ​​together in the channel dimension to obtain the visible light-infrared features and input them into the routing selector; then, the routing selector determines the expert network that needs to be activated in the sparse hybrid expert network and the corresponding expert weights according to the visible light-infrared features; the activated expert network in the sparse hybrid expert network takes the visible light-infrared features as input, and weights the outputs of each activated expert network according to the expert weights to obtain the confidence map of the visible light and the confidence map of the infrared features; finally, the confidence map of the visible light is multiplied and fused with the input visible light features to obtain the visible light fusion features, and the confidence map of the infrared features is multiplied and fused with the input infrared features to obtain the infrared fusion features, and the visible light fusion features and the infrared fusion features are element-by-element added and fused to obtain the fusion features of the current scale.

[0050] The role of the router is to determine which experts need to be activated and calculate the expert weights. Since directly calculating high-dimensional concatenated features will bring a large number of unnecessary parameters and calculations, Figure 4 As shown in the figure, in the router, first, a dimension reduction module R(·) consisting of a linear layer and a normalization layer is used to reduce the dimension of the visible light-infrared feature to obtain the reduced dimension visible light-infrared feature As shown below:

[0051]

[0052] Among them, Cat(·) represents the visible light feature F i rgb and infrared signature F i thr splicing in the channel dimension;

[0053] Next, the reduced visible light-infrared features are passed through the average pooling layer and the maximum pooling layer respectively to obtain the compressed average pooling output features And the maximum pooling output These two features are passed through a multi-layer perceptron with shared weights respectively, and then the two outputs of the multi-layer perceptron are added and normalized by the Sigmoid function to obtain the initial expert weights of all expert networks in the sparse mixed expert network; the calculation process is as follows:

[0054]

[0055] Where σ(·) represents the Sigmoid function, AvgPool(·) and MaxPool(·) represent average pooling and maximum pooling respectively; MLP(·) represents a multi-layer perceptron, which consists of two linear layers W down , W up and an activation layer, the activation layer uses the ReLU activation function, the first linear layer W down The input dimension is The number of channels of the second linear layer W up The input dimension of is the number of hidden layer channels, and the output dimension is the number of experts.

[0056] Then, the expert network corresponding to the largest k values ​​in the initial expert weights is used as the expert network that needs to be activated, k is a preset integer, and the remaining weight values ​​are set to negative infinity, and the experts corresponding to these values ​​will not be activated. Then, the expert weights corresponding to the expert network that needs to be activated are generated. Specifically, the Softmax function is applied to the selected top-k expert weights to obtain a probability distribution G with a sum of 1. These probabilities will be used as the weights of the sparse hybrid expert network. The calculation process is as follows:

[0057] G = Softmax(Topk(W))

[0058] Among them, Topk(·) means that the k weights with the largest values ​​are retained, and the corresponding experts will be activated; the weights of the remaining values ​​are set to negative infinity, so that after passing through the Softmax function, the weights will become 0, and the corresponding experts will not be activated.

[0059] The role of the sparse hybrid expert network is to generate confidence maps of complementary visible light and infrared features for feature fusion. The sparse hybrid expert network consists of several expert modules. Each expert module receives the visible light-infrared feature as input, and after the linear layer, RELU activation function and Softmax function, it outputs a pair of complementary visible light and infrared feature confidence maps for feature fusion.

[0060] Each activated expert calculates its output independently. Then, according to the expert weight output by the routing selector, the output of each activated expert is weighted and summed to obtain the final confidence map. This process only forwards the top k experts with the largest weights, avoiding unnecessary multiplication operations, so that the calculation of the entire expert network is sparse and the calculation cost is low. The calculation process is as follows:

[0061]

[0062] Among them, E j and G j represents the jth expert and its routing weight.

[0063] Finally, the confidence map and Multiply the visible light feature and infrared feature respectively, and then add them element by element to get the fused feature.

[0064] The decoder is a Mask2Former decoder. It takes multi-scale fusion features and a series of learnable query vectors as input. The query vectors and the multi-scale fusion features undergo multiple self-attention and cross-attention operations, so that the query can capture information from different regions in the image. These query vectors are processed through a multi-layer perceptron to predict the category of each region; at the same time, they are multiplied with the upsampled image features to predict the mask of each region.

[0065] (3) The visible light and infrared image pairs in the visible light-infrared dataset are input into the visible light-infrared semantic segmentation model based on mixed experts in a small batch training mode, and the model is trained using the ADAMW optimization algorithm, and the trained model parameters are saved to obtain the trained visible light-infrared semantic segmentation model;

[0066] (4) The visible light-infrared image pair to be segmented is input into the trained visible light-infrared semantic segmentation model, and the model outputs the semantic segmentation result.

[0067] In order to verify the effectiveness of the present invention, the present invention is experimentally verified on MFNet, the most representative visible light-infrared dataset in the field, and compared with the current cutting-edge visible light-infrared semantic segmentation method.

[0068] The MFNet dataset has a total of 784 pairs of visible-infrared images for training, 392 pairs for validation, and 393 pairs for testing. The resolution of both visible and infrared images is 640×480.

[0069] This embodiment mainly uses the mean intersection over union (mIoU), which is the most commonly used in the visible light-infrared semantic segmentation task, as the main indicator to evaluate the network performance. Its calculation method is:

[0070]

[0071] Where n is the number of categories, P ij represents the number of pixels that actually belong to category i but are predicted by the model to be category j, P ii represents the number of pixels that actually belong to category i but are predicted by the model to be category i, P ji Represents the number of pixels that actually belong to category j but are predicted by the model to be category i.

[0072] In order to achieve better performance of the network, the parameters in training are set as follows: the parameters of the ADAMW optimizer are set to (0.9, 0.99); the learning rate adjustment method is the cosine annealing method with warm-up, the initial learning rate is 0.0001, the warm-up coefficient is set to 0.1, and the number of warm-up rounds is set to 500; the total number of training rounds is 35,000, and the batch size is 15; the weight decay is set to 0.05; the backbone network is initialized using the pre-trained weights on ImageNet-1K.

[0073] Table 1 shows the comparison results of the network model of the present invention and other published advanced methods on the MFNet dataset. From the comparison results, it can be found that the present invention achieves the best performance in terms of the mIoU indicator.

[0074] Table 1 shows the comparison of the indicators of the present invention and the published advanced methods on the MFNet dataset.

[0075]

[0076]

[0077] The present invention also proposes a visible light-infrared semantic segmentation system based on hybrid experts, comprising:

[0078] A visible light-infrared data set input module is used to obtain a visible light-infrared data set;

[0079] A semantic segmentation module is used to train a visible light-infrared semantic segmentation model based on a hybrid expert using the acquired visible light-infrared data set, obtain the trained visible light-infrared semantic segmentation model and store it;

[0080] The semantic segmentation result output module is used to input the visible light-infrared image pair to be segmented into the trained visible light-infrared semantic segmentation model, and output the semantic segmentation result output by the model to the terminal device.

[0081] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a visible light-infrared semantic segmentation method based on mixed experts are implemented.

[0082] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a visible light-infrared semantic segmentation method based on mixed experts are implemented.

[0083] The present invention also proposes a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of a visible light-infrared semantic segmentation method based on mixed experts.

[0084] The above description is only a specific embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent example of equivalent changes. However, any simple modification, equivalent change and modification made to the above examples based on the technical essence of the present invention without departing from the technical solution of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A visible light-infrared semantic segmentation method based on mixed experts, characterized in that: The following steps are involved: (1) Obtaining a visible-infrared dataset; (2) Construct a visible light-infrared semantic segmentation model based on hybrid experts; (3) Inputting the visible light-infrared dataset into the visible light-infrared semantic segmentation model based on mixed experts and training the model to obtain a trained visible light-infrared semantic segmentation model; (4) The visible light-infrared image pair to be segmented is input into the trained visible light-infrared semantic segmentation model, and the model outputs the semantic segmentation result.

2. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 1, characterized in that: The (1) further comprises: Data enhancement processing is performed on the acquired visible light-infrared dataset to obtain the final visible light-infrared dataset.

3. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 1, characterized in that: In (2), the visible light-infrared semantic segmentation model based on hybrid experts includes a dual-branch encoder, a feature fusion module and a decoder; the aligned visible light image and infrared image pair are input into the visible light-infrared semantic segmentation model together, wherein the visible light image is passed through the visible light branch in the dual-branch image encoder to extract multi-scale visible light features, and the infrared image is passed through the infrared branch in the dual-branch image encoder to extract multi-scale infrared features; then the visible light features and infrared features of the same scale are input into a feature fusion module to obtain fused features of the same scale, and the multi-scale visible light features and the corresponding scale infrared features are traversed and fused respectively to obtain multi-scale fused features; finally, the multi-scale fused features are input into the decoder to obtain the final semantic segmentation result.

4. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 3, characterized in that: The feature fusion module includes a routing selector and a sparse hybrid expert network; First, the visible light features and infrared features of the same scale are concatenated in the channel dimension to obtain the visible light-infrared features and input them into the router. Then, the routing selector determines the expert network that needs to be activated in the sparse hybrid expert network and the corresponding expert weights according to the visible light-infrared features; the activated expert network in the sparse hybrid expert network takes the visible light-infrared features as input, and weights the outputs of each activated expert network according to each expert weight to obtain the confidence map of visible light and the confidence map of infrared features; finally, the confidence map of visible light is multiplied and fused with the input visible light features to obtain the visible light fusion feature, the confidence map of infrared features is multiplied and fused with the input infrared features to obtain the infrared fusion feature, and the visible light fusion feature and the infrared fusion feature are added and fused element by element to obtain the fusion feature of the current scale.

5. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 4, characterized in that: In the routing selector, first, a dimension reduction module composed of a linear layer and a normalization layer is used to reduce the dimension of the visible light-infrared feature to obtain the reduced dimension visible light-infrared feature; Next, the reduced visible light-infrared features are passed through the average pooling layer and the maximum pooling layer respectively to obtain the average pooling output feature And the maximum pooling output These two features are passed through a multi-layer perceptron with shared weights respectively, and then the two outputs of the multi-layer perceptron are added and normalized by the Sigmoid function to obtain the initial expert weights of all expert networks in the sparse mixture of expert networks; Then, the expert network corresponding to the largest k values ​​in the initial expert weights is used as the expert network that needs to be activated, and then the expert weights corresponding to the expert network that needs to be activated are generated.

6. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 3, characterized in that: The decoder is a Mask2Former decoder.

7. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 1, characterized in that: In (3), the ADAMW optimization algorithm is used for training during the model training process.

8. A visible light-infrared semantic segmentation system based on hybrid experts, characterized in that: include: A visible light-infrared data set input module is used to obtain a visible light-infrared data set; A semantic segmentation module is used to train a visible light-infrared semantic segmentation model based on a hybrid expert using the acquired visible light-infrared data set, obtain the trained visible light-infrared semantic segmentation model and store it; The semantic segmentation result output module is used to input the visible light-infrared image pair to be segmented into the trained visible light-infrared semantic segmentation model, and output the semantic segmentation result output by the model to the terminal device.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the visible light-infrared semantic segmentation method based on mixed experts described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visible light-infrared semantic segmentation method based on mixed experts described in any one of claims 1 to 7 are implemented.

Images