A visible-infrared semantic segmentation method based on hybrid experts

Abstract

The application discloses a visible light-infrared semantic segmentation method based on a mixed expert. The application establishes a neural network model based on a mixed expert, can effectively fuse visible light and infrared features, and thus obtains more accurate segmentation results. The application proposes a routing selector with channel attention, which can adaptively select a suitable expert network for different visible light-infrared image pairs, thereby improving the generalization performance of the model when dealing with different scenes. In addition, the feature fusion module based on the mixed expert designed by the application has sparsity, improves the expression ability of the model, only increases a small amount of calculation overhead, and achieves a good balance between the effect of the model and the inference speed. Experimental results prove that the application achieves advanced performance on the widely used MFNet dataset.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a visible light-infrared semantic segmentation method and system based on a hybrid expert. BACKGROUND

[0002] In the field of computer vision and image processing, semantic segmentation based on visible light and infrared image fusion is an important and active research direction. With the development of technology, more and more application fields require image processing systems that can work stably under different environmental conditions. These fields include autonomous driving, security monitoring, military reconnaissance, search and rescue, and industrial detection, etc. In these applications, the technology of effectively fusing visible light and infrared images will greatly improve the reliability and accuracy of the system. For example, under night or bad weather conditions, a visible light camera may not be able to capture clear images, while a thermal imager can detect image information due to temperature differences; conversely, visible light images can supplement the lack of detail information and color information of infrared images, thereby producing a more detailed semantic segmentation result.

[0003] How to utilize the complementarity of visible light and infrared images to improve the effect of semantic segmentation has always been the focus of research in the field of RGB-T semantic segmentation.

[0004] Deng et al. introduced a feature-enhanced attention module in "FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation" to mine and enhance multi-level features from both channel and spatial perspectives, thereby preserving spatial information, refining detailed features, and shifting more attention to the fused high-resolution features. Zhou et al. proposed a spatial attention mechanism module in "MFFENet: Multiscale Feature Fusion and Enhancement Network for RGB–Thermal Urban Road Scene Parsing" that gives higher weights to foreground regions and introduces multi-label supervision to optimize model parameters. Zhang et al. used a bidirectional image translation-based approach to reduce the modal difference between visible and infrared features in "ABMDRNet: Adaptive-weighted Bi-directional Modality Difference Reduction Network for RGB-T Semantic Segmentation" and then adaptively selected discriminative multimodal features for semantic segmentation through channel-weighted fusion. However, these methods often design complex attention-based feature fusion modules, significantly increasing the computational overhead of the model.

[0005] In "Spatial-aware Demand-guided Recursive Meshing for RGB-TSemantic Segmentation," Fan et al. proposed a spatially aware demand-guided recursive network. Using a demand-guided target masking algorithm, it actively compensates for insufficient contextual semantics in optically damaged regions and refines the fused semantic features through recursive meshing. In "Complementary Random Masking for RGB-Thermal Semantic Segmentation," Shin et al. proposed a complementary masking strategy for visible and infrared images, encouraging the network to learn complementary feature representations from a single modality or complementary masked modalities, improving pixel-level semantic segmentation performance. However, these methods do not consider the model's generalization capabilities across different scenarios. Summary of the Invention

[0006] In order to solve the problems in the background art, the application provides a visible-infrared semantic segmentation method and system based on hybrid experts. The application performs feature fusion between visible light and infrared modalities through a multi-modal fusion module based on hybrid experts, selects the most suitable fusion mode for different visible-infrared image pairs, and thus adapts to complex and variable application scenarios.

[0007] The technical scheme adopted by the application is:

[0008] One kind is based on hybrid expert's visible-infrared semantic segmentation method

[0009] (1) obtain a visible-infrared dataset;

[0010] (2) construct a visible-infrared semantic segmentation model based on hybrid experts;

[0011] (3) input the visible-infrared dataset into the visible-infrared semantic segmentation model based on hybrid experts, and train the model to obtain a trained visible-infrared semantic segmentation model;

[0012] (4) input the visible-infrared image pair to be segmented into the trained visible-infrared semantic segmentation model, and the model outputs the semantic segmentation result.

[0013] The (1) further comprises:

[0014] The obtained visible-infrared dataset is subjected to data enhancement processing, thereby obtaining a final visible-infrared dataset.

[0015] In the (2), the visible-infrared semantic segmentation model based on hybrid experts comprises a double-branch encoder, a feature fusion module and a decoder; the aligned visible light image and infrared image pair are input into the visible-infrared semantic segmentation model, wherein the visible light image extracts multi-scale visible light features after passing through the visible light branch in the double-branch image encoder, and extracts multi-scale infrared features after passing through the infrared branch in the double-branch image encoder; then the visible light features and the infrared features of the same scale are input into a feature fusion module to obtain the fusion features of the same scale, and after the multi-scale visible light features and the corresponding scale infrared features are subjected to traversal fusion processing, the multi-scale fusion features are obtained; finally, the multi-scale fusion features are input into the decoder to obtain the final semantic segmentation result.

[0016] The feature fusion module comprises a router selector and a sparse hybrid expert network.

[0017] Firstly, the visible light features and infrared features of the same scale are spliced in the channel dimension to obtain visible-infrared features and input into the router; then, the router determines the expert networks that need to be activated and the corresponding expert weights in the sparse mixed expert network according to the visible-infrared features; the activated expert networks in the sparse mixed expert network take the visible-infrared features as input, and obtain the confidence map of the visible light features and the confidence map of the infrared features by weighting the outputs of the activated expert networks according to the expert weights; finally, the visible light fusion features are obtained by multiplying the confidence map of the visible light features and the input visible light features, the infrared fusion features are obtained by multiplying the confidence map of the infrared features and the input infrared features, and the fusion features of the current scale are obtained by element-wise addition of the visible light fusion features and the infrared fusion features.

[0018] In the router, firstly, the visible-infrared features are reduced in dimension using a dimension reduction module composed of a linear layer and a normalization layer to obtain reduced visible-infrared features;

[0019] Then, the reduced visible-infrared features pass through an average pooling layer and a maximum pooling layer respectively to obtain average pooling output features and maximum pooling output The two features pass through a shared weight multi-layer perceptron respectively, and then the outputs of the multi-layer perceptron are added and normalized by a Sigmoid function to obtain the initial expert weights of all expert networks in the sparse mixed expert network.

[0020] Then, the expert networks corresponding to the largest k values in the initial expert weights are taken as the expert networks that need to be activated, and the expert weights corresponding to the expert networks that need to be activated are generated.

[0021] The decoder is a Mask2Former decoder.

[0022] In the (3), the ADAMW optimization algorithm is used for training in the training process of the model.

[0023] II. A visible-infrared semantic segmentation system based on mixed experts

[0024] A visible-infrared dataset input module for obtaining a visible-infrared dataset;

[0025] A semantic segmentation module for training a visible-infrared semantic segmentation model based on mixed experts using the obtained visible-infrared dataset, obtaining a trained visible-infrared semantic segmentation model and storing it;

[0026] The semantic segmentation result output module is configured to input the visible light-infrared image pair to be segmented into the trained visible light-infrared semantic segmentation model, and output the semantic segmentation result output by the model to a terminal device.

[0027] Three, a computer device

[0028] The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the mixed-expert-based visible light-infrared semantic segmentation method when executing the computer program.

[0029] Four, a computer readable storage medium

[0030] The computer readable storage medium stores a computer program, and the computer program implements the steps of the mixed-expert-based visible light-infrared semantic segmentation method when executed by a processor.

[0031] Five, a computer program product

[0032] The product comprises a computer program / instruction, which implements the steps of the mixed-expert-based visible light-infrared semantic segmentation method when executed by a processor.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1. The present application proposes an effective visible light-infrared feature fusion method (i.e., ), which can adapt to complex and variable application scenarios and has stronger robustness.

[0035] 2. The present application introduces channel attention in the router, which can better select appropriate expert networks for different visible light-infrared image pairs, thereby improving the generalization performance of the model in dealing with different scenes.

[0036] 3. The present application uses sparse mixed expert networks to improve the expression ability of the model while only increasing a small amount of calculation; it can effectively fuse visible light and infrared features to obtain more accurate segmentation results. The mixed-expert-based feature fusion module designed in the present application has sparsity, which improves the expression ability of the model while only increasing a small amount of calculation overhead, achieving a good balance between the effect of the model and the inference speed. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of an embodiment of the present application.

[0038] Figure 2 is a schematic diagram of a network model in an embodiment of the present application.

[0039] Figure 3is a schematic diagram of a feature fusion module used in a network model in an embodiment of the present application.

[0040] Figure 4 is a schematic diagram of a router used in a feature fusion module in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0042] Embodiments of the present application and the specific implementation process are as follows:

[0043] As shown in Figure 1 , the present application proposes a visible-infrared semantic segmentation method based on mixed experts, including the following steps:

[0044] (1) Obtain a visible-infrared dataset, and perform data enhancement processing on the images, including random flipping, brightness transformation, contrast transformation, and saturation transformation. Specifically: flip the visible light image and the infrared image randomly with a probability of 50%; multiply the brightness, contrast, and saturation of the two images by a uniformly sampled random number between 0.5 and 1.5.

[0045] (2) Construct a visible-infrared semantic segmentation model based on mixed experts, as shown in Figure 2 .

[0046] The visible-infrared semantic segmentation model based on mixed experts includes a double-branch encoder, a feature fusion module, and a decoder; the aligned visible light image and infrared image pair are input into the visible-infrared semantic segmentation model, wherein the visible light image extracts multi-scale visible light features after passing through the visible light branch in the double-branch image encoder, and extracts multi-scale infrared features after passing through the infrared branch in the double-branch image encoder; then the same scale visible light features and infrared features are input into a feature fusion module to obtain the same scale fusion features, and after traversing fusion processing of the multi-scale visible light features and the corresponding scale infrared features, multi-scale fusion features are obtained; finally, the multi-scale fusion features are input into the decoder to obtain the final semantic segmentation result.

[0047] The dual-branch image encoder is used to extract multiscale visible and infrared image features. In general, the image encoder can be any network that outputs image features of CxHxW. Since the semantic segmentation task relies on multiscale feature representation, the Swin Transformer is selected as the backbone network to construct the dual-branch image encoder. The Swin Transformers of the two branches respectively accept the visible light image and the infrared image as input. The input image is first divided into a plurality of small non-overlapping patches, which are used as the basic processing unit of the model. Each patch is flattened and converted into a series of tokens by a linear embedding layer, which are then sent into the Transformer model for processing. The model gradually reduces the scale of the feature map while increasing its channel dimension through layer-by-layer Transformer block processing, thereby gradually constructing deeper feature representation. Finally, the model outputs visible light and infrared feature maps with N = 4 different scales, respectively represented as

[0048] The feature fusion module is used to fuse the features of the visible light and infrared images by utilizing the complementary information of the two images. As shown in Figure 3 , the feature fusion module includes a router and a sparse mixing expert network (i.e., a sparse mixing expert network).

[0049] First, the visible light features and the infrared features of the same scale are concatenated in the channel dimension to obtain visible-infrared features and input into the router. Then, the router determines the expert networks that need to be activated in the sparse mixing expert network and the corresponding expert weights according to the visible-infrared features. The activated expert networks in the sparse mixing expert network take the visible-infrared features as input, and the outputs of the activated expert networks are weighted according to the expert weights to obtain the confidence map of the visible light features and the confidence map of the infrared features. Finally, the confidence map of the visible light features is multiplied with the input visible light features to obtain the visible light fusion features, the confidence map of the infrared features is multiplied with the input infrared features to obtain the infrared fusion features, and the visible light fusion features and the infrared fusion features are element-wise added to obtain the fusion features of the current scale.

[0050] The role of the router is to determine which experts need to be activated and calculate the expert weights. Since directly calculating the high-dimensional concatenated features will bring a large amount of unnecessary parameters and computational load, as shown in Figure 4 , in the router, first, a dimension reduction module R(·) composed of a linear layer and a normalization layer is used to reduce the dimension of the visible-infrared features to obtain the reduced visible-infrared features As shown in the following formula:

[0051]

[0052] Among them, Cat(·) represents the visible light feature F i rgb and infrared signature F i thr splicing in the channel dimension;

[0053] Then, the reduced visible light-infrared features are passed through the average pooling layer and the maximum pooling layer respectively to obtain the compressed average pooling output features and max pooling output These two features are passed through a multi-layer perceptron with shared weights respectively, and then the two outputs of the multi-layer perceptron are added and normalized by the Sigmoid function to obtain the initial expert weights of all expert networks in the sparse mixture of expert networks; the calculation process is as follows:

[0054]

[0055] Where σ(·) represents the Sigmoid function, AvgPool(·) and MaxPool(·) represent average pooling and maximum pooling respectively; MLP(·) represents a multi-layer perceptron, which consists of two linear layers W down 、W up and an activation layer, the activation layer uses the ReLU activation function, the first linear layer W down The input dimension is The number of channels of the second linear layer W up The input dimension is the number of hidden layer channels, and the output dimension is the number of experts.

[0056] Then, the expert networks corresponding to the largest k values ​​in the initial expert weights are used as the expert networks that need to be activated, where k is a preset integer. At the same time, the remaining weight values ​​are set to negative infinity, and the experts corresponding to these values ​​will not be activated. Then, the expert weights corresponding to the expert networks that need to be activated are generated. Specifically, the Softmax function is applied to the selected top-k expert weights to obtain a probability distribution G with a sum of 1. These probabilities will be used as the weights of the sparse hybrid expert network. The calculation process is as follows:

[0057] G = Softmax(Topk(W))

[0058] Among them, Topk(·) means that the k weights with the largest values ​​are retained, and the corresponding experts will be activated; the weights of the remaining values ​​are set to negative infinity, so that after passing through the Softmax function, the weights will become 0, and the corresponding experts will not be activated.

[0059] The role of the sparse hybrid expert network is to generate complementary confidence maps of visible light and infrared features for feature fusion. The sparse hybrid expert network is composed of several expert modules. Each expert module receives visible light-infrared features as input, and after passing through a linear layer, a RELU activation function and a Softmax function, outputs a pair of complementary confidence maps of visible light and infrared features for feature fusion.

[0060] Each activated expert independently calculates its output. Then, according to the expert weights output by the router, the outputs of the activated experts are weighted and summed to obtain the final confidence map. This process only forwards the top k experts with the largest weights, avoiding unnecessary multiplication operations, so that the entire expert network calculation is sparse and the calculation cost is low. The calculation process is as follows:

[0061]

[0062] wherein E j and G j represent the jth expert and its routing weight.

[0063] Finally, the confidence map and are multiplied by the visible light feature and the infrared feature respectively, and then element-wise addition is performed, to obtain the fused feature

[0064] The decoder is the Mask2Former decoder. The multi-scale fused features and a series of learnable query vectors are input, and the query vectors and the multi-scale fused features are subjected to multiple self-attention and cross-attention operations, so that the query can capture the information of different regions in the image. These query vectors are processed through a multi-layer perceptron to predict the class of each region; at the same time, they are multiplied by the upsampled image features to predict the mask of each region.

[0065] (3) The visible light and infrared picture pairs in the visible light-infrared dataset are input into the hybrid expert-based visible light-infrared semantic segmentation model in a small batch training manner, and the ADAMW optimization algorithm is used to train the model. Save the trained model parameters to obtain the trained visible light-infrared semantic segmentation model;

[0066] (4) The visible light-infrared image pair to be segmented is input into the trained visible light-infrared semantic segmentation model, and the model outputs the semantic segmentation result.

[0067] In order to verify the effectiveness of the present application, the present application is experimentally verified on the most representative visible-infrared dataset MFNet in the field, and compared with the current advanced visible-infrared semantic segmentation method.

[0068] The MFNet dataset has a total of 784 pairs of visible-infrared images for training, 392 pairs for verification, and 393 pairs for testing. The resolution of the visible light and infrared images is 640x480.

[0069] The present embodiment mainly takes the most commonly used mean intersection over union (mIoU) in the visible-infrared semantic segmentation task as the main index to evaluate the network performance. The calculation method is as follows:

[0070]

[0071] Wherein, n is the number of categories, P ij represents the number of pixels actually belonging to category i but predicted by the model as category j, P ii represents the number of pixels actually belonging to category i but predicted by the model as category i, P ji represents the number of pixels actually belonging to category j but predicted by the model as category i.

[0072] In order to make the network achieve better performance, the parameter settings in the training are as follows: the parameter settings of the ADAMW optimizer are (0.9, 0.99); the learning rate adjustment method is the cosine annealing method with warm-up, the initial learning rate is 0.0001, the warm-up coefficient is set to 0.1, and the warm-up round number is set to 500; the total training round number is 35000, and the batch size is 15; the weight decay is set to 0.05; the backbone network initialization adopts the pre-training weight on ImageNet-1K.

[0073] Table 1 shows the comparison results of the network model of the present application and other published advanced methods on the MFNet dataset. From the comparison results, it can be found that the present application achieves the optimal performance in the mIoU index.

[0074] Table 1 is the index comparison of the present application and the published advanced method on the MFNet dataset

[0075]

[0076]

[0077] The present application also proposes a visible-infrared semantic segmentation system based on mixed experts, comprising:

[0078] A visible-infrared dataset input module is used to obtain a visible-infrared dataset.

[0079] a semantic segmentation module, configured to train a hybrid-expert-based visible-infrared semantic segmentation model by using the acquired visible-infrared dataset, obtain a trained visible-infrared semantic segmentation model and store the trained visible-infrared semantic segmentation model;

[0080] a semantic segmentation result output module, configured to input a to-be-segmented visible-infrared image pair into the trained visible-infrared semantic segmentation model, and output a semantic segmentation result output by the model to a terminal device.

[0081] The application further provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the hybrid-expert-based visible-infrared semantic segmentation method when executing the computer program.

[0082] The application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the hybrid-expert-based visible-infrared semantic segmentation method when executed by a processor.

[0083] The application further provides a computer program product, which includes a computer program / instruction, and the computer program / instruction implements the steps of the hybrid-expert-based visible-infrared semantic segmentation method when executed by a processor.

[0084] The above is only a specific embodiment of the application, and does not limit the application in any form. Any person skilled in the art can modify or change the above disclosed technical content into equivalent examples, but any simple modification, equivalent change and modification within the technical solution content of the application and according to the technical essence of the application are still within the protection scope of the application.

Claims

1. A visible light-infrared semantic segmentation method based on hybrid experts, characterized in that: The following steps are involved: (1) Obtaining a visible-infrared dataset; (2) Constructing a visible-infrared semantic segmentation model based on hybrid experts; In (2), the visible light-infrared semantic segmentation model based on hybrid experts includes a dual-branch encoder, a feature fusion module and a decoder; the aligned visible light image and infrared image pair are input together into the visible light-infrared semantic segmentation model, wherein the visible light image is passed through the visible light branch in the dual-branch image encoder to extract multi-scale visible light features, and the visible light image is passed through the infrared branch in the dual-branch image encoder to extract multi-scale infrared features; then the visible light features and infrared features of the same scale are input into a feature fusion module to obtain fused features of the same scale, and the multi-scale visible light features and the infrared features of the corresponding scales are traversed and fused respectively to obtain multi-scale fused features; finally, the multi-scale fused features are input into the decoder to obtain the final semantic segmentation result; The feature fusion module includes a routing selector and a sparse hybrid expert network; First, the visible light features and infrared features of the same scale are concatenated in the channel dimension to obtain the visible light-infrared features and input them into the router. Then, the router determines the expert network that needs to be activated in the sparse hybrid expert network and the corresponding expert weights based on the visible light-infrared features. The activated expert network in the sparse hybrid expert network takes the visible light-infrared features as input, and weights the outputs of each activated expert network according to the expert weights to obtain the visible light confidence map and the infrared feature confidence map. Finally, the visible light confidence map is multiplied and fused with the input visible light features to obtain the visible light fusion feature. The infrared feature confidence map is multiplied and fused with the input infrared features to obtain the infrared fusion feature. The visible light fusion feature and the infrared fusion feature are then element-by-element added to obtain the fusion feature of the current scale. In the router, first, a dimension reduction module composed of a linear layer and a normalization layer is used to reduce the dimension of the visible light-infrared feature to obtain the reduced dimension visible light-infrared feature; Then, the reduced visible light-infrared features are passed through the average pooling layer and the maximum pooling layer respectively to obtain the average pooling output features and max pooling output , these two features are passed through a multi-layer perceptron with shared weights respectively, and then the two outputs of the multi-layer perceptron are added and normalized by the Sigmoid function to obtain the initial expert weights of all expert networks in the sparse mixture of experts network; Then, the expert network corresponding to the largest k values ​​in the initial expert weights is used as the expert network that needs to be activated, and then the expert weights corresponding to the expert network that needs to be activated are generated; (3) Input the visible light-infrared dataset into the visible light-infrared semantic segmentation model based on hybrid experts, and train the model to obtain a trained visible light-infrared semantic segmentation model; (4) The visible light-infrared image pair to be segmented is input into the trained visible light-infrared semantic segmentation model, and the model outputs the semantic segmentation result.

2. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 1, characterized in that: Said (1) further comprises: Data enhancement processing is performed on the obtained visible light-infrared dataset to obtain the final visible light-infrared dataset.

3. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 1, characterized in that: The decoder is a Mask2Former decoder.

4. The method for visible light-infrared semantic segmentation based on hybrid experts according to claim 1, characterized in that: In (3), the ADAMW optimization algorithm is used for training during the model training process.

5. A hybrid expert-based visible light-infrared semantic segmentation system for implementing the hybrid expert-based visible light-infrared semantic segmentation method according to any one of claims 1 to 4, characterized in that: include: Visible light-infrared dataset input module, used to obtain visible light-infrared dataset; A semantic segmentation module is used to train a visible light-infrared semantic segmentation model based on a hybrid expert using the acquired visible light-infrared dataset, obtain the trained visible light-infrared semantic segmentation model, and store it; The semantic segmentation result output module is used to input the visible light-infrared image pair to be segmented into the trained visible light-infrared semantic segmentation model, and output the semantic segmentation result output by the model to the terminal device.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the visible light-infrared semantic segmentation method based on hybrid experts described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visible light-infrared semantic segmentation method based on hybrid experts described in any one of claims 1 to 4 are implemented.

Images