A multi-modal image fusion recognition method and system based on multi-task collaborative learning

Abstract

The present invention discloses a multi-modal image fusion recognition method and system based on multi-task collaborative learning. The method of the present invention includes inputting a multi-modal image into a pre-trained multi-modal image fusion recognition network for image recognition to obtain a multi-modal image fusion recognition result, including: extracting a plurality of feature maps with different resolutions through an encoder; extracting feature maps of corresponding modalities and image fusion weights through an image fusion network, performing element-wise multiplication operations on the image fusion weights and the original images of the corresponding modalities and then adding them to generate a fused image; performing weight-guided feature fusion based on image fusion weights through a weight-guided feature fusion module to obtain fusion features of different resolutions; and decoding through a semantic segmentation decoder to obtain a multi-modal image fusion recognition result. The present invention aims to achieve a close combination of multi-modal image fusion and segmentation to simultaneously improve the performance of each, thereby achieving accurate estimation of the location of the target of interest and accurate recognition of the category.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-mode image fusion recognition method and system based on multi-task collaborative learning. Background Art

[0002] Images of different modalities can provide different information. For example, visible light images provide rich color and texture information, while thermal infrared images can capture the temperature information of objects. Thermal infrared images play an important role, especially at night or in poor lighting conditions. The fusion and segmentation of multi-source images such as visible light and thermal infrared images have become a research focus due to their excellent performance in complex environments, especially in poor lighting conditions. They are widely used in fields such as autonomous driving and robot perception. Existing methods are usually designed only for a single task. Specifically, methods are proposed for image fusion or strategies are designed for multimodal semantic segmentation. Existing multimodal semantic segmentation methods focus more on the design of multimodal feature fusion strategies and the use of edge information. For example, EGFNet (Zhou W, Dong S, Xu C, et al. Edge-aware guidance fusion network for rgb–thermal scene parsing [C]. Proceedings of the AAAI conference on artificial intelligence. 2022, 36 (3): 3571-3579.) uses the prior edge map to embed edge information into the feature map to generate a high-quality edge prediction map, and proposes a multimodal fusion module to explore the effectiveness and complementarity between the two modal features. SFAF-MA (He X, Wang M, Liu T, et al. SFAF-MA: Spatial feature aggregation and fusion with modality adaptation for RGB-thermal semantic segmentation [J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-10.) introduces a modality difference adaptive fusion module, which uses the corresponding weights generated by attention to adaptively fuse the features of multimodal images.SGFNet (Wang Y, Li G, Liu Z. SGFNet: semantic-guided fusion network for rgb-thermal semantic segmentation [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33 (12): 7737-7748.) proposes a multimodal coordination and distillation unit to fuse multimodal features, and uses cross-layer and semantic-guided enhancement units to enhance the cross-layer information and semantic information of the fused features to improve the segmentation performance. LASNet (Li G, Wang Y, Liu Z, et al. RGB-T semantic segmentation with location, activation, and sharpening [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 33 (3): 1223-1235.) processes the features at different levels differently according to the different characteristics of the features extracted by the encoder, and uses high, medium, and low-level fusion modules to locate potential objects, activate target areas, and refine boundaries. Existing image fusion methods mainly use deep learning techniques, such as convolutional neural networks and generative adversarial networks. For example, DDcGAN (Ma J, Xu H, Jiang J, et al. DDcGAN: A dual-discriminator conditional generative adversarial networkfor multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.) uses a dual-discriminator generative adversarial network, one for discriminating between fused images and visible light images, and the other for discriminating between fused images and infrared images, to improve the quality of multimodal image fusion.APWNet (Zhang X, Zhai H, Liu J, et al. Real-time infrared and visible image fusion network using adaptive pixel weighting strategy [J]. Information Fusion, 2023, 99: 101863.) uses convolutional neural networks to build a lightweight multimodal image fusion network, and uses a pixel-based fusion strategy to adaptively learn the weights of image fusion pixel by pixel to achieve efficient and accurate image fusion. Some works combine image fusion and segmentation tasks, such as SEAFusion (Tang L, Yuan J, Ma J. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network [J]. Information Fusion, 2022, 82: 28-42.) uses semantic loss to guide high-level semantic information to be injected into the fused image, effectively improving the performance of high-level vision tasks on the fused image. Although many advanced methods have been proposed for multimodal image fusion and segmentation, there are still some challenges: the two tasks of fusion and segmentation are only constrained by loss functions, lacking close interaction and making it difficult to improve the performance of both tasks at the same time. How to closely combine the two tasks to improve their respective performance at the same time has become a key technical problem to be solved. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a multimodal image fusion recognition method and system based on multi-task collaborative learning are provided. The present invention aims to achieve a close combination of multimodal image fusion and segmentation to simultaneously improve the performance of each, thereby achieving accurate estimation of the location of the target of interest and precise identification of the category.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A multimodal image fusion recognition method based on multi-task collaborative learning comprises inputting a multimodal image into a pre-trained multimodal image fusion recognition network for image recognition to obtain a multimodal image fusion recognition result, wherein the multimodal image fusion recognition network is composed of an encoder, an image fusion network, a weight-guided feature fusion module and a semantic segmentation decoder, and the multimodal image fusion recognition network performs image recognition to obtain the multimodal image fusion recognition result, comprising: extracting features of original images of each modality in the multimodal image by the encoder to obtain a plurality of feature maps with different resolutions; extracting feature maps of corresponding modalities by the image fusion feature extractor from original images of each modality in the multimodal image by the image fusion network, adding the feature maps of each modality, decoding by the image fusion decoder, and obtaining image fusion weights of corresponding modalities by a Sigmoid activation function, performing element-wise multiplication operation on the image fusion weights and the original images of the corresponding modalities, and then adding them to generate a fused image; performing weight-guided feature fusion on multiple feature maps of different resolutions of each modality based on the image fusion weights by the weight-guided feature fusion module to obtain fusion features with different resolutions; and aggregating fusion features of different resolutions by the semantic segmentation decoder to obtain the multimodal image fusion recognition result.

[0006] Optionally, the extracting features of the original images of each modality in the multimodal image using the backbone network to obtain multiple feature maps of different resolutions means extracting features of the original images of each modality in the multimodal image using the Segformer backbone network to obtain multiple feature maps of different resolutions, and the original images of different modalities use independent Segformer backbone networks that do not share network parameters.

[0007] Optionally, the image fusion feature extractor is composed of a convolution layer, a linear activation function layer and another convolution layer connected in sequence.

[0008] Optionally, the image fusion decoder is composed of two convolutional layers connected in sequence.

[0009] Optionally, the original images of each modality in the multimodal image include original images R and T of different modalities; when the weight-guided feature fusion module performs weight-guided feature fusion on multiple feature maps of different resolutions of each modality based on image fusion weights to obtain fusion features of different resolutions, the multiple feature maps of different resolutions of each modality include multiple feature maps of different resolutions corresponding to the modality of the original image R And multiple feature maps of different resolutions corresponding to the modality of the original image T , wherein the i in the subscript represents the i-th resolution, and the weight-guided feature fusion module performs weight-guided feature fusion on multiple feature maps of different resolutions of each modality based on the image fusion weight to obtain fusion features of different resolutions, including:

[0010] S101, for each feature map of resolution, multiple feature maps of different resolutions of each modality are input into a convolution module according to the following formula to improve feature learning ability and enhance generalization. The number of feature map channels is reduced through convolution operation to reduce the number of model parameters:

[0011] ,

[0012] ,

[0013] In the above formula, is the feature map of the i-th resolution of the original image R corresponding to the modality The enhanced feature map obtained by the convolution module, is the feature map of the i-th resolution of the original image T corresponding to the modality The enhanced feature map obtained by the convolution module, is a convolution operation with a convolution kernel of 3×3. For batch normalization, is linear activation;

[0014] S102, for each feature map of resolution, the feature map obtained by the convolution module is calculated according to the following formula after being guided by the image fusion weight:

[0015] ,

[0016] ,

[0017] In the above formula, and are the enhanced feature maps of the i-th resolution of the corresponding modality of the original image R and the original image T, respectively. Indicates a convolution operation with a convolution kernel of 1×1. and are the image fusion weights of the original image R and the original image T corresponding to the modality;

[0018] S103, for each feature map of resolution, an enhanced feature map of the i-th resolution of the corresponding modality of the obtained original image R and the original image T is obtained. and Perform the multimodal feature interaction operation shown in the following formula:

[0019] ,

[0020] ,

[0021] In the above formula, It indicates that the convolution kernel is 5×5 depth-separable convolution. They are the key matrix, query matrix and value matrix obtained by the interaction operation of multimodal features of the corresponding modality of the original image R; They are the key matrix, query matrix and value matrix obtained by the interaction operation of multimodal features of the corresponding modality of the original image T;

[0022] S104, for each feature map of each resolution, the key matrix, query matrix and value matrix obtained by the multimodal feature interaction operation of the two modes are subjected to two multimodal feature interactions according to the following formula:

[0023] ,

[0024] ，

[0025] In the above formula, and They are the feature maps obtained after two multimodal feature interactions of the original image R and the original image T corresponding to the modality. is the Sigmoid activation function, is a linear layer, is the GELU activation function, is the global average pooling layer;

[0026] S105, for each feature map of resolution, combine the original image R and the enhanced feature map of the i-th resolution of the corresponding modality of the original image T and , the feature maps obtained after the interaction of the two multimodal features of the original image R and the original image T corresponding to the modality are fused according to the following formula to obtain the fusion feature of the i-th resolution:

[0027] ，

[0028] In the above formula, is the fusion feature of the i-th resolution, is the Relu activation function.

[0029] Optionally, the semantic segmentation decoder is a multi-layer perceptron decoder, and the multi-modal image recognition result obtained by the multi-layer perceptron decoder is a segmentation map of the same size as the multi-modal image, in which each pixel is assigned a category label.

[0030] Optionally, before inputting the multimodal image into a pre-trained multimodal image fusion recognition network for image recognition to obtain a multimodal image fusion recognition result, the method further includes training the multimodal image fusion recognition network and using a preset loss function To optimize and update the network parameters of the multi-modal image fusion recognition network, and the preset loss function The function expression is:

[0031] ,

[0032] In the above formula, is the image fusion loss, is the semantic segmentation loss, and:

[0033] ，

[0034] ,

[0035] ，

[0036] ，

[0037] ，

[0038] ,

[0039] ,

[0040] ，

[0041] in, is the strength loss, is the texture loss, and are the height and width of the multi-mode image, respectively. To fuse the images, and are the original images of different modes in the multi-modal image, is L1 regularization, To obtain the maximum value, is the gradient operator, is the weighted cross entropy loss, For Lovasz loss, is the Dice loss, is the number of categories, represents the weight given to each category, represents the true label, Prediction probability, Represents a collection of categories, Lovasz extension of the Jaccard index, is the Jaccard index, is the predicted probability, is a category in the category set, is the true label of category c. If category c is correct, then 1, otherwise is 0; , , and are all weight parameters.

[0042] In addition, the present invention also provides a multimodal image fusion recognition system based on multi-task collaborative learning, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal image fusion recognition method based on multi-task collaborative learning.

[0043] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multi-modal image fusion recognition method based on multi-task collaborative learning through a processor.

[0044] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multi-modal image fusion recognition method based on multi-task collaborative learning through a processor.

[0045] Compared with the prior art, the present invention mainly has the following advantages: the present invention designs a multi-task collaborative framework in which image fusion and segmentation promote each other, and promotes the improvement of their respective performances through the close interaction of the two tasks of image fusion and segmentation; designs a feature fusion module aggregation guided by fusion weights, and uses the image fusion weights generated in the image fusion process to guide the multimodal features in the segmentation process, enhance the information-rich areas of each modality, suppress the interference of irrelevant information, and then use the rich multimodal feature interaction operations to fully fuse the information of the two modalities, so as to achieve accurate recognition and analysis of the scene, thereby being able to achieve a close combination of multimodal image fusion and segmentation to simultaneously improve their respective performances, thereby achieving accurate estimation of the location of the target of interest and accurate recognition of the category. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0047] Figure 2Schematic diagram of the network structure of the multi-modal image fusion recognition network in an embodiment of the present invention.

[0048] Figure 3 The figure is a schematic diagram of the principle of weight-guided feature fusion performed by the weight-guided feature fusion module in an embodiment of the present invention.

[0049] Figure 4 Schematic diagram of the network structure of the weight-guided part in the weight-guided feature fusion module in an embodiment of the present invention.

[0050] Figure 5 Schematic diagram of the network structure of the feature fusion part in the weight-guided feature fusion module in an embodiment of the present invention.

[0051] Figure 6 Schematic diagram of the structure of the semantic segmentation decoder in an embodiment of the present invention.

[0052] Figure 7 is an example of a multimodal image in an embodiment of the present invention, wherein (a) to (e) are original images of the first modality of the multimodal image; (a-1) to (e-1) are original images of the second modality of the multimodal image, and (a-2) to (e-2) are label images of the multimodal image.

[0053] Figure 8 : This is a comparison of the semantic segmentation visualization results in the embodiments of the present invention, where (a-3) to (e-3) are the semantic segmentation visualization results of the EGFNet method, (a-4) to (e-4) are the semantic segmentation visualization results of the SFAF-MA method, (a-5) to (e-5) are the semantic segmentation visualization results of the SGFNet method, and (a-6) to (e-6) are the semantic segmentation visualization results of the method in the embodiments of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] like Figure 1As shown, this embodiment provides a multimodal image fusion recognition method based on multi-task collaborative learning, including inputting a multimodal image into a pre-trained multimodal image fusion recognition network for image recognition to obtain a multimodal image fusion recognition result, wherein the multimodal image fusion recognition network is composed of an encoder, an image fusion network, a weight-guided feature fusion module and a semantic segmentation decoder, and the multimodal image fusion recognition network performs image recognition to obtain a multimodal image fusion recognition result, including: extracting features of the original images of each mode in the multimodal image by the encoder to obtain multiple feature maps with different resolutions; extracting features of each mode in the multimodal image by the image fusion network The original images of the modalities are extracted by the image fusion feature extractor respectively, and the feature maps of each modality are added together and then decoded by the image fusion decoder. After a Sigmoid activation function, the image fusion weights of the corresponding modalities are obtained respectively. The image fusion weights and the original images of the corresponding modalities are element-wise multiplied and added together to generate a fused image; the feature fusion module guides the feature fusion of multiple feature maps of different resolutions of each modality based on the image fusion weights to obtain fusion features of different resolutions; the semantic segmentation decoder aggregates the fusion features of different resolutions to obtain the multi-modal image fusion recognition result.

[0056] See also Figure 2 , the network framework of the multimodal image fusion recognition network in this embodiment is divided into two parts: semantic segmentation and image fusion. For the image fusion process, a fusion network based on an adaptive pixel weighting strategy is used to fuse the modality 1 image (taking the visible light image as an example) and the modality 2 image (taking the infrared image as an example) to generate a fused image. First, two layers of convolution and one layer of linear activation function are used as feature extractors to extract the feature maps of the two modalities respectively, and then they are added and input into the two layers of convolution layers as decoders. Finally, a Sigmoid activation function is used to obtain the image fusion weights of the two modalities respectively. The weight corresponding to the area containing rich information is large, and vice versa, the corresponding weight is small. Then, the obtained weights are element-wise multiplied with the original image of the corresponding modality and then the two are added to generate a fused image. The image fusion weights of different modalities generated in the above process will be used to guide the fusion of multimodal features of different resolutions in the semantic segmentation process.

[0057] As an optional implementation, in this embodiment, the original images of each modality in the multimodal image are respectively subjected to feature extraction using a backbone network to obtain multiple feature maps of different resolutions, which means that the original images of each modality in the multimodal image are respectively subjected to feature extraction using a Segformer backbone network to obtain multiple feature maps of different resolutions, and the original images of different modalities use independent Segformer backbone networks that do not share network parameters. The Segformer backbone network includes a hierarchical Transformer encoder, which downsamples the output feature map of the previous stage before inputting the next stage, and finally obtains four feature maps of different resolutions. In addition, the Segformer backbone network does not use position encoding, thereby avoiding the problem of performance degradation when the test resolution is different from the training resolution due to the interpolation problem of position encoding. For input images of two modalities, two Segformer backbone networks are used for feature extraction. The two backbone networks do not share parameters to obtain multi-scale features R of modality 1 image (taking visible light image as an example). 1 , R 2 , R 3 , R 4 and the multi-scale features T of modality 2 images (taking infrared images as an example) 1 , T 2 , T 3 , T 4 .

[0058] As an optional implementation, Figure 2 As shown in FIG. 1 , the image fusion feature extractor in this embodiment is composed of a convolution layer, a linear activation function layer and another convolution layer connected in sequence. As an optional implementation, Figure 2 As shown, the image fusion decoder in this embodiment is composed of two layers of convolution layers connected in sequence. In this embodiment, a fusion network based on an adaptive pixel weighting strategy is used to fuse multimodal images to generate a fused image. First, two layers of convolution and one layer of linear activation function are used as feature extractors to respectively propose feature maps of the two modalities, which are added and then input into two layers of convolution layers as decoders. Finally, a Sigmoid activation function is used to obtain the image fusion weights of the two modalities. The weight corresponding to the information-rich area is large, and vice versa. The corresponding weight is small. The obtained weights are then element-wise multiplied with the original images of the corresponding modalities and then the two are added to generate a fused image. The image fusion weights of the two modalities generated in the above process will be used to guide the fusion of multimodal features of different resolutions in the semantic segmentation process.

[0059] like Figure 2 As shown, in this embodiment, the original images of each mode in the multi-mode image include original images R and T of different modes. Figure 3 As shown, the weight-guided feature fusion module in this embodiment includes two parts: weight guidance and feature fusion. Weight guidance uses the fusion weights of different modalities generated in the image fusion process to guide the multi-scale features of the corresponding modality images respectively, enhance the areas rich in feature information of each modality, and suppress the interference of irrelevant information. Specifically, when the weight-guided feature fusion module performs weight-guided feature fusion based on the image fusion weights on multiple feature maps of different resolutions of each modality to obtain fusion features of different resolutions, the multiple feature maps of different resolutions of each modality include multiple feature maps of different resolutions corresponding to the modality of the original image R And multiple feature maps of different resolutions corresponding to the modality of the original image T , where the i in the subscript represents the i-th resolution, i = 1, 2, 3, 4, such as Figure 4 As shown, in this embodiment, the weight-guided feature fusion module performs weight-guided feature fusion on multiple feature maps of different resolutions of each modality based on the image fusion weight to obtain fusion features of different resolutions, including:

[0060] S101, for each feature map of resolution, multiple feature maps of different resolutions of each modality are input into a convolution module according to the following formula to improve feature learning ability and enhance generalization. The number of feature map channels is reduced through convolution operation to reduce the number of model parameters:

[0061] ,

[0062] ,

[0063] In the above formula, is the feature map of the i-th resolution of the original image R corresponding to the modality The enhanced feature map obtained by the convolution module, is the feature map of the i-th resolution of the original image T corresponding to the modality The enhanced feature map obtained by the convolution module, is a convolution operation with a convolution kernel of 3×3. For batch normalization, is linear activation;

[0064] S102, for each feature map of resolution, then the fusion weight of the modality 1 image is and the fusion weights of modality 2 images The number of channels is adjusted through a 1×1 convolution layer respectively, so that the number of channels of the two weight feature maps is equal to and The number of channels is consistent, and the adjusted weight feature map is respectively and Perform element-wise multiplication to achieve weight guidance, and then adjust the number of channels to the original value through a 1×1 convolution layer. and After the number of channels is increased, the residual connection is introduced to and Perform element addition operation to obtain the final features guided by image fusion weights and , which can be specifically expressed as: for the feature map obtained by the convolution module, the feature map guided by the image fusion weight is calculated according to the following formula:

[0065] ,

[0066] ,

[0067] In the above formula, and are the enhanced feature maps of the i-th resolution of the corresponding modality of the original image R and the original image T, respectively. Indicates a convolution operation with a convolution kernel of 1×1. and are the image fusion weights of the original image R and the original image T corresponding to the modality;

[0068] S103, for each resolution feature map, see Figure 5 In this embodiment, the weight-guided feature fusion module achieves full fusion of the information of the two modalities after weight guidance through rich multi-modal feature interaction operations. First, a 1×1 convolutional layer is used to generate and , and then input the obtained six features into the depth-separable convolution to aggregate the local features to obtain and , which can be specifically expressed as: for the obtained original image R and the original image T corresponding to the i-th resolution enhanced feature map of the modality and Perform the multimodal feature interaction operation shown in the following formula:

[0069] ,

[0070] ,

[0071] In the above formula, It indicates that the convolution kernel is 5×5 depth-separable convolution. They are the key matrix, query matrix and value matrix obtained by the interaction operation of multimodal features of the corresponding modality of the original image R; They are the key matrix, query matrix and value matrix obtained by the interaction operation of multimodal features of the corresponding modality of the original image T;

[0072] S104, for each feature map of each resolution, the key matrix, query matrix and value matrix obtained by the multimodal feature interaction operation of the two modes are subjected to two multimodal feature interactions according to the following formula:

[0073] ,

[0074] ，

[0075] In the above formula, and They are the feature maps obtained after two multimodal feature interactions of the original image R and the original image T corresponding to the modality. is the Sigmoid activation function, is a linear layer, is the GELU activation function, is the global average pooling layer; according to the above function expression, the two multimodal feature interactions include: first, the acquired multimodal features and Perform element-wise multiplication to achieve the first multimodal feature interaction, perform global average pooling on the interacted features to obtain the region of interest in the channel dimension, and then input it into a linear layer for channel dimensionality reduction to achieve compression and condensation of the input features. Enhance the nonlinearity of the features through the GELU activation function, and then use a linear layer to raise the channel back to its original dimension to map the features back to the original spatial dimension. Finally, use the Sigmoid activation function to calculate the attention weight of each channel, and combine the obtained weight information with the feature map. Multiply to achieve the second interaction of multimodal features. and as well as The specific interaction process is exactly the same as above, the only difference is that the corresponding The modal information contained is different;

[0076] S105: For each feature map of each resolution, the features after the second interaction are passed through a convolution module to keep the features in the same distribution range, and then a residual connection is introduced to the original input features. Adding enhances the stability of the model. and as well as The specific interaction process is exactly the same as above, and the final fusion feature is obtained , which can be specifically expressed as: combining the original image R and the enhanced feature map of the i-th resolution of the corresponding modality of the original image T and , the feature maps obtained after the interaction of the two multimodal features of the original image R and the original image T corresponding to the modality are fused according to the following formula to obtain the fusion feature of the i-th resolution:

[0077] ，

[0078] In the above formula, is the fusion feature of the i-th resolution, is the Relu activation function.

[0079] like Figure 6 As shown, the semantic segmentation decoder used in this embodiment is a multi-layer perceptron decoder, and the multi-modal image recognition result obtained by the multi-layer perceptron decoder is a segmentation map of the same size as the multi-modal image, in which each pixel is assigned a category label.

[0080] In addition, before the multi-modal image is input into the pre-trained multi-modal image fusion recognition network for image recognition to obtain the multi-modal image fusion recognition result, the present embodiment also includes training the multi-modal image fusion recognition network and using a preset loss function To optimize and update the network parameters of the multi-modal image fusion recognition network, and the preset loss function The function expression is:

[0081] ,

[0082] In the above formula, is the image fusion loss, is the semantic segmentation loss, and:

[0083] ，

[0084] ,

[0085] ，

[0086] ，

[0087] ，

[0088] ,

[0089] ,

[0090] ，

[0091] in, is the strength loss, is the texture loss, and are the height and width of the multi-mode image, respectively. To fuse the images, and are the original images of different modes in the multi-modal image, is L1 regularization, To obtain the maximum value, is the gradient operator, is the weighted cross entropy loss, For Lovasz loss, is the Dice loss, is the number of categories, represents the weight given to each category, represents the true label, Prediction probability, Represents a collection of categories, Lovasz extension of the Jaccard index, is the Jaccard index, is the predicted probability, is a category in the category set, is the true label of category c. If category c is correct, then 1, otherwise is 0; , , and They are all weight parameters. Specifically, in this embodiment, the values ​​are 0.1, 0.1, 0.5 and 0.5 respectively.

[0092] In order to verify the multimodal image fusion recognition method based on multi-task collaborative learning in this embodiment, this embodiment uses images of visible light and infrared modes as examples for experimental verification. Figure 7 is an example of a multimodal image in this embodiment, where (a) to (e) are original images of the first mode of the multimodal image; (a-1) to (e-1) are original images of the second mode of the multimodal image, and (a-2) to (e-2) are label images of the multimodal image. Figure 8This is a comparison of the semantic segmentation visualization results in this embodiment, where (a-3) to (e-3) are the semantic segmentation visualization results of the EGFNet (see the literature: Zhou W, Dong S, Xu C, et al. Edge-aware guidance fusion network for rgb–thermal scene parsing[C]. Proceedingsof the AAAI conference on artificial intelligence. 2022, 36(3): 3571-3579.) method, (a-4) to (e-4) are the semantic segmentation visualization results of the SFAF-MA (see the literature: He X, Wang M, Liu T, etal. SFAF-MA: Spatial feature aggregation and fusion with modality adaptationfor RGB-thermal semantic segmentation[J]. IEEE Transactions onInstrumentation and Measurement, 2023, 72: 1-10.) method, and (a-5) to (e-5) are the semantic segmentation visualization results of the SGFNet (see the literature: Wang Y, Li G, Liu Z. SGFNet: semantic-guidedfusion network for rgb-thermal semantic segmentation[J]. IEEE Transactions onCircuits and Systems for Video Technology, 2023, 33(12): 7737-7748.) The semantic segmentation visualization results of the method in (a-6) to (e-6) are the semantic segmentation visualization results of the method in this embodiment. Figure 8 It can be seen that, no matter in the daytime or nighttime scenes, the method of this embodiment achieves more precise scene analysis and identifies corresponding objects more accurately and finely.

[0093] In addition, this embodiment takes the two modalities of infrared and visible light as an example, uses the public dataset MFNet for training and testing, and compares it with the methods proposed in recent years for multimodal semantic segmentation. The image contains 1569 pairs of infrared and visible light images with an image resolution of 480×640, including 9 categories. This experiment uses the AdamW optimizer, with an initial learning rate of 0.0001 and a batch size of 4. For the multi-classification task of semantic segmentation, the average accuracy mAcc and the average intersection-over-union ratio mIoU are used to quantitatively measure the performance of semantic segmentation. mAcc is calculated as the ratio of the number of correct pixels for each category to the number of all predicted pixels for that category, and then the average is calculated by summing. mIoU is the average result of adding the ratio of the intersection of the true values ​​predicted for each class to the sum. The expressions of each indicator are as follows:

[0094] ,

[0095] ,

[0096] In the above formula, represents the number of categories, Indicates that it belongs to the class Predicted as class The number of pixels, Indicates that it belongs to the class Predicted as class Table 1 compares the experimental results of the recognition accuracy of the four algorithms and the method of this embodiment on the MFNet dataset.

[0097] Table 1 Comparison of experimental results between the method in this embodiment and the existing method on the MFNet dataset (%)

[0098]

[0099] In Table 1, Acc is the accuracy, and IoU is the intersection over union ratio. As shown in Table 1, for the average accuracy mAc, the method of this embodiment is 1.7%, 4.8%, and 0.8% higher, respectively; for the average intersection over union ratio mIoU, the method of this embodiment is 3.2%, 2.5%, and 0.4% higher, respectively, which proves the superiority of the method of this embodiment.

[0100] In summary, this embodiment designs a multi-task collaborative learning multi-modal image fusion recognition framework based on a multi-task collaborative learning multi-modal image fusion recognition method, which can simultaneously perform image fusion and target recognition multi-task collaborative learning reasoning on images of different modalities (for example: visible light, infrared images). First, in the image fusion task, an adaptive weighted fusion network is designed to perform pixel-level fusion on multi-modal images, achieve pixel-level adaptive image fusion results, and obtain image fusion weights of different modalities; secondly, in the target recognition task, the Segformer backbone network is used to extract multi-level features from shallow to deep layers of different modalities, and a fusion weight-guided multi-modal image weight-guided feature fusion module is designed. The image fusion weight is used to guide the fusion process of multi-modal features to achieve feature enhancement effects of detail information enhancement and interference information suppression for each modality image. At the same time, a weight-guided feature fusion module based on rich multi-modal feature interaction operations is designed to achieve effective fusion of complementary information of different modalities, thereby improving the ability to distinguish different semantic information; finally, a multi-layer perceptron decoder is used to aggregate multi-level features and obtain the prediction results of image semantic segmentation, so as to achieve accurate estimation of the location of the target of interest and accurate recognition of the category.

[0101] In addition, this embodiment also provides a multimodal image fusion recognition system based on multi-task collaborative learning, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal image fusion recognition method based on multi-task collaborative learning.

[0102] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multi-modal image fusion recognition method based on multi-task collaborative learning through a processor.

[0103] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multi-modal image fusion recognition method based on multi-task collaborative learning through a processor.

[0104] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multi-modal image fusion recognition method based on multi-task collaborative learning, characterized in that: The method includes inputting a multimodal image into a pre-trained multimodal image fusion recognition network for image recognition to obtain a multimodal image fusion recognition result. The multimodal image fusion recognition network is composed of an encoder, an image fusion network, a weight-guided feature fusion module and a semantic segmentation decoder. The multimodal image fusion recognition network performs image recognition to obtain a multimodal image fusion recognition result, including: extracting features of original images of each modality in the multimodal image by the encoder to obtain a plurality of feature maps with different resolutions; extracting feature maps of corresponding modalities from original images of each modality in the multimodal image by an image fusion feature extractor by the image fusion network, adding the feature maps of each modality, decoding by an image fusion decoder, and obtaining image fusion weights of corresponding modalities by a Sigmoid activation function, performing element-wise multiplication operation on the image fusion weights and the original images of the corresponding modalities, and then adding them to generate a fused image; performing weight-guided feature fusion on multiple feature maps of different resolutions of each modality based on the image fusion weights by the weight-guided feature fusion module to obtain fusion features of different resolutions; and aggregating fusion features of different resolutions by the semantic segmentation decoder to obtain a multimodal image fusion recognition result.

2. The multi-modal image fusion recognition method based on multi-task collaborative learning according to claim 1 is characterized in that: The method of extracting features from the original images of each modality in the multimodal image using a backbone network to obtain multiple feature maps of different resolutions means extracting features from the original images of each modality in the multimodal image using a Segformer backbone network to obtain multiple feature maps of different resolutions, and original images of different modalities use independent Segformer backbone networks that do not share network parameters.

3. The multi-modal image fusion recognition method based on multi-task collaborative learning according to claim 1 is characterized in that: The image fusion feature extractor is composed of a convolution layer, a linear activation function layer and another convolution layer which are connected in sequence.

4. The multi-modal image fusion recognition method based on multi-task collaborative learning according to claim 1 is characterized in that: The image fusion decoder is composed of two convolutional layers connected in sequence.

5. The multi-modal image fusion recognition method based on multi-task collaborative learning according to claim 1 is characterized in that: The original images of each modality in the multimodal image include original images R and T of different modalities; when the weight-guided feature fusion module performs weight-guided feature fusion on multiple feature maps of different resolutions of each modality based on image fusion weights to obtain fusion features of different resolutions, the multiple feature maps of different resolutions of each modality include multiple feature maps of different resolutions corresponding to the modality of the original image R And multiple feature maps of different resolutions corresponding to the modality of the original image T , wherein the i in the subscript represents the i-th resolution, and the weight-guided feature fusion module performs weight-guided feature fusion on multiple feature maps of different resolutions of each modality based on the image fusion weight to obtain fusion features of different resolutions, including: S101, for each feature map of resolution, multiple feature maps of different resolutions of each modality are input into a convolution module according to the following formula to improve feature learning ability and enhance generalization. The number of feature map channels is reduced through convolution operation to reduce the number of model parameters: ， ， In the above formula, is the feature map of the i-th resolution of the original image R corresponding to the modality The enhanced feature map obtained by the convolution module, is the feature map of the i-th resolution of the original image T corresponding to the modality The enhanced feature map obtained by the convolution module operation, is a convolution operation with a convolution kernel of 3×3. For batch normalization, is linear activation; S102, for each feature map of the resolution, the feature map obtained by the convolution module operation is calculated according to the following formula after being guided by the image fusion weight: ， ， In the above formula, and are the enhanced feature maps of the i-th resolution of the corresponding modality of the original image R and the original image T, respectively. Indicates a convolution operation with a convolution kernel of 1×1. and are the image fusion weights of the corresponding modalities of the original image R and the original image T respectively; S103, for each feature map of resolution, an enhanced feature map of the i-th resolution of the corresponding modality of the obtained original image R and the original image T is obtained. and Perform the multimodal feature interaction operation shown in the following formula: ， ， In the above formula, It indicates that the convolution kernel is 5×5 depth-separable convolution. They are the key matrix, query matrix and value matrix obtained by the interaction operation of multimodal features of the corresponding modality of the original image R; They are the key matrix, query matrix and value matrix obtained by the interaction operation of multimodal features of the corresponding modality of the original image T; S104, for each feature map of each resolution, the key matrix, query matrix and value matrix obtained by the multimodal feature interaction operation of the two modes are subjected to two multimodal feature interactions according to the following formula: ， ， In the above formula, and They are the feature maps obtained after two multimodal feature interactions of the original image R and the original image T corresponding to the modality. is the Sigmoid activation function, is a linear layer, is the GELU activation function, is the global average pooling layer; S105, for each feature map of resolution, combine the original image R and the enhanced feature map of the i-th resolution of the corresponding modality of the original image T and , the feature maps obtained after the interaction of the two multimodal features of the original image R and the original image T corresponding to the modality are fused according to the following formula to obtain the fusion feature of the i-th resolution: ， In the above formula, is the fusion feature of the i-th resolution, is the Relu activation function.

6. The multi-modal image fusion recognition method based on multi-task collaborative learning according to claim 1 is characterized in that: The semantic segmentation decoder is a multi-layer perceptron decoder, and the multi-modal image recognition result obtained by the multi-layer perceptron decoder is a segmentation map of the same size as the multi-modal image, in which each pixel is assigned a category label.

7. The multi-modal image fusion recognition method based on multi-task collaborative learning according to claim 1 is characterized in that: Before inputting the multimodal image into the pre-trained multimodal image fusion recognition network for image recognition to obtain the multimodal image fusion recognition result, it also includes training the multimodal image fusion recognition network and using a preset loss function To optimize and update the network parameters of the multi-modal image fusion recognition network, and the preset loss function The function expression is: ， In the above formula, is the image fusion loss, is the semantic segmentation loss, and: ， ， ， ， ， ， ， ， in, is the strength loss, is the texture loss, and are the height and width of the multi-mode image, respectively. To fuse the images, and are the original images of different modes in the multi-modal image, is L1 regularization, To obtain the maximum value, is the gradient operator, is the weighted cross entropy loss, For Lovasz loss, is the Dice loss, is the number of categories, represents the weight given to each category, represents the true label, Prediction probability, Represents a collection of categories, Lovasz extension of the Jaccard index, is the Jaccard index, is the predicted probability, is a category in the category set, is the true label of category c. If category c is correct, then 1, otherwise is 0; , , and are all weight parameters.

8. A multi-mode image fusion recognition system based on multi-task collaborative learning, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the multi-modal image fusion recognition method based on multi-task collaborative learning as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the multi-modal image fusion recognition method based on multi-task collaborative learning described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the multi-modal image fusion recognition method based on multi-task collaborative learning described in any one of claims 1 to 7 through a processor.

Images