A salient target segmentation method and system based on multi-level modal interaction

Abstract

This invention relates to a salient target segmentation method and system based on multi-level modal interaction, belonging to the field of computer vision technology. It includes: acquiring RGB and thermal infrared images of a test image and extracting multiple hierarchical feature maps; inputting the multi-modal feature maps of the same level into a triple attention fusion module, where difference feature maps of different modalities at the same level are acquired, and a first fusion feature map is obtained based on the difference feature maps; inputting the multi-modal feature maps of the same level into a dual-path grouping fusion module, where RGB feature groups and thermal infrared feature groups are obtained, and a second fusion feature map is obtained based on these two groups; and obtaining the salient target of the test image based on the first and second fusion feature maps. This invention fully utilizes features from different levels and modalities, fuses complementary features between the RGB and thermal infrared images, avoids redundancy and interference from non-salient features, and improves the performance and generalization ability of salient target detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a salient target segmentation method and system based on multi-level modal interaction. Background Technology

[0002] With the continuous development of computer vision technology, salient object detection has gradually become a research hotspot. In recent years, this task has entered its third stage based on deep learning models. Deep learning models based on convolutional neural networks (CNNs) have made significant progress in the field of salient object detection due to their powerful feature extraction capabilities and model complexity. These models typically contain hundreds of thousands of adjustable parameters and neurons with receptive fields of varying sizes. Large receptive field neurons provide global information, helping the model identify the most salient regions in an image; small receptive field neurons provide local information, used to refine the saliency mapping. This combination of multi-scale information lays the foundation for introducing more modalities and more complex models for salient object detection.

[0003] Meanwhile, hardware such as thermal infrared sensors continues to develop and become more widespread. Existing thermal infrared information can effectively solve the target blurring problem caused by factors such as illumination, occlusion, and complex backgrounds. Although the RGB-T fusion method improves detection performance to some extent, it still has some shortcomings. Most current models directly extract and fuse the original features from the backbone, which is easily limited by low-quality modal data and redundant cross-modal features. The operation of fusing multimodal features through simple concatenation or element-wise summation does not consider the importance of different modal features and fails to explore the complementary information features between RGB images and thermal infrared images well. This fusion method allows the inclusion of redundant and non-salient features, making it difficult for the fusion process to achieve complementarity well. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the limitations of the fusion process in the prior art, which is limited by low-quality modal data and redundant cross-modal features, as well as the poor fusion effect.

[0005] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a salient target segmentation method based on multi-level modal interaction, comprising:

[0006] Acquire the RGB image and thermal infrared image of the image to be tested, and extract multiple hierarchical feature maps from the RGB image and the thermal infrared image;

[0007] A triple attention fusion module is constructed, and multimodal feature maps at the same level are input into the triple attention fusion module; in the triple attention fusion module, difference feature maps of different modal feature maps at the same level are obtained, and a first fusion feature map is obtained based on the difference feature maps;

[0008] A dual-path grouping fusion module is constructed, and multimodal feature maps at the same level are input into the dual-path grouping fusion module; in the dual-path grouping fusion module, RGB feature groups and thermal infrared feature groups are obtained, and a second fused feature map is obtained based on the RGB feature groups and the thermal infrared feature groups;

[0009] The salient targets of the image under test are obtained based on the first fused feature map and the second fused feature map.

[0010] In one embodiment of the present invention, the step of obtaining the difference feature map of the multimodal feature map at the same level and obtaining the first fused feature map based on the difference feature map in the triple attention fusion module is as follows:

[0011] Obtain the first and second feature maps at the same level, and calculate the difference feature map between the first and second feature maps;

[0012] The first feature map, the second feature map, and the difference feature map are compressed into channel-level vectors respectively to obtain the first global feature, the second global feature, and the third global feature;

[0013] The first global feature, the second global feature, and the third global feature are input into the attention sub-network and dynamically weighted and fused to obtain the first fused feature map.

[0014] In one embodiment of the present invention, the step of inputting the first global feature, the second global feature, and the third global feature into the attention sub-network and performing dynamic weighted fusion to obtain the first fused feature map is as follows:

[0015] Perform a first convolution operation on the first global feature, the second global feature, and the third global feature respectively to extract enhanced feature maps;

[0016] A second convolution operation is performed on the enhanced feature map to obtain a weight coefficient map;

[0017] Based on the weight coefficient map, the first global feature, the second global feature, and the third global feature are dynamically weighted and fused to obtain the first fused feature map.

[0018] In one embodiment of the present invention, the steps of the first convolution operation are: compressing the spatial dimension of the input global features and extracting key information by halving the number of channels; and using an activation function to enhance the key information to obtain an enhanced feature map.

[0019] In one embodiment of the present invention, the steps of the second convolution operation are as follows: output a single-channel feature map according to the input enhanced feature map; process the single-channel feature map using an activation function to obtain a weight coefficient map.

[0020] In one embodiment of the present invention, in the dual-path grouping fusion module, the steps of obtaining the RGB feature group and the thermal infrared feature group and obtaining the second fusion feature map according to the RGB feature group and the thermal infrared feature group are as follows:

[0021] Obtain multi-modal feature maps at the same level, group the RGB features and the thermal infrared features to obtain the RGB feature group and the thermal infrared feature group;

[0022] Perform cross-modal splicing on the RGB feature group and the thermal infrared feature group to obtain a spliced feature group;

[0023] Perform dimensionality reduction processing on the spliced feature group to obtain a fine-grained fusion feature map.

[0024] In one embodiment of the present invention, when performing cross-modal splicing on the RGB feature group and the thermal infrared feature group to obtain a spliced feature group, it also includes directly splicing the ungrouped RGB features and thermal infrared features to obtain a coarse-grained fusion feature; splicing the coarse-grained fusion feature and the fine-grained fusion feature to obtain the second fusion feature map.

[0025] In one embodiment of the present invention, obtain the significant target of the待测图像 according to the second fusion feature map and the first fusion feature map.

[0026] In one embodiment of the present invention, the steps of obtaining the significant target of the待测图像 according to the second fusion feature map and the first fusion feature map are as follows:

[0027] Perform feature extraction and upsampling processing on the second fusion feature map in the nth layer feature map to obtain a first enhanced feature map; where n is a positive integer that decreases from a value greater than 4;

[0028] Perform splicing processing on the first enhanced feature map and the second fusion feature map in the (n-1)th layer feature map to obtain a first spliced feature map;

[0029] Perform feature extraction and upsampling processing on the first spliced feature map to obtain a second enhanced feature map;

[0030] Perform splicing processing on the second enhanced feature map and the first fusion feature map in the mth layer feature map to obtain a second spliced feature map; where 1 m < n - 1, and m is a positive integer;

[0031] The second stitched feature map is subjected to feature extraction and upsampling to obtain the salient target of the image to be tested.

[0032] Secondly, to solve the above-mentioned technical problems, the present invention provides a salient target segmentation system based on multi-level modal interaction, comprising:

[0033] The feature extraction module is used to acquire the RGB image and thermal infrared image of the image to be tested, and extract multiple layers of feature maps from the RGB image and the thermal infrared image;

[0034] The first fusion feature map acquisition module is used to construct a triple attention fusion module and input multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, difference feature maps of different modal feature maps of the same level are acquired, and the first fusion feature map is obtained based on the difference feature maps;

[0035] The second fusion feature map acquisition module is used to construct a dual-path grouping fusion module and input multimodal feature maps of the same level into the dual-path grouping fusion module; in the dual-path grouping fusion module, RGB feature groups and thermal infrared feature groups are obtained, and a second fusion feature map is obtained based on the RGB feature groups and the thermal infrared feature groups;

[0036] A salient target output module is used to obtain the salient targets of the image under test based on the first fused feature map and the second fused feature map.

[0037] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0038] This invention discloses a salient target segmentation method and system based on multi-level modal interaction, which constructs a triple attention fusion module and a dual-path grouping fusion module. The triple attention fusion module cleverly fuses RGB images, thermal infrared images, and their differential features through a dynamic weighting mechanism, and effectively compensates for the shortcomings of lightweight backbone networks in detail processing by employing a channel attention mechanism, thus enhancing the expressive power of spatial detail features. The dual-path grouping fusion module adopts a dual-path strategy combining grouped convolution and direct concatenation to perform both fine and coarse fusion of deep semantic features. This optimizes the capture of local feature differences while ensuring global semantic consistency, achieving deep complementarity and optimization of semantic features. These two modules work together to construct a multi-level interactive framework, opening up new design ideas for multi-modal feature fusion. It fully leverages the unique advantages of each modality and achieves complementary advantages, making the entire system more flexible and efficient in the feature fusion process, unaffected by low-quality data and redundant features. Attached Figure Description

[0039] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0040] Figure 1 Flowcharts for early-stage and late-stage fusion in existing technologies;

[0041] Figure 2 A flowchart of hybrid fusion processes in existing technologies;

[0042] Figure 3 This is a flowchart of a salient target segmentation method based on multi-level modal interaction in a preferred embodiment of the present invention;

[0043] Figure 4 This is a network architecture diagram in a preferred embodiment of the present invention;

[0044] Figure 5 This is a flowchart of the triple attention fusion module in a preferred embodiment of the present invention;

[0045] Figure 6 This is a flowchart of the attention sub-network in the triple attention fusion module of the present invention in a preferred embodiment;

[0046] Figure 7 This is a flowchart of the dual-path grouping fusion module in a preferred embodiment of the present invention;

[0047] Figure 8 The images show the salient target detection results of different models in a preferred embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0049] The technologies involved in this invention mainly include salient target detection and modality fusion.

[0050] In the field of salient object detection, current deep learning-based models are mainly divided into two categories: models based on convolutional neural networks (CNN) and models based on vision transformers (ViT).

[0051] For CNN models, the ability of their convolutional neurons to sensitively capture local information enables them to achieve significant performance improvements over traditional machine learning methods in salient object detection tasks. MFSR was the first salient object detection model to adopt a CNN architecture. With the continuous advancement of deep learning technology, CNN-based models have further evolved and subdivided, mainly falling into the following three categories:

[0052] The first type of method is based on superpixel extraction. In traditional machine learning algorithms, manually designing the feature extraction process is not only time-consuming but also laborious. To address this issue, researchers have begun to utilize convolutional neural networks (CNNs) to automatically learn the relationships between superpixels, thereby leveraging the deep features extracted by CNNs to overcome the limitations of traditional manual feature extraction methods. However, this method still has some shortcomings, such as low detection accuracy and unclear target boundaries.

[0053] The second type of method is feature enhancement. This method performs a series of transformations on RGB or thermal infrared images to generate more discriminative feature maps. These enhanced feature maps help the model learn more efficiently, thereby improving the accuracy and speed of salient object detection.

[0054] The third type of method is feature fusion (modal interaction). This method aims to improve the learning efficiency of the model by integrating complementary information between different modalities. However, simply fusing different modalities directly may introduce noise, which can negatively impact the model's performance.

[0055] For ViT-based salient object detection, the first ViT-based model, SwinNet, was proposed in 2021. ViT models convert input images into sequences, add positional encoding to preserve sequence relationships, and then utilize a self-attention mechanism to calculate the correlation between elements in the sequence, thereby completing the image classification task. This approach gives the model a stronger global feature modeling capability than CNN-based models, making it perform better in terms of detecting the completeness of objects. However, because the image is segmented into multiple patches, this method still has shortcomings in capturing local detail information. This lack of local information may affect the model's accurate identification and localization of object details.

[0056] In modality fusion, current research typically classifies modality fusion techniques based on whether a method depends on the implementation of a specific model. If it does not depend on a specific deep learning model, it is a model-independent method; otherwise, it is a model-dependent method.

[0057] For model-independent methods, these methods are mainly divided into three strategies: early fusion, late fusion, and hybrid fusion.

[0058] Early fusion merges multimodal data directly in the input layer or shallow network, i.e., by directly experimenting with channel concatenation (e.g., RGB-T four-channel input) or weighted superposition to fuse different feature maps. The advantage of this method is that it preserves the physical properties of the original modalities and can directly combine the temperature sensitivity of thermal infrared with the texture information of RGB. However, it is susceptible to modal noise interference. Because the correlation between different modalities is difficult to extract, the complementarity between different modules cannot be fully reflected. The specific process is as follows... Figure 1 As shown in (a) in the figure.

[0059] Late fusion employs a multi-stream independent coding structure, training different modalities with different models and finally aggregating features only at the decision layer; hence, it is also known as a decision-level fusion method. The fusion process of this method is independent of the features of each modality, and the results of different modalities trained by different models often have low correlation, thus effectively avoiding modal interference. Currently, late fusion strategies mainly determine the output results of different models by specifying different rules, thereby changing the fusion result. Examples include rule-based fusion methods such as maximum value fusion, average value fusion, Bayesian rule fusion, and ensemble learning. Figure 1 As shown in (b) of the diagram.

[0060] Hybrid fusion employs a multi-layered, progressive interaction strategy, combining early and late fusion strategies, such as... Figure 2 As shown, a typical example is the cross-modal interaction network under the Swin-Transformer architecture, which achieves feature complementarity through cross-attention, while the decoder introduces a multi-scale fusion module. High-level layers employ semantic fusion guided by channel attention, while low-level layers enhance detail alignment through spatial attention. This strategy combines the advantages of both, but inevitably increases the structural complexity and training difficulty of the model. With the rapid development and widespread application of deep learning models, model structures have become more diverse and flexible. Although hybrid fusion methods are more complex to implement, they remain a widely used modal fusion strategy because they can fully utilize the advantages of different models.

[0061] The three strategies mentioned above are common fusion methods, and they show a clear technological evolution: early fusion methods focused on data merging but lacked robustness; later methods focused on maintaining the independence of each modality, which may sacrifice some of the advantages of information complementarity; while hybrid fusion strategies, by establishing hierarchical interactive networks, have performed well in improving detection accuracy, but this strategy inevitably increases the complexity of the model, which may affect the efficiency of computation and training.

[0062] Model-related methods can be seen as extensions based on the characteristics of specific deep learning models. They utilize the inherent features of deep learning models to solve fusion problems, including multiple kernel learning (MKL), graphical model (GM), and neural network (NN) methods.

[0063] To address the shortcomings of existing technologies, this invention designs a highly efficient RGB-T salient target detection network architecture based on modal interaction. This architecture allows for cross-modal, cross-path, and cross-layer information interaction, fully utilizing the complementary information between RGB and thermal imaging. This architecture enables the complementarity and fusion of features across different modalities at multiple levels, allowing deep networks to learn more comprehensive multimodal features.

[0064] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0065] Example 1: Refer to Figure 3 As shown, this embodiment of the invention provides a salient target segmentation method based on multi-level modal interaction, including but not limited to the following steps:

[0066] S1. Obtain the RGB image and thermal infrared image of the image to be tested, and extract multiple layers of feature maps from the RGB image and thermal infrared image;

[0067] S2. Construct a triple attention fusion module and input the multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, obtain the difference feature maps of different modal feature maps of the same level, and obtain the first fusion feature map based on the difference feature maps;

[0068] S3. Construct a dual-path grouping fusion module and input the multimodal feature maps of the same level into the dual-path grouping fusion module; in the dual-path grouping fusion module, obtain RGB feature groups and thermal infrared feature groups, and obtain the second fused feature map based on the RGB feature groups and thermal infrared feature groups;

[0069] S4. Based on the first fusion feature map and the second fusion feature map, obtain the salient targets of the image to be tested.

[0070] This invention provides a salient object segmentation method based on multi-level modal interaction, constructing a multi-level modal interaction network (MMINet) architecture. This architecture incorporates a triple attention fusion module and a dual-path grouping fusion module, operating on the shallow and deep layers of the multimodal model respectively, forming a hierarchical interaction framework that enhances spatial details and complements semantic features, providing a new design paradigm for multimodal feature fusion. For the triple attention fusion module, dynamic weighted fusion of RGB, thermal infrared, and their differential features utilizes a channel attention mechanism to address the limitations of lightweight backbone networks in edge detail processing. For the dual-path grouping fusion module, a dual-path strategy combining grouped convolution and direct concatenation is employed to achieve fine-grained and coarse-grained fusion of deep semantic features, effectively balancing the capture of local modal differences with the maintenance of global semantic consistency. During feature fusion, it is not limited by low-quality modal data and redundant cross-modal features, thus achieving superior fusion results. This method not only improves the accuracy of salient object segmentation but also enhances the model's adaptability and generalization performance in complex scenes.

[0071] Specifically, in step S1, the RGB image and thermal infrared image of the image to be tested are acquired. The RGB image can be captured by common devices such as ordinary digital cameras, SLR cameras, and mobile phone cameras, or by professional devices such as industrial cameras and panoramic cameras. Alternatively, an image signal from an external camera can be converted into a digital signal using a USB or PCIe image acquisition card. The thermal infrared image can be captured by a handheld, fixed, or drone-mounted thermal infrared camera, or by a portable or vehicle-mounted thermal infrared imaging system, as well as MEMS thermal infrared sensors, thermopile sensors, etc. Furthermore, multimodal imaging devices such as dual-spectrum cameras and multispectral cameras can be used to simultaneously acquire RGB and thermal infrared images, or a synchronous acquisition system can be used to ensure that both modal images are acquired at the same time.

[0072] Specifically, to address issues such as insufficient modal interaction and poor feature complementarity, this invention proposes a Triple Stream Attention Fusion Module (TSAFM) and a Dual-path Group Fusion Module (DGFM). TSAFM utilizes a channel attention mechanism to dynamically weight and fuse RGB, thermal infrared, and their differential features, emphasizing the complementarity of shallow spatial information. DGFM, on the other hand, achieves efficient fusion of deep semantic features through a combination of grouped convolution and direct concatenation, cleverly balancing the consistency between local details and global semantics. These two modules are applied to the shallow and deep multimodal features of the backbone network, respectively, employing targeted modal interaction strategies for different feature levels, thereby comprehensively improving the performance of model feature fusion and object detection.

[0073] Existing technologies employ direct element-wise summation in multimodal feature fusion, which easily leads to model performance limitations imposed by low-quality modal data and redundant cross-modal features. This direct element-wise summation method fails to deeply explore the correlations and importance between different modalities and carries a large amount of redundant information, making it unable to effectively achieve feature complementarity, thus impacting model performance. To address this issue, this invention, based on the lightweight MobileNetV2 backbone network, designs TSAFM and DGFM modules to construct a hierarchical multimodal interaction framework, which can be referred to... Figure 4 As shown in the diagram. The TSAFM module is used to fuse the first few layers (e.g., the first three layers), while the DGFM module is specifically used to fuse the last few layers (e.g., the last two layers).

[0074] Specifically, the first three layers, as shallow features, have high resolution and contain rich spatial structural information such as edges, contours, and textures; however, their semantic abstraction level is relatively low. Therefore, by leveraging the differential feature calculation in the TSAFM module, the differences between RGB and thermal infrared images in terms of edge positions and texture details can be directly captured, thereby enhancing spatial consistency. Simultaneously, the channel attention weighting mechanism can dynamically adjust the contribution ratio of different modalities for local regions of low-level features. This mechanism effectively avoids the spatial detail blurring problem caused by the use of depthwise separable convolutions in the lightweight backbone network MobileNetV2, thus providing clearer and more accurate edge cues for subsequent decoders.

[0075] Specifically, the latter two layers, as deep features, have lower resolution but higher channel count, primarily focusing on semantic abstraction, while spatial location information is relatively weakened. To address this, the DGFM module divides the high-channel features into two groups, processing local semantics independently for each group, thus avoiding the smoothing of details caused by global operations. Simultaneously, this module also processes the ungrouped original concatenated features to preserve the overall consistency of high-level semantics and prevent semantic fragmentation caused by grouping operations. Therefore, in the DGFM module, the combination of fine-grained local semantics and coarse-grained global semantics preserves component details while maintaining the overall semantics of the object, effectively solving the localization ambiguity problem caused by low resolution of high-level features.

[0076] Specifically, in step S2, the constructed TSAFM module, such as Figure 5 As shown, its operation process revolves closely around the dynamic weighted fusion of cross-modal features. The constructed TSAFM module introduces a channel attention mechanism, which enables adaptive integration of complementary features between different modalities. The TSAFM module also includes an attention sub-network.

[0077] Furthermore, the TSAFM module is used to obtain the difference feature maps of different modalities at the same level. And based on the difference feature map The first fusion feature map is obtained. The specific steps are as follows:

[0078] S210. Obtain the first feature map at the same level. Second feature map Calculate the first feature map Second feature map Difference feature map between The calculation formula is:

[0079] ;

[0080] in, The value can be 1, 2, 3, 4, 5, etc., and this difference feature captures the uniqueness of the two modalities in terms of spatial structure and semantic information.

[0081] S220, respectively, the first feature map Second feature map and difference feature map Compressed into channel-level vectors, the first global feature is obtained. Second global feature and third global features .

[0082] For example, the above three feature maps ( , , Each feature map is compressed into a channel-level vector using a global average pooling layer to obtain the corresponding global feature. The specific expression is:

[0083] ;

[0084] ;

[0085] ;

[0086] in, This indicates that the feature map is compressed to a spatial size of 1×1.

[0087] S230, First global feature Second global feature and third global features The input is fed into the attention subnetwork and dynamically weighted and fused to obtain the first fused feature map. The attention subnetwork consists of two convolutional layers, which can be referenced... Figure 6 As shown, dynamic weighted fusion is performed in the attention subnetwork to obtain the first fused feature map. The specific steps are as follows:

[0088] S231, respectively, for the first global feature Second global feature and third global features Perform the first convolution operation to extract the enhanced feature map. The first convolution operation involves the following steps: processing the input global features (…). Spatial dimensionality is compressed, and key information is extracted by halving the number of channels; activation functions are used to enhance the key information, resulting in an enhanced feature map. ).

[0089] S232, Enhance the feature map ( Perform a second convolution operation to obtain the weight coefficient map. The second convolution operation involves the following steps: based on the input enhanced feature map (... Output single-channel feature map ( ); Activation functions are used to process single-channel feature maps ( The weighted coefficient graph is obtained by processing the graph. .

[0090] S233, According to the weighting coefficient diagram For the first global feature Second global feature and third global features Dynamic weighted fusion is performed to obtain the first fused feature map. .

[0091] For example, the obtained global features ( The input consists of an attention subnetwork with two convolutional layers. The first convolutional layer halves the number of channels and performs ReLU activation, while the second convolutional layer restores the number of channels to 1 and generates corresponding weight coefficients using the sigmoid function, denoted as follows: , and The mathematical expression is:

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] .

[0101] Based on these weights, the three types of features are dynamically weighted and fused to obtain the final first fused feature map. Its expression is:

[0102] ;

[0103] Here, ⊙ represents element-wise multiplication (and the scalar weights are expanded to the same size as the feature map through a broadcast mechanism).

[0104] The above steps adaptively adjust the contributions of each modality at the same level through an attention mechanism (e.g., in low-light scenes, the weight of thermal infrared features automatically increases, dominating the localization of salient regions), resulting in a difference feature map. It can also supplement the module with unique information between modes, ensuring complementarity between different modes.

[0105] Specifically, in step S3, the constructed DGFM module, such as Figure 7As shown. This module achieves effective fusion of RGB and thermal infrared features through a combination of grouped convolution and direct convolution. The DGFM module is used to obtain RGB feature groups ( ) and thermal infrared feature grouping ( ), and grouped according to RGB features ( ) and thermal infrared feature grouping ( The second fusion feature map is obtained. The specific steps are as follows:

[0106] S310. Obtain multimodal feature maps at the same level. ), RGB features ( ) and thermal infrared characteristics ( Grouping them together yields RGB feature groups. ) and thermal infrared feature grouping ( ).

[0107] Specifically, for features of a specific layer, the module first groups them by channel, separating RGB features ( ) and thermal infrared characteristics ( They were all divided into two groups on average. ),( Its purpose is to facilitate the independent processing of feature information in local areas, thereby capturing the differences in RGB and thermal infrared features in different local areas.

[0108] S320, Group the RGB features ( ) and thermal infrared feature grouping ( Cross-modal concatenation is performed to obtain concatenated feature groups. , ), ( , Specifically, RGB features are grouped ( ) and corresponding thermal infrared feature groups ( ) pieced together, we get ( , ), ( , ).

[0109] S330. Perform dimensionality reduction on the spliced ​​feature groups to obtain a fine-grained fused feature map. .

[0110] For example, a preferred method is to use a 1×1 convolution to reduce the dimensionality of the concatenated features, resulting in a fine-grained fused feature map. Its expression is:

[0111] ;

[0112] Here, [;] represents a stitching operation. This fine-grained feature can well reflect the relationship between RGB and thermal infrared features in a local area, preserving more local detail information.

[0113] S340, the module will also process the ungrouped raw RGB features ( ) and thermal infrared characteristics ( Direct concatenation is performed on ungrouped RGB features. Direct concatenation preserves global feature information and avoids the loss of overall semantic consistency due to grouping operations. Specifically, for ungrouped RGB features (… ) and thermal infrared characteristics ( Direct splicing and convolution processing are performed to obtain coarse-grained fused features. Coarse-grained features focus more on reflecting the overall semantics and macroscopic structure of the features. Integrating coarse-grained features... and fine-grained fusion features The two maps are then stitched together to obtain the second fused feature map. .

[0114] For example, after splicing, a 1×1 convolution is preferably performed to obtain coarse-grained fused features. Its expression is:

[0115] .

[0116] coarse-grained fusion features and fine-grained fusion features The two maps are then stitched together to obtain the second fused feature map. The specific expression is:

[0117] .

[0118] Through the above steps, the DGFM module will fuse features at a fine-grained level. Features of coarse-grained fusion This process involves splicing the data together to organically combine local details with global semantics. The final feature... It contains rich local information and also demonstrates good global semantic consistency.

[0119] Specifically, in step S4, when there are ungrouped original RGB features ( ) and thermal infrared characteristics ( When, according to the second fusion feature map and the first fused feature map This yields salient targets in the image under test. When there are no ungrouped original RGB features ( ) and thermal infrared characteristics ( When, according to the first fusion feature map Second fusion feature map , the prominent object of the image to be measured is obtained.

[0120] Specifically, referring to Figure 4 as shown, according to the second fusion feature map and the first fusion feature map , the specific steps to obtain the prominent object of the image to be measured are as follows:

[0121] S410. Feature extraction and upsampling are performed on the second fusion feature map in the nth layer feature map to obtain the first enhanced feature map ; where n is a positive integer that decreases and is greater than 4, i is a positive integer that increases starting from 1, and the specific expression is:

[0122] .

[0123] S420. The first enhanced feature map is concatenated with the second fusion feature map in the (n - 1)th layer feature map to obtain the first concatenated feature map , i is a positive integer that increases starting from 1, and the specific expression is:

[0124] .

[0125] S430. Feature extraction and upsampling are performed on the first concatenated feature map to obtain the second enhanced feature map , and the specific expression is:

[0126] .

[0127] Steps S420 - S430 are repeated to obtain different enhanced feature maps ; [[ID=**56**]]

[0128] S440. The second enhanced feature map is concatenated with the first fusion feature map in the mth layer feature map to obtain the second concatenated feature map ; where 1 <= m < n - 1, and m is a decreasing positive integer; the specific expression of the second concatenated feature map is:

[0129] .

[0130] S450. Feature extraction and upsampling are performed on the second concatenated feature map to obtain a new enhanced feature map . Note: There seems to be a numbering issue in the original text where step S440's description has an incorrect numbering for the variable related to the second concatenated feature map's expression. I've translated it as presented but it might need to be corrected in the source text for proper understanding. Also, step ID 56 was marked as bold as it seems to be an incorrect or out-of-order ID in the original sequence.

[0131] Repeat steps S440-S450 to obtain different enhanced feature maps. Finally, the salient targets in the image under test are obtained.

[0132] It should be noted that when there are no ungrouped original RGB features ( ) and thermal infrared characteristics ( When obtaining the salient target of the image to be tested, the specific steps are similar to steps S410 to S450 above, and will not be repeated here.

[0133] To more clearly illustrate the steps of the method described in the embodiments of the present invention, the following specific steps are given as an example, with the first three layers as shallow features and the last two layers as deep features. Please refer to the following examples. Figure 4 As shown:

[0134] Step 1: Acquire the RGB image of the image to be tested using the mobile phone camera, and acquire the thermal infrared image of the image to be tested using the MEMS thermal infrared sensor; extract five-level feature maps from the RGB image and the thermal infrared image using the MobileNetV2 backbone network.

[0135] Specifically, each level of multimodal feature map includes two feature maps, with the first level feature map including the first feature map. Second feature map The second-level feature map includes the first feature map. Second feature map The third-level feature map includes the first-level feature map. Second feature map The fourth-level feature map includes the first-level feature map. Second feature map The fifth-level feature map includes the first-level feature map. Second feature map .

[0136] Step 2: In the first 3 layers, respectively, the first feature map of the same layer... ( i =1, 2, 3) and the second feature map ( i Inputting (1, 2, 3) into three triple attention fusion modules yields the first-level fusion feature map. Second-level fusion feature map Feature maps fused with the third level It should be noted that the first-level fused feature map Second-level fusion feature map Feature maps fused with the third level Both can be regarded as the first fused feature map Specific form.

[0137] Step 3: In layers 4 and 5, respectively, the first feature map of the same layer... ( i =4, 5) and the second feature map ( i =4, 5) are input into two dual-path grouping fusion modules to obtain the fourth-level fusion feature map. fused feature map with the fifth level It should be noted that the fourth level fusion feature map fused feature map with the fifth level Both can be regarded as the second fusion feature map Specific form.

[0138] Step 4: Fuse the fifth-level feature map Input to the CBG 3x3 module and the Unsample 2x module to generate enhanced feature maps. The CBG 3x3 module is a convolutional block containing 3x3 convolutional kernels, followed by a Batch Normalization (BN) layer, which aims to accelerate the training process and enhance the model's stability. Then, the GELU (Gaussian Error Linear Unit) activation function is applied, a non-linear activation function that helps the network capture more complex features.

[0139] Furthermore, the Unsample 2x module performs an upsampling operation using bilinear interpolation, with an upsampling ratio of 2. This upsampling technique increases the resolution of the feature map by inserting new pixels between the original pixels, thus providing a higher-resolution feature map for subsequent processing steps. The specific expression is as follows:

[0140] .

[0141] Step 5: Enhance the feature map Feature map fusion with fourth level The splicing process is performed to obtain the spliced ​​feature map. The feature maps will be stitched together. The inputs are fed into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature maps. The specific expression is:

[0142] ;

[0143] .

[0144] Step 6: Enhance Feature Maps Feature map fused with third level Perform splicing processing to obtain splicing features. The feature maps will be stitched together. The inputs are fed into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature maps. The specific expression is:

[0145] ;

[0146] .

[0147] Step 7: Enhance Feature Maps Feature maps fused with the second level The splicing process is performed to obtain the spliced ​​feature map. The feature maps will be stitched together. The inputs are fed into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature maps. The specific expression is:

[0148] ;

[0149] .

[0150] Step 8: Enhance the feature map Feature map fused with first level The splicing process is performed to obtain the spliced ​​feature map. The feature maps will be stitched together. The inputs are fed into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature maps. The specific expression is:

[0151] ;

[0152] .

[0153] Step 9: Based on the enhanced feature map The final salient target of the image under test is obtained.

[0154] This invention presents a multi-layered modal interaction network architecture, deploying TSAFM and DGFM modules layer by layer, acting on shallow layers (e.g., the first three layers) and deep layers (e.g., the last two layers), respectively. This forms a hierarchical interaction framework of spatial detail enhancement and semantic feature complementarity, providing a new design paradigm for multi-modal feature fusion. The designed TSAFM module addresses the shortcomings of lightweight backbone networks in edge detail processing by dynamically weighting the channel attention of three features (RGB, thermal infrared, and differential features). The designed DGFM module employs a dual-path strategy combining grouped convolution and direct concatenation to achieve fine-grained and coarse-grained fusion of deep semantic features, balancing local modal difference capture with global semantic consistency maintenance.

[0155] This invention utilizes multi-level feature extraction and modal interaction to achieve accurate and robust salient target segmentation. Multi-level feature extraction enriches semantic information, modal interaction fuses complementary information, an attention mechanism highlights key features, and dual-path grouping enhances fusion flexibility, significantly improving segmentation accuracy and robustness. To verify the effectiveness of the method, comparative experiments were designed and analyzed in depth. The specific experimental content is as follows:

[0156] Obtaining the experimental dataset: In the experiment, the preferred datasets used were VT817, VT1000, and VT5000. The specific data allocation of the three datasets used in this experiment is shown in Table 1. During the data preprocessing stage, to ensure the convenience and efficiency of model processing, all input images were adjusted to a uniform size of 224×224 pixels, effectively reducing computational complexity. Simultaneously, to enhance the model's generalization ability and robustness, data augmentation operations such as random flipping, rotation, and cropping were performed on the training set.

[0157] Table 1. Data Allocation for Training and Testing Sets

[0158]

[0159] Experimental environment and parameter settings, specifically including:

[0160] Hardware configuration: Equipped with an 11th Gen Intel® Core™ i7-11800H processor (2.30GHz) and an NVIDIA GeForce RTX 3060 Laptop GPU.

[0161] Software environment: Based on Python 3.9 and the PyTorch framework.

[0162] The parameter settings are shown in Table 2.

[0163] Table 2 Experimental parameters

[0164]

[0165] Set evaluation metrics: Key metrics used to evaluate the performance of the detection model include S-measure (S-measure), E-measure (E-measure), F-measure (F-measure), and Mean Absolute Error (MAE).

[0166] The S-measure measures the structural similarity between a predicted map and a ground truth map; it is a structural similarity metric. It comprehensively considers both region-aware and object-aware factors, effectively assessing structural integrity and more closely reflecting the sensitivity of the human visual system to object structure. The specific calculation formula is as follows:

[0167] ;

[0168] in, It represents the perceived structural similarity of objects and is used to calculate the structural similarity between the saliency map and the object regions in the ground truth. It represents region-aware structural similarity and is used to calculate the structural similarity between the saliency map and each superpixel region in the ground truth. This indicates the weight used to balance the first two, which is preferably 0.5 in this experiment.

[0169] The E-measure is a region-based evaluation metric that assesses overall quality by calculating the local alignment between the saliency map and the ground truth. It effectively simulates human visual discrimination of image targets in salient object detection. Its adaptive version (adpE) uses the mean of the saliency map as a threshold to generate a binary map, simultaneously capturing the global and local similarity between the saliency map and the ground truth, resulting in more accurate evaluation of salient object detection in complex scenes. The specific formula for calculating the E-measure is as follows:

[0170] ;

[0171] in, This represents the enhanced alignment matrix, used to reflect the saliency map and the true value at the pixel level. The correlation at each point is calculated. An adaptive threshold, set to twice the mean of the saliency map, is used to convert the saliency map into a binary map, thereby calculating the alignment. Indicates the height of the image; Indicates the width of the image; H × W This represents the total number of pixels in the image.

[0172] The F-measure is the harmonic mean of precision and recall, used to comprehensively evaluate the detection accuracy of the saliency map. Its adaptive version (adpF) uses twice the mean of the saliency map as a threshold to generate a binary map. This metric quantifies the proportion of correctly detected pixels in the saliency map by balancing precision and recall, thus evaluating the overall detection performance of the model at different thresholds. The specific formula for calculating the F-measure is:

[0173] ;

[0174] In this experiment, Setting it to 0.3 emphasizes the importance of precision. and The expression is:

[0175] ;

[0176] ;

[0177] in, , , These represent the number of pixels for true positives, false positives, and false negatives, respectively.

[0178] MAE is a pixel-level error metric that calculates the sum of the absolute differences between the saliency map and the ground truth at each pixel, and then averages these sums. It directly reflects the overall consistency between the saliency map and the ground truth. A smaller MAE value indicates a smaller global error distribution and higher consistency between the saliency map and the ground truth, thus providing a direct assessment of the model's global performance. The specific formula for calculating MAE is:

[0179] ;

[0180] in, Is the saliency map at the pixel The value at that location, Is it the actual value in pixels? The value at that location.

[0181] Based on the aforementioned experimental dataset, experimental environment, parameter settings, and evaluation metrics, the final experimental results were obtained. This experiment evaluated the performance of the present invention and compared it with several common RGB-T-based salient object detection models. Detailed results are shown in Table 3. These models include: Large-Small Network (LSNet), Few-shot Multimodal Aspect-based Sentiment Analysis Framework based on Contrastive Finetuning (FMCF), Adaptive Fusion Network (ADF), and Multimodal Independent Decoding and Decoding (MIDD). The results show that the embodiment of the present invention is optimal across all evaluation metrics on all datasets.

[0182] Table 3 Experimental Results

[0183]

[0184] Furthermore, the effectiveness of the TSAFM and DGFM modules was further confirmed through ablation experiments in this embodiment of the invention, as shown in Table 4. When these two modules work together, they can significantly enhance the model's feature learning ability, thereby improving the accuracy of target boundary localization and the ability to detect semantic integrity in complex scenes.

[0185] Table 4 Ablation Experiment Results

[0186]

[0187] Furthermore, the modules designed in this embodiment of the invention exhibit differentiated performance for scenarios of varying complexity and data volumes. Ablation experiments clearly demonstrate that the DGFM module performs better on small datasets and simpler scenarios, while the TSAFM module excels at handling complex scenarios. Therefore, this invention comprehensively enhances model performance by applying these two modules to feature maps at different levels and modalities.

[0188] Furthermore, the model's qualitative results in salient object detection are applicable to, for example... Figure 8 As shown in the diagram, the performance differences between different models are clearly visible through the intuitive effect diagrams. The embodiments of this invention can more accurately identify salient regions in a scene and more effectively complete boundary segmentation tasks, while other models exhibit varying degrees of shortcomings. Specifically, in... Figure 8In this context, RGB represents an RGB image; T represents a thermal infrared image; Ours represents the method described in this invention; and GT represents correctly labeled data.

[0189] Based on a comprehensive analysis of the experimental results, this invention demonstrates a significant improvement over other models in salient target detection, enabling more accurate localization of salient regions in a scene and better completion of boundary segmentation tasks.

[0190] This invention proposes a high-efficiency RGB-T salient target detection network architecture based on modal interaction. This architecture fully utilizes the complementary information of RGB and thermal imaging through cross-modal, cross-path, and cross-layer information interaction, achieving deep complementarity and fusion of features across different modalities at multiple levels. This allows the deep network to learn more comprehensive multimodal features. Simultaneously, this invention delves into the information interaction mechanisms between different layers, ensuring the model accurately captures multimodal features of the target at multiple levels. The proposed method significantly improves the model's ability to learn multimodal features through cross-modal, cross-path, and cross-layer information interaction, and demonstrates superior performance in experiments.

[0191] Example 2: Based on the same inventive concept, this example provides a salient target segmentation system based on multi-level modal interaction. The principle of solving the problem is similar to that of the salient target segmentation method based on multi-level modal interaction provided in Example 1, and the repeated parts will not be described again.

[0192] This embodiment provides a salient target segmentation system based on multi-level modal interaction, including:

[0193] The feature extraction module is used to acquire the RGB image and thermal infrared image of the image to be tested, and extract multiple layers of feature maps from the RGB image and thermal infrared image;

[0194] The first fusion feature map acquisition module is used to construct the triple attention fusion module and input the multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, the difference feature maps of different modal feature maps of the same level are obtained, and the first fusion feature map is obtained based on the difference feature maps;

[0195] The second fusion feature map acquisition module is used to construct the dual-path grouping fusion module and input the multimodal feature maps of the same level into the dual-path grouping fusion module; in the dual-path grouping fusion module, RGB feature groups and thermal infrared feature groups are obtained, and the second fusion feature map is obtained based on the RGB feature groups and thermal infrared feature groups;

[0196] The salient target output module is used to obtain the salient targets of the image under test based on the first fused feature map and the second fused feature map.

[0197] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A salient target segmentation method based on multi-level modal interaction, characterized in that... ,include: Acquire the RGB image and thermal infrared image of the image to be tested, and extract multiple hierarchical feature maps from the RGB image and the thermal infrared image; A triple attention fusion module is constructed, and multimodal feature maps at the same level are input into the triple attention fusion module; in the triple attention fusion module, difference feature maps of different modal feature maps at the same level are obtained, and a first fused feature map is obtained based on the difference feature maps; wherein, the step of obtaining the first fused feature map is as follows: Obtain the first and second feature maps at the same level, and calculate the difference feature map between the first and second feature maps; The first feature map, the second feature map, and the difference feature map are compressed into channel-level vectors respectively to obtain the first global feature, the second global feature, and the third global feature; The first global feature, the second global feature, and the third global feature are input into the attention sub-network and dynamically weighted and fused to obtain the first fused feature map; A dual-path grouping fusion module is constructed, and multimodal feature maps at the same level are input into the dual-path grouping fusion module; in the dual-path grouping fusion module, RGB feature groups and thermal infrared feature groups are obtained, and a second fused feature map is obtained based on the RGB feature groups and the thermal infrared feature groups; wherein, the step of obtaining the second fused feature map is as follows: Obtain multimodal feature maps at the same level, and group RGB features and thermal infrared features to obtain RGB feature groups and thermal infrared feature groups; The RGB feature group and the thermal infrared feature group are spliced ​​across modes to obtain a spliced ​​feature group; The spliced ​​feature group is subjected to dimensionality reduction processing to obtain a fine-grained fused feature map; Ungrouped RGB features and thermal infrared features are directly spliced ​​together to obtain coarse-grained fused features; the coarse-grained fused features and the fine-grained fused features are spliced ​​together to obtain a second fused feature map; The salient targets of the image under test are obtained based on the first fused feature map and the second fused feature map.

2. The salient target segmentation method based on multi-level modal interaction according to claim 1, characterized in that, The steps for inputting the first global feature, the second global feature, and the third global feature into the attention subnetwork and performing dynamic weighted fusion to obtain the first fused feature map are as follows: Perform a first convolution operation on the first global feature, the second global feature, and the third global feature respectively to extract enhanced feature maps; A second convolution operation is performed on the enhanced feature map to obtain a weight coefficient map; Based on the weight coefficient map, the first global feature, the second global feature, and the third global feature are dynamically weighted and fused to obtain the first fused feature map.

3. The salient target segmentation method based on multi-level modal interaction according to claim 2, characterized in that, The steps of the first convolution operation are as follows: compress the spatial dimension of the input global features and extract key information by halving the number of channels; use an activation function to enhance the key information to obtain an enhanced feature map.

4. The salient target segmentation method based on multi-level modal interaction according to claim 2, characterized in that, The steps of the second convolution operation are as follows: output a single-channel feature map based on the input enhanced feature map; process the single-channel feature map using an activation function to obtain a weight coefficient map.

5. The salient target segmentation method based on multi-level modal interaction according to claim 1, characterized in that, The steps for obtaining the salient targets in the image under test based on the first fused feature map and the second fused feature map are as follows: Feature extraction and upsampling are performed on the second fused feature map in the nth layer feature map to obtain the first enhanced feature map; where n is a positive integer greater than 4 and decreasing in value. The first enhanced feature map is concatenated with the second fused feature map in the (n-1)th layer feature map to obtain the first concatenated feature map; The first spliced ​​feature map is subjected to feature extraction and upsampling to obtain the second enhanced feature map; The second enhanced feature map is subjected to splicing processing with the first fusion feature map in the m-th layer feature map to obtain a second spliced feature map; where 1 m < n - 1, and m is a positive integer; The second stitched feature map is subjected to feature extraction and upsampling to obtain the salient target of the image to be tested.

6. A salient target segmentation system based on multi-level modal interaction, used to implement the method according to any one of claims 1 to 5, characterized in that, include: The feature extraction module is used to acquire the RGB image and thermal infrared image of the image to be tested, and extract multiple layers of feature maps from the RGB image and the thermal infrared image; The first fusion feature map acquisition module is used to construct the triple attention fusion module and input the multimodal feature maps of the same level into the triple attention fusion module; In the triple attention fusion module, the difference feature map of different modal feature maps at the same level is obtained, and the first fused feature map is obtained based on the difference feature map; The second fusion feature map acquisition module is used to construct the dual-path grouping fusion module and input the multimodal feature maps of the same level into the dual-path grouping fusion module; In the dual-path grouping fusion module, RGB feature groups and thermal infrared feature groups are obtained, and a second fused feature map is obtained based on the RGB feature groups and the thermal infrared feature groups; A salient target output module is used to obtain the salient targets of the image under test based on the first fused feature map and the second fused feature map.

Images