Significant target segmentation method and system based on multilevel modal interaction

Abstract

The invention relates to a salient target segmentation method and system based on multilevel modal interaction, and belongs to the technical field of computer vision. Comprising the following steps: acquiring an RGB image and a thermal infrared image of a to-be-detected image, and extracting a plurality of hierarchical feature maps; inputting the multi-modal feature maps of the same level into a triple attention fusion module, obtaining a difference feature map of different modal feature maps of the same level in the triple attention fusion module, and obtaining a first fusion feature map according to the difference feature map; inputting the multi-modal feature maps of the same level into a dual-path grouping fusion module, obtaining an RGB feature group and a thermal infrared feature group in the dual-path grouping fusion module, and obtaining a second fusion feature map according to the RGB feature group and the thermal infrared feature group; and obtaining a salient target of the to-be-detected image according to the first and second fusion feature maps. According to the method, features of different levels and modes are fully utilized, complementary features between the RGB image and the thermal infrared image are fused, redundant and non-significant feature interference is avoided, and the performance and generalization ability of significant target detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a salient object segmentation method and system based on multi-level modal interaction. Background Art

[0002] With the continuous advancement of computer vision technology, salient object detection has gradually become a research hotspot. In recent years, this task has entered its third phase, based on deep learning models. Deep learning models based on convolutional neural networks (CNNs) have made significant progress in the field of salient object detection due to their powerful feature extraction capabilities and model complexity. These models typically contain hundreds of thousands of adjustable parameters and have neurons with receptive fields of varying sizes. Neurons with large receptive fields provide global information, helping the model identify the most salient areas in the image; neurons with small receptive fields provide local information, which is used to refine the saliency map. This combination of multi-scale information lays the foundation for the introduction of more modalities and more complex models to the salient object detection task.

[0003] At the same time, hardware such as thermal infrared sensors continues to develop and become more popular. Existing thermal infrared information can effectively solve the problem of target blur caused by factors such as lighting, occlusion, and complex background. Although the RGB-T fusion method has improved the detection performance to a certain extent, it still has some shortcomings. Most current models directly extract and fuse original features from the backbone, which is susceptible to the limitations of low-quality modal data and redundant cross-modal features. The operation of fusing multimodal features through simple concatenation or element-by-element summation does not consider the importance of different modal features and fails to explore the complementary information features between RGB images and thermal infrared images. This fusion method allows the inclusion of redundant and non-salient features, making the fusion process unable to achieve good complementarity. Summary of the Invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the limitations of the fusion process in the prior art due to low-quality modal data and redundant cross-modal features, as well as the poor fusion effect.

[0005] In a first aspect, to solve the above technical problems, the present invention provides a salient object segmentation method based on multi-level modal interaction, comprising: Acquire an RGB image and a thermal infrared image of the image to be measured, and extract multiple hierarchical feature maps of the RGB image and the thermal infrared image; Constructing a triple attention fusion module and inputting the multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, obtaining a difference feature map of the feature maps of different modalities at the same level, and obtaining a first fused feature map based on the difference feature map; Constructing a dual-path grouping fusion module and inputting the multimodal feature maps of the same level into the dual-path grouping fusion module; obtaining RGB feature grouping and thermal infrared feature grouping in the dual-path grouping fusion module, and obtaining a second fused feature map based on the RGB feature grouping and the thermal infrared feature grouping; Obtaining a salient object in the image to be tested according to the first fused feature map and the second fused feature map.

[0006] In one embodiment of the present invention, in the triple attention fusion module, the steps of obtaining the difference feature map of the multimodal feature map at the same level and obtaining the first fusion feature map according to the difference feature map are as follows: Obtaining a first feature map and a second feature map at the same level, and calculating a difference feature map between the first feature map and the second feature map; Compressing the first feature map, the second feature map, and the difference feature map into channel-level vectors respectively to obtain a first global feature, a second global feature, and a third global feature; The first global feature, the second global feature, and the third global feature are input into the attention subnetwork and dynamically weighted fused to obtain a first fused feature map.

[0007] In one embodiment of the present invention, the first global feature, the second global feature, and the third global feature are input into the attention subnetwork and dynamically weighted fused to obtain a first fused feature map. The steps are as follows: performing a first convolution operation on each of the first global feature, the second global feature, and the third global feature to extract an enhanced feature map; Performing a second convolution operation on the enhanced feature map to obtain a weight coefficient map; According to the weight coefficient map, the first global feature, the second global feature and the third global feature are dynamically weighted fused to obtain a first fused feature map.

[0008] In one embodiment of the present invention, the steps of the first convolution operation are: compressing the spatial dimension of the input global features and extracting key information by halving the number of channels; and enhancing the key information using an activation function to obtain an enhanced feature map.

[0009] In one embodiment of the present invention, the steps of the second convolution operation are: outputting a single-channel feature map based on the input enhanced feature map; and processing the single-channel feature map using an activation function to obtain a weight coefficient map.

[0010] In one embodiment of the present invention, in the dual-path grouping fusion module, the steps of obtaining the RGB feature grouping and the thermal infrared feature grouping, and obtaining the second fusion feature map according to the RGB feature grouping and the thermal infrared feature grouping are as follows: Obtain multi-modal feature maps at the same level, group the RGB features and the thermal infrared features to obtain the RGB feature grouping and the thermal infrared feature grouping; Perform cross-modal splicing on the RGB feature grouping and the thermal infrared feature grouping to obtain a spliced feature group; Perform dimensionality reduction processing on the spliced feature group to obtain a fine-grained fusion feature map.

[0011] In one embodiment of the present invention, when performing cross-modal splicing on the RGB feature grouping and the thermal infrared feature grouping to obtain a spliced feature group, it further includes directly splicing the ungrouped RGB features and thermal infrared features to obtain a coarse-grained fusion feature; splicing the coarse-grained fusion feature and the fine-grained fusion feature to obtain the second fusion feature map.

[0012] In one embodiment of the present invention, according to the second fusion feature map and the first fusion feature map, the significant target of the待测图像 (to be measured image) is obtained.

[0013] In one embodiment of the present invention, the steps of obtaining the significant target of the待测图像 (to be measured image) according to the second fusion feature map and the first fusion feature map are as follows: Perform feature extraction and upsampling processing on the second fusion feature map in the nth layer feature map to obtain a first enhanced feature map; where n is a positive integer that decreases from a value greater than 4; Perform splicing processing on the first enhanced feature map and the second fusion feature map in the (n - 1)th layer feature map to obtain a first spliced feature map; Perform feature extraction and upsampling processing on the first spliced feature map to obtain a second enhanced feature map; Perform splicing processing on the second enhanced feature map and the first fusion feature map in the mth layer feature map to obtain a second spliced feature map; where 1 m < n - 1, and m is a positive integer; Perform feature extraction and upsampling processing on the second spliced feature map to obtain the significant target of the待测图像 (to be measured image).

[0014] Second, to solve the above technical problems, the present invention provides a significant target segmentation system based on multi-level modal interaction, including: A feature extraction module, configured to obtain the RGB image and the thermal infrared image of the待测图像 (to be measured image), and extract multiple-level feature maps of the RGB image and the thermal infrared image; a first fusion feature map acquisition module, configured to construct a triple attention fusion module and input the multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, obtaining a difference feature map of the feature maps of different modalities at the same level, and obtaining a first fusion feature map based on the difference feature map; A second fusion feature map acquisition module is used to construct a dual-path grouping fusion module and input the multimodal feature map at the same level into the dual-path grouping fusion module; in the dual-path grouping fusion module, RGB feature grouping and thermal infrared feature grouping are obtained, and a second fusion feature map is obtained based on the RGB feature grouping and the thermal infrared feature grouping; The salient target output module is used to obtain the salient target of the image to be tested according to the first fused feature map and the second fused feature map.

[0015] The above technical solution of the present invention has the following beneficial effects compared with the prior art: The present invention discloses a method and system for salient target segmentation based on multi-level modal interaction, which constructs a triple attention fusion module and a dual-path grouping fusion module. The triple attention fusion module cleverly fuses RGB images, thermal infrared images and their difference features through a dynamic weighting mechanism, and uses a channel attention mechanism to effectively make up for the shortcomings of the lightweight backbone network in detail processing, thereby enhancing the expression ability of spatial detail features. The dual-path grouping fusion module adopts a dual-path strategy that combines group convolution with direct splicing to perform a detailed and coarse dual fusion of deep semantic features, which not only optimizes the capture of local feature differences, but also ensures the consistency of global semantics, and realizes deep complementarity and optimization of semantic features. The two modules work together to construct a multi-level interaction framework, which opens up new design ideas for multimodal feature fusion, can fully tap the unique advantages of each modality, and realize complementary advantages, making the entire system more flexible and efficient in the feature fusion process, and not affected by low-quality data and redundant features. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of early fusion and late fusion in the prior art; Figure 2 This is a flow chart of hybrid fusion in the prior art; Figure 3 Flowchart of a salient object segmentation method based on multi-level modal interaction in a preferred embodiment of the present invention; Figure 4 This is a diagram of the network architecture structure in a preferred embodiment of the present invention; Figure 5 This is a flow chart of the triple attention fusion module in the preferred embodiment of the present invention; Figure 6 Flowchart of the attention sub-network in the triple attention fusion module in the preferred embodiment of the present invention; Figure 7 This is a flow chart of a dual-path packet fusion module in a preferred embodiment of the present invention; Figure 8 Graphs showing salient target detection results using different models in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0018] The technologies involved in the present invention mainly include salient target detection and modality fusion.

[0019] In terms of salient object detection, current deep learning-based models are mainly divided into two categories: models based on convolutional neural networks (CNN) and models based on vision transformers (ViT).

[0020] CNN models, through their convolutional neurons' ability to sensitively capture local information, achieve significant performance improvements over traditional machine learning methods in salient object detection tasks. MFSR was the first salient object detection model to adopt a CNN architecture. With the continuous advancement of deep learning technology, CNN-based models have further evolved and become subdivided, primarily into the following three categories: The first category of methods is based on superpixels. In traditional machine learning algorithms, manually designing feature extraction is both time-consuming and laborious. To address this issue, researchers have begun using convolutional neural networks (CNNs) to automatically learn the relationships between superpixels, leveraging the deep features extracted by CNNs to overcome the limitations of traditional manual feature extraction methods. However, this approach still suffers from some drawbacks, such as low detection accuracy and unclear object boundaries.

[0021] The second category involves feature enhancement. These methods perform a series of transformations on RGB or thermal infrared images to generate more discriminative feature maps. These enhanced feature maps help the model learn more efficiently, thereby improving the accuracy and speed of salient object detection.

[0022] The third category of methods is feature fusion (modal interaction). This approach aims to improve model learning efficiency by integrating complementary information between different modalities. However, simply fusing different modalities directly can introduce noise, which can negatively impact model performance.

[0023] In 2021, SwinNet, the first ViT-based salient object detection model, was proposed for the ViT model. The ViT model converts the input image into a sequence, adds positional encoding to preserve the image sequence relationship, and then uses the self-attention mechanism to calculate the correlation between each element in the sequence to complete the image classification task. This approach gives the model stronger global feature modeling capabilities than CNN-based models, making it perform better in detecting the completeness of the target. However, because the image is divided into multiple small patches (patches), this method still has shortcomings in capturing local detail information. This lack of local information may affect the model's accurate recognition and positioning of target details.

[0024] In terms of modal fusion, current research usually uses whether a method depends on the implementation of a specific model to classify modal fusion technology. If it does not depend on a specific deep learning model, it is a model-independent method; otherwise, it is a model-dependent method.

[0025] For model-independent methods, such methods are mainly divided into three strategies: early fusion, late fusion and hybrid fusion.

[0026] Early fusion directly merges multimodal data at the input layer or shallow network layer, that is, directly splicing experimental channels (such as RGB-T four-channel input) or weighted superposition to fuse different feature maps. The advantage of this method is that it retains the physical properties of the original modality and can directly combine the temperature sensitivity of thermal infrared with the texture information of RGB. However, it is susceptible to modal noise interference. Since the correlation between different modalities is difficult to extract, the complementarity between different modules cannot be fully reflected. The specific process is as follows: Figure 1 As shown in (a) in .

[0027] Late fusion adopts a multi-stream independent coding structure, trains different modalities through different models, and finally aggregates features only at the decision layer, so it is also called a decision-level fusion method. The fusion process of this method is independent of the features of each modality, and the results of different modalities trained by different models are often less correlated, so this strategy effectively avoids the problem of modal interference. At present, the late fusion strategy mainly determines the output results of different models by specifying different rules, thereby achieving the purpose of changing the fusion results, such as maximum fusion, average fusion, Bayesian rule fusion, and ensemble learning. Figure 1 As shown in (b) in .

[0028] Hybrid Fusion adopts a multi-level progressive interaction strategy, combining early and late fusion strategies, such as Figure 2 A typical example is the cross-modal interaction network under the Swin-Transformer architecture, which achieves feature complementarity through cross-attention and introduces a multi-scale fusion module into the decoder. High-level semantic fusion is guided by channel attention, while low-level spatial attention enhances detail alignment. This strategy combines the advantages of both, but inevitably increases the structural complexity and training difficulty of the model. With the rapid development and widespread application of deep learning models, model structures have become more diverse and flexible. Although hybrid fusion methods are relatively complex to implement, they remain a widely used modal fusion strategy because they can fully utilize the advantages of different models.

[0029] The above three strategies are common fusion methods, and there is a clear technical evolutionary thread between them: early fusion methods focused on data-level merging, but lacked robustness; later methods focused on maintaining the independence of each modality, which may sacrifice some advantages of information complementarity; and hybrid fusion strategies excel in improving detection accuracy by establishing a hierarchical interaction network, but this strategy also inevitably increases the complexity of the model, which may affect the efficiency of calculation and training.

[0030] Model-related methods can be seen as extensions based on the characteristics of specific deep learning models. They use the characteristics of deep learning models to solve fusion problems, including multiple kernel learning (MKL) methods, graphical models (GM) methods, and neural networks (NN) methods.

[0031] To address the shortcomings of the aforementioned existing technologies, this paper designs a network architecture for efficient RGB-T salient object detection based on modal interaction. This architecture allows for cross-modal, cross-path, and cross-layer information exchange, fully leveraging the complementary information between RGB and thermal imaging. This architecture enables multi-level complementarity and fusion of features from different modalities, enabling deep networks to learn more comprehensive multimodal features.

[0032] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0033] Example 1: Reference Figure 3As shown, an embodiment of the present invention provides a salient object segmentation method based on multi-level modal interaction, including but not limited to the following steps: S1. Obtain an RGB image and a thermal infrared image of the image to be tested, and extract multiple hierarchical feature maps of the RGB image and the thermal infrared image; S2. Construct a triple attention fusion module and input the multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, obtain the difference feature maps of the different modal feature maps of the same level, and obtain the first fused feature map based on the difference feature maps; S3. Construct a dual-path grouping fusion module and input the multimodal feature map of the same level into the dual-path grouping fusion module; in the dual-path grouping fusion module, obtain RGB feature grouping and thermal infrared feature grouping, and obtain a second fused feature map based on the RGB feature grouping and the thermal infrared feature grouping; S4. Obtain a salient object in the image to be tested according to the first fused feature map and the second fused feature map.

[0034] The embodiment of the present invention provides a salient object segmentation method based on multi-level modal interaction and constructs a multi-level modal interaction network (MMINet) architecture. This architecture designs a triple attention fusion module and a dual-path group fusion module, which act on the shallow and deep layers of the multimodality, respectively, forming a hierarchical interaction framework of spatial detail enhancement and semantic feature complementarity, providing a new design paradigm for multimodal feature fusion. For the triple attention fusion module, by dynamically weighted fusion of RGB, thermal infrared and their difference features, the channel attention mechanism is used to address the limitations of lightweight backbone networks in edge detail processing. For the dual-path group fusion module, a dual-path strategy combining group convolution and direct splicing is adopted to achieve fine-grained and coarse-grained fusion of deep semantic features, effectively balancing the capture of local modal differences and the maintenance of global semantic consistency. During the feature fusion process, it is not limited by low-quality modal data and redundant cross-modal features, thereby achieving a better fusion effect. This method not only improves the accuracy of salient object segmentation, but also enhances the model's adaptability and generalization performance to complex scenes.

[0035] Specifically, in step S1, an RGB image and a thermal infrared image of the image to be measured are obtained. RGB images can be captured using common devices such as ordinary digital cameras, SLR cameras, and mobile phone cameras, or can be acquired using specialized equipment such as industrial cameras and panoramic cameras. USB or PCIe image acquisition cards can also be used to convert image signals from external cameras into digital signals. Thermal infrared images can be captured using handheld, fixed, or drone-mounted thermal infrared cameras, or can be acquired using portable or vehicle-mounted thermal infrared imaging systems, as well as MEMS thermal infrared sensors, thermopile sensors, and the like. Furthermore, multimodal imaging devices such as dual-spectral cameras and multispectral cameras can be used to simultaneously acquire RGB and thermal infrared images, or a synchronous acquisition system can be used to ensure that both modal images are acquired at the same time.

[0036] Specifically, to address issues such as insufficient modal interaction and poor feature complementarity, the present invention employs a Triple Stream Attention Fusion Module (TSAFM) and a Dual-path Group Fusion Module (DGFM). TSAFM leverages a channel-attention mechanism to dynamically weighted-fuse RGB, thermal infrared, and their differential features, emphasizing the complementarity of shallow spatial information. DGFM, on the other hand, combines grouped convolution with direct concatenation to efficiently fuse deep semantic features, cleverly balancing local details with global semantic consistency. These two modules, respectively, are applied to the shallow and deep multimodal features of the backbone network, employing targeted modal interaction strategies for features at different levels, thereby comprehensively improving the performance of model feature fusion and object detection.

[0037] The existing technology adopts a direct element-by-element summation method for multimodal feature fusion, which can easily limit the model performance by low-quality modal data and redundant cross-modal features. This direct element-by-element summation fusion method fails to deeply explore the correlation and importance between different modalities, and carries a large amount of redundant information, which cannot effectively achieve feature complementarity, thus having a certain impact on model performance. To address this problem, the embodiment of the present invention is based on the lightweight MobileNetV2 backbone network, designs the TSAFM module and the DGFM module, and constructs a hierarchical multimodal interaction framework, which can be referred to Figure 4 Among them, the TSAFM module is used to fuse the first few layers (such as the first three layers), while the DGFM module is specifically used to fuse the last few layers (such as the last two layers).

[0038] Specifically, the first three layers, as shallow features, have high resolution and contain rich spatial structural information such as edges, contours, and textures. However, their semantic abstraction is relatively low. Therefore, with the help of the difference feature calculation in the TSAFM module, the differences between RGB and thermal infrared images in terms of edge position and texture details can be directly captured, thereby enhancing spatial consistency. At the same time, the channel attention weighting mechanism can dynamically adjust the contribution ratio of different modalities for local areas of low-level features. This mechanism effectively avoids the problem of blurred spatial details caused by the use of depthwise separable convolution in the lightweight backbone network MobileNetV2, thereby providing clearer and more accurate edge cues for the subsequent decoder.

[0039] Specifically, the latter two layers, as deep features, have lower resolution but higher number of channels, and mainly focus on semantic abstraction, but the spatial position information is relatively weakened. To this end, the DGFM module divides the high-channel features into two groups, each of which processes local semantics independently, thereby avoiding the smoothing of details by global operations. At the same time, the module also processes the ungrouped original spliced ​​features to retain the overall consistency of high-level semantics and prevent semantic fragmentation caused by grouping operations. Therefore, in the DGFM module, fine-grained local semantics are combined with coarse-grained global semantics, which not only retains the details of the components but also maintains the overall semantics of the object, effectively solving the positioning ambiguity problem caused by the low resolution of high-level features.

[0040] Specifically, in step S2, the constructed TSAFM module is as follows: Figure 5 As shown in Figure 2, its operational process closely revolves around the dynamic weighted fusion of cross-modal features. The constructed TSAFM module introduces a channel attention mechanism, enabling the adaptive integration of complementary features across modalities. The TSAFM module also includes an attention subnetwork.

[0041] Furthermore, the TSAFM module is used to obtain the difference feature maps of different modal feature maps at the same level. , and according to the difference feature map , get the first fusion feature map The specific steps are: S210, obtain the first feature map at the same level and the second feature map , calculate the first feature map and the second feature map The difference feature map between , the calculation formula is: ; in, The value of can be 1, 2, 3, 4, 5, etc. This difference feature captures the uniqueness of the two modalities in spatial structure and semantic information.

[0042] S220, respectively, the first feature map , the second feature map and difference feature maps Compressed into channel-level vectors, the first global feature is obtained , the second global feature and the third global feature .

[0043] For example, the above three feature maps ( , , ) compresses each feature map into a channel-level vector through the global average pooling layer to obtain the corresponding global feature The specific expression is: ; ; ; in, Indicates compressing the feature map to a spatial size of 1×1.

[0044] S230, the first global feature , the second global feature and the third global feature Input into the attention sub-network and perform dynamic weighted fusion to obtain the first fusion feature map Among them, the attention sub-network includes two layers of convolution, which can be referred to Figure 6 As shown. In the attention sub-network, dynamic weighted fusion is performed to obtain the first fusion feature map The specific steps are: S231, respectively, the first global feature , the second global feature and the third global feature Perform the first convolution operation to extract the enhanced feature map ( ). Among them, the first convolution operation steps are: the global features of the input ( ) to compress the spatial dimension and extract key information by halving the number of channels; use the activation function to enhance the key information and obtain the enhanced feature map ( ).

[0045] S232, enhance the feature map ( ) Perform the second convolution operation to obtain the weight coefficient map Among them, the second convolution operation steps are: according to the input enhanced feature map ( ), output single channel feature map ( ); Use activation function to transform single channel feature map ( ) is processed to obtain the weight coefficient graph .

[0046] S233, according to the weight coefficient diagram , for the first global feature , the second global feature and the third global feature Perform dynamic weighted fusion to obtain the first fusion feature map .

[0047] For example, the obtained global features ( ) Input the attention sub-network consisting of two layers of convolution. The first layer of convolution reduces the number of channels by half and performs ReLU activation. The second layer of convolution restores the number of channels to 1 and generates the corresponding weight coefficients through the Sigmoid function, which are expressed as 、 and , the mathematical expression is: ; ; ; ; ; ; ; ; .

[0048] Based on these weights, the three types of features are dynamically weighted and fused to obtain the final first fusion feature map , whose expression is: ; where ⊙ denotes element-wise multiplication (and the scalar weights are expanded to the same size as the feature map via broadcasting).

[0049] In the above steps, the contribution of each modality at the same level is adaptively adjusted through the attention mechanism (for example, in low-light scenes, the weight of thermal infrared features will automatically increase to dominate the positioning of significant areas), and the difference feature map The module can also be supplemented with unique information between modalities to ensure complementarity between different modalities.

[0050] Specifically, in step S3, the constructed DGFM module is as follows: Figure 7As shown. This module realizes the effective fusion of RGB and thermal infrared features by combining group convolution and direct convolution. The RGB feature grouping is obtained by using the DGFM module ( ) and thermal infrared feature grouping ( ), and grouped by RGB features ( ) and thermal infrared feature grouping ( ), get the second fusion feature map The specific steps are: S310, obtaining a multimodal feature map at the same level ( ), the RGB features ( ) and thermal infrared characteristics ( ) to group and get RGB feature grouping ( ) and thermal infrared feature grouping ( ).

[0051] Specifically, for the features of a specific layer, the module will first group them by channel, and group the RGB features ( ) and thermal infrared characteristics ( ) were divided equally into two groups ( ),( ). Its purpose is to facilitate the independent processing of feature information of local areas, and then capture the differences in RGB and thermal infrared features in different local areas.

[0052] S320, grouping RGB features ( ) and thermal infrared feature grouping ( ) to perform cross-modal splicing and obtain the splicing feature group ( , ), ( , Specifically, the RGB features are grouped ( ) and the corresponding thermal infrared feature grouping ( ) are spliced ​​together to obtain ( , ), ( , ).

[0053] S330, perform dimensionality reduction processing on the spliced ​​feature group to obtain a fine-grained fusion feature map .

[0054] For example, it is preferred to perform dimensionality reduction processing on the concatenated features using a 1×1 convolution to obtain a fine-grained fusion feature map. , whose expression is: ; Here, [;] represents a concatenation operation. This fine-grained feature can well reflect the relationship between RGB and thermal infrared features in a local area and retain more local detail information.

[0055] S340, the module also performs ungrouped raw RGB features ( ) and thermal infrared characteristics ( ) are directly spliced. Direct splicing can retain the global information of the features and avoid the loss of overall semantic consistency due to grouping operations. Specifically, for the ungrouped RGB features ( ) and thermal infrared characteristics ( ) to directly splice and convolve to obtain coarse-grained fusion features Coarse-grained features focus more on reflecting the overall semantics and macroscopic structure of features. and fine-grained fusion features Splice to get the second fusion feature map .

[0056] For example, after splicing, 1×1 convolution is preferably performed to obtain coarse-grained fusion features , whose expression is: .

[0057] Fusing coarse-grained features and fine-grained fusion features Splice to get the second fusion feature map , the specific expression is: .

[0058] Through the above steps, the DGFM module combines fine-grained features Fusion with coarse-grained features Splicing is performed to organically combine local details with global semantics. The final feature It not only contains rich local information, but also shows good global semantic consistency.

[0059] Specifically, in step S4, when there are ungrouped original RGB features ( ) and thermal infrared characteristics ( ), according to the second fusion feature map and the first fusion feature map , get the salient target of the image to be tested. When there is no ungrouped original RGB feature ( ) and thermal infrared characteristics ( ), according to the first fusion feature map and the second fusion feature map , and obtain the salient targets of the image to be tested.

[0060] Specifically, referring to Figure 4 shown, according to the second fused feature map and the first fused feature map , the specific steps to obtain the salient object of the image to be measured are as follows: S410. Extract features and perform upsampling on the second fused feature map in the nth layer of feature maps to obtain the first enhanced feature map ; where n is a positive integer that decreases from a value greater than 4, i is a positive integer that increases from 1, and the specific expression is: .

[0061] S420. Concatenate the first enhanced feature map with the second fused feature map in the (n - 1)th layer of feature maps to obtain the first concatenated feature map , i is a positive integer that increases from 1, and the specific expression is: .

[0062] S430. Extract features and perform upsampling on the first concatenated feature map to obtain the second enhanced feature map , and the specific expression is: .

[0063] Repeat steps S420 - S430 to obtain different enhanced feature maps ; S440. Concatenate the second enhanced feature map with the first fused feature map in the mth layer of feature maps to obtain the second concatenated feature map ; where 1 <= m < n - 1, and m is a decreasing positive integer; the specific expression of the second concatenated feature map is: .

[0064] S450. Extract features and perform upsampling on the second concatenated feature map to obtain a new enhanced feature map .

[0065] Repeat steps S440 - S450 to obtain different enhanced feature maps , and finally obtain the salient object of the image to be measured. [[ID=It should be noted that when there is no ungrouped original RGB feature ( ) and thermal infrared characteristics ( ), the specific steps of obtaining the salient targets in the image to be tested are similar to the above steps S410 to S450 and will not be repeated here.

[0067] In order to more clearly illustrate the steps of the method of the embodiment of the present invention, the following specific steps are given by taking the first three layers as shallow features and the last two layers as deep features as an example. Figure 4 As shown: Step 1: Use the mobile phone camera to obtain the RGB image of the image to be tested, and the MEMS thermal infrared sensor to obtain the thermal infrared image of the image to be tested; use the MobileNetV2 backbone network to extract the five-level feature maps of the RGB image and the thermal infrared image.

[0068] Specifically, each level of multimodal feature map includes two feature maps. The first level feature map includes the first feature map and the second feature map ; The second level feature map includes the first feature map and the second feature map ; The third level feature map includes the first feature map and the second feature map ; The fourth level feature map includes the first feature map and the second feature map ; The fifth level feature map includes the first feature map and the second feature map .

[0069] Step 2: In the first three layers, the first feature map of the same layer is ( i =1, 2, 3) and the second feature map ( i =1, 2, 3) are input to three triple attention fusion modules to obtain the first-level fusion feature map , the second level fusion feature map And the third level fusion feature map It should be noted that the first level fusion feature map , the second level fusion feature map And the third level fusion feature map Both can be regarded as the first fusion feature map Specific form.

[0070] Step 3: In the 4th and 5th layers, the first feature map of the same layer is ( i =4, 5) and the second feature map (i =4, 5) are input to two dual-path group fusion modules to obtain the fourth-level fusion feature map And the fifth level fusion feature map It should be noted that the fourth level fusion feature map And the fifth level fusion feature map Both can be regarded as the second fusion feature map Specific form.

[0071] Step 4: Fuse the fifth-level feature map Input to the CBG 3x3 module and the Unsample 2x module to generate enhanced feature maps The CBG 3x3 module is a convolutional block containing a 3x3 convolution kernel, followed by a batch normalization (BN) layer designed to accelerate training and enhance model stability. The GELU (Gaussian Error Linear Unit) activation function is then applied, a nonlinear activation function that helps the network capture more complex features.

[0072] Furthermore, the Unsample 2x module performs an upsampling operation using bilinear interpolation with an upsampling ratio of 2. This upsampling technique increases the resolution of the feature map by inserting new pixels between the original pixels, thereby providing a higher resolution feature map for subsequent processing steps. The specific expression is: .

[0073] Step 5: Enhance the feature map Fusion feature map with the fourth level Perform splicing processing to obtain the splicing feature map . The spliced ​​feature map Input into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature map , the specific expression is: ; .

[0074] Step 6: Enhance the feature map Fusion feature map with the third level Perform splicing processing to obtain splicing features . The spliced ​​feature map Input into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature map , the specific expression is: ; .

[0075] Step 7: Enhance the feature map Fusion feature map with the second level Perform splicing processing to obtain the splicing feature map . The spliced ​​feature map Input into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature map , the specific expression is: ; .

[0076] Step 8: Enhance the feature map Fusion feature map with the first level Perform splicing processing to obtain the splicing feature map . The spliced ​​feature map Input into the CBG 3x3 module and the Unsample 2x module to obtain the enhanced feature map , the specific expression is: ; .

[0077] Step 9: Based on the enhanced feature map Get the final salient target of the image to be tested.

[0078] The embodiment of the present invention designs a network architecture for multi-level modal interaction, with TSAFM modules and DGFM modules deployed in layers, acting on shallow layers (e.g., the first three layers) and deep layers (e.g., the last two layers), respectively. This forms a hierarchical interactive framework for spatial detail enhancement and semantic feature complementarity, providing a new design paradigm for multimodal feature fusion. The designed TSAFM module addresses the shortcomings of lightweight backbone networks in processing edge details through dynamic weighting of channel attention of triple features (RGB, thermal infrared, and difference features). The designed DGFM module adopts a dual-path strategy combining grouped convolution and direct splicing to achieve fine-grained and coarse-grained fusion of deep semantic features, balancing the capture of local modal differences with the maintenance of global semantic consistency.

[0079] The present invention utilizes multi-level feature extraction and modal interaction to achieve accurate and robust salient object segmentation. Multi-level feature extraction enriches semantic information, modal interaction fuses complementary information, the attention mechanism highlights key features, and dual-path grouping enhances fusion flexibility, significantly improving segmentation accuracy and robustness. To verify the effectiveness of this method, comparative experiments were designed and analyzed in depth. The specific experimental contents are as follows: Obtaining Experimental Datasets: The datasets used in this experiment are preferably VT817, VT1000, and VT5000. The specific data allocations for the three datasets used in this experiment are shown in Table 1. During data preprocessing, to ensure ease and efficiency of model processing, all input images were resized to a uniform size of 224×224 pixels, effectively reducing computational complexity. Furthermore, to enhance the model's generalization and robustness, data augmentation operations such as random flipping, rotation, and cropping were performed on the training set.

[0080] Table 1 Data distribution of training set and test set

[0081] Experimental environment and parameter settings, including: Hardware: 11th Gen Intel (R) Core (TM) i7-11800H processor (2.30GHz) and NVIDIA GeForce RTX 3060 Laptop GPU.

[0082] Software environment: Based on Python 3.9 and Pytorch framework.

[0083] The parameter settings are shown in Table 2.

[0084] Table 2 Experimental parameters

[0085] Set evaluation metrics: The key metrics used to evaluate the performance of the detection model include S-measure (abbreviated as S-measure), E-measure (abbreviated as E-measure), F-measure (abbreviated as F-measure), and Mean Absolute Error (MAE).

[0086] The S measure is used to measure the structural similarity between the predicted image and the true image. It is a structural similarity metric. It comprehensively considers the factors of region-awareness and object-awareness, can effectively evaluate the integrity of the structure, and is closer to the sensitivity of the human visual system to object structure. The specific calculation formula is: ; in, Represents the object perception structural similarity, which is used to calculate the structural similarity between the object area in the saliency map and the ground truth; Represents region-aware structural similarity, which is used to calculate the structural similarity between each superpixel region in the saliency map and the ground truth; It represents the weight used to balance the first two, and is preferably 0.5 in this experiment.

[0087] The E-measure is a region-based evaluation metric that assesses overall quality by calculating the local alignment between the saliency map and the ground truth. It can effectively simulate the human eye's discrimination of image objects in salient object detection. Its adaptive version (adpE) uses the mean of the saliency map as the threshold to generate a binary image. It can simultaneously capture the global and local similarities between the saliency map and the ground truth, providing more accurate evaluation of salient object detection in complex scenes. The specific calculation formula of the E-measure is: ; in, Represents the enhanced alignment matrix, which is used to reflect the difference between the saliency map and the true value in pixels The adaptive threshold is set to twice the mean of the saliency map, which is used to convert the saliency map into a binary map to calculate the alignment. Indicates the height of the image; Indicates the width of the image; H × W Indicates the total number of pixels in the image.

[0088] The F-measure is the harmonic mean of precision and recall, used to comprehensively evaluate the detection accuracy of a saliency map. Its adaptive version (adpF) uses twice the mean of the saliency map as the threshold to generate a binary map. This metric balances precision and recall, quantifying the proportion of correctly detected pixels in the saliency map, thereby evaluating the model's overall detection performance at different thresholds. The specific calculation formula for the F-measure is: ; In this experiment, Set to 0.3 to emphasize the importance of accuracy. and The expression is: ; ; in, 、 、 are the number of true positive, false positive, and false negative pixels, respectively.

[0089] MAE is a pixel-level error metric that calculates the sum of the absolute differences between the saliency map and the true value at each pixel and then averages them. It directly reflects the overall consistency between the saliency map and the true value. The smaller the MAE value, the smaller the error distribution of the model globally and the higher the consistency between the saliency map and the true value, thus intuitively evaluating the global performance of the model. The specific calculation formula for MAE is: ; in, is the saliency map at pixel The value at is the true value in pixel The value at .

[0090] Based on the above experimental data sets, experimental environment, parameter settings and evaluation indicators, the final experimental results were obtained. This experiment evaluated the performance of the present invention and compared it with several common salient target detection models based on RGB-T. The specific results are detailed in Table 3. These models include: Large-Small Network (LSNet), Few-shot Multimodal Aspect-based Sentiment Analysis Framework based on Contrastive Finetuning (FMCF), Adaptive Fusion Network (ADF) and Multimodal Independent Decoding and Decoding (MIDD). The results show that the embodiment of the present invention is the best in all evaluation indicators of all data sets.

[0091] Table 3 Experimental results

[0092] In addition, the present invention further demonstrates the effectiveness of the TSAFM and DGFM modules through ablation experiments, as shown in Table 4. When these two modules work together, they can significantly enhance the model's feature learning capabilities, thereby improving the accuracy of object boundary positioning and the ability to detect semantic integrity in complex scenarios.

[0093] Table 4 Ablation experiment results

[0094] Furthermore, the modules designed in this embodiment demonstrate differentiated performance for scenarios of varying complexity and data volumes. Ablation experiments clearly demonstrate that the DGFM module performs better in small datasets and simpler scenarios, while the TSAFM module excels in complex scenarios. Therefore, by applying these two modules to different modal feature maps at different levels, this invention comprehensively enhances model performance.

[0095] Furthermore, the qualitative results of the model in salient object detection are compared with Figure 8 As shown. The performance differences between different models can be clearly seen through the intuitive effect diagram. The embodiment of the present invention can more accurately identify the salient areas in the scene and more effectively complete the boundary segmentation task, while other models have different degrees of deficiencies. Figure 8 In FIG, the symbol RGB represents an RGB image; the symbol T represents a thermal infrared image; the symbol Ours represents the method of the present invention; and GT represents correctly labeled data.

[0096] Comprehensive analysis of the experimental results shows that the present invention has a significant improvement in salient object detection compared with other models. It can more accurately locate salient areas in the scene and better complete the boundary segmentation task.

[0097] The present invention proposes an efficient RGB-T salient target detection network architecture based on modal interaction. This architecture fully utilizes the complementary information of RGB and thermal imaging through cross-modal, cross-path, and cross-layer information interaction, realizes deep complementarity and fusion of features between different modalities at multiple levels, and enables deep networks to learn more comprehensive multimodal features. At the same time, the present invention deeply explores the information interaction mechanism between different levels, ensuring that the model accurately captures the multimodal features of the target at multiple levels. The proposed method significantly improves the model's ability to learn multimodal features with the help of cross-modal, cross-path, and cross-layer information interaction, and demonstrates superior performance in experiments.

[0098] Example 2: Based on the same inventive concept, this example provides a salient target segmentation system based on multi-level modal interaction. The principle of solving the problem is similar to the salient target segmentation method based on multi-level modal interaction provided in Example 1, and the repeated parts will not be repeated.

[0099] This embodiment provides a salient object segmentation system based on multi-level modal interaction, including: A feature extraction module is used to obtain the RGB image and thermal infrared image of the image to be tested, and extract multiple layers of feature maps of the RGB image and the thermal infrared image; The first fusion feature map acquisition module is used to construct a triple attention fusion module and input the multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, a difference feature map of the feature maps of different modalities at the same level is obtained, and a first fusion feature map is obtained based on the difference feature map; The second fusion feature map acquisition module is used to construct a dual-path grouping fusion module and input the multimodal feature map of the same level into the dual-path grouping fusion module; in the dual-path grouping fusion module, RGB feature grouping and thermal infrared feature grouping are obtained, and the second fusion feature map is obtained based on the RGB feature grouping and thermal infrared feature grouping; The salient target output module is used to obtain the salient target of the image to be tested based on the first fused feature map and the second fused feature map.

[0100] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0104] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A salient object segmentation method based on multi-level modal interaction, characterized by ,include: Acquire an RGB image and a thermal infrared image of the image to be measured, and extract multiple hierarchical feature maps of the RGB image and the thermal infrared image; Constructing a triple attention fusion module and inputting the multimodal feature maps of the same level into the triple attention fusion module; in the triple attention fusion module, obtaining a difference feature map of the feature maps of different modalities at the same level, and obtaining a first fused feature map based on the difference feature map; Constructing a dual-path grouping fusion module and inputting the multimodal feature maps of the same level into the dual-path grouping fusion module; obtaining RGB feature grouping and thermal infrared feature grouping in the dual-path grouping fusion module, and obtaining a second fused feature map based on the RGB feature grouping and the thermal infrared feature grouping; Obtaining a salient object in the image to be tested according to the first fused feature map and the second fused feature map.

2. The method for salient object segmentation based on multi-level modal interaction according to claim 1, characterized in that: In the triple attention fusion module, the steps of obtaining a difference feature map of feature maps of different modalities at the same level and obtaining a first fusion feature map based on the difference feature map are as follows: Obtaining a first feature map and a second feature map at the same level, and calculating a difference feature map between the first feature map and the second feature map; Compressing the first feature map, the second feature map, and the difference feature map into channel-level vectors respectively to obtain a first global feature, a second global feature, and a third global feature; The first global feature, the second global feature, and the third global feature are input into the attention subnetwork and dynamically weighted fused to obtain a first fused feature map.

3. The method for salient object segmentation based on multi-level modal interaction according to claim 2, characterized in that: The first global feature, the second global feature, and the third global feature are input into the attention subnetwork for dynamic weighted fusion to obtain a first fused feature map. performing a first convolution operation on each of the first global feature, the second global feature, and the third global feature to extract an enhanced feature map; Performing a second convolution operation on the enhanced feature map to obtain a weight coefficient map; According to the weight coefficient map, the first global feature, the second global feature and the third global feature are dynamically weighted fused to obtain a first fused feature map.

4. The method for salient object segmentation based on multi-level modal interaction according to claim 3, characterized in that: The steps of the first convolution operation are: compressing the spatial dimension of the input global features and extracting key information by halving the number of channels; and enhancing the key information using an activation function to obtain an enhanced feature map.

5. The method for salient object segmentation based on multi-level modal interaction according to claim 3, characterized in that: The steps of the second convolution operation are: outputting a single-channel feature map according to the input enhanced feature map; and processing the single-channel feature map using an activation function to obtain a weight coefficient map.

6. The method for salient object segmentation based on multi-level modal interaction according to claim 1, characterized in that: In the dual-path grouping fusion module, the steps of obtaining RGB feature grouping and thermal infrared feature grouping, and obtaining a second fusion feature map based on the RGB feature grouping and the thermal infrared feature grouping are as follows: Obtain multimodal feature maps at the same level, group RGB features and thermal infrared features, and obtain RGB feature groups and thermal infrared feature groups; Cross-modally splicing the RGB feature grouping and the thermal infrared feature grouping to obtain a spliced ​​feature group; The concatenated feature group is subjected to dimensionality reduction processing to obtain a fine-grained fusion feature map.

7. The method for salient object segmentation based on multi-level modal interaction according to claim 6, characterized in that: The RGB feature grouping and the thermal infrared feature grouping are cross-modally spliced ​​to obtain a spliced ​​feature group, and the ungrouped RGB features and thermal infrared features are directly spliced ​​to obtain a coarse-grained fusion feature; the coarse-grained fusion feature and the fine-grained fusion feature are spliced ​​to obtain a second fusion feature map.

8. The method for salient object segmentation based on multi-level modal interaction according to claim 7, characterized in that: A salient object in the image to be tested is obtained according to the second fused feature map and the first fused feature map.

9. The method for salient object segmentation based on multi-level modal interaction according to claim 8, characterized in that: The step of obtaining a salient object in the image to be tested according to the second fused feature map and the first fused feature map is as follows: Performing feature extraction and upsampling processing on the second fused feature map in the nth layer feature map to obtain a first enhanced feature map; wherein n is a decreasing positive integer greater than 4; Splicing the first enhanced feature map with the second fused feature map in the n-1th layer feature map to obtain a first spliced ​​feature map; Extracting features and performing upsampling processing on the first spliced ​​feature map to obtain a second enhanced feature map; The second enhanced feature map is subjected to a splicing process with the first fusion feature map in the m-th layer feature map to obtain a second spliced feature map; where 1 m < n - 1, and m is a positive integer; Feature extraction and upsampling are performed on the second spliced ​​feature map to obtain salient objects in the image to be tested.

10. A salient object segmentation system based on multi-level modal interaction, characterized in that: include: A feature extraction module is used to obtain an RGB image and a thermal infrared image of the image to be tested, and extract multiple hierarchical feature maps of the RGB image and the thermal infrared image; A first fusion feature map acquisition module is used to construct a triple attention fusion module and input the multimodal feature maps of the same level into the triple attention fusion module; In the triple attention fusion module, a difference feature map of feature maps of different modalities at the same level is obtained, and a first fusion feature map is obtained based on the difference feature map; A second fusion feature map acquisition module is used to construct a dual-path grouping fusion module and input the multimodal feature maps of the same level into the dual-path grouping fusion module; In the dual-path grouping fusion module, RGB feature grouping and thermal infrared feature grouping are obtained, and a second fusion feature map is obtained based on the RGB feature grouping and the thermal infrared feature grouping; The salient target output module is used to obtain the salient target of the image to be tested according to the first fused feature map and the second fused feature map.

Images