A tamper detection and positioning method for generating images for scene text

Abstract

The application discloses a tampering detection and positioning method for scene text generated images, which is detected and positioned by a model composed of a deep feature branch, a multi-scale feature branch and a double attention fusion module. The deep feature branch captures the structural relationship and high-level semantic information of the whole scene text generated image, and outputs the deep feature map to the double attention fusion module. The multi-scale feature branch extracts the multi-scale feature map of the scene text generated image by a step-by-step feature extraction method from the shallow layer to the deep layer, and outputs the multi-scale feature map to the double attention fusion module. The double attention fusion module fully fuses the deep feature map and the multi-scale feature map in local details and global semantic information, and obtains the tampering detection result and the classification prediction result. The method has the advantages of strong positioning ability for the tampering area, strong adaptability to complex backgrounds, strong detection ability for small-scale tampering areas, strong detection ability for irregular tampering texts, and can accurately position the tampering area in a complex background.

Description

Technical Field

[0001] This invention relates to methods for tamper detection and localization, and in particular to a method for tamper detection and localization of scene-based text-generated images. Background Technology

[0002] Digital image tampering refers to the use of various tools to modify, edit, or alter the content of images, thereby changing their original semantics or visual effects. Based on the different tampering methods, digital image tampering techniques can be broadly divided into two categories: content-unchanging tampering and content-altering tampering. Content-unchanging tampering mainly involves adjusting the statistical characteristics of the image, such as Gaussian blur, contrast adjustment, and JPEG recompression. This type of tampering usually does not directly change the perceptible content of the image, but it can interfere with subsequent image processing or analysis. In contrast, content-altering tampering directly modifies the image content. Typical methods include copy-paste, splicing, object removal, compositing, and computer-generated images. These tampering methods are visually more deceptive; therefore, recent research has focused on the detection and pixel-level localization of content-altering tampering, particularly with significant progress made in research on traditional tampering methods such as copy-paste, splicing, and object removal.

[0003] Text image generation techniques have demonstrated remarkable capabilities in image manipulation. Generative techniques such as diffusion models and generative adversarial networks (GANs) generate high-quality, realistic images from random input. For example, the Denoising Diffusion Probabilistic Model (DDPM) proposed by Ho et al. in reference 1, "Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. ArXiv, abs / 2006.11239.", uses a Markov chain trained by variational inference to generate samples that highly match the distribution of real data in a finite amount of time. The GAN proposed by Goodfellow et al. in reference 2, "Goodfellow, IJ, Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, AC, & Bengio, Y. (2014). Generative adversarial networks. Communications of the ACM, 63, 139-144." utilizes adversarial training between the generator and the discriminator to generate realistic samples from random vectors and effectively distinguish between real and fake data. Furthermore, regarding text-to-image generation, Hertz et al. in reference 3, "Hertz, A., Mokady, R., Tenenbaum, JM, Aberman, K., Pritch, Y., & Cohen-Or, D. (2022). Prompt-to-Prompt Image Editing with Cross Attention Control. ArXiv, abs / 2208.01626." proposed a text control framework that utilizes a cross-attention mechanism to achieve a precise correspondence between text and images. Yang et al. in reference 4, "Yang, Y., Gui, D., Yuan, Y., Ding, H., Hu, H., & Chen, K. (2023). GlyphControl: GlyphConditional Control for Visual Text Generation. ArXiv, abs / 2305.18259." further improved the control over text content and the visual accuracy of generated images through the GlyphControl method. With the rapid development of generative artificial intelligence (AIGC) technology, visual scene generation technology has made significant progress in fields such as image synthesis and augmented reality.

[0004] However, this has also led to a proliferation of highly realistic fake images, increasing the risk of spreading misinformation. For example, on social media, some individuals may use AIGC technology to generate inauthentic news images, such as simulating "scenes after natural disasters"; text-to-image generation technology can be used to create non-existent "virtual characters" and impersonate real social media users for other illegal activities; in sensitive areas, such as public safety or military intelligence, some individuals may use generation techniques to tamper with key details (such as vehicles or buildings) in satellite images or surveillance images, thereby altering the semantics of the scene. Against this backdrop, effectively identifying and detecting image tampering caused by AIGC has become a critical technical challenge that urgently needs to be addressed.

[0005] Currently, the application of generation technology in scene text tampering is becoming increasingly widespread, but existing research on tampering detection and localization mainly focuses on general image tampering (such as face tampering detection and full image generation detection), while research on tampering detection of scene text generated images is relatively scarce.

[0006] Against the backdrop of the rapid development of generative artificial intelligence technology, generative models are capable of various modifications to scene text in images, such as font style modification, text replacement, or region manipulation. Figure 1 As shown. Existing methods for tampering detection and localization of scene-generated images based on text, such as the Difference-Aware Forensics (DAF) framework disclosed in reference 5, "C. Qu, Y. Zhong, F. Guo, and L. Jin, Generalized Tampered Scene Text Detection in the Era of Generative AI," ArXiv, vol.abs / 2407.21422, 2024," are among the few frameworks currently available for tampering detection of scene-generated images based on text. It achieves the detection of scene text tampering through feature difference comparison, and exhibits good generalization ability, especially in unseen tampering types. However, in-depth analysis reveals several shortcomings of the DAF framework, limiting its practical application in complex scenes:

[0007] I. Insufficient ability to locate tampered regions: The primary goal of the DAF framework is to detect tampered text, not to precisely locate the tampered regions. DAF uses feature comparison to detect tampered regions, but feature comparison operations are based on global features, not pixel-level operations. Although it extracts multi-scale features, its goal is to optimize the classification performance of global features, rather than to locate the specific boundaries or details of the tampered regions.

[0008] Second, insufficient adaptability to complex backgrounds: The feature comparison mechanism of the DAF framework relies on the feature differences between real and tampered texts, but complex backgrounds may affect the extraction of real text features, thereby reducing robustness.

[0009] Third, its ability to detect small-scale tampering regions is relatively weak: The DAF framework is designed to handle medium to large-scale tampering regions, and its feature extraction relies on the Faster R-CNN detection framework and the pyramid structure of FPN. For small-scale tampering regions, features may be masked by background or other real text features, leading to a decrease in detection performance.

[0010] IV. Limited Detection Capability of Irregularly Tampered Text: The DAF framework is based on Faster R-CNN feature extraction and the RoIPooling method. This method performs well when processing regular rectangular text regions, but it may not be able to extract accurate features for curved or deformed text. The distribution of real text features is calculated based on the average features of real text in the training set. Its generalization ability may be limited for tampered text with irregular shapes or varied styles. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a scene-oriented text-generated image tampering detection and localization method that has strong localization ability of tampered areas, strong adaptability to complex backgrounds, strong detection ability of small-scale tampered areas, and strong detection ability of irregular tampered text, and can accurately locate tampered areas in complex backgrounds.

[0012] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a method for tamper detection and localization of scene text-generated images. First, a model for tamper detection and localization of scene text-generated images is constructed. Then, the model is trained to obtain a trained model. When performing tamper detection and localization on scene text-generated images, the trained model is used to perform tamper detection and localization, obtaining detection and localization results. The model is based on a two-stream network structure and consists of a deep feature branch, a multi-scale feature branch, and a dual-attention fusion module. The deep feature branch is used to capture the overall structural relationships and high-level semantic information of the scene text-generated image, obtaining deep features of the scene text-generated image and outputting them to the dual-attention fusion module. The multi-scale feature branch is used to extract multi-scale features of the scene text-generated image through a step-by-step feature extraction method from shallow to deep layers and output them to the dual-attention fusion module. The dual-attention fusion module is used to fully fuse the deep features of the scene text-generated image output from the deep feature branch and the multi-scale features of the scene text-generated image output from the multi-scale feature branch, combining local details and global semantic information to obtain detection and localization results.

[0013] Compared with existing technologies, the advantages of this invention lie in its model construction using deep feature branches, multi-scale feature branches, and a dual-attention fusion module. The dual-attention fusion module significantly enhances the model's feature representation capabilities in both spatial and channel dimensions, ensuring spatial resolution consistency and achieving pixel-level precise localization of tampered regions. The multi-scale feature branches extract multi-scale feature information from shallow to deep layers, focusing on local texture and edge changes, exhibiting stronger robustness against tampered regions in complex backgrounds. The dual-attention fusion module adaptively adjusts the contribution ratio of different feature branches based on the feature distribution of the input image, weightedly fusing multi-scale and deep features to effectively reduce the interference of complex backgrounds on single feature branches. Furthermore, the dual-attention fusion module further strengthens the important characteristics of each feature map, ensuring the model can accurately perceive tampered regions in complex backgrounds. Through the joint extraction of deep and multi-scale feature branches, combined with the dual-attention fusion module, comprehensive capture of global semantic information and local detail features is achieved, thereby enhancing the detection capability of small-scale tampered regions. The dual-attention fusion module utilizes adaptive attention... The attention mechanism dynamically assigns higher weights to key regions in the input feature map, enhancing the model's ability to perceive tampered text content and its semantic relationship with the background. This enables the model to keenly detect tampering issues caused by scene-generated text. Combining deep and multi-scale features, the deep feature branch captures global semantic information and excels at identifying overall anomalies in scene-generated images, while the multi-scale feature branch focuses on local details, particularly suitable for capturing edge details and local changes. The extracted deep and multi-scale features are further enhanced through a dual-attention fusion module, improving spatial and channel perception and ensuring more comprehensive feature extraction for irregular text. During the fusion process, the dual-attention fusion module dynamically balances the contributions of global semantic information and local details through an adaptive weighting mechanism, enabling the model to flexibly adapt to diverse tampering scenarios and significantly improving the detection capability for irregular tampered text. Therefore, this invention demonstrates strong localization capabilities for tampered regions, strong adaptability to complex backgrounds, strong detection capabilities for small-scale tampered regions, and strong detection capabilities for irregular tampered text, accurately locating tampered regions in complex backgrounds.

[0014] Furthermore, the deep feature branch includes four ResNet blocks and one CBAM module. The four ResNet blocks are referred to as the first ResNet block, the second ResNet block, the third ResNet block, and the fourth ResNet block, respectively, and the CBAM module is referred to as the first CBAM module. The first ResNet block is used to access the scene text generated image and extract the shallow features of the scene text generated image to obtain the feature map F. R1The output is fed into the second ResNet block; the second ResNet block is used to extract the feature map F from the output of the first ResNet block. R1 The high-level semantic features are used to obtain the feature map F. R2 The output is fed into the third ResNet block; the third ResNet block is used to extract the feature map F from the output of the second ResNet block. R2 Abstract features are used to obtain feature map F. R3 The output is fed into the fourth ResNet block; the fourth ResNet block is used to extract the feature map F output from the third ResNet block. R3 The deep features are used to obtain the feature map F. R4 The output is fed to the first CBAM module; the first CBAM module is used to process the feature map F output to the fourth ResNet block. R4 Feature enhancement is performed to obtain the enhanced deep feature map F”. R4 The output is sent to the dual attention fusion module.

[0015] Furthermore, the multi-scale feature branch includes an EfficientNet-B3 architecture and four CBAM modules. The EfficientNet-B3 architecture is composed of multiple MBConv blocks, and these four CBAM modules are respectively referred to as the second CBAM module, the third CBAM module, the fourth CBAM module, and the fifth CBAM module. The EfficientNet-B3 architecture is used to access scene text generated images and extract multi-scale feature maps from the scene text generated images. Specifically, the EfficientNet-B3 architecture obtains the multi-scale feature map F through its three MBConv blocks. E1 ; and multi-scale feature map F E1 The output is sent to the second CBAM module, where the EfficientNet-B3 architecture obtains multi-scale feature maps F through its four MBConv blocks. E2 and multi-scale feature maps F E2 The output is sent to the third CBAM module, where the EfficientNet-B3 architecture obtains multi-scale feature maps F through its six MBConv blocks. E3 and, and the multi-scale feature map F E3 The output is sent to the fourth CBAM module, where the EfficientNet-B3 architecture obtains multi-scale feature maps F through its three MBConv blocks. E4 and multi-scale feature maps F E4 The output is sent to the fifth CBAM module; the second CBAM module is used to process the multi-scale feature map F output to the EfficientNet-B3 architecture. E1Feature enhancement is performed to obtain the enhanced multi-scale feature map F”. E1 The output is sent to the dual attention fusion module; the third CBAM module is used to process the multi-scale feature map F output by the EfficientNet-B3 architecture. E2 Feature enhancement is performed to obtain the enhanced multi-scale feature map F”. E2 The output is sent to the dual attention fusion module; the fourth CBAM module is used to process the multi-scale feature map F output by the EfficientNet-B3 architecture. E3 Feature enhancement is performed to obtain the enhanced multi-scale feature map F”. E3 The output is sent to the dual attention fusion module; the fifth CBAM module is used to process the multi-scale feature map F output by the EfficientNet-B3 architecture. E4 Feature enhancement is performed to obtain the enhanced multi-scale feature map F”. E4 The output is sent to the dual attention fusion module.

[0016] Furthermore, the dual attention fusion module includes an AttentionFusion module and a dual attention head module; the AttentionFusion module is used to access the deep feature map F” output by the first CBAM module. R4 The second CBAM module outputs a multi-scale feature map F” E1 The third CBAM module outputs a multi-scale feature map F” E2 The multi-scale feature map F output by the fourth CBAM module E3 And the multi-scale feature map F output by the fifth CBAM module E4 And for the deep feature map F” R4 Multi-scale feature map F” E1 Multi-scale feature map F” E2 Multi-scale feature map F” E3 and multi-scale feature maps F” E4 Weighted fusion is performed to obtain a fused feature map, which is then output to the dual attention head module. The dual attention head module is used to classify and predict the fused feature map output by the AttentionFusion module to obtain the tampering detection result and classification prediction result of the scene text generated image.

[0017] Furthermore, the AttentionFusion module includes a bilinear interpolation layer, five SE modules, and a feature fusion layer. These five SE modules are designated as the first SE module, the second SE module, the third SE module, the fourth SE module, and the fifth SE module, respectively. The bilinear interpolation layer is used to access the multi-scale feature map F".E1 Multi-scale feature map F” E2 Multi-scale feature map F” E3 and multi-scale feature maps F” E4 And the multi-scale feature map F” is obtained by bilinear interpolation. E1 Multi-scale feature map F” E2 Multi-scale feature map F” E3 and multi-scale feature maps F” E4 Adjust to the deep feature map F” R4 To ensure consistent resolution across feature maps before fusion, the same spatial dimensions are used to combine the multi-scale feature maps F". E1 The adjusted feature map is denoted as F1. E1 Multi-scale feature map F” E2 The adjusted feature map is denoted as F1. E2 Multi-scale feature map F” E3 The adjusted feature map is denoted as F1. E3 Multi-scale feature map F” E4 The adjusted feature map is denoted as F1. E4 And the deep feature map F” R4 The output is sent to the first SE module and the aforementioned feature fusion layer, and the adjusted feature map F1” is then processed. E1 The output is sent to the second SE module and the aforementioned feature fusion layer, and the adjusted feature map F1” is then processed. E2 The output is sent to the third SE module and the aforementioned feature fusion layer, and the adjusted feature map F1” is then processed. E3 The output is sent to the fourth SE module and the aforementioned feature fusion layer, and the adjusted feature map F” is then processed. E4 The output is sent to the fifth SE module and the aforementioned feature fusion layer; the first SE module is used to process the deep feature map F input thereto. R4 Attention weights are calculated to obtain an attention weight map W1, which is then output to the feature fusion layer. The second SE module is used to process the input feature map F1. E1 Attention weights are calculated to obtain attention weight map W2, which is then output to the feature fusion layer. The third SE module is used to process the input feature map F1. E2 Attention weights are calculated to obtain attention weight map W3, which is then output to the feature fusion layer. The fourth SE module is used to process the input feature map F1. E3 Attention weights are calculated to obtain the attention weight map W4, which is then output to the feature fusion layer. The fifth SE module is used to process the input feature map F”. E4Attention weights are calculated to obtain an attention weight map W5, which is then output to the feature fusion layer. The feature fusion layer is used to process the attention weight map W1 and the feature map F1 respectively. R4 The weighted feature map is obtained by performing weighted processing. Attention weight map W2 and feature map F1 E1 The weighted feature map is obtained by performing weighted processing. Attention weight map W3 and feature map F1” E2 The weighted feature map is obtained by performing weighted processing. Attention weight map W4 and feature map F1” E3 The weighted feature map is obtained by performing weighted processing. Attention weight map W5 and feature map F1 are respectively E4 The weighted feature map is obtained by performing weighted processing. and the weighted feature map The fusion is performed to obtain the fused feature map F. combined The output is sent to the aforementioned dual attention head module. Attached Figure Description

[0018] Figure 1 Generate an example image of the altered image for the scene text;

[0019] Figure 2 This is a structural diagram of the model of the scene-oriented text-to-image tampering detection and localization method of the present invention;

[0020] Figure 3 This is a framework diagram of the AttentionFusion module of the model of the scene-oriented text-to-image tampering detection and localization method of the present invention.

[0021] Figure 4 A statistical chart showing the localization results of all models for the tampered images in the comparative experiment;

[0022] Figure 5 A statistical chart of F1 scores obtained by different tamper detection and localization models under post-processing operations. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] Example 1: As Figure 2As shown, a method for tamper detection and localization of scene-generated text images is proposed. First, a model for tamper detection and localization of scene-generated text images is constructed, and then the model is trained to obtain the trained model. When performing tamper detection and localization on scene-generated text images, the trained model is used to perform tamper detection and localization, obtaining the detection and localization results. The model is based on a two-stream network structure, consisting of a deep feature branch, a multi-scale feature branch, and a dual-attention fusion module. The deep feature branch is used to capture the overall structural relationships and high-level semantic information of the scene-generated text image, obtaining a deep feature map of the scene-generated text image and outputting it to the dual-attention fusion module. The multi-scale feature branch is used to extract multi-scale feature maps of the scene-generated text image through a step-by-step feature extraction method from shallow to deep layers and output them to the dual-attention fusion module. The dual-attention fusion module is used to fully fuse the deep feature maps of the scene-generated text image output by the deep feature branch and the multi-scale feature maps of the scene-generated text image output by the multi-scale feature branch, fully integrating local details and global semantic information to obtain tamper detection results and classification prediction results.

[0025] In this embodiment, the model is constructed using deep feature branches, multi-scale feature branches, and a dual-attention fusion module. The dual-attention fusion module significantly enhances the model's feature representation capabilities in both spatial and channel dimensions, ensuring spatial resolution consistency and achieving pixel-level precise localization of tampered regions. The multi-scale feature branches extract multi-scale feature information from shallow to deep layers, focusing on local texture and edge changes, exhibiting stronger robustness to tampered regions in complex backgrounds. The dual-attention fusion module adaptively adjusts the contribution ratio of different feature branches based on the feature distribution of the input image, weightedly fusing multi-scale and deep features to effectively reduce the interference of complex backgrounds on individual feature branches. Furthermore, the dual-attention fusion module further strengthens the important characteristics of each feature map, ensuring the model can accurately perceive tampered regions in complex backgrounds. Through the joint extraction of deep and multi-scale feature branches, combined with the dual-attention fusion module, comprehensive capture of global semantic information and local detail features is achieved. This enhances the detection capability for small-scale tampering regions. The dual-attention fusion module utilizes an adaptive attention mechanism to dynamically assign higher weights to key regions in the input feature map. This mechanism strengthens the model's perception of tampered text content and its semantic relationship with the background, enabling the model to keenly detect tampering issues caused by scene text generation. Combining deep features and multi-scale features, the deep feature branch captures global semantic information and excels at identifying overall anomalies in scene text-generated images, while the multi-scale feature branch focuses on local detail information, particularly suitable for capturing edge details and local changes. The dual-attention fusion module further enhances the spatial and channel perception capabilities of the extracted deep and multi-scale features, ensuring more comprehensive feature extraction for irregular text. During the fusion process, the dual-attention fusion module dynamically balances the contributions of global semantic information and local detail information through an adaptive weight mechanism, enabling the model to flexibly adapt to diverse tampering scenarios and significantly improving the detection capability for irregular tampered text.

[0026] Example 2: This example is basically the same as Example 1, except that: In this example, the deep feature branch includes four ResNet blocks and one CBAM module. The four ResNet blocks are referred to as the first ResNet block, the second ResNet block, the third ResNet block, and the fourth ResNet block, respectively, and the CBAM module is referred to as the first CBAM module. The first ResNet block is used to input the scene text generated image and extract the shallow features (such as edges, textures, etc.) of the scene text generated image to obtain the feature map F. R1 The output is fed into the second ResNet block; the second ResNet block is used to extract the feature map F from the output of the first ResNet block. R1 The high-level semantic features (such as local object parts and structures) are used to obtain the feature map F.R2 The output is fed into the third ResNet block; the third ResNet block is used to extract the feature map F from the output of the second ResNet block. R2 Abstract features (such as object shape and scene structure) are used to obtain feature map F. R3 The output is fed into the fourth ResNet block; the fourth ResNet block is used to extract the feature map F output from the third ResNet block. R3 The deep features (such as global image content and semantic information) are used to obtain the feature map F. R4 The output is fed to the first CBAM module; the first CBAM module is used to process the feature map F output to the fourth ResNet block. R4 Feature enhancement is performed to obtain the enhanced deep feature map F”. R4 Output to the dual attention fusion module.

[0027] In this embodiment, the deep feature branch, through layer-by-layer feature extraction from four ResNet blocks, can gradually learn more complex semantic features from simple local information. In the complex tampering detection of scene text generated images, extracting these features is crucial for distinguishing normal and tampered regions. The deep features contain high-level semantic information, which provides strong support for detecting anomalies in the global structure of scene text generated images (such as tampering, splicing, etc.). The first CBAM module emphasizes those more critical features in tampering detection through channel attention and spatial attention mechanisms, helping the deep feature branch focus on tampered regions. By focusing on key regions and channels in the scene text generated image, the expressive power of the deep feature branch for these deep features can be further enhanced, making the model more focused on salient feature regions related to tampering in the scene text generated image.

[0028] Example 3: This example is basically the same as Example 2, except that: In this example, the multi-scale feature branch includes the EfficientNet-B3 architecture and four CBAM modules. The EfficientNet-B3 architecture consists of multiple MBConv blocks, and these four CBAM modules are referred to as the second CBAM module, the third CBAM module, the fourth CBAM module, and the fifth CBAM module, respectively. The EfficientNet-B3 architecture is used to access the scene text generated image and extract the multi-scale feature map of the scene text generated image. Specifically, the EfficientNet-B3 architecture obtains the multi-scale feature map F through its three MBConv blocks. E1 and multi-scale feature maps F E1 The output is fed into the second CBAM module, where the EfficientNet-B3 architecture obtains multi-scale feature maps F through its four MBConv blocks. E2 and multi-scale feature maps FE2 The output is sent to the third CBAM module. The EfficientNet-B3 architecture obtains multi-scale feature maps F through its six MBConv blocks. E3 and multi-scale feature maps F E3 The output is sent to the fourth CBAM module. The EfficientNet-B3 architecture obtains multi-scale feature maps F through its three MBConv blocks. E4 and multi-scale feature maps F E4 The output is sent to the fifth CBAM module; the second CBAM module is used to process the multi-scale feature map F output by the EfficientNet-B3 architecture. E1 Feature enhancement is performed to obtain the enhanced multi-scale feature map F'. E '1' output is fed to the dual attention fusion module; the third CBAM module is used to process the multi-scale feature map F output by the EfficientNet-B3 architecture. E2 Feature enhancement is performed to obtain the enhanced multi-scale feature map F'. E '2' outputs to the dual attention fusion module; the fourth CBAM module is used to process the multi-scale feature map F output by the EfficientNet-B3 architecture. E3 Feature enhancement is performed to obtain the enhanced multi-scale feature map F'. E '3' outputs to the dual attention fusion module; the fifth CBAM module is used to process the multi-scale feature map F output by the EfficientNet-B3 architecture. E4 Feature enhancement is performed to obtain the enhanced multi-scale feature map F'. E '4 Output to the dual attention fusion module.'

[0029] In this embodiment, the EfficientNet-B3 architecture extracts multi-scale features of the scene text generated image through its inconvolutional network architecture in the multi-scale feature branch. This effectively captures local details and large-scale background information of the scene text generated image, enabling the model to focus on the details and structural information of the scene text generated image at different scales. This compensates for the lack of detection capability of a single deep feature branch for small-scale or detailed information (such as small object tampering or minor image modification), thereby improving the model's ability to detect complex tampering patterns, especially in terms of accuracy when dealing with local tampering and detail distortion.

[0030] Example 4: This example is basically the same as Example 3, except that: in this example, the dual attention fusion module includes an AttentionFusion module and a dual attention head module; the AttentionFusion module is used to access the deep feature map F” output by the first CBAM module. R4The second CBAM module outputs a multi-scale feature map F” E1 The third CBAM module outputs a multi-scale feature map F” E2 The multi-scale feature map F output by the fourth CBAM module E3 And the multi-scale feature map F output by the fifth CBAM module E4 And for the deep feature map F” R4 Multi-scale feature map F” E1 Multi-scale feature map F” E2 Multi-scale feature map F” E3 and multi-scale feature maps F” E4 Weighted fusion is performed to obtain a fused feature map, which is then output to the dual attention head module. The dual attention head module is used to classify and predict the fused feature map output by the AttentionFusion module, thereby obtaining the tampering detection result and classification prediction result of the scene text generated image.

[0031] In this embodiment, the dual-attention fusion module, through the AttentionFusion module, receives the outputs of multiple CBAM modules and efficiently fuses these feature maps from different scales and levels. This allows for a more comprehensive understanding of image content, particularly in complex tamper detection tasks, enhancing sensitivity to tampered regions. The main task of the AttentionFusion module is to fuse feature maps from various CBAM modules. It receives and integrates feature maps from different scales and levels, using an attention mechanism to weightedly fuse the feature maps. This helps the model focus on key regions in the image and ignore irrelevant information, thereby improving the model's detection accuracy. The dual-attention head module incorporates channel attention and spatial attention mechanisms. Channel attention allows the model to focus on the most important feature channels, while spatial attention helps the model focus on key regions in the image. The combination of these two mechanisms enables the model to focus more intently on tampered regions or key parts of the image, improving sensitivity to tamper detection. The dual-attention head module uses the fused feature maps for classification prediction, thereby generating tamper detection results and classification predictions.

[0032] Example 5: This example is basically the same as Example 4, except that: Figure 3 As shown, in this embodiment, the AttentionFusion module includes a bilinear interpolation layer, five SE modules, and a feature fusion layer. The five SE modules are designated as the first SE module, the second SE module, the third SE module, the fourth SE module, and the fifth SE module, respectively. The bilinear interpolation layer is used to access the multi-scale feature map F". E1 Multi-scale feature map F” E2 Multi-scale feature map F”E3 and multi-scale feature maps F” E4 And the multi-scale feature map F” is obtained by bilinear interpolation. E1 Multi-scale feature map F” E2 Multi-scale feature map F” E3 and multi-scale feature maps F” E4 Adjust to the deep feature map F” R4 To ensure consistent resolution across feature maps before fusion, the same spatial dimensions are used to combine the multi-scale feature maps F". E1 The adjusted feature map is denoted as F1. E1 Multi-scale feature map F” E2 The adjusted feature map is denoted as F1. E2 Multi-scale feature map F” E3 The adjusted feature map is denoted as F1. E3 Multi-scale feature map F” E4 The adjusted feature map is denoted as F1. E4 And the deep feature map F” R4 The output is fed to the first SE module and the feature fusion layer, and the adjusted feature map is then processed using the F1" method. E1 The output is fed to the second SE module and the feature fusion layer, and the adjusted feature map F1” is then processed. E2 The output is fed to the third SE module and the feature fusion layer, and the adjusted feature map F1” is then processed. E3 The output is sent to the fourth SE module and the feature fusion layer, and the adjusted feature map F” is then processed. E4 The output goes to the fifth SE module and the feature fusion layer; the first SE module is used to process the deep feature map F” input thereto. R4 Attention weights are calculated, and the resulting attention weight map W1 is output to the feature fusion layer; the second SE module is used to process the input feature map F1. E1 Attention weights are calculated, and the resulting attention weight map W2 is output to the feature fusion layer; the third SE module is used to process the input feature map F1. E2 Attention weights are calculated, and the resulting attention weight map W3 is output to the feature fusion layer; the fourth SE module is used to process the input feature map F1”. E3 Attention weights are calculated, and the resulting attention weight map W4 is output to the feature fusion layer; the fifth SE module is used to process the input feature map F”. E4 Attention weights are calculated, and the resulting attention weight map W5 is output to the feature fusion layer. The feature fusion layer is used to process the attention weight map W1 and the feature map F1 respectively. R4 The weighted feature map is obtained by performing weighted processing. Attention weight map W2 and feature map F1 E1The weighted feature map is obtained by performing weighted processing. Attention weight map W3 and feature map F1” E2 The weighted feature map is obtained by performing weighted processing. Attention weight map W4 and feature map F1” E3 The weighted feature map is obtained by performing weighted processing. Attention weight map W5 and feature map F1 are respectively E4 The weighted feature map is obtained by performing weighted processing. and the weighted feature map The fusion is performed to obtain the fused feature map F. combined Output to the dual attention head module.

[0033] In this embodiment, the bilinear interpolation layer uses bilinear interpolation to unify feature maps of different sizes into the same spatial dimension. This method is computationally efficient and suitable for feature map size adjustment, avoiding difficulties in feature fusion caused by size inconsistencies. Furthermore, by unifying the spatial dimension, it ensures that all feature maps can be effectively weighted and fused in subsequent fusion operations. The attention weights are calculated using the SE module to learn the importance of each feature map, thus assigning different weights to different feature maps during subsequent fusion. The fusion feature layer combines feature maps from different sources to provide a more comprehensive feature representation, i.e., the fused feature map F. combined This can effectively enhance the expressive power of key features, improve the model's ability to detect tampered regions, and enhance the model's robustness and generalization ability.

[0034] In practical applications of the scene-based text-generated image tampering detection and localization method of the present invention, the model in the method can be implemented in environments such as cloud servers, edge computing devices, or local computers. The model is trained, packaged, and an interactive interface associated with the model is set up for easy user operation. When the user provides a scene-based text-generated image to be detected, the model receives the image through the interactive interface and performs detection and localization. If the detection result is a genuine image, a report indicating that the image has not been tampered with is generated and displayed on the interactive interface. If the detection result is a tampered image, possible tampered areas are marked in the original scene-based text-generated image, and a detection report including the location of the tampered area and the probability of tampering is generated and displayed on the interactive interface.

[0035] To verify the performance of the scene-oriented text-generated image tampering detection and localization method of the present invention, experimental verification was conducted on the scene-oriented text-generated image tampering detection and localization method of the present invention, specifically as follows:

[0036] I. Experimental Setup

[0037] The proposed model for tamper detection and localization of scene-based text-generated images is implemented in Python and trained on an NVIDIA GeForce RTX 4090 graphics card to achieve tamper detection and localization of scene-based text-generated images.

[0038] Due to the scarcity of datasets for detecting text tampering in scene text images, this experiment uses an existing scene text generation image model to construct the training dataset. Specifically, the dataset is generated based on the AnyText method proposed in reference 6, "Tuo, Y., Xiang, W., He, J., Geng, Y., & Xie, X. (2023). AnyText: Multilingual Visual Text Generation And Editing. ArXiv, abs / 2311.03054." Two subsets, AnyText-11k and AnyText-20k, are constructed and partitioned using this method to explore the impact of dataset size on model performance. The AnyText-11k dataset contains 11,456 tampered image text samples, covering both Chinese and English languages, aiming to provide diverse training and testing data for multilingual text tampering detection tasks. The AnyText-20k dataset contains 20,023 manipulated images. It uses the same generation method as AnyText-11k, but is larger in scale to provide richer training samples, thereby further improving the model's generalization ability.

[0039] II. Experimental Indicators:

[0040] To comprehensively evaluate the model's performance in tampering detection, we use the following core metrics: Intersection over Union (IoU), Area Under the Curve (AUC), and F1 score. IoU measures the overlap between the model's predicted tampered regions and the actual tampered regions, ranging from 0 to 1. A higher IoU value indicates more accurate detection of tampered regions. AUC evaluates the model's overall classification ability at different detection thresholds; an AUC value close to 1 indicates excellent performance in distinguishing tampered and non-tampered regions. The F1 score combines Precision and Recall to balance detection accuracy and coverage; a higher F1 score indicates a more balanced performance in both accurate detection and comprehensive coverage of tampered regions. Precision reflects the proportion of correctly predicted tampered regions out of all predicted tampered regions; a higher Precision value indicates fewer false positives. Recall reflects the proportion of actually tampered regions successfully detected; a higher Recall value indicates fewer missed detections. When the F1 score is close to 1, it indicates that the model has achieved a good balance between accuracy and coverage, while a lower F1 score may mean that the model has a high false positive or false negative rate. IoU and F1 are calculated using formulas (1) and (2), respectively:

[0041]

[0042] Wherein, TP (True Positive) represents the number of correctly predicted tampered pixels, FP (False Positive) represents the number of falsely detected non-tampered pixels, and FN (False Negative) represents the number of falsely detected tampered pixels.

[0043] III. Experimental Procedure:

[0044] The model input size is 512×512 pixels to ensure performance on high-resolution images. The AdamW optimizer is used during model training, with a learning rate set to 10. -4 To achieve more stable gradient updates, the entire training process lasted 100 epochs with a batch size of 8. To enhance the model's generalization ability and avoid overfitting, various data augmentation techniques were introduced, including horizontal flipping, vertical flipping, scaling, compression, and noise addition.

[0045] 1. Comparative Experiment: The tampering detection and localization method for scene-oriented text-generated images of the present invention was fairly compared with several state-of-the-art deep learning-based methods. These state-of-the-art deep learning-based methods include: the method using the MVSS-Net model disclosed in reference 7, “Chen, X., Dong, C., Ji, J., Cao, J., & Li, X. (2021). Image Manipulation Detection by Multi-View Multi-Scale Supervision. 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 14165-14173.”; and the method using the MVSS-Net model disclosed in reference 8, “Zhuang, P., Li, H., Tan, S., Li, B., & Huang, J. (2021). Image Tampering Localization Using a Dense Fully Convolutional Network. IEEE Transactions on Information Forensics and…”. The method of the Dense-FCN model disclosed in Security, 16, 2986-2999, uses the method of the SE-Network model disclosed in reference 9, "Zhang, Y., Zhu, G., Wu, L., Kwong, ST, Zhang, H., & Zhou, Y. (2022). Multi-Task SE-Network for Image Splicing Localization. IEEE Transactions on Circuits and Systems for Video Technology, 32, 4828-4840.", and uses the method of the SE-Network model disclosed in reference 10, "Islam, A., Long, C., Basharat, A., & Hoogs, AJ (2020). DOA-GAN: Dual-Order Attentive Generative Adversarial Network for Image Copy-Move Forgery Detection and Localization. 2020 IEEE / CVF Conference on Computer Vision and Pattern The method of the DOA-GAN model disclosed in Recognition (CVPR), 4675-4684, uses reference 11 "

[37] Bi,X.,Wei,Y.The method of the RRU-Net model disclosed in "RRU-Net: The Ringed Residual U-Net for Image Splicing Forgery Detection. 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 30-39" is compared with the method of the CFL-Net model disclosed in "Niloy, FF, Bhaumik, KK, & Woo, SS (2022). CFL-Net: Image Forgery Localization Using Contrastive Learning. 2023 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), 4631-4640"". All experiments were trained and tested on the Anytext-11k and Anytext-20k datasets under the same experimental settings. To ensure the reliability of the evaluation results, the best performance of each experiment was selected in the final evaluation.

[0046] 2. Robustness Experiment: To conceal traces of tampering, tampered images typically undergo a series of post-processing operations. These operations aim to reduce the detection capability of tampering detection algorithms, thereby improving the concealment of tampered images. To comprehensively evaluate the robustness of different methods under these operations, four common image post-processing techniques were applied to the test set of the Anytext-20k dataset: JPEG compression, Gaussian filtering, image scaling, and image cropping.

[0047] 3. Ablation Experiments: The deep feature branch is mainly used to extract high-level semantic features from the original input image, while the multi-scale feature branch focuses on capturing details and local information. The dual attention fusion module achieves more accurate classification prediction by fusing the features extracted from these two branches at different levels. Meanwhile, the convolutional block attention module (CBAM) is used for feature enhancement, particularly guiding the model to pay more attention to tampered regions in the image, thereby improving the model's performance in tamper detection tasks. To evaluate the effectiveness of the deep feature branch, multi-scale feature branch, dual attention fusion module, and CBAM, they were individually removed from the model, and the tampered text detection performance on the Anytext-20k dataset was evaluated.

[0048] IV. Analysis of Experimental Results:

[0049] 1. Comparative Experiments: The experimental results are shown in Table 1. The tamper detection and localization method for scene-based text-generated images according to the present invention demonstrates superior advantages in the task of tamper detection and localization of generated images, especially in the F1 score and IOU metrics, significantly outperforming several existing methods. In contrast, although the method using the MVSS-Net model performs well in some tasks, it is still inferior to the tamper detection and localization method for scene-based text-generated images according to the present invention. The methods using the Dense-FCN model and the SE-Network model perform even weaker in the core metrics, reflecting their limitations in this task. Further experiments show that the tests on the Anytext-11k and Anytext-20k datasets verify the model's generalization ability, and the IOU score on the Anytext-20k dataset reaches 0.9514, an improvement of 0.0203 compared to Anytext-11k. In addition, the tamper detection and localization method for scene-based text-generated images according to the present invention also shows significant improvements in other key metrics, indicating that expanding the dataset size helps improve the model's adaptability in complex scenarios. Similarly, other comparison methods have also improved on larger datasets, particularly in terms of performance on F1 and IOU metrics. Figure 4 The results of all models for tampering image localization are shown, demonstrating the superior tampering localization performance of the scene-oriented text-generated image tampering detection and localization method of the present invention.

[0050] Table 1: Comparison of comparative test indicators

[0051]

[0052] 2. Robustness test: Figure 5 The F1 scores of different models under these post-processing operations are shown, providing an intuitive basis for further analysis of their robustness. Robustness evaluation for JPEG compression: To evaluate the model's adaptability to JPEG compression, we reduced the compression quality of the dataset from 100 (highest quality) to 30 (lowest quality) and compared the detection performance of each model under different compression intensities. Experimental results are as follows... Figure 5 (a) shows that the proposed model of the scene-based text-generated image tampering detection and localization method of the present invention maintains stable and excellent detection performance across the entire range. This indicates that the model of the scene-based text-generated image tampering detection and localization method of the present invention has strong resistance to JPEG compression distortion, can effectively cope with the loss of details caused by compression, and ensures robustness under various tampering concealment strategies. Robustness evaluation for Gaussian filtering: The model detection performance under different blur intensities was evaluated to address the impact of image blur. Experimental results are as follows: Figure 5As shown in (b), the proposed model of the scene-based text-generated image tampering detection and localization method of this invention, along with the RRU-Net model, exhibits excellent robustness at various blur levels. In contrast, other models show a significant decrease in detection performance as the blur level increases, indicating their weak resistance to image blur distortion. This further verifies the stability and adaptability of the scene-based text-generated image tampering detection and localization method of this invention in handling blurred images, making it more advantageous in real-world applications. Robustness evaluation for image scaling: Image scaling changes the geometric structure and spatial resolution, thus affecting the performance of the detection model. Experimental results are as follows... Figure 5 As shown in (c), across all scaling factors, the proposed model of the scene-based text-generated image tampering detection and localization method of this invention maintains optimal performance, with the F1 score consistently at the highest level. When the scaling factor is greater than 0.75, the F1 score tends to saturate, stabilizing around 0.95, indicating that regardless of image enlargement or reduction, the model of the scene-based text-generated image tampering detection and localization method of this invention maintains excellent detection performance, demonstrating extremely strong scaling robustness. Robustness evaluation for image cropping: By adjusting the cropping ratio, the detection capabilities of each model under different cropping intensities are analyzed. Experimental results are as follows... Figure 5 As shown in (d), the F1 score of most models increases with the increase of the cropping ratio, reaching its optimum near 1.0. However, when the cropping ratio is small, the F1 scores of all models are low, reflecting the limited detection performance under conditions of severe information loss. However, the model proposed in this invention's method for tamper detection and localization of scene-based text-generated images exhibits stronger robustness at different cropping ratios compared to other methods, with a significantly higher F1 score. This indicates that the model of this invention's method for tamper detection and localization of scene-based text-generated images can maintain high detection accuracy even when image content is limited, further verifying its effectiveness and stability.

[0053] 3. Ablation Experiments: The results of the ablation experiments are shown in Table 2. Removing the deep feature branch significantly reduced F1, IOU, and AUC scores, especially F1 and IOU. Removing the multi-scale feature branch, while maintaining high F1 and IOU scores, showed a performance decrease compared to the complete model. Removing the dual-attention fusion module, replacing the AttentionFusion module with concatenation for multi-scale feature fusion, and using a convolutional layer to predict the fused feature map, resulted in a slight decrease in overall model performance, particularly noticeable in IOU. Removing CBAM reduced F1, IOU, and AUC scores, with a significant decrease in IOU. By comparing the ablation experiment results, it can be seen that deep feature branches, multi-scale feature branches, dual-attention fusion modules, and CBAM each play important roles in improving model performance. Deep feature branches and multi-scale feature branches, in particular, make significant contributions to the extraction of complex features and the capture of global semantics. The dual attention fusion module and CBAM further enhance feature fusion and attention mechanisms, improving the model's accuracy and robustness in tamper detection tasks.

[0054] Table 2: Ablation Experiment Results

[0055]

[0056]

[0057] In summary, this invention proposes a novel and efficient detection model for tampering detection and localization in scene-generated text images, addressing the problem of region tampering detection in such images. The model demonstrates superior detection performance across multiple evaluation metrics, significantly outperforming traditional methods and validating its effectiveness in complex generated image tampering detection tasks. Therefore, this invention's tampering detection and localization method for scene-generated text images exhibits strong localization capabilities for tampered regions, strong adaptability to complex backgrounds, strong detection capabilities for small-scale tampered regions, and strong detection capabilities for irregularly tampered text, enabling accurate localization of tampered regions in complex backgrounds.

Claims

1. A method for tamper detection and localization of scene-generated text images, comprising: firstly, constructing a model for tamper detection and localization of scene-generated text images; then training the model to obtain a trained model; and when performing tamper detection and localization on scene-generated text images, using the trained model to perform tamper detection and localization on the scene-generated text images to obtain detection and localization results, characterized in that... The model is based on a two-stream network structure and consists of a deep feature branch, a multi-scale feature branch, and a dual-attention fusion module. The deep feature branch is used to capture the overall structural relationship and high-level semantic information of the scene text generated image, and outputs the deep feature map of the scene text generated image to the dual-attention fusion module. The multi-scale feature branch is used to extract the multi-scale feature map of the scene text generated image through a step-by-step feature extraction method from shallow to deep layers and outputs it to the dual-attention fusion module. The dual-attention fusion module is used to fully fuse the deep feature map of the scene text generated image output by the deep feature branch and the multi-scale feature map of the scene text generated image output by the multi-scale feature branch to obtain tampering detection results and classification prediction results. The deep feature branch comprises four ResNet blocks and one CBAM module. The four ResNet blocks are referred to as the first ResNet block, the second ResNet block, the third ResNet block, and the fourth ResNet block, respectively. The CBAM module is referred to as the first CBAM module. The first ResNet block is used to access the scene text generated image and extract shallow features from the scene text generated image to obtain a feature map. The output is fed into the second ResNet block; the second ResNet block is used to extract the feature map from the output of the first ResNet block. High-level semantic features are used to obtain feature maps. The output is fed into a third ResNet block; the third ResNet block is used to extract the feature map from the output of the second ResNet block. Abstract features are used to obtain feature maps. The output is fed into the fourth ResNet block; the fourth ResNet block is used to extract the feature map output from the third ResNet block. The deep features are used to obtain the feature map. The output is fed to the first CBAM module; the first CBAM module is used to process the feature map output from the fourth ResNet block. Feature enhancement is performed to obtain the enhanced deep feature map. The output is sent to the dual attention fusion module. The multi-scale feature branch includes the EfficientNet-B3 architecture and four CBAM modules. The EfficientNet-B3 architecture consists of multiple MBConv blocks, and the four CBAM modules are referred to as the second CBAM module, the third CBAM module, the fourth CBAM module, and the fifth CBAM module, respectively. The EfficientNet-B3 architecture is used to input scene text generated images and extract multi-scale feature maps from the scene text generated images. Specifically, the EfficientNet-B3 architecture obtains multi-scale feature maps through three of its internal MBConv blocks. and multi-scale feature maps The output is sent to the second CBAM module, where the EfficientNet-B3 architecture obtains multi-scale feature maps through its four MBConv blocks. and multi-scale feature maps The output is sent to the third CBAM module, where the EfficientNet-B3 architecture obtains multi-scale feature maps through its six MBConv blocks. and multi-scale feature maps The output is sent to the fourth CBAM module, where the EfficientNet-B3 architecture obtains multi-scale feature maps through its three MBConv blocks. and multi-scale feature maps The output is sent to the fifth CBAM module; the second CBAM module is used to process the multi-scale feature maps output to the EfficientNet-B3 architecture. Feature enhancement is performed to obtain enhanced multi-scale feature maps. The output is sent to the dual attention fusion module; the third CBAM module is used to process the multi-scale feature maps output to the EfficientNet-B3 architecture. Feature enhancement is performed to obtain enhanced multi-scale feature maps. The output is sent to the dual attention fusion module; the fourth CBAM module is used to process the multi-scale feature map output to the EfficientNet-B3 architecture. Feature enhancement is performed to obtain enhanced multi-scale feature maps. The output is sent to the dual attention fusion module; the fifth CBAM module is used to process the multi-scale feature map output to the EfficientNet-B3 architecture. Feature enhancement is performed to obtain enhanced multi-scale feature maps. The output is sent to the dual attention fusion module.

2. The method for tamper detection and localization of scene-oriented text-generated images according to claim 1, characterized in that... The dual attention fusion module includes an AttentionFusion module and a dual attention head module; the AttentionFusion module is used to access the deep feature map output by the first CBAM module. The second CBAM module outputs a multi-scale feature map. The third CBAM module outputs a multi-scale feature map. The multi-scale feature map output by the fourth CBAM module and the multi-scale feature map output by the fifth CBAM module and deep feature maps Multi-scale feature maps Multi-scale feature maps Multi-scale feature maps and multi-scale feature maps The output is sent to the dual attention head module, which is used to classify and predict the fused feature map output by the AttentionFusion module to obtain the tampering detection result and classification prediction result of the scene text generated image.

3. The method for tamper detection and localization of scene-oriented text-generated images according to claim 2, characterized in that... The AttentionFusion module includes a bilinear interpolation layer, five SE modules, and a feature fusion layer. These five SE modules are designated as the first SE module, the second SE module, the third SE module, the fourth SE module, and the fifth SE module. The bilinear interpolation layer is used to access multi-scale feature maps. Multi-scale feature maps Multi-scale feature maps and multi-scale feature maps And multi-scale feature maps are obtained through bilinear interpolation. Multi-scale feature maps Multi-scale feature maps and multi-scale feature maps Adjust to deep feature map To ensure consistent resolution across feature maps before fusion, the same spatial dimensions are used to combine multi-scale feature maps. The adjusted feature map is denoted as Multi-scale feature maps The adjusted feature map is denoted as Multi-scale feature maps The adjusted feature map is denoted as Multi-scale feature maps The adjusted feature map is denoted as and deep feature maps The output is sent to the first SE module and the aforementioned feature fusion layer, and the adjusted feature map is... The output is sent to the second SE module and the aforementioned feature fusion layer, and the adjusted feature map is... The output is sent to the third SE module and the aforementioned feature fusion layer, and the adjusted feature map is... The output is sent to the fourth SE module and the aforementioned feature fusion layer, and the adjusted feature map is... The output is sent to the fifth SE module and the aforementioned feature fusion layer; the first SE module is used to process the deep feature map input thereto. Attention weights are calculated to obtain an attention weight map. The output is sent to the feature fusion layer; the second SE module is used to process the feature map input thereto. Attention weights are calculated to obtain an attention weight map. The output is sent to the feature fusion layer; the third SE module is used to process the feature map input thereto. Attention weights are calculated to obtain an attention weight map. The output is sent to the feature fusion layer; the fourth SE module is used to process the feature map input thereto. Attention weights are calculated to obtain an attention weight map. The output is sent to the feature fusion layer; the fifth SE module is used to process the feature map input thereto. Attention weights are calculated to obtain an attention weight map. The output is fed to the feature fusion layer; the feature fusion layer is used to process the attention weight map respectively. With feature map The weighted feature map is obtained by performing weighted processing. Attention weight map With feature map The weighted feature map is obtained by performing weighted processing. Attention weight map With feature map The weighted feature map is obtained by performing weighted processing. Attention weight map With feature map The weighted feature map is obtained by performing weighted processing. Attention weight map Separately and feature maps The weighted feature map is obtained by performing weighted processing. And the weighted feature map , , , The fusion is performed to obtain a fused feature map. The output is sent to the aforementioned dual attention head module.

Images