A multimodal image tampering localization method and system based on edge guidance

Abstract

The present invention discloses a multimodal image tampering localization method and system based on edge guidance, comprising the following steps: acquiring an RGB spatial domain image and a noise domain image; using a pyramid visual transformer as an encoder to perform feature extraction on the RGB spatial domain image and the noise domain image to obtain a multi-level coding feature map, and fusing the multi-level coding feature map to obtain a multimodal fusion feature map; performing edge prior feature extraction on the multi-level coding feature map to obtain a boundary prior feature map; fusing the boundary prior feature map with the multimodal fusion feature map to extract semantic features and obtain a multi-level saliency feature coding map; inputting the multi-level saliency feature coding map into a multi-level and multi-scale feature aggregation module, adopting a group-by-group interaction mechanism and a channel attention mechanism to mine discriminative and valuable semantic information from different channels, and finally outputting an image tampering localization result image.

Description

Technical Field

[0001] The present invention relates to the technical field of image tampering positioning detection, and in particular to a multimodal image tampering positioning method and system based on edge guidance. Background Art

[0002] In today's rapidly developing era of artificial intelligence, deep learning has profoundly impacted computer vision, significantly impacting daily life, academia, and national security. The widespread application of deep learning has also driven the development of image tampering detection technology. In recent years, the abuse of digital image manipulation methods, such as fake face scams and fake news, has become increasingly common, and the application of image tampering detection technology has played a crucial role.

[0003] Initially, image tampering localization detection was primarily based on manual feature extraction. However, this approach significantly degrades in performance when subjected to post-processing attacks (such as compression, filtering, and color adjustment), making it insufficient for current image tampering techniques. Therefore, deep learning-based methods are gradually becoming mainstream. This approach, supported by large-scale pre-training data covering a wide range of tampering categories, effectively improves the accuracy of image tampering localization tasks and has been widely used in the field of image tampering localization detection.

[0004] Although deep learning methods have made significant progress in image tampering location detection, tamperers are also constantly improving their technology. There are already some tampering methods that can hide tampering traces to a certain extent, making tampering detection more difficult. Therefore, although deep learning methods have improved the performance of tampering location detection to a certain extent, these methods are not perfect. When facing some tampering techniques, the accuracy of existing tampering location detection methods may still be insufficient. Therefore, how to further improve the accuracy of tampering location detection has become an urgent problem to be solved. Summary of the Invention

[0005] In view of the above-mentioned prior art, the present invention provides a multimodal image tampering location method and system based on edge guidance, which mainly solves the technical problems existing in the above-mentioned background technology.

[0006] To achieve the above-mentioned purpose, the technical solution of the embodiment of the present invention is implemented as follows:

[0007] In a first aspect, the present invention provides a multimodal image tampering localization method based on edge guidance, comprising the following steps:

[0008] Obtain RGB spatial domain image and noise domain image;

[0009] Using a pyramid visual transformer as an encoder, feature extraction is performed on the RGB spatial domain image and the noise domain image to obtain a multi-level encoding feature map, and the multi-level encoding feature map is fused to obtain a multimodal fusion feature map;

[0010] Performing edge prior feature extraction on the multi-level coding feature map to obtain a boundary prior feature map;

[0011] Fusing the boundary prior feature map with the multimodal fusion feature map to extract semantic features and obtain a multi-level saliency feature encoding map;

[0012] The multi-level saliency feature encoding map is input into a multi-level and multi-scale feature aggregation module, and a group-by-group interaction mechanism and a channel attention mechanism are adopted to mine the discriminative and valuable semantic information from different channels, and finally the image tampering localization result image is output.

[0013] Optionally, the using a pyramid visual transformer as an encoder to perform feature extraction on the RGB spatial domain image and the noise domain image to obtain a multi-level encoding feature map, and fusing the multi-level encoding feature map to obtain a multimodal fusion feature map, includes:

[0014] Performing feature extraction on the RGB spatial domain image and the noise domain image to obtain multi-level coding feature maps based on the RGB spatial domain image and the noise domain image respectively;

[0015] The multi-level coding feature map based on the RGB spatial domain image includes a first-level RGB coding feature map, a second-level RGB coding feature map, a third-level RGB coding feature map, and a fourth-level RGB coding feature map;

[0016] The multi-level coding feature map based on the noise domain image includes a first-level noise coding feature map, a second-level noise coding feature map, a third-level noise coding feature map, and a fourth-level noise coding feature map;

[0017] The multi-level encoded feature maps of the RGB spatial domain image and the noise domain image are used to utilize four trainable dual attention modules consisting of a channel attention mechanism and a position attention mechanism to adaptively explore pixel dependencies at the spatial and channel levels and selectively update features to obtain a four-layer multimodal fusion feature map.

[0018] Optionally, performing edge prior feature extraction on the multi-level coding feature map to obtain a boundary prior feature map includes:

[0019] Performing feature fusion on the first-level RGB coding feature map and the second-level RGB coding feature map based on the RGB spatial domain image using a gated fusion strategy to obtain a gated fusion feature;

[0020] The gated fusion features and the fourth-level RGB encoding feature map based on the RGB spatial domain image are used to extract edge prior features using a Sobel filter, and the edge information of low-level features and high-level features is combined, and then a convolution operation and Sigmoid function calculation are performed to obtain a boundary prior feature map.

[0021] Optionally, fusing the boundary prior feature map with the multimodal fusion feature map to extract semantic features and obtain a multi-level saliency feature encoding map includes:

[0022] Obtaining a significant boundary information map according to the boundary prior feature map and the multimodal fusion feature map;

[0023] Inputting the salient boundary information map into an edge-guided module based on separation attention guidance composed of two parallel residual channel attention modules to extract the foreground image and the background image respectively;

[0024] Based on the foreground image and the background image, a multi-level saliency feature coding image is obtained;

[0025] The step of obtaining a multi-level saliency feature coding map based on the foreground image and the background image includes:

[0026] The foreground image and the background image are fused and input into a multi-scale channel attention module to obtain a composite image;

[0027] The synthetic image is interacted with the foreground image to obtain a first branch feature map, the synthetic image is interacted with the background image and then subjected to a dense combination block to obtain a second branch feature map, and then the first branch feature map and the second branch feature map are fused and subjected to a dense combination block to obtain a multi-level saliency feature coding map.

[0028] Optionally, the edge prior feature extraction is performed on the multi-level coding feature map to obtain a boundary prior feature map, and the calculation formula is:

[0029]

[0030]

[0031]

[0032] in, is the gated fusion feature map, is the boundary prior feature map, It is the significant boundary information feature map obtained after the gated fusion feature map passes through the Sobel filter. is the fourth-level RGB encoding feature map, It is the feature map obtained by passing the fourth-level RGB encoding feature map through the Sobel filter; is convolution, and Cat() is connection operation.

[0033] Optionally, the significant boundary information map is obtained according to the boundary prior feature map and the multimodal fusion feature map, and the calculation formula is:

[0034]

[0035] in, For the significant boundary information map, is the boundary prior feature map, is the multimodal fusion feature map, , is element-wise multiplication;

[0036] The salient boundary information map is input into an edge guidance module based on separation attention guidance composed of two parallel residual channel attention modules to extract the foreground image and the background image respectively. The calculation formula is:

[0037]

[0038]

[0039] in, Foreground image, For the background image, is the residual channel attention module, and All are learnable parameters;

[0040] The multi-level saliency feature coding map is obtained based on the foreground image and the background image, and the calculation formula is:

[0041]

[0042] in, is a multi-level saliency feature encoding map, For batch normalization and combination.

[0043] Optionally, the multi-level saliency feature encoding map is input into a multi-level and multi-scale feature aggregation module, and a group-by-group interaction mechanism and a channel attention mechanism are used to mine discriminative and valuable semantic information from different channels, including:

[0044] Adopting a group-by-group interaction mechanism, the number of channels of the multi-level saliency feature encoding map is expanded through convolution operations; the expanded feature set is divided into multiple groups according to the channel dimension, and feature interaction is performed between the groups in an iterative manner;

[0045] The specific calculation formula is:

[0046]

[0047] in, For the final image tampering positioning result image, is a multi-level saliency feature encoding map, is the convolution operation, is the group interaction feature, where G is the number of groups, Act () is the ReLU activation function, Nor is the normalization operation, is the characteristic modulation vector.

[0048] Optionally, the method further includes: training an edge-guided multimodal image tampering localization model using a total loss function composed of an edge loss function and an image tampering localization supervision loss function;

[0049] The edge loss function is Dice loss;

[0050] The image tampering positioning supervision loss function includes weighted cross entropy loss and weighted IOU loss;

[0051] The total loss is calculated as follows:

[0052]

[0053] in, is the total loss function, is the weighted cross entropy loss, is the weighted IOU loss, is the Dice loss, To balance the parameters, To tamper with the regional prediction map, This is the real tampering area positioning map, is the boundary prior feature map, The actual boundary map.

[0054] In a second aspect, the present invention further provides a multimodal image tampering localization system based on edge guidance, comprising:

[0055] Noise feature extraction unit, multimodal feature extraction unit, multimodal feature fusion unit, edge prediction unit, edge-guided evolution unit, image decoding generation unit;

[0056] The noise feature extraction unit is used to process the RGB domain image to obtain a noise domain image;

[0057] The multimodal feature extraction unit uses a pyramid visual transformer as an encoder to extract features from the RGB spatial domain image and the noise domain image to obtain a four-level encoding feature map;

[0058] The multimodal feature fusion unit includes a trainable dual attention module composed of a channel attention mechanism and a position attention mechanism, which is used to perform feature fusion on the four-level encoding feature maps to obtain a four-layer multimodal fusion feature map;

[0059] The edge prediction unit includes a gated edge perception module, a Sobel filter, a convolution module, and a Sigmoid function; the gated edge perception module is used to extract edge prior features, and a gated fusion strategy is adopted to suppress noise that may exist in low-level features to obtain gated fusion features; the gated fusion features and the fourth-level RGB encoding feature map based on the RGB spatial domain image are input into the Sobel filter, the convolution module, and the Sigmoid function to obtain a boundary prior feature map;

[0060] The edge-guided evolution unit includes an edge-guided module based on separable attention guidance, a multi-scale channel attention module, and two dense combination blocks, which are composed of two parallel residual channel attention modules; it is used to sequentially input the four-layer multimodal fusion feature map into the edge-guided module based on separable attention guidance, and introduce the boundary prior feature map as a priori guidance to obtain a foreground image and a background image, input the foreground image and the background image into the multi-scale channel attention module and the two dense combination blocks, perform feature enhancement and extraction on the boundary information of the tampered area in the image, and obtain a multi-level saliency feature encoding map;

[0061] The image decoding generation unit includes a multi-level and multi-scale feature aggregation module, which is used to perform information interaction and feature refinement on the multi-level saliency feature encoding map and the output feature maps of different levels to obtain the final image tampering positioning result image.

[0062] The beneficial effects of the present invention are: a multimodal image tampering localization method based on edge guidance provided by the present invention introduces high-frequency information as a supplement to RGB images, uses PVT2 as the backbone network, and performs multi-level feature extraction on RGB images and noise images; uses a trainable dual attention mechanism to fuse multimodal features; makes full use of low-level features containing rich texture information and high-level features containing rich semantic information, extracts edge prior feature maps under explicit edge supervision, and mines edge semantic information as a priori guidance for the edge-guided fusion module; designs an edge guidance module based on split attention, adopts a split attention mechanism to generate foreground features and background features, and locates more complete tampering objects through the synergy of foreground and background information, and at the same time fuses boundary information to generate features with clearer boundaries; designs a multi-level and multi-scale feature aggregation module, and gradually integrates multi-level features in a top-down manner to enhance the mixed-scale feature representation.

[0063] The present invention extracts the multimodal information and edge information of the tampered image in sequence, and effectively eliminates artifacts through encoding and decoding conversion of a multi-level multi-scale feature aggregation module. In addition, the present invention utilizes an edge-guided method based on separated attention, takes the foreground as the seed point and the background as the irrelevant area, and performs segmentation near the seed point, effectively mining the key features and edge information in the tampered image, while also reducing the influence of boundary blur, and more effectively and accurately locating and detecting the tampered area of ​​the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of a flow chart of a multimodal image tampering localization method based on edge guidance provided in an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of the process of noise feature extraction in the present invention;

[0066] Figure 3 Schematic diagram of the process of the multimodal feature extraction unit and the multimodal feature fusion unit in the present invention;

[0067] Figure 4 Schematic diagram of the process of the edge prediction unit in the present invention;

[0068] Figure 5 Schematic diagram of the process of the edge-guided evolution unit in the present invention;

[0069] Figure 6 Schematic diagram of the process of the multi-level and multi-scale feature aggregation module in the present invention. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is further elaborated in detail below in conjunction with the drawings and specific embodiments of the specification. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In the following description, reference is made to "some embodiments", which describes a subset of all possible embodiments, but it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0071] In the following description, numerous specific details are provided to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without one or more of these details. In other instances, certain technical features well known in the art are not described to avoid confusion with the present invention.

[0072] It should be understood that the present invention can be implemented in different forms and should not be interpreted as being limited to the embodiments proposed herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. And the purpose of the terms used herein is only to describe specific embodiments and is not intended to limit the present invention. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "comprising" and / or "comprising" when used in this specification determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0073] It should also be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "inner," "outer," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only implementation methods.

[0074] In order to fully understand the present invention, a detailed structure will be provided in the following description to illustrate the technical solution proposed by the present invention. Optional embodiments of the present invention are described in detail below. However, in addition to these detailed descriptions, the present invention may also have other implementations.

[0075] Example 1

[0076] Please refer to the attached Figure 1 , the present application provides a multimodal image tampering location method based on edge guidance, comprising the following steps:

[0077] Obtain RGB spatial domain image and noise domain image;

[0078] Using a pyramid visual transformer as an encoder, feature extraction is performed on the RGB spatial domain image and the noise domain image to obtain a multi-level encoding feature map, and the multi-level encoding feature map is fused to obtain a multimodal fusion feature map;

[0079] Performing edge prior feature extraction on the multi-level coding feature map to obtain a boundary prior feature map;

[0080] Fusing the boundary prior feature map with the multimodal fusion feature map to extract semantic features and obtain a multi-level saliency feature encoding map;

[0081] The multi-level saliency feature encoding map is input into a multi-level and multi-scale feature aggregation module, and a group-by-group interaction mechanism and a channel attention mechanism are adopted to mine the discriminative and valuable semantic information from different channels, and finally the image tampering localization result image is output.

[0082] Specifically, an RGB spatial domain image is first obtained. The RGB spatial domain image refers to the original image represented in the RGB (red, green, and blue) color space. The noise domain image is generated by extracting noise information from the RGB spatial domain image. The noise domain image can be represented as the same size as the RGB spatial domain image. The noise domain image can reveal some subtle changes that are difficult to detect in the RGB spatial domain image. The tampered image is different from the real image in noise distribution. Therefore, by using the RGB spatial domain image and the noise domain image for feature extraction, it is beneficial to the subsequent identification of the tampered image.

[0083] In this embodiment, please refer to the attached Figure 2 , using the Bayar filter to extract noise information, the Bayar filter can adaptively learn tampering features that are independent of semantic content from the image. Using it to extract noise modality can avoid interference from image semantics and promote the versatility and robustness of the model.

[0084] Using the pyramid visual transformer (PVT2) as an encoder, multi-level feature extraction can be performed on RGB spatial domain images and noise domain images to combine high-level features and low-level features in the image, make full use of low-level features containing rich texture information and high-level features containing rich semantic information, and obtain a multimodal fusion feature map; edge prior feature extraction is performed on the multi-level encoding feature map, edge semantic information is mined, and a more complete tampering object is located. At the same time, boundary information is fused to generate features with clearer boundaries, and a multi-level saliency feature encoding map is obtained; the multi-level saliency feature encoding map is input into the multi-level multi-scale feature aggregation module, and multi-level features are gradually integrated in a top-down manner to enhance the mixed-scale feature representation, and the final image tampering localization result image is obtained; the final image tampering localization result image obtained by this method has high accuracy.

[0085] As an optional implementation, please refer to the attached Figure 3 The method uses a pyramid visual transformer as an encoder to extract features from the RGB spatial domain image and the noise domain image to obtain a multi-level encoding feature map, and fuses the multi-level encoding feature map to obtain a multimodal fusion feature map, including:

[0086] Performing feature extraction on the RGB spatial domain image and the noise domain image to obtain multi-level coding feature maps based on the RGB spatial domain image and the noise domain image respectively;

[0087] The multi-level coding feature map based on the RGB spatial domain image includes a first-level RGB coding feature map, a second-level RGB coding feature map, a third-level RGB coding feature map, and a fourth-level RGB coding feature map;

[0088] The multi-level coding feature map based on the noise domain image includes a first-level noise coding feature map, a second-level noise coding feature map, a third-level noise coding feature map, and a fourth-level noise coding feature map;

[0089] The multi-level encoded feature maps of the RGB spatial domain image and the noise domain image are used to utilize four trainable dual attention modules consisting of a channel attention mechanism and a position attention mechanism to adaptively explore pixel dependencies at the spatial and channel levels and selectively update features to obtain a four-layer multimodal fusion feature map.

[0090] Specifically, the Pyramid Visual Transformer (PVT2) combines the pyramid structure of CNN and the global self-attention mechanism of Transformer. The pyramid structure achieves multi-level expression of image information by gradually reducing the resolution of feature maps and increasing feature dimensions. The global self-attention mechanism reduces computational complexity, enabling PVT2 to efficiently process high-resolution feature maps, thereby improving computational efficiency and performance.

[0091] The pyramid visual transformer extracts multi-scale long-distance dependency features from RGB spatial domain images and noise domain images. The pyramid structure is used to calculate its representation through spatial reduction attention operations, thereby reducing resource consumption and explicitly extracting more powerful features. Finally, four levels of coded feature maps based on RGB spatial domain images and four levels of coded feature maps based on noise domain images are obtained respectively.

[0092] The trainable dual-attention module is based on two parallel attention mechanisms: channel attention (CA) and position attention (PA), which adaptively explore pixel dependencies at the spatial and channel levels. CA flattens multimodal fusion features into a one-dimensional patch embedding and adds a learnable position embedding, which is randomly initialized to compensate for the spatial information destroyed by serialization, to generate the final sequence embedding. PA selectively updates the features of each position by taking the weighted sum of all position features. Finally, the outputs of CA and PA are added together.

[0093] The four-level coding feature maps based on the RGB spatial domain image and the four-level coding feature maps based on the noise domain image are respectively sent to four trainable dual attention modules for multimodal feature fusion to obtain four layers of multimodal fusion feature maps, namely: the first-level multimodal fusion feature map is obtained by fusing the first-level RGB coding feature map with the first-level noise coding feature map, the second-level multimodal fusion feature map is obtained by fusing the second-level RGB coding feature map with the second-level noise coding feature map, the third-level multimodal fusion feature map is obtained by fusing the third-level RGB coding feature map with the third-level noise coding feature map, and the fourth-level multimodal fusion feature map is obtained by fusing the fourth-level RGB coding feature map with the fourth-level noise coding feature map.

[0094] As an optional implementation, please refer to the attached Figure 4 , performing edge prior feature extraction on the multi-level coding feature map to obtain a boundary prior feature map, including:

[0095] Performing feature fusion on the first-level RGB coding feature map and the second-level RGB coding feature map based on the RGB spatial domain image using a gated fusion strategy to obtain a gated fusion feature;

[0096] The gated fusion features and the fourth-level RGB encoding feature map based on the RGB spatial domain image are used to extract edge prior features using a Sobel filter, and the edge information of low-level features and high-level features is combined, and then a convolution operation and Sigmoid function calculation are performed to obtain a boundary prior feature map.

[0097] Specifically, a gated fusion strategy is adopted to fuse the first-level RGB encoding feature map and the second-level RGB encoding feature map. The two feature maps are respectively integrated into a 3 × 3 convolution layer, and then batch normalization and activation function are performed respectively. The two feature maps are then connected to obtain fused low-level features, which effectively fuses the low-level features of the first two layers while suppressing the noise that may exist in the low-level features. Subsequently, the Sobel filter is used to extract the boundary prior feature map from the gated fusion features obtained by fusion of the first-level RGB encoding feature map and the second-level RGB encoding feature map, and the fourth-level multimodal fusion encoding feature map, so as to combine low-level features and high-level features to model edge information related to the object.

[0098] As an optional implementation, please refer to the attached Figure 5 , fusing the boundary prior feature map with the multimodal fusion feature map to extract semantic features and obtain a multi-level saliency feature encoding map, including:

[0099] Obtaining a significant boundary information map according to the boundary prior feature map and the multimodal fusion feature map;

[0100] Inputting the salient boundary information map into an edge-guided module based on separation attention guidance composed of two parallel residual channel attention modules to extract the foreground image and the background image respectively;

[0101] Based on the foreground image and the background image, a multi-level saliency feature coding image is obtained;

[0102] The step of obtaining a multi-level saliency feature coding map based on the foreground image and the background image includes:

[0103] The foreground image and the background image are fused and input into a multi-scale channel attention module to obtain a composite image;

[0104] The synthetic image is interacted with the foreground image to obtain a first branch feature map, the synthetic image is interacted with the background image and then subjected to a dense combination block to obtain a second branch feature map, and then the first branch feature map and the second branch feature map are fused and subjected to a dense combination block to obtain a multi-level saliency feature coding map.

[0105] Specifically, the boundary prior feature map is a representation of the extracted image boundary information. The boundary prior feature map is used as a priori guidance for the edge guidance module based on separated attention guidance, and the multimodal fusion feature map is used as input to extract the foreground image and the background image. The boundary prior feature map can provide clues about the target edge, which can better help the extraction of the foreground image and the background image. Moreover, the boundary information can enhance the robustness to noise and complex background, especially when the target edge is not obvious. Therefore, the contextual information in the boundary information feature map is used for implicit edge guidance, so as to achieve the extraction of rich semantic features without additional explicit boundary information supervision, and enhance the perception of the edge information of the tampered area.

[0106] The edge-guided module based on separated attention guidance mainly includes two parallel residual channel attention modules, which enable the module to focus more on the information channels and high-frequency information in the feature map. At the same time, a multi-scale channel attention module is applied when fusing foreground features and background features to further strengthen the information interaction between the two branches. The features that have undergone information interaction between the two branches are fused to obtain a multi-level saliency feature encoding map.

[0107] As an optional implementation, the edge prior feature extraction is performed on the multi-level coding feature map to obtain a boundary prior feature map, and the calculation formula is:

[0108]

[0109]

[0110]

[0111] in, is the gated fusion feature map, is the boundary prior feature map, It is the significant boundary information feature map obtained after the gated fusion feature map passes through the Sobel filter. is the fourth-level RGB encoding feature map, It is the feature map obtained by passing the fourth-level RGB encoding feature map through the Sobel filter; is convolution, and Cat() is connection operation.

[0112] As an optional implementation, the significant boundary information map is obtained according to the boundary prior feature map and the multimodal fusion feature map, and the calculation formula is:

[0113]

[0114] in, For the significant boundary information map, is the boundary prior feature map, is the multimodal fusion feature map, , is element-wise multiplication;

[0115] The salient boundary information map is input into an edge guidance module based on separation attention guidance composed of two parallel residual channel attention modules to extract the foreground image and the background image respectively. The calculation formula is:

[0116]

[0117]

[0118] in, Foreground image, For the background image, is the residual channel attention module, and All are learnable parameters;

[0119] The multi-level saliency feature coding map is obtained based on the foreground image and the background image, and the calculation formula is:

[0120]

[0121] in, is a multi-level saliency feature encoding map, For batch normalization and combination.

[0122] As an optional implementation, please refer to the attached Figure 6 The multi-level saliency feature encoding map is input into the multi-level and multi-scale feature aggregation module, and a group-by-group interaction mechanism and a channel attention mechanism are used to mine the discriminative and valuable semantic information from different channels, including:

[0123] Adopting a group-by-group interaction mechanism, the number of channels of the multi-level saliency feature encoding map is expanded through convolution operations; the expanded feature set is divided into multiple groups according to the channel dimension, and feature interaction is performed between the groups in an iterative manner;

[0124] The specific calculation formula is:

[0125]

[0126] in, For the final image tampering positioning result image, is a multi-level saliency feature encoding map, For the convolution operation, is the group interaction feature, where G is the number of groups, Act() is the ReLU activation function, Nor is the normalization operation, feature modulation vector.

[0127] Specifically, since different channels carry different semantic information, it becomes crucial to deeply挖掘 the valuable clues contained in each channel; using a multi-level multi-scale feature aggregation module, adopting a group-by-group interaction mechanism and a channel attention mechanism,挖掘 the discriminative and valuable semantic information from different channels, and promoting information exchange and feature refinement between different channels;

[0128] Specifically: First, adopt the group-by-group interaction mechanism, use 1x1 convolution to expand the channel number of the multi-level saliency feature encoding map of, and then divide the set of multi-level saliency feature encoding maps after expanding the channel number into G groups according to the channel dimension , and the feature interaction between groups is carried out in an iterative manner; the first group is divided into three feature sets of convolutional blocks, where is used for information exchange with the next group, and the other two groups are used for channel modulation; in the j-th group (1 < j < G), the feature is connected along the channel with the feature of the previous group , and this feature group is also divided into three feature sets and then convolutional and splitting operations are carried out;

[0129] Subsequently, adopt the channel attention mechanism, the feature sets are spliced through a small convolutional network and then converted into a feature modulation vector , which is used to weight another feature , and the weighted feature is sent to the convolutional layer for processing, foreground and background label segmentation is carried out, and the output feature map is obtained.

[0130] The multi-level saliency feature encoding map has four levels. Therefore, passing the first-level multi-level saliency feature encoding map through the multi-level multi-scale feature aggregation module for feature interaction will obtain the first-level output feature map, and then inputting the first-level output feature map and the second-level multi-level saliency feature encoding map into the multi-level multi-scale feature aggregation module for feature interaction will obtain the second-level output feature map. Then, inputting the second-level output feature map and the third-level multi-level saliency feature encoding map into the multi-level multi-scale feature aggregation module for feature interaction will obtain the third-level output feature map. Next, inputting the third-level output feature map and the fourth-level multi-level saliency feature encoding map into the multi-level multi-scale feature aggregation module for feature interaction will obtain the fourth-level output feature map. Finally, the obtained fourth-level output feature map is the final image tampering localization result image.

[0131] Note: The Chinese words "挖掘" are used in the original text and are left untranslated here as there is no clear indication of their exact English equivalent in the context. You may need to replace them with appropriate English words according to the specific meaning intended in the patent context.As an optional implementation, the method further includes: training the edge-guided multimodal image tampering localization model using a total loss function composed of an edge loss function and an image tampering localization supervision loss function;

[0132] The edge loss function is Dice loss;

[0133] The image tampering positioning supervision loss function includes weighted cross entropy loss and weighted IOU loss;

[0134] The total loss is calculated as follows:

[0135]

[0136] in, is the total loss function, is the weighted cross entropy loss, is the weighted IOU loss, is the Dice loss, To balance the parameters, To tamper with the regional prediction map, This is the real tampering area positioning map, is the boundary prior feature map, The actual boundary map.

[0137] Specifically, in the process of training the recognition model, the total loss function is also used for training. The total loss function includes two parts: the edge loss function and the image tampering localization supervision loss function; the image tampering localization supervision loss function is used to generate the image tampering localization feature map, including the weighted cross entropy loss ( ) and weighted IOU loss ( ) for supervision; for edge supervision, the edge loss function ( ), the marginal loss function can handle the strong imbalance between positive and negative samples.

[0138] It should be noted that tampered images refer to images that have been artificially modified or edited, such as images generated by image editing software (such as Photoshop) or artificial intelligence technology (such as generative adversarial networks, GANs);

[0139] Tampered area location images: refers to images with tampered areas marked; these images clearly indicate the specific location where the tampering occurred.

[0140] Example 2

[0141] A multimodal image tampering localization system based on edge guidance, comprising:

[0142] Noise feature extraction unit, multimodal feature extraction unit, multimodal feature fusion unit, edge prediction unit, edge-guided evolution unit, image decoding generation unit;

[0143] The noise feature extraction unit is used to process the RGB domain image to obtain a noise domain image;

[0144] The multimodal feature extraction unit uses a pyramid visual transformer as an encoder to extract features from the RGB spatial domain image and the noise domain image to obtain a four-level encoding feature map;

[0145] The multimodal feature fusion unit includes a trainable dual attention module composed of a channel attention mechanism and a position attention mechanism, which is used to perform feature fusion on the four-level encoding feature maps to obtain a four-layer multimodal fusion feature map;

[0146] The edge prediction unit includes a gated edge perception module, a Sobel filter, a convolution module, and a Sigmoid function; the gated edge perception module is used to extract edge prior features, and a gated fusion strategy is adopted to suppress noise that may exist in low-level features to obtain gated fusion features; the gated fusion features and the fourth-level RGB encoding feature map based on the RGB spatial domain image are input into the Sobel filter, the convolution module, and the Sigmoid function to obtain a boundary prior feature map;

[0147] The edge-guided evolution unit includes an edge-guided module based on separable attention guidance, a multi-scale channel attention module, and two dense combination blocks, which are composed of two parallel residual channel attention modules; it is used to sequentially input the four-layer multimodal fusion feature map into the edge-guided module based on separable attention guidance, and introduce the boundary prior feature map as a priori guidance to obtain a foreground image and a background image, input the foreground image and the background image into the multi-scale channel attention module and the two dense combination blocks, perform feature enhancement and extraction on the boundary information of the tampered area in the image, and obtain a multi-level saliency feature encoding map;

[0148] The image decoding generation unit includes a multi-level and multi-scale feature aggregation module, which is used to perform information interaction and feature refinement on the multi-level saliency feature encoding map and the output feature maps of different levels to obtain the final image tampering positioning result image.

[0149] Specifically, an RGB domain image is obtained, and a noise feature extraction unit is used to extract noise features from the RGB domain image and generate a corresponding noise domain image. Both the RGB domain image and the noise domain image are input into a multimodal feature extraction unit. In the multimodal feature extraction unit, the RGB domain image and the noise domain image are subjected to their respective corresponding pyramid visual transformers for feature extraction of RGB spatial domain and noise modality to obtain four levels of coding feature maps corresponding to the RGB domain and four levels of coding feature maps corresponding to the noise domain. The four levels of coding feature maps corresponding to the RGB domain and the four levels of coding feature maps corresponding to the noise domain are then input into a multimodal feature fusion unit for feature fusion, namely: the first-level RGB coding feature map based on the RGB spatial domain image is fused with the first-level noise coding feature map based on the noise domain image, the second-level RGB coding feature map based on the RGB spatial domain image is fused with the second-level noise coding feature map based on the noise domain image, and the third-level RGB coding feature map based on the RGB spatial domain image is fused with the third-level noise coding feature map based on the noise domain image. The fourth-level RGB coding feature map based on the RGB spatial domain image is fused with the fourth-level noise coding feature map based on the noise domain image to obtain the multimodal fusion feature map corresponding to the four levels; the first-level RGB coding feature map and the second-level RGB coding feature map are then fused using the gated edge perception module to obtain the gated fusion feature, and the gated fusion feature and the fourth-level RGB coding feature map are subjected to the Sobel filter, the convolution module and the Sigmoid function to obtain the boundary prior feature map; the multimodal fusion feature map and the boundary prior feature map corresponding to the four levels are then input into the edge-guided evolution unit, and the foreground map and background map are obtained after the edge-guided module based on the separation attention guidance. The multi-scale channel attention module and the two dense combination blocks are then used to enhance and extract the boundary information of the tampered area in the image to obtain a multi-level saliency feature coding map. Finally, the multi-level saliency feature coding map is input into the image decoding generation unit for information interaction and feature refinement of the four-level saliency feature coding maps and the output feature maps of different levels, and the image tampering positioning result is finally output.

[0150] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. The scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A multimodal image tampering localization method based on edge guidance, characterized in that: The following steps are involved: Obtain RGB spatial domain image and noise domain image; Using a pyramid visual transformer as an encoder, feature extraction is performed on the RGB spatial domain image and the noise domain image to obtain a multi-level encoding feature map, and the multi-level encoding feature map is fused to obtain a multimodal fusion feature map; Performing edge prior feature extraction on the multi-level coding feature map to obtain a boundary prior feature map; Fusing the boundary prior feature map with the multimodal fusion feature map to extract semantic features and obtain a multi-level saliency feature encoding map; The multi-level salient feature encoding map is input into a multi-level and multi-scale feature aggregation module, and a group-by-group interaction mechanism and a channel attention mechanism are adopted to mine the discriminative and valuable semantic information from different channels, and finally an image tampering localization result image is output; The step of fusing the boundary prior feature map with the multimodal fusion feature map to extract semantic features and obtain a multi-level saliency feature encoding map includes: Obtaining a significant boundary information map according to the boundary prior feature map and the multimodal fusion feature map; Inputting the salient boundary information map into an edge-guided module based on separation attention guidance composed of two parallel residual channel attention modules to extract the foreground image and the background image respectively; The foreground image and the background image are fused and input into a multi-scale channel attention module to obtain a composite image; The synthetic image is interacted with the foreground image to obtain a first branch feature map, the synthetic image is interacted with the background image and then subjected to a dense combination block to obtain a second branch feature map, and then the first branch feature map and the second branch feature map are fused and subjected to a dense combination block to obtain a multi-level saliency feature coding map.

2. The edge-guided multimodal image tampering location method according to claim 1, characterized in that: The method uses a pyramid visual transformer as an encoder to perform feature extraction on the RGB spatial domain image and the noise domain image to obtain a multi-level encoding feature map, and fuses the multi-level encoding feature map to obtain a multimodal fusion feature map, including: Performing feature extraction on the RGB spatial domain image and the noise domain image to obtain multi-level coding feature maps based on the RGB spatial domain image and the noise domain image respectively; The multi-level coding feature map based on the RGB spatial domain image includes a first-level RGB coding feature map, a second-level RGB coding feature map, a third-level RGB coding feature map, and a fourth-level RGB coding feature map; The multi-level coding feature map based on the noise domain image includes a first-level noise coding feature map, a second-level noise coding feature map, a third-level noise coding feature map, and a fourth-level noise coding feature map; The multi-level encoded feature maps of the RGB spatial domain image and the noise domain image are used to utilize four trainable dual attention modules consisting of a channel attention mechanism and a position attention mechanism to adaptively explore pixel dependencies at the spatial and channel levels and selectively update features to obtain a four-layer multimodal fusion feature map.

3. The edge-guided multimodal image tampering location method according to claim 1, characterized in that: The step of extracting edge prior features from the multi-level coding feature maps to obtain a boundary prior feature map includes: Performing feature fusion on the first-level RGB coding feature map and the second-level RGB coding feature map based on the RGB spatial domain image using a gated fusion strategy to obtain a gated fusion feature; The gated fusion features and the fourth-level RGB encoding feature map based on the RGB spatial domain image are used to extract edge prior features using a Sobel filter, and the edge information of low-level features and high-level features is combined, and then a convolution operation and Sigmoid function calculation are performed to obtain a boundary prior feature map.

4. The edge-guided multimodal image tampering location method according to claim 1, characterized in that: The edge prior feature extraction is performed on the multi-level coding feature map to obtain the boundary prior feature map, and the calculation formula is: in, is the gated fusion feature map, is the boundary prior feature map, It is the significant boundary information feature map obtained after the gated fusion feature map passes through the Sobel filter. is the fourth-level RGB encoding feature map, It is the feature map obtained by passing the fourth-level RGB encoding feature map through the Sobel filter; is convolution, and Cat() is connection operation.

5. The edge-guided multimodal image tampering location method according to claim 1, characterized in that: The significant boundary information map is obtained according to the boundary prior feature map and the multimodal fusion feature map, and the calculation formula is: in, For the significant boundary information map, is the boundary prior feature map, is the multimodal fusion feature map, , is element-wise multiplication; The salient boundary information map is input into an edge guidance module based on separation attention guidance composed of two parallel residual channel attention modules to extract the foreground image and the background image respectively. The calculation formula is: in, Foreground image, For the background image, is the residual channel attention module, and All are learnable parameters; Based on the foreground image and the background image, a multi-level saliency feature coding map is obtained, and the calculation formula is: in, is a multi-level saliency feature encoding map, For batch normalization and combination.

6. The edge-guided multimodal image tampering location method according to claim 5, characterized in that: The multi-level saliency feature encoding map is input into a multi-level and multi-scale feature aggregation module, and a group-by-group interaction mechanism and a channel attention mechanism are used to mine discriminative and valuable semantic information from different channels, including: Adopting a group-by-group interaction mechanism, the number of channels of the multi-level saliency feature encoding map is expanded through convolution operations; the expanded feature set is divided into multiple groups according to the channel dimension, and feature interaction is performed between the groups in an iterative manner; The specific calculation formula is: in, For the final image tampering positioning result image, is a multi-level saliency feature encoding map, is the convolution operation, is the group interaction feature, where G is the number of groups, Act () is the ReLU activation function, Nor is the normalization operation, is the characteristic modulation vector.

7. The edge-guided multimodal image tampering location method according to claim 1, characterized in that: The method also includes: training an edge-guided multimodal image tampering localization model using a total loss function composed of an edge loss function and an image tampering localization supervision loss function; The edge loss function is Dice loss; The image tampering positioning supervision loss function includes weighted cross entropy loss and weighted IOU loss; The total loss is calculated as follows: in, is the total loss function, is the weighted cross entropy loss, is the weighted IOU loss, is the Dice loss, To balance the parameters, To tamper with the regional prediction map, This is the real tampering area positioning map, is the boundary prior feature map, The actual boundary map.

8. A multimodal image tampering localization system based on edge guidance, characterized in that: include: Noise feature extraction unit, multimodal feature extraction unit, multimodal feature fusion unit, edge prediction unit, edge-guided evolution unit, image decoding generation unit; The noise feature extraction unit is used to process the RGB domain image to obtain a noise domain image; The multimodal feature extraction unit uses a pyramid visual transformer as an encoder to extract features from the RGB spatial domain image and the noise domain image to obtain a four-level encoding feature map; The multimodal feature fusion unit includes a trainable dual attention module composed of a channel attention mechanism and a position attention mechanism, which is used to perform feature fusion on the four-level encoding feature maps to obtain a four-layer multimodal fusion feature map; The edge prediction unit includes a gated edge perception module, a Sobel filter, a convolution module, and a Sigmoid function; the gated edge perception module is used to extract edge prior features, and a gated fusion strategy is adopted to suppress noise that may exist in low-level features to obtain gated fusion features; the gated fusion features and the fourth-level RGB encoding feature map based on the RGB spatial domain image are input into the Sobel filter, the convolution module, and the Sigmoid function to obtain a boundary prior feature map; The edge-guided evolution unit includes an edge-guided module based on separable attention guidance, a multi-scale channel attention module, and two dense combination blocks, which are composed of two parallel residual channel attention modules; it is used to sequentially input the four-layer multimodal fusion feature map into the edge-guided module based on separable attention guidance, and introduce the boundary prior feature map as a priori guidance to obtain a foreground image and a background image, input the foreground image and the background image into the multi-scale channel attention module and the two dense combination blocks, perform feature enhancement and extraction on the boundary information of the tampered area in the image, and obtain a multi-level saliency feature encoding map; The image decoding generation unit includes a multi-level and multi-scale feature aggregation module, which is used to perform information interaction and feature refinement on the multi-level saliency feature encoding map and the output feature maps of different levels to obtain the final image tampering positioning result image.

Images