Image tampering positioning method based on memory guide adjacent feature fusion

Abstract

The invention discloses an image tampering positioning method based on memory guide adjacent feature fusion, and belongs to the technical field of image tampering positioning methods. According to the method, a memory guiding and positioning module is provided, semantic clustering is used for accumulating consistency and abnormal features between a tampered area and an image background, memory prior is formed, image tampering positioning expression learning is guided, and therefore the positioning accuracy of the tampered area is improved; an adjacent feature interaction module is also provided, features of different scales are fused through expansion convolution, and more stable and effective feature representation is generated by utilizing jump connection; in view of the importance of global semantic information and detail information on accurate positioning and tampering region segmentation, the adjacent feature interaction module enhances the feature representation learning ability of image tampering positioning by fusing adjacent features; the invention further provides a verification fusion module, and channel connection between high-level and low-level features is established through a weight map, so that more effective multi-level feature fusion is realized; and the verification fusion module verifies the feature masks output at different levels through a multi-level supervision strategy, and finally generates a fine tampering region mask.

Description

Technical Field

[0001] The present invention relates to the technical field of image tampering positioning methods, and in particular to an image tampering positioning method based on memory-guided adjacent feature fusion. Background Art

[0002] In recent years, with the rapid popularization of digital acquisition equipment and the widespread use of powerful image editing software, it has become easier to edit and modify digital images. In order to meet certain needs or achieve certain purposes, people can modify and edit digital photos at will, which greatly reduces the credibility of digital photos and subverts the traditional concept of "seeing is believing". The existence of tampered and forged images has made people lose trust in multimedia data.

[0003] Nowadays, many image tampering localization methods based on deep learning have been proposed. For example, some studies have shown that the edges of tampered areas can be constructed under explicit supervision, and the boundary representation can be enhanced by leveraging the detailed information contained in low-level features. Other related studies have shown that the generalization ability of the model can be improved by introducing noisy views to provide new semantically unrelated elements. In addition, some works capture subtle traces of tampering that are difficult to detect in the RGB domain by extracting frequency information as supplementary data. However, these works often ignore the importance of feature fusion. Therefore, some methods based on feature fusion have been proposed to enrich contextual information and improve segmentation accuracy, such as Figure 1 As shown in (a) and (b) in the figure, existing methods use a bottom-up approach to perform feature fusion and enrich context information.

[0004] Although existing methods have made significant progress in performance, most feature fusion methods simply use a bottom-up approach to perform fusion. This may lead to the propagation of irrelevant semantic features, which in turn causes confusion of important information between different feature layers. In addition, locating the initial position of subtle tampering traces only from a global perspective may lead to overconfident mispredictions. In view of this, the present invention proposes an image tampering localization method based on memory-guided neighbor feature fusion to solve the above problems. Summary of the invention

[0005] 1. Technical Problems to be Solved by the Present Invention

[0006] The purpose of the present invention is to propose an image tampering localization method based on memory-guided adjacent feature fusion to solve the following problems existing in the prior art:

[0007] (1) Existing image tampering localization methods only fuse features in a bottom-up manner, which leads to the propagation of irrelevant semantic features and causes confusion between the target and the background. Some semantic features are introduced into shallow features, making it difficult to fully represent detailed information. The same problem also exists in deep features.

[0008] (2) Existing techniques only locate the initial position of subtle tampering traces in an image from a global perspective, which can lead to overconfident prediction errors;

[0009] (3) The existing prototype memory technology stores images at the instance level in target recognition, while image tampering detection is a binary segmentation task with only target and background categories. This technology has not yet been introduced into the field of image tampering, making the construction of the memory library too limited.

[0010] 2. Technical solution

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] The image tampering localization method based on memory-guided adjacent feature fusion includes the following steps:

[0013] S1, perform data enhancement processing on the tampered image;

[0014] S2, input the data enhanced image into the image editor PVTv2, and extract the tampering features and background features of the input image;

[0015] S3, inputting the tampering features and background features into the memory guided positioning module to obtain positioning features for subsequent model training;

[0016] S4, inputting the adjacent backbone features into the adjacent feature interaction module for fusion to obtain fusion features;

[0017] S5, inputting the fused backbone features into the verification fusion module to further enrich the upper and lower information, and obtaining the resulting binary image by compressing the number of channels;

[0018] S6. Input the resulting binary image into the training target to complete the training of the image tampering localization model, and then generate an accurate object prediction mask.

[0019] Preferably, the data enhancement processing in S1 specifically refers to: rotating, cutting and rotating the object image.

[0020] Preferably, S2 specifically includes the following contents:

[0021] S2.1. Input the image into the image encoder PVTv2, and convert the data into a high-dimensional vector representation through the embedding layer;

[0022] S2.2, input the high-dimensional vector representation into multiple Transformer blocks to capture the features in the input data and model the features at different levels; the Transformer blocks include a multi-head self-attention mechanism and a feedforward neural network;

[0023] S2.3. Based on the modeling results, complete the extraction of the tampering characteristics of the input image and the image background features.

[0024] Preferably, the training objectives in S3 include: using semantic clustering to accumulate the consistency and abnormal features between the tampered area and the image background to form a memory prior, and guide feature representation learning.

[0025] Preferably, the S4 specifically includes the following contents:

[0026] A series of continuous dilated convolutions in a neighboring feature interaction module are used to fuse features of different scales, and skip connections are used to generate more robust and effective feature representations. For neighboring shallow features, the neighboring feature interaction module adopts a dual-branch cross optimization strategy to process them so as to retain detail information. For neighboring deep features, the neighboring feature interaction module combines contextual information through feature weighting and captures global semantic information for processing. The neighboring feature interaction module enhances the feature representation learning ability of image tampering localization by fusing neighboring shallow features and deep features.

[0027] Preferably, the positioning features and object prediction masks are based on the cross entropy loss L em The specific calculation formula is:

[0028]

[0029] Among them, N represents the total number of samples; i represents the sample index; exp represents the exponential function with the natural constant e as the base; R represents the prediction result of the model; w represents the weight; and b represents the bias.

[0030] Preferably, the total loss L of the image tampering localization model is all The function is expressed as:

[0031]

[0032] Among them, P i Represents the prediction map output by each fusion module; G represents the true value.

[0033] 3. Beneficial effects

[0034] (2) This paper proposes a memory-guided localization module that uses semantic clustering to accumulate the consistency and abnormal features between the tampered area and the image background. These features form a memory prior to guide the learning of image tampering localization representation, thereby improving the localization accuracy of the tampered area.

[0035] (3) The present invention proposes a neighboring feature interaction module, which fuses features of different scales through a series of continuous dilated convolutions and uses skip connections to generate more robust and effective feature representations. In addition, the neighboring feature interaction module adopts a dual-branch cross optimization strategy to process neighboring shallow features, significantly retaining detail information. For neighboring deep features, the neighboring feature interaction module combines contextual information through feature weighting and captures global semantic information. Given the importance of global semantic information and detail information for accurately locating and segmenting tampered areas, the neighboring feature interaction module enhances the feature representation learning ability of image tampering localization by fusing neighboring features.

[0036] (4) In order to further enrich the contextual semantic information and improve the integrity and accuracy of the segmentation results, the present invention also proposes a verification fusion module, which establishes channel connections between high-level and low-level features through a weight map, thereby achieving more effective multi-level feature fusion; the feature masks output by the verification fusion module at different levels are verified through a multi-level supervision strategy, and finally a refined tampering area mask is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A diagram of a common feature fusion architecture in the prior art proposed in the background technology of the present invention;

[0038] Figure 2 This is an overall flow chart of the image tampering localization method based on memory-guided adjacent feature fusion proposed in Example 1 of the present invention;

[0039] Figure 3 This is a principle structure diagram of the image tampering localization method based on memory-guided adjacent feature fusion proposed in Example 2 of the present invention;

[0040] Figure 4 This is the visualization result diagram proposed in Example 2 of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0043] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0044] Embodiment 1:

[0045] See also Figure 2 ,This invention proposes an image tampering location method based on memory-guided adjacent feature fusion, the specific process is as follows:

[0046] S1. Perform data enhancement processing on the tampered image, specifically: rotating, cutting and rotating the object image;

[0047] S2: Input the data-enhanced image into the image editor PVTv2 to extract the tampering features and background features of the input image, including the following:

[0048] S2.1. Input the image into the image encoder PVTv2, and convert the data into a high-dimensional vector representation through the embedding layer;

[0049] S2.2, input the high-dimensional vector representation into multiple Transformer blocks to capture the features in the input data and model the features at different levels; the Transformer blocks include a multi-head self-attention mechanism and a feedforward neural network;

[0050] S2.3. According to the modeling results, the tampering characteristics of the input image and the extraction of the image background features are completed;

[0051] S3, inputting the tampering features and background features into the memory-guided positioning module to obtain positioning features, using semantic clustering to accumulate the consistency and abnormal features between the tampered area and the image background to form a memory prior, and guide feature representation learning;

[0052] S4. Input the adjacent backbone features into the adjacent feature interaction module for fusion to obtain fusion features, which specifically includes the following contents:

[0053] A series of continuous dilated convolutions in the neighboring feature interaction module are used to fuse features of different scales, and skip connections are used to generate more robust and effective feature representations. For neighboring shallow features, the neighboring feature interaction module uses a dual-branch cross optimization strategy to process them so as to retain detail information. For neighboring deep features, the neighboring feature interaction module combines contextual information through feature weighting and captures global semantic information for processing. The neighboring feature interaction module enhances the feature representation learning ability of image tampering localization by fusing neighboring shallow features and deep features.

[0054] S5, inputting the fused backbone features into the verification fusion module to further enrich the upper and lower information, and obtaining the resulting binary image by compressing the number of channels;

[0055] S6. Input the resulting binary image into the training target to complete the training of the image tampering localization model, and then generate an accurate object prediction mask.

[0056] The above positioning features and object prediction masks are based on the cross entropy loss L em The specific calculation formula is:

[0057]

[0058] Among them, N represents the total number of samples; i represents the sample index; exp represents the exponential function with the natural constant e as the base; R represents the prediction result of the model; w represents the weight; and b represents the bias.

[0059] The total loss L of the above image tampering localization model is all The function is expressed as:

[0060]

[0061] Among them, P i Represents the prediction map output by each fusion module; G represents the true value.

[0062] Embodiment 2:

[0063] Based on Example 1, but different in that, the image tampering positioning method based on memory-guided neighboring feature fusion proposed by the present invention is described below in conjunction with specific examples and drawings, and the specific content is as follows.

[0064] See also Figure 3 The structure of the image tampering localization method based on memory-guided neighboring feature fusion proposed in the present invention is as follows: Figure 3 The present invention uses PVTv2 as the backbone network.

[0065] First, the backbone network is used to extract multi-level features from the input image, denoted as f i, i={1,...,4};

[0066] Then, f2, f3, and f4 are passed to the memory-guided localization module to generate a representation with higher-level semantics, denoted as f5;

[0067] Next, the present invention designs multiple neighboring feature interaction modules (NFIMs) to generate fusion features F by integrating valuable information in neighboring features. i ;

[0068] Finally, the verification fusion module enriches the contextual information to generate accurate prediction masks.

[0069] See also Figure 4 , and compare the visualization results. Specifically, our method performs well in the small-scale tampered regions in the first row and the multi-tampered regions in the third row. This is due to our carefully designed memory-guided localization module and the clever processing of deep semantic features. For the large-scale tampered regions in the second row, our model generates more refined masks by leveraging shallow detail features to detect local artifacts. In all these scenarios, we can see that our model achieves significantly more accurate results when referring to the true labels.

[0070] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and improved concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the protection scope of the present invention.

Claims

1. An image tampering localization method based on memory-guided neighboring feature fusion, characterized in that: The following steps are involved: S1, perform data enhancement processing on the tampered image; S2, input the data enhanced image into the image editor PVTv2, and extract the tampering features and background features of the input image; S3, inputting the tampering features and background features into the memory guided positioning module to obtain positioning features for subsequent model training; S4, inputting the adjacent backbone features into the adjacent feature interaction module for fusion to obtain fusion features; S5, inputting the fused backbone features into the verification fusion module to further enrich the upper and lower information, and obtaining the resulting binary image by compressing the number of channels; S6. Input the resulting binary image into the training target to complete the training of the image tampering localization model, and then generate an accurate object prediction mask.

2. The image tampering location method based on memory-guided adjacent feature fusion according to claim 1 is characterized in that: The data enhancement processing described in S1 specifically refers to: rotating, cutting and rotating the object image.

3. The image tampering location method based on memory-guided adjacent feature fusion according to claim 1 is characterized in that: The S2 specifically includes the following contents: S2.

1. Input the image into the image encoder PVTv2, and convert the data into a high-dimensional vector representation through the embedding layer; S2.2, input the high-dimensional vector representation into multiple Transformer blocks to capture the features in the input data and model the features at different levels; the Transformer blocks include a multi-head self-attention mechanism and a feedforward neural network; S2.

3. Based on the modeling results, complete the extraction of the tampering characteristics of the input image and the image background features.

4. The image tampering location method based on memory-guided adjacent feature fusion according to claim 1 is characterized in that: The training objectives described in S3 include: using semantic clustering to accumulate the consistency and abnormal features between the tampered area and the image background to form a memory prior and guide feature representation learning.

5. The image tampering location method based on memory-guided adjacent feature fusion according to claim 1 is characterized in that: The S4 specifically includes the following contents: A series of continuous dilated convolutions in a neighboring feature interaction module are used to fuse features of different scales, and skip connections are used to generate more robust and effective feature representations. For neighboring shallow features, the neighboring feature interaction module adopts a dual-branch cross optimization strategy to process them so as to retain detail information. For neighboring deep features, the neighboring feature interaction module combines contextual information through feature weighting and captures global semantic information for processing. The neighboring feature interaction module enhances the feature representation learning ability of image tampering localization by fusing neighboring shallow features and deep features.

6. The image tampering location method based on memory-guided adjacent feature fusion according to claim 1 is characterized in that: The positioning features and object prediction masks are based on the cross entropy loss L em The specific calculation formula is: Among them, N represents the total number of samples; i represents the sample index; exp represents the exponential function with the natural constant e as the base; R represents the prediction result of the model; w represents the weight; and b represents the bias.

7. The image tampering location method based on memory-guided adjacent feature fusion according to claim 6 is characterized in that: The total loss L of the image tampering localization model all The function is expressed as: Among them, P i Represents the prediction map output by each fusion module; G represents the true value.

Images