Image tampering positioning method and device, equipment and medium

Abstract

The invention relates to an image tampering positioning method and device, equipment and a storage medium, and the method comprises the steps: extracting two types of initial features from an input image, namely a spatial feature and a high-frequency feature; respectively sending the two types of features into a pre-trained visual backbone network with frozen parameters, introducing two groups of learnable prompt word vectors in front of each layer of the visual backbone network to respectively act on a spatial branch and a high-frequency branch, and carrying out multi-view feature adaptation and enhancement; s3, feature alignment and fusion: in the alignment stage, respectively calculating cross-branch channel attention and space attention and carrying out interactive weighting, and in the fusion stage, applying cross-branch deformable attention by adopting multi-expansion-rate cavity convolution; and S4, sending the multi-scale features output by the visual backbone network into a tampering region positioning decoder to generate a tampering mask. Compared with the prior art, the method has the advantages that the prior knowledge of the pre-training model can be fully mobilized under the condition that the training data is limited, and the higher tampering positioning capability is obtained.

Description

Technical Field

[0001] This invention relates to the field of digital image forensics and content security technology, and in particular to a method, apparatus, device and medium for locating image tampering. Background Technology

[0002] The advancement of image editing technology has lowered the barrier to forgery. Copy-move, splicing, and restoration techniques alter semantics and introduce social risks, making image tampering localization an important research direction. Image tampering localization differs from classification / detection / segmentation in that it focuses on non-semantic cues and low-level discontinuities, such as high-frequency information from differences in camera noise patterns and jagged edges / color inconsistencies at forgery boundaries. Meanwhile, the scarcity of publicly available large-scale image tampering localization training sets limits the performance of methods. Commonly used image tampering localization datasets are limited in size, and researchers often rely on private augmented data or adversarial / style transfer-generated data, but these methods face issues of availability and cost.

[0003] To address the aforementioned issues, some studies have proposed using pre-trained visual backbone networks and fine-tuning them on public datasets to alleviate the data scarcity problem, but this ignores high-frequency information. Other studies have used learnable cue words for low-level structure segmentation, but their simple addition and fusion of multi-view features limits the fusion efficiency. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an image tampering localization method, apparatus, device, and medium. Under conditions of limited training data, it fully utilizes the prior knowledge of the pre-trained model and achieves high-quality multi-view feature alignment and fusion without altering the backbone network structure, thereby obtaining robust and generalized tampering localization capabilities.

[0005] The objective of this invention can be achieved through the following technical solutions: According to a first aspect of the present invention, an image tampering location method is provided, comprising: S1. Extract two types of initial features from the input image: spatial features that characterize the discontinuity of the image structure, and high-frequency features that characterize the statistical differences in frequency domain noise. S2. The two types of initial features are fed into the pre-trained visual backbone network with frozen parameters, and two sets of learnable cue word vectors are introduced in front of each layer of the visual backbone network to act on the spatial branch and the high-frequency branch respectively, so as to perform multi-view feature adaptation and enhancement. S3. Feature Alignment and Fusion: In the alignment stage, channel attention and spatial attention across branches are calculated separately and interactively weighted. In the fusion stage, deformable attention across branches is applied using multi-dilation rate dilated convolution. S4. Input the multi-scale features output by the visual backbone network into the tamper region localization decoder to generate a tamper mask.

[0006] Preferably, the spatial features are obtained through block partitioning and position embedding, and the high-frequency features are extracted through multi-scale Bayar convolution kernels.

[0007] Preferably, the pre-trained visual backbone network is a hierarchical visual encoder with window attention and inter-layer downsampling to generate multi-scale feature maps.

[0008] Preferably, the learnable cue words are set in two groups in each layer of the visual backbone network, and are respectively concatenated with spatial features and high-frequency features before entering the corresponding layer's self-attention structure for joint updating.

[0009] Preferably, the alignment stage calculates cross-branch channel attention and spatial attention respectively and performs interactive weighting, specifically as follows: After globally averaging and pooling the features of the two branches, channel attention weights are generated by a multilayer perceptron, and spatial attention weights are generated by two 1×1 convolutions. Then, a cross-weighting strategy is used to update the features of each branch.

[0010] Preferably, the fusion stage employs multi-dilation rate dilated convolution to apply deformable attention across branches, specifically including: Features are subjected to dilated convolutions with dilation rates of 1, 3, and 5 and concatenated. After integration by 1×1 convolution, information is exchanged through deformable attention between branches. The fused output is a linear combination of the two branches after weighting by deformable attention, and the combination coefficients are learnable parameters.

[0011] Preferably, the tampered region localization decoder uses multi-scale features and query embedding to generate a mask.

[0012] According to a second aspect of the present invention, an image tampering location apparatus for implementing any of the methods is provided, comprising a processor, a memory and program instructions stored therein, the processor being configured to execute any of the method steps when the instructions are executed.

[0013] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.

[0014] According to a fourth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention significantly improves the tampering localization in cross-distribution scenarios by combining a pre-trained backbone network of learnable prompt words and frozen parameters, and aligning and fusing features between layers. It is also more robust to JPEG compression and Gaussian blur attacks, and its robustness to blur is significantly better than the control method.

[0016] (2) Considering that in the image tampering localization task, the tampering traces are mainly reflected in two types of the most critical and stable visual cues: one is the structural discontinuity in the spatial domain, and the other is the statistical difference in noise in the frequency domain. The former is due to the destruction of the original image structure by the tampering process, such as jagged edges, inconsistent textures, or local geometric abrupt changes; the latter is due to the inherent differences in high-frequency noise, compression features, and local residual patterns of different camera imaging processes or generation models. Therefore, high-frequency features can effectively reveal the abrupt changes in noise distribution in the tampered area.

[0017] (3) The present invention uses multi-scale Bayar convolution to extract noise and boundary information and then maps it to the initial high-frequency features of the same dimension as the spatial branch. It has the following advantages: Bayar convolution has strong constraints that the center weight is -1 and the sum of the other weights is +1, so that the convolution kernel naturally has the function of predicting residuals. It can explicitly suppress semantic content and retain only the local strongly correlated high-frequency perturbations. These high-frequency residuals are highly correlated with the tampered area. By expanding Bayar convolution to multi-scale settings, it can capture multi-level noise inconsistencies under different receptive fields, and can effectively cover everything from fine-grained pixel-level residuals to larger-scale texture destruction.

[0018] (4) In this invention, two sets of learnable cue word vectors are introduced in front of each layer of the visual backbone network and applied to the spatial branch and the high-frequency branch respectively to perform multi-view feature adaptation and enhancement. The number of learnable cue word parameters (0.09M) is much lower than that of the dual-branch backbone network with full fine-tuning (93.14M), which is more conducive to adapting to large-scale pre-trained priors. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the image tampering location data stream of the present invention.

[0021] Figure 3 The graph shows the performance of this invention under JPEG compression and Gaussian blur attack, with the purple curve representing the performance of this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] Example like Figure 1 and Figure 2 As shown, this embodiment provides a method for locating image tampering, which includes the following steps: S1. Extract two types of initial features from the input image: spatial features that characterize the discontinuity of the image structure and high-frequency features that characterize the statistical differences in frequency domain noise.

[0024] In image tampering localization tasks, tampering traces are mainly manifested in two types of key and stable visual cues: one is the structural discontinuity in the spatial domain, and the other is the statistical difference in noise in the frequency domain. The former stems from the destruction of the original image structure by the tampering process, such as jagged edges, inconsistent textures, or local geometric abrupt changes; the latter stems from the inherent differences in high-frequency noise, compression features, and local residual patterns of different camera imaging processes or generation models. Therefore, high-frequency features can effectively reveal abrupt changes in noise distribution in the tampered area.

[0025] The specific implementation includes the following sub-steps: S1-1: Receive the input image, complete normalization and block division (preferably block size P=16), and add learnable location embedding to preserve spatial location information.

[0026] S1-2, Constructing multi-view features: (1) Spatial features: Spatial features are obtained by mapping through block partitioning and position embedding.

[0027] (2) High-frequency features: High-frequency features are obtained by multi-scale Bayar convolution kernels. Specifically, noise and boundary information are extracted by multi-scale Bayar convolution (convolution kernel size is 3×3, 5×5, 7×7) and then mapped to high-frequency features of the same dimension as the spatial branch.

[0028] In this embodiment, high-frequency residuals are extracted using multi-scale Bayar convolution kernels, which has the following advantages: First, Bayar convolution, through its strong constraint characteristic of a center weight of -1 and the sum of the remaining weights of +1, enables the convolution kernel to naturally have the function of predicting residuals, which can explicitly suppress semantic content and retain only locally strongly correlated high-frequency perturbations. These high-frequency residuals are highly correlated with the tampered region. Second, by expanding Bayar convolution to multi-scale settings (3×3, 5×5, 7×7), multi-level noise inconsistencies can be captured under different receptive fields, effectively covering everything from fine-grained pixel-level residuals to larger-scale texture destruction.

[0029] S2. Construct a pre-trained backbone network with frozen parameters and inject learnable cues: A visual encoder with frozen parameters is used as a single backbone network. At the input of each layer of the backbone network, learnable cues (preferably 8 words in length) are injected into the spatial and high-frequency branches respectively. The two sets of learnable cues are concatenated with the spatial / high-frequency input features respectively, and then fed into the self-attention and feedforward units of that layer for updating. Only the learnable cues and downstream modules are trained, and the backbone network weights remain frozen, thus adapting to the pre-trained prior with very few new parameters.

[0030] S3. Inter-layer Feature Alignment and Fusion: A feature alignment and fusion module is inserted between adjacent layers of the backbone network to align and fuse the spatial and high-frequency features output from each layer of the backbone network. The aligned features serve as the input to the next layer of the backbone network, and the fused features serve as the input to the decoding network.

[0031] S3-1, Feature alignment process; For the backbone network i Spatial features of layer output F i RGB High frequency characteristics F i HFQ First, a global average is performed at the channel level to obtain the channel statistical vector for each branch. Then, the channel attention weights are obtained through a fully connected neural network and a sigmoid function. W C RGB and W C HFQ Meanwhile, spatial features F i RGB High frequency characteristics F i HFQ Spatial attention weights are obtained by using two 1×1 convolutions after concatenation. W S RGB and W SHFQ .

[0032] Combining channel weights W C RGB and W C HFQ and spatial weights W S RGB and W S HFQ Aligned features are obtained by cross-weighting the initial features. Fa i RGB and Fa i HFQ As the first i+1 The process of cross-weighting the input features of the layer backbone network is as follows: , , Here, "⊙" represents element-wise multiplication.

[0033] S3-2, The process of feature fusion: For the backbone network i Spatial features of layer output F i RGB High frequency characteristics F i HFQ First, multi-scale dilated convolution is used to extract features from different receptive fields. The extracted features from different receptive fields are then concatenated and subjected to a 1×1 convolution. Finally, these features are concatenated with the original features and subjected to a 1×1 convolution to integrate them, resulting in the integrated spatial features. F ^ i RGB and high frequency characteristics F ^ i HFQ Then, cross-branch deformable attention is performed, using one branch of spatial / high-frequency features as the query and the other as the key / value pair to obtain cross-branch attention features. attn RGB and attn HFQ By fusing the above features, we obtain the first... i The fusion method for layer output features is as follows: , in, γ1 and γ2These are learnable parameters.

[0034] S4. The features output from the multi-scale process are fed into the tampering region localization decoder to generate a tampering mask.

[0035] Figure 3 The figure shows the performance under JPEG compression and Gaussian blur attack. The purple curve represents the performance of the solution in this embodiment. It can be seen that the method in this embodiment is more robust under JPEG compression and Gaussian blur attack, and its robustness to blur is significantly better than the control method.

[0036] In this embodiment, the learnable prompt words require a parameter size adjustment of 0.09M, while the dual-branch backbone network with full fine-tuning requires a parameter size adjustment of 93.14M. Thus, the number of learnable prompt word parameters is much lower than that of the dual-branch backbone network with full fine-tuning.

[0037] This embodiment also provides an image tampering location device, including a processor and a memory for storing executable instructions of the processor, the processor being configured to perform the image tampering location method as described above by executing the executable instructions.

[0038] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0039] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0040] The processing unit executes the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S4 by any other suitable means (e.g., by means of firmware).

[0041] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0042] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0043] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0044] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for locating image tampering, characterized in that, include: S1. Extract two types of initial features from the input image: spatial features that characterize the discontinuity of the image structure and high-frequency features that characterize the statistical differences in frequency domain noise. S2. The two types of initial features are fed into the pre-trained visual backbone network with frozen parameters, and two sets of learnable cue word vectors are introduced in front of each layer of the visual backbone network to act on the spatial branch and the high-frequency branch respectively, so as to perform multi-view feature adaptation and enhancement. S3. Feature Alignment and Fusion: In the alignment stage, channel attention and spatial attention across branches are calculated separately and interactively weighted. In the fusion stage, deformable attention across branches is applied using multi-dilation rate dilated convolution. S4. Input the multi-scale features output by the visual backbone network into the tamper region localization decoder to generate a tamper mask.

2. The image tampering location method according to claim 1, characterized in that, The spatial features are obtained through block partitioning and position embedding, and the high-frequency features are extracted through multi-scale Bayar convolution kernels.

3. The image tampering location method according to claim 1, characterized in that, The pre-trained visual backbone network is a hierarchical visual encoder with window attention and inter-layer downsampling to generate multi-scale feature maps.

4. The image tampering location method according to claim 1, characterized in that, The learnable cue words are set in two sets in each layer of the visual backbone network. They are concatenated with spatial features and high-frequency features respectively and then entered into the self-attention structure of the corresponding layer for joint updating.

5. The image tampering location method according to claim 1, characterized in that, The alignment phase calculates cross-branch channel attention and spatial attention separately and performs interactive weighting, specifically as follows: After globally averaging and pooling the features of the two branches, channel attention weights are generated by a multilayer perceptron, and spatial attention weights are generated by two 1×1 convolutions. Then, a cross-weighting strategy is used to update the features of each branch.

6. The image tampering location method according to claim 1, characterized in that, The fusion phase employs multi-dilation rate dilated convolution to apply deformable attention across branches, specifically including: Features are subjected to dilated convolutions with dilation rates of 1, 3, and 5 and concatenated. After integration by 1×1 convolution, information is exchanged through deformable attention between branches. The fused output is a linear combination of the two branches after weighting by deformable attention, and the combination coefficients are learnable parameters.

7. The image tampering location method according to claim 1, characterized in that, The tampered region localization decoder uses multi-scale features and query embedding to generate a mask.

8. An image tampering location device for implementing the method of any one of claims 1 to 7, characterized in that, It includes a processor, a memory, and program instructions stored therein, wherein the processor is configured to perform the method steps of any one of claims 1 to 7 when executing the instructions.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images