Weak supervision image tampering detection positioning method and system based on multiple modes and multiple scales

Through the weakly supervised image tampering detection method of multimodal and multi-scale, the feature fusion and correction are used to fusion and correction of multiple types of features, and the problems of difficulty in labeling and limited single modal features in the prior art are solved, and efficient image tampering detection and position.

CN119027789APending Publication Date: 2024-11-26SHANGHAI JIAOTONG UNIV

Patent Information

Application Number
CN202411023149.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing image tamper detection and positioning techniques rely on a large number of pixel-level annotations, which are time-consuming and labor-intensive, and a single modal feature is limited by specific conditions or interference, affecting the accuracy of detection and positioning.

Method used

Using a weakly supervised image tampering detection and positioning method based on multimodal and multi-scale, multi-scale feature fusion is performed by extracting multiple types of complementary noise features and edge enhancement features, and performing feature stitching and dimensionality reduction, combining high-resolution networks and feature pyramid networks to perform multi-scale feature fusion , and generate tamper probability mask information for detection and positioning through cross-modal correction and feature fusion.

Benefits of technology

It reduces the cost and time of data labeling, improves the robustness and accuracy of detection, and can effectively capture different types of tampering traces in complex scenarios, improving the accuracy of detection and positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027789A_ABST
    Figure CN119027789A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-modal and multi-scale-based weak supervision image tampering detection and positioning method and system, and the method comprises three parts: a multi-modal feature extractor, a feature coding and double-branch feature alignment fusion part, a region extraction network and a multi-task decoder. Firstly, a noise filter and an edge enhancement module are used for extracting multi-modal features from an input image, then a double-branch architecture is used for encoding RGB domain features and auxiliary modal features, then alignment fusion is carried out on representations of the two modals, a region suggestion network is used for an obtained feature map to obtain a candidate frame, and the candidate frame is used for carrying out multi-modal feature extraction. And finally, the tampered area is positioned and detected in the multi-task decoder. According to the method, weak supervised learning of frame labeling is utilized, multi-modal feature extraction, alignment fusion and HRNet and FPN network coding are combined, the accuracy and robustness of detection are improved, and in addition, the method based on the weak supervised learning has the advantages that the data labeling cost is reduced, and the practical application flexibility is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image tampering detection and positioning, and in particular to a multi-modal and multi-scale weakly supervised image tampering detection and positioning method and system. Background Art

[0002] With the continuous development of digital image processing technology, image tampering technology has become increasingly mature. Common types of tampering include copy-move, splicing, removal, etc. The widespread use of tools such as Photoshop has brought challenges to the authenticity and credibility of images. In addition, generative models based on deep learning, such as generative adversarial networks (GANs), have been able to generate realistic images and even create non-existent scenes and characters. Although the development of this technology has brought revolutionary changes in the fields of artistic creation and entertainment, it has also brought new challenges to the field of multimedia forensics. The generated image tampering is almost visually indistinguishable from the real thing, and this type of tampering has also received widespread attention in the field of multimedia forensics.

[0003] Image tampering detection and location technology aims to detect whether an image has been tampered with and find out the tampered area. This is of great significance for maintaining the authenticity of information, combating the spread of false information, and protecting the rights and interests of individuals and enterprises. It has broad application prospects in many fields such as justice, news, and social media.

[0004] Current image tampering detection and localization technologies have the following shortcomings:

[0005] (1) Strongly supervised learning methods rely on a large amount of pixel-level fine annotations, which are time-consuming and laborious.

[0006] (2) Single modal features are often limited by specific conditions or interference, which in turn affects the accuracy of detection and positioning.

[0007] Currently, there is a lack of an image tampering detection and positioning method to solve or partially solve the aforementioned problems. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multi-modal and multi-scale weakly supervised image tampering detection and positioning method and system to solve or partially solve the problem that training data is difficult to label and is greatly affected by single modality features.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] In one aspect of the present invention, a multi-modal and multi-scale weakly supervised image tampering detection and positioning method is provided, wherein the image to be detected is used as the input of a detection and positioning model that has been pre-trained with weak supervision, and tampering probability mask information is predicted to achieve image tampering detection and positioning, wherein the training process of the detection and positioning model includes the following steps:

[0011] Step S1, for the training image, extract multiple types of complementary noise features and edge enhancement features respectively, and obtain edge enhancement noise mixed features through splicing and dimensionality reduction;

[0012] Step S2, taking the training image and the edge-enhanced noise mixed feature as two branches, performing encoding and multi-scale feature fusion processing based on a feature pyramid on each branch in turn, and obtaining fused features by performing cross-modal correction and feature fusion on the features processed by the two branches;

[0013] Step S3, based on the fusion features, generate multiple candidate regions and perform bounding box regression for each candidate region, calculate the bounding box loss, perform data enhancement by constructing example packages of positive and negative samples for the candidate region features, calculate the segmentation loss, and combine the bounding box loss and the segmentation loss to realize the training of the detection and positioning model.

[0014] As a preferred technical solution, the step S1 includes:

[0015] Step S101, for the training image, extract multiple types of complementary noise features through SRM filtering, Bayar convolution, and Nosieprint++ respectively;

[0016] Step S102, for the training image, edge enhancement features are obtained through frequency domain transformation, high-pass filtering and frequency domain inverse transformation;

[0017] Step S103, concatenating the multiple types of complementary noise features and the edge enhancement features, and obtaining edge enhancement noise mixed features through convolution dimensionality reduction.

[0018] As a preferred technical solution, the step S2 includes:

[0019] Step S201, respectively extracting multi-scale features for the training image and the edge-enhanced noise mixed feature using high-resolution networks of respective branches;

[0020] Step S202, based on the multi-scale features, using the feature pyramid network of each branch to perform multi-scale feature fusion processing;

[0021] Step S203, performing channel correction and space correction on the features of the two branches obtained by multi-scale feature fusion, respectively, to obtain the corrected features of the two branches;

[0022] Step S204, calculating the attention weights for the corrected features of the two branches respectively, and fusing the corrected features of the two branches to obtain the final fused features.

[0023] As a preferred technical solution, the channel correction includes the following steps:

[0024] The features obtained by multi-scale feature fusion of the two branches are respectively subjected to average pooling, maximum pooling and splicing processing, and the multi-layer perception mechanism is used to perform channel correction on the two branches.

[0025] As a preferred technical solution, the spatial correction includes the following steps:

[0026] The features obtained by multi-scale feature fusion of the two branches after channel correction are calculated through the spatial level attention mechanism to achieve spatial correction.

[0027] As a preferred technical solution, the step S3 includes the following steps:

[0028] Step S301, based on the fusion features, using the region proposal network, generating multiple candidate regions, calculating the possibility of the candidate regions as tampered regions by scoring, performing non-maximum suppression processing on high-scoring candidate regions, and obtaining the final candidate regions;

[0029] Step S302, mapping the candidate regions obtained in step S302 to the training image, performing feature extraction on each candidate region, aligning the features of the candidate region with those on the training image, performing bounding box regression on each candidate region, and calculating bounding box loss;

[0030] Step S303, performing pixel-level segmentation on the candidate area obtained in step S302, performing data enhancement by constructing example packages of positive and negative samples, constructing a set of positive and negative example packages for a single image, and calculating the segmentation loss;

[0031] Step S304, combining the bounding box loss, the segmentation loss and the binary classification loss, calculates the total loss function value to implement training of the detection and positioning model.

[0032] As a preferred technical solution, the loss function value The calculation is done using the following formula:

[0033]

[0034] in, They are binary classification loss, bounding box loss and segmentation loss, λ cls , reg , seg is the coefficient.

[0035] Another aspect of the present invention provides a multi-modal and multi-scale weakly supervised image tampering detection and positioning system, comprising:

[0036] The multimodal feature extraction module is used to extract multiple types of complementary noise features and edge enhancement features for the input image, and obtain edge enhancement noise mixed features through splicing and dimensionality reduction;

[0037] A feature multi-scale encoding and dual-branch feature fusion alignment module is used to take the input image and the edge-enhanced noise mixed feature as two branches, perform encoding and feature pyramid-based multi-scale feature fusion processing on each branch in turn, and obtain fusion features by performing cross-modal correction and feature fusion on the features processed by the two branches;

[0038] The multi-task decoding module is used in the training stage to generate multiple candidate regions based on the fusion features and perform bounding box regression for each candidate region, calculate the bounding box loss, perform data enhancement by constructing example packages of positive and negative samples for the candidate region features, calculate the segmentation loss, and combine the bounding box loss and the segmentation loss to implement weak supervision training of the detection and positioning model. In the inference stage, image tampering positioning is implemented based on the tampering probability mask map information, and image tampering detection is implemented based on the tampering classification prediction score.

[0039] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned multi-modal and multi-scale based weakly supervised image tampering detection and positioning method.

[0040] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned multi-modal and multi-scale weakly supervised image tampering detection and positioning method.

[0041] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0042] (1) Weakly supervised learning based on box annotation: In order to solve the problem that traditional strongly supervised learning methods rely on a large number of pixel-level fine annotations, and the annotation process is time-consuming and laborious, the present invention adopts a weakly supervised learning method, which only requires box-annotated data for training, greatly reducing the annotation workload and cost. In addition, box annotation is simpler than pixel-level annotation, and annotators can complete data annotation work faster and more efficiently, thereby accelerating model training and deployment. In addition, the box annotation-based method is more flexible in practical applications, avoiding the shortcomings of low accuracy and inaccurate segmentation of image-level annotation methods, which is conducive to promotion and use in different application scenarios and can reduce data annotation costs.

[0043] (2) Complementary multimodal feature extraction: In view of the problem that single modal features are often limited by specific conditions or interference in previous tampering detection and positioning methods, the present invention can effectively reduce the dependence on a certain feature and improve the detection performance in complex scenarios through a more comprehensive multimodal feature extraction. By combining multiple modal features (such as RGB features, SRM noise features, Bayar convolution features, Noiseprint++ features and edge enhancement features), the present invention can capture different information in the image, thereby enhancing the robustness of tampering detection. Different modal features complement each other and can capture different types of tampering traces, improving the accuracy and precision of overall detection.

[0044] (3) Modal feature fusion and alignment: Different from the traditional method that usually uses information from other modalities to assist RGB features, the present invention realizes deep fusion and alignment of multimodal features from dual branches through cross-modal feature correction and feature fusion, ensuring the effective combination of different modal features and improving the integrity and consistency of feature expression. The method in the present invention utilizes high resolution and feature pyramid network structure to effectively extract and fuse multi-scale information, so that the detection system has higher accuracy when processing tampered areas of different scales. Then, through cross-modal feature correction, the features of different modalities can be carefully corrected to eliminate interference and inconsistency between features, further improving the accuracy of detection and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of a multi-modal and multi-scale weakly supervised image tampering detection and positioning system in an embodiment;

[0046] Figure 2 Flow chart of multi-task decoding in an embodiment. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0048] Example 1

[0049] In response to the problems existing in the aforementioned prior art, the present embodiment provides a weakly supervised image tampering detection and positioning method and an image tampering detection and positioning system based on multimodal and multi-scale, aiming to fully utilize complementary multimodal information, and utilize noise information, edge enhancement information and RGB image information to interact, so as to improve the robustness and forensic ability of the model. At the same time, through multi-scale encoding and multi-task decoding, the tampering features at different levels are fully mined. In addition, the weakly supervised learning model training with bounding box annotation does not require expensive pixel-level annotation data, and the cost of obtaining training data is lower. At the same time, the problem of high false alarm rate of image-level labels is avoided, and the detection and positioning performance is better.

[0050] See also Figure 1 , the image tampering detection and positioning system includes a multimodal feature extractor, a feature encoder, a feature fusion module, a regional feature extraction module, and a detection and segmentation branch network. This method uses a dual-branch architecture, and the first branch encodes the RGB image embedding. The second branch encodes the noise embedding extracted by the noise extractor such as SRM and the edge enhancement embedding, obtains the representation of the two branches through the FPN network, and uses the modal fusion alignment network to align and fuse the features of the two branches to obtain the fusion features, and then uses the RPN network to obtain the regional features, which are input into the detection and segmentation network to realize tampering detection and tampering area segmentation.

[0051] The weakly supervised image tampering detection and positioning method based on multi-modal and multi-scale includes the following steps:

[0052] Step S1: Extraction of multimodal features. By extracting complementary noise information and edge information, the artifacts and traces related to tampering and the rich edge information are fully highlighted to meet the tampering detection and positioning requirements in different detection domains. Specifically, this step includes the following sub-steps:

[0053] Step S101, extraction of noise features, using SRM filter, Bayar convolution, Nosieprint++ to extract complementary noise features of RGB images to suppress semantic content and highlight artifacts and traces related to editing and tampering.

[0054] Specifically, this sub-step uses a static high-pass filter to extract the noise feature N from the image to be detected. SRMNoise features of RGB images are extracted using an SRM filter, which can identify edges and boundaries without relying on the camera or compression / editing artifacts.

[0055] Use restricted convolution to extract noise features N from the image to be detected Bayar The pre-trained Bayar convolution module is used to extract the noise features of the RGB image. The Bayar convolution restricts the convolution kernel. In this way, Bayar convolution suppresses the semantic content of the image and pays more attention to tampering traces and artifacts, which has been shown to be robust in previous studies.

[0056] Use Noiseprint++ to extract noise features N from the image to be detected nosieprint++ The self-supervised learning Noiseprint++ module is used to extract noise features in the image and obtain traces and artifact features generated by editing.

[0057] Step S102, using an edge enhancement module to extract edge enhancement components of the image. Since clues of image tampering are usually abundant at the boundaries, the features processed by DCT transform, high-pass filter, and inverse DCT transform can better express edge information.

[0058] Specifically, this sub-step uses frequency domain transformation and high-pass filter to obtain edge enhancement component H. By performing DCT transformation on the RGB image, using a high-pass filter to obtain the high-frequency component of the transformed component, and then using inverse DCT transformation to obtain the edge enhancement component, that is, the frequency domain feature Edge Enhancement Component in Represents DCT transform and inverse DCT transform, respectively. represents a high-pass filter, and α represents the parameter of the high-pass filter.

[0059] In step S103, the three noise features and the edge enhancement component are concatenated, and a convolution layer is used to obtain edge enhancement noise mixed features. Vector concatenation and convolution dimensionality reduction can reduce the amount of calculation and improve the model training and inference speed.

[0060] Specifically, this substep converts the feature {N SRM , N Bayar , N nosieprint++ , H} for vector concatenation, and then use the convolution block to obtain the edge enhanced noise mixed feature X mix .

[0061] Step S2: Feature multi-scale encoding and dual-branch feature fusion alignment. Use the pre-trained HRNet and FPN multi-scale encoding network structure to fully capture the tampering features of different scales and improve the robustness of the model. Use the multimodal feature alignment fusion model to align and fuse the feature maps of the two independent branches, and perform subsequent multi-task decoding on the fused features. Specifically, this step includes the following sub-steps:

[0062] Step S201: In two symmetrical branches, use independent HRNet encoders to encode the image RGB features X RGB and edge-enhanced noise mixed feature X MIX Encoding is performed to obtain features of four scales.

[0063] Specifically, this substep uses the HRNet encoder to encode the image RGB feature X RGB and edge-enhanced noise mixed feature X mix Encoding. HRNet (High Resolution Network) achieves effective fusion of multi-scale features by maintaining high-resolution representation and frequently exchanging information between different resolutions. Specifically, the HRNet encoder includes multiple parallel sub-networks, each of which operates at a different resolution. After multiple exchanges and fusions, the generated features contain rich multi-scale information. After encoding by the HRNet network, features of four scales are obtained for each of the two branches.

[0064] In step S202, two independent FPN networks are used to process the multi-scale features encoded by the two branches respectively. FPN fully integrates the representation encoded by HRNet by constructing a feature pyramid from high level to low level.

[0065] Specifically, this substep processes the features encoded by the two branches through FPN (Feature Pyramid Network). FPN can effectively fuse information from different scales of HRNet and enhance the diversity and robustness of feature representation by constructing a feature pyramid from high to low levels. The specific process is as follows: using the multi-scale features generated by the HRNet encoder, the pyramid structure is constructed step by step, starting from high-level features and merging step by step to low-level features. After each layer is fused, upsampling (bilinear interpolation) is used to enlarge the feature map to the size of the adjacent high-level feature map, and feature weighted fusion is performed to retain information of features of different scales. The feature map finally generated contains information of different scales, which can effectively improve the accuracy of tampering detection and positioning.

[0066] Step S203, using the CM-FRM cross-modal feature correction module to align the features of the two branches (RGB features and edge enhancement noise mixed enhancement);

[0067] Specifically, this substep uses the CM-FRM cross-modal feature correction module (Cross-ModalFeatureRectification Module) to align RGB features and noise enhancement features. The two input features are RGB features and noise enhancement mixed features, and their sizes are both C×H×W. First, average pooling and maximum pooling are performed on these two features respectively, and the features are pooled into 1×1×C dimensional vectors. Then, the pooled vectors are concatenated into a 1×1×4C vector, and the features of the upper and lower branches are corrected through a multi-layer perceptron (MLP) and a Sigmoid function. Subsequently, a spatial-level attention calculation is performed on the corrected features, that is, the attention weights of the RGB features are used to correct the mixed features, and vice versa. Finally, the corrected RGB features and mixed features are output.

[0068] Step S204 uses the FFM feature fusion module to further fuse the RGB features and edge-enhanced noise mixed features aligned by the CM-FRM cross-modal correction module, and outputs the final fused features.

[0069] Specifically, this substep uses the FFM feature fusion module. Based on the features corrected by CM-FRM, FFM is used for further fusion processing. FFM is based on the Transformer architecture. Unlike traditional methods, it treats RGB and mixed features equally. Specifically, the calculation steps of FFM are as follows: QKV calculations are performed on RGB features and mixed features respectively to obtain query, key, and value vectors. An efficient attention mechanism is used for calculation to obtain the fused features. Finally, the fused features are processed through deep separable convolution and residual connection to output the final fused feature map F. fused .

[0070] Step S3: Use RPN to extract the target area and obtain the positioning and detection results through the multi-task decoding head. Use the RPN network to extract the subsequent target area and screen it, and decode the tampering classification, tampering area border detection, and tampering area weak supervision segmentation tasks at the same time. During the training process, the classification loss, border detection regression loss, and segmentation loss are used as optimization targets to update the model parameters. In the inference stage, the classification results of whether it has been tampered and the predicted probability distribution map of the tampered area are obtained. Specifically, this step includes the following sub-steps:

[0071] Step S301, input the fused feature map to RPN, generate potential target areas, generate candidate areas through a sliding window mechanism, obtain several candidate boxes, and retain high-quality region proposals through non-maximum suppression.

[0072] Specifically, in this sub-step, first, the obtained fused feature map is input into the region proposal network (RPN). The main function of RPN is to generate a series of potential target regions (Region Proposals). RPN generates candidate regions on the feature map through a sliding window mechanism. Each sliding window center generates multiple candidate regions of different scales and aspect ratios, scores each candidate region, and calculates its possibility as a foreground (ie, target region) and background. Two 1x1 convolutions are used to calculate the classification scores of the foreground and background, as well as the bounding box regression of the candidate region. Through non-maximum suppression (NMS), non-maximum suppression is applied to candidate regions with higher scores to remove redundant and overlapping regions, and finally retain high-quality region proposals to obtain the final result RoI.

[0073] In step S302, the candidate region generated by the RPN is then subjected to feature extraction through the RoI Align operation to obtain the tampered region bounding box RoI, which is then input into the fully connected layer for bounding box regression.

[0074] Specifically, this substep maps the candidate region RoI generated by RPN back to the original image, extracts features from each candidate region through the RoI Align operation, ensures that the candidate region is aligned with the features on the feature map, performs bounding box regression on each candidate region to obtain an accurate bounding box, and constructs a bounding box loss in the training phase.

[0075] In step S303, the RoI features of the candidate regions generated by RPN and RoIAlign are input into the multi-layer convolutional layer segmentation branch for pixel-level segmentation. In the training phase, the weakly supervised learning method of box annotation is used for training, and in the inference phase, a pixel-level tampering probability mask map is generated.

[0076] Specifically, this substep inputs the candidate region feature RoI generated by RPN into the segmentation branch for more fine-grained pixel-level segmentation. This branch includes multiple convolutional layers, which use the convolutional layers to generate segmentation masks for each candidate region and predict the precise shape and position of the target region. In the training phase, data enhancement is performed by constructing bags of positive and negative samples, and the loss function is optimized using the enhanced training data. Specifically, for each bounding box label of each training image, the row and column pixel sets in the bbox are constructed as positive bags (i.e., sample bags), denoted as Randomly select the same number of row and column pixel sets outside the bbox to construct negative bags, denoted as The above data augmentation method is used to construct a set of positive and negative bags for a single photo. This constructs the weakly supervised segmentation loss of one-way and two-way terms

[0077] In the inference phase, the segmentation mask image is obtained as the tampering localization result. The segmentation mask represents the probability that each pixel belongs to the target. The segmentation mask is combined with the detection result to further adjust and refine the candidate area to ensure the consistency and accuracy of the detection and segmentation results.

[0078] Step S304, comprehensively analyzing the image-level tampering binary cross entropy loss, the bounding box regression loss, and the weakly supervised segmentation loss to construct a multi-task decoding joint loss, and training the model by optimizing the joint loss.

[0079] Specifically, in the training phase, this substep constructs a multi-task decoding loss through three subtasks: image tampering binary classification, tampering region bounding box regression, and tampering region box annotation weak supervision segmentation. Multi-task decoding can effectively improve the robustness and generalization ability of the model. In the inference stage, the image tampering binary classification result is used as the tampering detection result to predict whether the image has been tampered with, and the tampering mask map output by weakly supervised segmentation is used as the tampering segmentation result to predict the pixel-level probability of the image tampering area.

[0080] The following article will use a specific example to illustrate the above solution:

[0081] See also Figure 1 In the step S1 of this embodiment, the multimodal feature extraction uses three different noise filters and frequency domain enhancement modules to obtain edge enhancement noise mixed features, which form two branches with the RGB spatial domain features.

[0082] S100: Perform preprocessing such as resizing the image to be detected to obtain the input image embedding I. Use the SRM system to extract the noise feature R using a static high-pass filter SRM The filter is able to identify edges and boundaries in images without relying on the camera or compression / editing artifacts. A pre-trained Bayar restricted convolution is applied to extract the noise feature R Bayar , extract noise features in the image, suppress semantic content, and focus on tampering traces. Use the Noiseprint++ module that has undergone self-supervised learning to extract noise features R noiseprint , obtain features related to traces and artifacts produced by editing. Obtain edge enhancement components through frequency domain transformation and high-pass filter The specific process is DCT transformation, high-pass filtering, and inverse DCT transformation to enhance the edge information of the image.

[0083] S110: The feature set {R SRM , R Bayar , R noiseprint , H} for vector concatenation, and extract edge enhancement noise mixed features X through convolution blocksMIX .

[0084] See also Figure 1 In step S2 of this embodiment, two independent branches of HRNet encoder and FPN decoder network are used to encode and decode the features obtained in step 1 to obtain the features of RGB spatial domain branch and edge enhancement noise mixed branch. HRNet realizes the effective fusion of multi-scale features through parallel sub-networks and information exchange, and FPN enhances the diversity and robustness of features by constructing feature pyramids and fusing information of different scales.

[0085] S120: In branch 1, use HRNet encoder and FPN decoder to represent the RGB spatial domain X RGB Perform encoding and decoding to obtain representation C1.

[0086] S130: In branch 2, the edge-enhanced noise mixed feature X is obtained by using the HRNet encoder and FPN decoder of the same architecture (the encoder and decoder in branch 1 do not share parameters). MIX Encode and decode to obtain representation C2.

[0087] See also Figure 1 In step S3 of this embodiment, the CMX architecture is used to perform multi-modal feature alignment and fusion on the representations C1 and C2 obtained in step 2. In order to fully align and fuse the two modalities, the cross-modal correction module CM-FRM is first used to align the representations of the two modalities, and then the feature fusion module FFM is used to perform context exchange.

[0088] S140: Using CM-FRM module to characterize RGB spatial domain and edge-enhanced noise characterization First, perform channel correction, perform global maximum pooling and global average pooling for C1 and C2, and then concatenate the four vectors obtained to obtain Then use MLP and Sigmoid function on feature Y, then perform vector splitting, perform channel correction, and obtain the corrected features In addition, spatial correction is performed, the inputs C1 and C2 are concatenated, and then encoded through two 1×1 convolutions and activation functions, that is, softmax is used to obtain the feature map Then separate the feature map, perform spatial correction, and obtain the spatially corrected features Get the final output Setting the parameter λ C =0.5,λ S =0.5.

[0089] S150: Apply FFM module to the features after CM-FRM correction Based on the Transformer architecture, FFM achieves in-depth feature fusion through QKV calculation and efficient attention mechanism. The final fusion feature M is obtained.

[0090] See also Figure 2 In step S4 of this embodiment, a region proposal network (RPN) is used to extract target region features and generate candidate regions. RPN (Region Proposal Network) is mainly used to quickly generate candidate regions that may be tampered with.

[0091] S160: Input the fused feature map M into RPN, and the input image is subjected to feature extraction through the convolution layer. In this embodiment, Renset50 is used as the feature extractor, and then a sliding window operation is performed on the feature map to generate anchor boxes of different sizes and proportions. For the predicted bounding box (position and size) of each anchor point, the candidate regions are screened according to the predicted score of the target existence, and the overlapping candidate regions are removed using non-maximum suppression (NMS), retaining high-quality region suggestions. Finally, several bounding boxes are obtained.

[0092] See also Figure 2 In this embodiment, step S5 includes a multi-task decoding head, which includes segmentation, object detection, and image level classification. In step S5, the final loss function is constructed through multiple tasks. in Represents the cross-entropy loss of the classification task (i.e., the binary classification loss), represents the MIL loss of the positioning and segmentation task, represents the loss of the object detection task.

[0093] S170: The localization branch is composed of MLP layers. For the nth input image In, the predicted tampering score map is calculated. During the training process, the segmentation loss is calculated based on the expanded data set. The method of expanding the data is as follows: Among them I n represents the nth image, B n I n The box annotation, That is I n K n For each bounding box, collect all horizontal and vertical intersection lines to construct positive bags, defined as pi represents the th pixel on each line. Then the same number of negative bags are randomly selected, defined as Construct positive and negative bags as Get the enhanced dataset The formula for calculating the loss is One of the unary items yes is the estimated probability of being positive; for binary terms, φ(S)=∑ (p,p′)∈ε ||S(p)-S(p′)||, ε is the set of all adjacent pixel pairs. By modeling the segmentation task with the above loss, SGD can be used for effective optimization.

[0094] S180: The bounding box regression branch consists of an MLP group. For each bounding box t=(t x , t y , t w , t h ), whose border is labeled as v, and the corresponding border regression is loss is a robust L1 loss, and then the regression loss of each border is summed to get the final regression loss

[0095] S190: Combine three loss training models, namely Use SGD to optimize the loss function, update the model parameters, and obtain the final model.

[0096] In summary, the multimodal and multi-scale weakly supervised image tampering detection and positioning method of this embodiment can effectively locate the image tampering area and classify whether it has been tampered with through multimodal feature extraction, HRNet and FPN network coding, dual-branch multi-scale feature alignment fusion, region extraction and multi-task decoding. In the image tampering detection task, compared with previous methods, this method has lower requirements for data annotation, and at the same time utilizes a variety of complementary modal information to achieve better detection results.

[0097] (1) Advantages of weakly supervised learning based on box annotation

[0098] The weakly supervised learning method can reduce the cost of data annotation. The traditional strongly supervised learning method relies on a large number of pixel-level fine annotations, which are time-consuming and laborious. The present invention adopts weakly supervised learning, and only requires box-annotated data for training, which greatly reduces the workload and cost of annotation. Moreover, box annotation is simpler than pixel-level annotation, and annotators can complete data annotation work faster and more efficiently, thereby accelerating the training and deployment of the model. In addition, the box-annotation-based method is more flexible in practical applications, avoiding the shortcomings of low accuracy and inaccurate segmentation of image-level annotation methods, which is conducive to promotion and use in different application scenarios.

[0099] (2) Advantages of complementary multimodal feature extraction

[0100] In previous tampering detection and positioning methods, single modal features are often limited by specific conditions or interference. However, the method of the present invention can effectively reduce the dependence on a certain feature and improve the detection performance in complex scenarios through more comprehensive multimodal feature extraction. By combining multiple modal features (such as RGB features, SRM noise features, Bayar convolution features, Noiseprint++ features and edge enhancement features), the present invention can capture different information in the image, thereby enhancing the robustness of tampering detection. Different modal features complement each other and can capture different types of tampering traces, improving the accuracy and precision of overall detection.

[0101] (3) Advantages of modal feature fusion alignment

[0102] Traditional methods usually use information from other modalities to assist RGB features, while the present invention uses CM-FRM and FFM modules to achieve deep fusion and alignment of multimodal features from dual branches, ensuring the effective combination of features of different modalities and improving the integrity and consistency of feature expression. The method in the present invention uses HRNet and FPN network structures to effectively extract and fuse multi-scale information, so that the detection system has higher accuracy when processing tampered areas of different scales. Then, through the cross-modal feature correction module (CM-FRM), different modal features can be carefully corrected to eliminate interference and inconsistency between features, further improving the accuracy of detection and positioning.

[0103] Example 2

[0104] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the multi-modal and multi-scale weakly supervised image tampering detection and positioning method as described in Example 1.

[0105] Example 3

[0106] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the multi-modal and multi-scale weakly supervised image tampering detection and positioning method as described in Example 1.

[0107] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-modal and multi-scale weakly supervised image tampering detection and positioning method, characterized in that: The image to be detected is used as the input of the detection and positioning model that has been pre-trained with weak supervision, and the tampering probability mask information is predicted to achieve image tampering detection and positioning, wherein the training process of the detection and positioning model includes the following steps: Step S1, for the training image, extract multiple types of complementary noise features and edge enhancement features respectively, and obtain edge enhancement noise mixed features through splicing and dimensionality reduction; Step S2, taking the training image and the edge-enhanced noise mixed feature as two branches, performing encoding and multi-scale feature fusion processing based on a feature pyramid on each branch in turn, and obtaining fused features by performing cross-modal correction and feature fusion on the features processed by the two branches; Step S3, based on the fusion features, generate multiple candidate regions and perform bounding box regression for each candidate region, calculate the bounding box loss, perform data enhancement by constructing example packages of positive and negative samples for the candidate region features, calculate the segmentation loss, and combine the bounding box loss and the segmentation loss to achieve weak supervision training of the detection and positioning model.

2. According to claim 1, a multi-modal and multi-scale weakly supervised image tampering detection and positioning method is characterized in that: The step S1 comprises: Step S101, for the training image, extract multiple types of complementary noise features through SRM filtering, Bayar convolution, and Nosieprint++ respectively; Step S102, for the training image, edge enhancement features are obtained through frequency domain transformation, high-pass filtering and frequency domain inverse transformation operations; Step S103, concatenating the multiple types of complementary noise features and the edge enhancement features, and obtaining edge enhancement noise mixed features through convolution dimensionality reduction.

3. According to claim 1, a multi-modal and multi-scale weakly supervised image tampering detection and positioning method is characterized in that: The step S2 comprises: Step S201, respectively extracting multi-scale features for the training image and the edge-enhanced noise mixed feature using high-resolution networks of respective branches; Step S202, based on the multi-scale features, using the feature pyramid network of each branch to perform multi-scale feature fusion processing; Step S203, performing channel correction and space correction on the features of the two branches obtained by multi-scale feature fusion, respectively, to obtain the corrected features of the two branches; Step S204, calculating the attention weights for the corrected features of the two branches respectively, and fusing the corrected features of the two branches to obtain the final fused features.

4. According to claim 3, a multi-modal and multi-scale weakly supervised image tampering detection and positioning method is characterized in that: The channel correction comprises the following steps: The features obtained by multi-scale feature fusion of the two branches are respectively subjected to average pooling, maximum pooling and splicing processing, and the multi-layer perception mechanism is used to perform channel correction on the two branches.

5. According to claim 3, a multi-modal and multi-scale weakly supervised image tampering detection and positioning method is characterized in that: The spatial correction comprises the following steps: The features obtained by multi-scale feature fusion of the two branches after channel correction are calculated through the spatial level attention mechanism to achieve spatial correction.

6. According to claim 1, a multi-modal and multi-scale weakly supervised image tampering detection and positioning method is characterized in that: The step S3 comprises the following steps: Step S301, based on the fusion features, using the region proposal network, generating multiple candidate regions, calculating the possibility of the candidate regions as tampered regions by scoring, performing non-maximum suppression processing on high-scoring candidate regions, and obtaining the final candidate regions; Step S302, mapping the candidate regions obtained in step S302 to the training image, performing feature extraction on each candidate region, aligning the features of the candidate region with those on the training image, performing bounding box regression on each candidate region, and calculating bounding box loss; Step S303, performing pixel-level segmentation on the candidate area obtained in step S302, performing data enhancement by constructing example packages of positive and negative samples, constructing a set of positive and negative example packages for a single image, and calculating the segmentation loss; Step S304, combining the bounding box loss, the segmentation loss and the binary classification loss, calculates the total loss function value to implement training of the detection and positioning model.

7. The multi-modal and multi-scale weakly supervised image tampering detection and positioning method according to claim 6 is characterized in that: The loss function value The calculation is done using the following formula: in, They are binary classification loss, bounding box loss and segmentation loss, λ cls , reg , seg is the coefficient.

8. A multi-modal and multi-scale weakly supervised image tampering detection and positioning system, characterized in that: include: The multimodal feature extraction module is used to extract multiple types of complementary noise features and edge enhancement features for the input image, and obtain edge enhancement noise mixed features through splicing and dimensionality reduction; A feature multi-scale encoding and dual-branch feature fusion alignment module is used to take the input image and the edge-enhanced noise mixed feature as two branches, perform encoding and feature pyramid-based multi-scale feature fusion processing on each branch in turn, and obtain fusion features by performing cross-modal correction and feature fusion on the features processed by the two branches; The multi-task decoding module is used in the training stage to generate multiple candidate regions based on the fusion features and perform bounding box regression for each candidate region, calculate the bounding box loss, perform data enhancement by constructing example packages of positive and negative samples for the candidate region features, calculate the segmentation loss, and combine the bounding box loss and the segmentation loss to implement weak supervision training of the detection and positioning model. In the inference stage, image tampering positioning is implemented based on the tampering probability mask map information, and image tampering detection is implemented based on the tampering classification prediction score.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the multi-modal and multi-scale weakly supervised image tampering detection and positioning method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the multi-modal and multi-scale weakly supervised image tampering detection and positioning method as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Tampered image detection method based on deep learning

    CN110349136A

  • Image tampering detection method based on CNN multi-scale feature extraction and hierarchical fusion

    CN115294096A

  • Face deep false detection method based on multi-modal feature fusion

    CN115880749A

  • Weak supervision depth restoration image tampering positioning method and system based on tampering region separation and region constraint loss

    CN116912184A

  • Weak supervision image tampering and forgery detection method based on mask consistency

    CN118155055A

Cited By

  • Document image tampering detection method and system based on double-branch feature fusion

    CN119904879A

  • Image tampering detection method based on multi-stream feature fusion

    CN120147841A

  • Multi-source unsupervised track anomaly detection method and system

    CN120411893A

  • Image forgery positioning detection method based on collaborative difference optimization and multi-modal perception

    CN120708040A