A Method for Constructing an Image Tampering Detection Model Based on Weak Supervision

Abstract

This invention belongs to the field of computer multimedia security, specifically involving a method for constructing an image tampering detection model based on weak supervision. The method includes: extracting RGB features and noise-sensitive features of each image through a region feature construction unit and fusing them; semantically segmenting each image to obtain multiple binary mask images for the corresponding image; obtaining region features of different regions of the corresponding image based on the product of the fused features and each binary mask image; introducing a normal feature memory unit and a tampering feature memory unit; outputting the matching results of the features stored in each unit with each region feature and the enhanced features of each region feature through query and read operations of each unit; adding the two enhanced features output by the two units and concatenating them with the corresponding region features; performing detection and classification based on the concatenated features; and creatively introducing a constraint-based memory unit loss for loss calculation. This invention can achieve high-precision image tampering detection without pixel-level label information.

Description

Technical Field

[0001] This invention belongs to the field of computer multimedia security, and more specifically, relates to a method for constructing an image tampering detection model based on weak supervision. Background Technology

[0002] With the rapid development of science and technology, digital media technology has advanced by leaps and bounds, and digital products have become commonplace in ordinary households. Ordinary people can now use their mobile phones to record fleeting moments, generating massive amounts of media data that are transmitted to the internet. However, alongside the development and maturation of digital media technology, numerous image editing tools have also gradually appeared in the public eye, such as open-source editing software like Meitu and Photoshop, and even mobile phone cameras often come with built-in beauty filters. These software programs are very simple and easy to use, allowing people to modify digital images according to their own ideas at any time, often with effects that are difficult to discern with the naked eye.

[0003] However, some malicious individuals may tamper with images and publish them online, which not only affects people's judgment of the authenticity of information but may even have adverse social impacts. In recent years, malicious incidents caused by image tampering have frequently occurred, posing a significant threat to multimedia information security, mainly involving piracy, the spread of rumors, and disinformation. Therefore, image forensics research has become a crucial aspect of multimedia security. The field of online public opinion analysis now places great emphasis on image authenticity, and news media and judicial departments have specialized professionals responsible for verifying the authenticity of images. The authenticity of images in the process of online information dissemination affects people's daily lives and is related to social stability; ensuring the authenticity of the image information obtained has become a very important need in modern life.

[0004] Since image tampering poses significant security risks, the ability to obtain forensic evidence from image data to verify image authenticity is crucial. Existing image forensics technologies are mainly divided into two types based on their detection methods: active forensics and passive forensics. Active image forensics methods involve embedding digital signatures, digital watermarks, and other verification information into the image before extracting and verifying the information. However, the high requirements for the authenticity of the embedded information and the high cost of embedding it have limited the widespread application of active forensics technologies.

[0005] Blind image forensics, also known as passive image forensics, directly determines the source and integrity of an image, broadening its application. With the development of information technology, the efficiency of deep learning in image processing has become increasingly apparent. In recent years, blind image tampering forensics has primarily employed deep neural network-based methods to detect image tampering. However, existing methods mainly perform tampering detection at the image level. While image-level labeling can detect whether an image has been tampered with, it cannot detect specific tampered areas. To detect specific tampered areas, pixel-level labels are required, which necessitates significant human resources and increases the complexity of applying the model in real-world environments. Summary of the Invention

[0006] To address the shortcomings and improvement needs of existing technologies, this invention provides a method for constructing an image tampering detection model based on weak supervision, aiming to achieve high-precision tampering image detection based on image-level labels.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for constructing an image tampering detection model based on weak supervision is provided, comprising:

[0008] The sample images are input into an image tampering detection network. This network includes: a region feature construction unit, used to extract RGB features and noise-sensitive features from each image and fuse them to obtain fused features; semantic segmentation of each image to obtain multiple binary mask images covering different regions of the corresponding image; and region features of different regions of the corresponding image obtained based on the product of the fused features and each binary mask image; a dual-memory unit framework, including a normal feature memory unit and a tampering feature memory unit, used to output the matching results between the features stored in each unit and each region feature, as well as the enhanced features of each region feature, through query and read operations of each unit; and to add the two enhanced features output by the two units and concatenate them with the corresponding region feature to obtain the concatenated features; and a classifier used to perform tampering image recognition based on the concatenated feature set of each sample image.

[0009] The detection loss is calculated, and the parameters of the image tampering detection network are iteratively trained to complete the model construction.

[0010] The detection loss includes a memory unit loss based on constraints and the matching results. The constraints include: the features stored in the tampered feature memory unit do not match the features of each region of the real sample image, but match at least one region feature of the tampered sample image; the features stored in the normal feature memory unit match the features of each region of the real sample image, but do not match at least one region feature of the tampered sample image; the features of the real sample image are close to the regional features of the real region in the tampered sample image, while the regional features of the tampered region are far away.

[0011] Furthermore, the matching result between the features stored in each memory cell and the features of each region is a three-dimensional vector with the length and width dimensions matching the features of that region and the height dimension matching the number of feature blocks stored in that memory cell;

[0012] The calculation of the memory unit loss is implemented as follows:

[0013] The matching result between the features stored in each memory cell and the features of each region is converted into a matching score between the features of that region and the features stored in that memory cell;

[0014] The memory cell loss is calculated based on the matching score between the features of each region and the features stored in each memory cell, as well as the constraints.

[0015] Furthermore, the loss, constrained by the fact that the features of the real sample image are similar to the regional features of the real region in the tampered sample image, while the regional features of the tampered region are far away, is as follows:

[0016] The ternary loss L is calculated based on the region features in the tampered sample image that have the highest matching score with the tampered feature memory unit, the region features in the tampered sample image that have the highest matching score with the normal feature memory unit, and the region features of any region in the real sample image. trip .

[0017] Furthermore, the conversion is implemented as follows:

[0018] Based on the matching results between the features stored in each memory unit and the features of each region, and the binary mask image corresponding to the features of that region, according to the formula... Calculate the similarity score vector S between the features of the region and the features stored in the memory unit, where T(i,j) represents the matching result vector corresponding to pixel (i,j) in the region feature in the corresponding matching result, and the number of elements is the same as the number of feature blocks stored in the memory unit; M(i,j) represents the pixel value corresponding to the pixel in the binary mask image corresponding to the region feature; N m This represents the total number of pixels in the corresponding image region within the region feature; the average of the top K similarity scores from the similarity score vector S between each region feature and the features stored in each memory unit is taken as the matching score between the region feature and the features stored in the memory unit.

[0019] Furthermore, a matching score of 0 indicates that the features stored in the memory unit do not match the features of the region at all, while a matching score of 1 indicates that the features stored in the memory unit match the features of the region at all.

[0020] The constraints include: the matching scores between the features stored in the tampered feature memory unit and the features of each region of the real sample image should all be 0, and the maximum matching score among the features of each region of the tampered sample image should be 1; the matching scores between the features stored in the normal feature memory unit and the features of each region of the real sample image should all be 1, and the minimum matching score among the features of each region of the tampered sample image should be 0; the features of the real sample image are close to the regional features of the real region in the tampered sample image, while the regional features of the tampered region are far away, with 0 indicating a complete mismatch and 1 indicating a complete match;

[0021] The memory cell loss includes:

[0022]

[0023]

[0024] In the formula, This represents the matching loss between the features stored in the tampered feature memory unit and the real sample image. This represents the matching loss between the features stored in the real feature memory unit and the real sample image. This represents the matching loss between the features stored in the tampered feature memory unit and the tampered sample image. This represents the matching loss between the features stored in the true feature memory unit and the tampered sample image. This represents a vector composed of the matching scores of the features of each region of the real sample image and the features stored in the tampered feature memory unit. Indicates length same A vector in which all values ​​are 0. This represents a vector composed of the matching scores between the features of each region of the real sample image and the features stored in the real feature memory unit. Indicates length same A vector in which all values ​​are 1, and BCE represents the binary cross-entropy loss; This represents the largest matching score in the vector composed of the matching scores of the features of each region of the tampered sample image and the features stored in the tampered feature memory unit. The value is 1. This represents the smallest matching score in the vector composed of matching scores between the features of each region of the tampered sample image and the features stored in the memory units of the true features. The value is 0.

[0025] Furthermore, the detection loss is:

[0026] L = L cls +λ1L m +λ2L trip

[0027] In the formula, L represents the detection loss. L cls λ1 and λ2 represent the image-level label-based classification loss of the classifier, where λ1 and λ2 are hyperparameters.

[0028] Furthermore, a noise view of the sample image is extracted to extract the noise-sensitive features from the noise view; wherein, the noise view is extracted using BayarConv convolution, and the RGB features are extracted from the sample image and the noise-sensitive features are extracted from the noise view using a ResNet-50 backbone network;

[0029] The specific method for obtaining the multiple binary mask images is as follows:

[0030] SAM is used to perform semantic segmentation on each sample image, resulting in a set of binary mask images;

[0031] By performing region cross-removal and small region deletion on the binary mask image set, multiple binary mask images covering different regions of the corresponding image and covering the sample image are obtained.

[0032] Furthermore, the calculation method for the regional features is as follows:

[0033] G m =G f ×(p+Maxpooling(M));

[0034] In the formula, G f The fusion feature represents the fusion feature of each sample image, where M represents any binary mask image corresponding to that sample image; G m This represents the region features corresponding to M in the sample image, Maxpooling(·) represents the max pooling operation, and p represents the hyperparameter.

[0035] The query and the read operations are respectively represented as follows:

[0036]

[0037] G aug =TM b ;

[0038] In the formula, G m M represents regional characteristics. b G represents the features stored in the memory unit, C represents the dimension of the output features of the region feature construction unit, σ(x) refers to the sigmoid function, T represents the matching result, and G represents the feature stored in the memory unit. aug This refers to the enhanced feature.

[0039] The present invention also provides an image tampering detection method, comprising:

[0040] The image to be tested is input into the image tampering detection model constructed by the weakly supervised image tampering detection model construction method described above, and the result of whether the image to be tested is a tampered image is obtained. If it is a tampered image, the tampered area is output.

[0041] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to execute a method for constructing an image tampering detection model based on weak supervision as described above.

[0042] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0043] (1) This invention extracts the RGB features and noise-sensitive features of each image through a region feature construction unit and fuses them to obtain fused features. Semantic segmentation is performed on each image to obtain multiple binary mask images covering different regions of the corresponding image and the entire image. Based on the product of the fused features and each binary mask image, the region features of different regions of the corresponding image are obtained. Through a dual memory unit framework, which includes a normal feature memory unit and a tampering feature memory unit, the matching results of the features stored in the unit and each region feature, as well as the enhanced features of each region feature, are output through the query and read operations of each unit. The two enhanced features output by the two units are added together and then concatenated with the corresponding region features to obtain the concatenated features. Detection and classification are performed based on the concatenated features. The loss calculation creatively introduces a constraint-based memory unit loss. The entire process only uses image-level supervision information to complete the identification and detection of tampered images. At the same time, it can also complete the extraction and division of specific tampered regions in the tampered image. That is, this invention can achieve high-precision image tampering detection without pixel-level label information. Therefore, it is a high-precision image tampering detection under weak supervision. The method of this invention can improve the deployability and detection accuracy of tampered images, and ensure the authenticity of the evidence images.

[0044] (2) After obtaining the score and enhanced features, since some interfering pixels may be introduced when calculating the region features based on the mask image, affecting the score of subsequent memory units, another problem that needs to be solved is that pixels in other tampered region proposals may have a significant interference effect on the score of the current tampered region proposal in two memory units. To solve this problem, this method further proposes to optimize the matching results between the features stored in each memory unit and each region feature, as well as the binary mask image corresponding to the region feature, according to the formula Calculate the similarity score vector S between the feature of the region and the feature stored in the memory cell. Each memory cell contains multiple feature blocks. To improve accuracy, take the average of the top K similarity scores between each region feature and the feature stored in each memory cell as the matching score between the region feature and the feature stored in the memory cell. Attached Figure Description

[0045] Figure 1 A framework diagram of a method for constructing an image tampering detection model based on weak supervision provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of SAM binary mask generation provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of memory unit training provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Example 1

[0050] A method for constructing an image tampering detection model based on weak supervision, such as Figure 1 As shown, it includes:

[0051] The sample images are input into the image tampering detection network, which includes: a region feature construction unit, used to extract RGB features and noise-sensitive features of each image and fuse them to obtain fused features; semantic segmentation of each image to obtain multiple binary mask images covering different regions of the corresponding image and the entire image; and region features of different regions of the corresponding image obtained based on the product of the fused features and each binary mask image; a dual memory unit framework, which includes a normal feature memory unit and a tampering feature memory unit, used to output the matching results of the features stored in each unit with each region feature and the enhanced features of each region feature through query and read operations of each unit, and to add the two enhanced features output by the two units and concatenate them with the corresponding region features to obtain the concatenated features; and a classifier used to perform tampering image recognition based on the concatenated feature set of each sample image.

[0052] The detection loss is calculated, and the parameters of the image tampering detection network are iteratively trained to complete the model construction.

[0053] The aforementioned detection loss includes a memory unit loss based on constraints and the aforementioned matching results. The constraints include: the features stored in the tampered feature memory unit do not match any region features of the real sample image, but match at least one region feature of the tampered sample image; the features stored in the normal feature memory unit match all region features of the real sample image, but do not match at least one region feature of the tampered sample image; the features of the real sample image are close to the region features of the real region in the tampered sample image, while the region features of the tampered region are far away.

[0054] With an increasing number of post-processing techniques aimed at masking forgery traces in tampered images, the task of detecting subtle signs of tampering in the RGB space is becoming increasingly complex. Current image tampering detection techniques typically rely on time-consuming manual annotation to establish authenticity standards, as evidenced by well-known datasets such as CASIA and IMD. Some methods even require pixel-level annotation for supervision and training, which demands significant human resources and increases the complexity of applying the model in real-world environments. To address these challenges, this invention introduces a weakly supervised image tampering detection method (WSISD). This method only needs to know the authenticity status of the image, i.e., whether the image is forged or genuine. The core idea of ​​the model is that forged regions in an image will exhibit features that are significantly different from normal regions. If this information can be captured through memory units, the tampered image region can be identified. This method, through region feature construction, the introduction of a dual-memory unit framework, and the constraint design of memory unit loss, reduces the dependence of deep learning-based blind forensics schemes on pixel-level labels.

[0055] In the specific scheme, for the feature extraction part, this method uses features from the noise domain as supplementary information to enhance the detection effect. The noise domain can better capture the texture information of the tampered area of ​​the image, thereby enhancing the tamper detection capability of the model.

[0056] For the input image X, as a preferred implementation, the region feature construction unit can use a ResNet-50 network to extract relevant RGB features, denoted as G. r For noisy streams, the region feature construction unit first uses BayarConv convolution to extract the noisy view of the input image and extract noise-sensitive features. Then, the ResNet-50 network is used to extract the noise-sensitive features in the noisy view, which can be expressed by the formula:

[0057] G n =Re sNet(BayarConv(X));

[0058] In order to combine multi-view information, this invention will use G r and G n Connect them to generate the fused feature Gf .

[0059] However, since weakly supervised methods lack pixel-level tampering labels, directly applying the fused features G of the entire image would be problematic. f Inputting the data into the memory unit for detection would cause a mixture of real and tampered features from the tampered image, thus failing to achieve the detection objective.

[0060] This method leverages the inherent uniqueness of tampered regions in images, which are often semantically informative regions—that is, tampered regions in an image can be considered independent entities. This forms the basis of the invention. Therefore, this method proposes semantic segmentation for each image to generate refined tampered region suggestions. As a preferred implementation, SAM (Semantic Aspect-Oriented Segmentation Model) can be used for semantic segmentation. SAM is a powerful semantic segmentation model, and in this method, it is used to segment the input image into multiple precise tampered region suggestions. The previously extracted overall image features (i.e., fused features G) are then combined. f The image is segmented into sets of regional features in different regions. Since some regions are real regions and some are tampered regions, the tampered features in the image can be separated from the real features, which makes it easier for the model to make subsequent distinctions.

[0061] Semantic segmentation can generate a set of binary mask images, denoted by M, such as... Figure 2 As shown, multiple mask images (containing background information) can be generated for each image to be tested. The resulting set of binary mask images is preprocessed, during which smaller regions and regions with high repetition are deleted. For each input image, multiple binary mask images covering different regions of the corresponding image are generated and cover the entire image.

[0062] By combining these multiple binary mask images with the aforementioned extracted fusion feature G... f By performing product calculations, a set of region features can be generated for each input image. This set contains the fused feature information of all possible tampered regions in the image (i.e., all proposed tampered regions). In other words, multiple binary mask images of each input image cover all semantic regions of that input image. These masks are multiplied element-wise with the aforementioned fused features to obtain a set of refined fused features, highlighting the vivid features of each region and focusing the image features within each region. As a preferred implementation method, to ensure sufficient feature information within each region, this method proposes formula G. m =G f The method shown in ×(p+Maxpooling(M)) obtains the feature information of each region, that is, it obtains the regional features of each region, where G mThe SAM-optimized feature set represents the set of region features composed of the regional features of each region in the input image. `p` represents a hyperparameter used to control the influence of non-semantic information features on the fused features, preventing excessive semantic information from affecting the tampering detection. Max pooling is used on the mask image `M` to preserve the edge information of the regions. The features at the edges of each tampered region will differ significantly from the true features, while minimizing information loss. In this process, the noise and RGB fusion features and the mask generated by semantic segmentation are considered. Figure 1 This feature filtering mechanism is used to highlight multiple specific regions in an image. It ensures the separation of tampered regions and real regions in a tampered image, i.e., tampered features and real features, thereby mitigating the potential confusion introduced by non-tampered regions in the aforementioned fused features.

[0063] The feature fusion method based on semantic segmentation can flexibly acquire key visual cues within tampered regions of an image and separate the tampered region from the real region. This allows image features to be accurately parsed into different regions, avoiding difficulties in subsequent classification due to feature confusion. This method not only reduces the need for pixel-level label information in detection methods but also significantly improves the overall efficiency and accuracy of tamper detection mechanisms.

[0064] The above method uses semantic segmentation to divide the input image into multiple regions and obtains the region features of each region by fusing noise features and RGB features. Based on this, in order to enhance the discriminative ability and realize image tampering detection and localization without pixel-level supervision, a dual-memory unit framework is introduced. This framework includes a tampering feature-based unit and a real feature memory unit. One unit is used to store general tampering features, and the other unit is used to store general real features. By having different memory units store the corresponding feature information and compare and match it with the region features of each region, this method can accurately distinguish the fine-grained region proposals obtained by semantic segmentation and identify tampered regions and real regions.

[0065] In this embodiment, to achieve high-precision tamper detection, the pixel-level tamper detection task is transformed into the identification and authentication of tampered areas and real areas.

[0066] Specifically, during training, two memory units respectively memorize the features of the tampered region and the features of the real region in the image, thereby distinguishing between the two. For example... Figure 3As shown, to facilitate the accurate memorization of different feature information by the memory units, the framework employs query and retrieval operations for training. This operation is similar to the attention mechanism in deep learning. The query operation involves activating the region feature by multiplying it with the feature stored in the memory unit, and using the activation result as the matching result. The retrieval operation involves multiplying the matching result with the feature stored in the memory unit to obtain the feature information enhanced by the memory unit. This allows the dual-memory unit framework to evaluate the similarity between the input feature and the information stored in the memory unit by querying the memory unit during testing.

[0067] As a preferred implementation, the above query and read operations are respectively represented as follows:

[0068]

[0069] G aug =TM b ;

[0070] In the formula, G m M represents regional characteristics. b Let C represent the features stored in the memory unit, C represent the dimension of the output features of the backbone network, σ(x) refers to the sigmoid function, which is a monotonically increasing activation function with a range of (0,1), T represent the matching result, and G represent the matching result. aug This refers to the enhanced feature.

[0071] It should be noted that the matching result between the feature stored in each memory cell and the feature of each region is a three-dimensional vector with the same length and width as the feature of that region and the same height as the number of feature blocks stored in that memory cell. As a preferred implementation, the memory cell loss is calculated as follows: first, the matching result between the feature stored in each memory cell and the feature of each region is converted into a matching score between the feature of that region and the feature stored in that memory cell; then, based on the matching score between the feature of each region and the feature stored in each memory cell and the above constraints, the memory cell loss is calculated.

[0072] As a further preferred implementation, the loss constraint based on the similarity between the features of the real sample image and the regional features of the real region in the tampered sample image, and the greater distance between the regional features of the tampered region and the real region, is as follows:

[0073] The ternary loss L is calculated based on the region features in the tampered sample image that have the highest matching score with the tampered feature memory unit, the region features in the tampered sample image that have the highest matching score with the normal feature memory unit, and the region features of any region in the real sample image. trip .

[0074] After obtaining the score and enhancing the features, since some interfering pixels may be introduced when calculating the region features based on the mask image, affecting the score of subsequent memory units, another problem that needs to be addressed is that pixels in other tampered region proposals may significantly interfere with the score of the current tampered region proposal in two memory units. To solve this problem, this method preferably proposes the following implementation of the above transformation:

[0075] Based on the matching results between the features stored in each memory unit and the features of each region, and the binary mask image corresponding to the features of that region, according to the formula... Calculate the similarity score vector S between the features of the region and the features stored in the memory unit, where T(i,j) represents the matching result vector corresponding to pixel (i,j) in the region feature in the corresponding matching result, and the number of elements is the same as the number of feature blocks stored in the memory unit; M(i,j) represents the pixel value corresponding to the pixel in the binary mask image corresponding to the region feature; N m This represents the total number of pixels in the corresponding image region within the region feature. Each memory unit contains multiple feature blocks. To improve accuracy, the average of the top K similarity scores between each region feature and the features stored in each memory unit is taken as the matching score between the region feature and the features stored in the memory unit.

[0076] During training, features from both real and manipulated images undergo processing using a dual-memory unit architecture to enhance discriminative power. Since weakly supervised tasks lack detailed pixel-level labels, the memory units require strong constraints. The loss function is designed as follows: when comparing region features from the real image with features stored in the real feature memory unit, the desired matching score is... The value will be uniformly set to 1 (1 is an example, other values ​​are also acceptable), because all regions in a real image contain real features, so they should be completely consistent with the information in the real memory unit. The subscript 'a' represents the score of the real image, and the superscript 'a' indicates that the score is the result of matching with the real memory unit. Similarly, when these images are input into the tampered feature memory unit, the resulting matching score... All values ​​should be displayed as 0, because their features should be completely inconsistent with the information in the feature memory unit that has been tampered with. Through the constraints of these two loss functions, it can be ensured that the two memory units learn the corresponding features during training.

[0077] Similarly, unlike features in real images, since the tampered region in a tampered image only appears in one location within the image, when matching this feature with the tampered feature memory unit, the maximum matching score must be 1. This means that at least one tampered region feature in the set of region features across multiple regions of the tampered image must match the tampered feature memory unit. Correspondingly, during the matching process with real feature memory units, the minimum matching score should be 0, meaning that at least one region in the image has been tampered with and does not match the real memory unit.

[0078] Therefore, memory cell loss may include:

[0079]

[0080]

[0081] In the formula, This represents the matching loss between the features stored in the tampered feature memory unit and the real sample image. This represents the matching loss between the features stored in the real feature memory unit and the real sample image. This represents the matching loss between the features stored in the tampered feature memory unit and the tampered sample image. This represents the matching loss between the features stored in the true feature memory unit and the tampered sample image. This represents a vector composed of the matching scores of the features of each region of the real sample image and the features stored in the tampered feature memory unit. Indicates length same A vector in which all values ​​are 0. This represents a vector composed of the matching scores between the features of each region of the real sample image and the features stored in the real feature memory unit. Indicates length same A vector in which all values ​​are 1, and BCE represents the binary cross-entropy loss; This represents the largest matching score in the vector composed of the matching scores of the features of each region of the tampered sample image and the features stored in the tampered feature memory unit. The value is 1. This represents the smallest matching score in the vector composed of matching scores between the features of each region of the tampered sample image and the features stored in the memory units of the true features. The value is 0.

[0082] This method employs weak supervision for image tampering detection. In this approach, the only available supervisory information for the network is the image-level label indicating whether the image has been tampered with, rather than pixel-level information about the tampered area. Therefore, to achieve image tampering detection and localization on the internet where the source cannot be traced, this method requires a finely defined loss function as a constraint. This loss function consists of two main parts: a memory unit loss function and an image-level supervised tampering classifier loss function.

[0083] The dual-memory unit framework utilizes dual memory units to perform weakly supervised tamper detection. To ensure effective differentiation between tampered and genuine features, this embodiment designs a loss function consisting of four memory unit loss functions, used to train two memory units respectively. Each memory unit loss function is determined by calculating the similarity between the input image feature set and the memory unit entries. The goal of this loss function is to improve the similarity score of regions matching the memory units and reduce the similarity score of non-matching regions, thereby enhancing the accuracy of memory unit retrieval.

[0084]

[0085] The first two terms on the right-hand side of the formula are designed to constrain the learning of tampered feature memory units on the tampered features of the image tampered region, while the last two terms are designed to facilitate the learning of the real feature memory units on the real region features of the image.

[0086] To ensure that both the tampered feature memory units and the true feature memory units can learn effective features during training and to enhance the distinguishability of the contents of the two memory units, this embodiment also introduces a ternary loss function. This loss function encourages the matched features to show significant differences between the tampered feature memory units and the true feature memory units, thereby better identifying tampered regions.

[0087]

[0088] In the formula, L trip Indicates a three-element loss. The region feature in the tampered sample image that has the highest matching score with the tampered feature memory unit can be regarded as representative of the tampered region features in the tampered image; The region feature in the tampered sample image that has the highest matching score with the normal feature memory unit can be considered representative of the true region features in the tampered image. The three features represent the regional features of any region in the real sample image. These features can be regarded as the representative of the real image features. The three features participate in the calculation of the ternary loss, so that the features of the real image are close to the real regional features in the tampered image, while being far away from the tampered regional features of the tampered image, thus making the two memory units present a large difference during the training process.

[0089] During testing, the test image is processed by a region feature construction unit to obtain a region feature set. This set is then input into a dual-memory unit framework for querying and retrieving features. The enhanced features of each region feature are combined with the original region feature and input into a trained classifier for score calculation. In the final evaluation, if any region in the image is identified as tampered, the entire image is marked as tampered; otherwise, the image is considered genuine. If the image is considered genuine, pixel-level tampered regions are not output. However, if the image is identified as tampered, the model uses the region that best matches the tampering memory unit as the pixel-level location of the tampering when outputting the pixel-level tampering location. This method effectively identifies and locates tampered regions in images, providing valuable information for subsequent analysis and investigation.

[0090] This method adds the enhanced features G obtained by enhancing the features of each region through two memory units. aug By connecting it with the features of that region, a unified feature representation is formed. This step not only ensures the integrity of the image feature information but also enhances the expressive power of information within the memory unit.

[0091] After feature concatenation, this method uses a linear classifier to perform image-level classification. The loss function of this classifier is designed to calculate a binary cross-entropy loss between the image classification score and the corresponding label. This loss function design is the specific implementation strategy of this method for training with image-level supervised information.

[0092] L cls =BCE(linear(G m ⊕G aug ),y)

[0093] In the formula, y represents the image-level label information of the training images, and linear represents a linear classifier.

[0094] To improve the learning performance of the network, this invention preferably employs a composite loss function design, combining memory unit loss, triplet loss, and CLS loss. The two losses of the memory units are primarily used to ensure that the memory units can effectively store and retain relevant image feature information during the learning process, thereby promoting the self-learning and optimization of the dual memory units. The classifier loss, on the other hand, mainly guides the network to utilize label information to achieve better classification results in image tampering classification tasks, thus realizing more accurate image tampering detection.

[0095] L = L cls +λ1L m +λ2L trip

[0096] In the formula, λ1 and λ2 are balancing hyperparameters. The method of using multiple loss functions in combination during the design of the total loss function can optimize the tamper detection performance of WSITD from multiple perspectives and improve the accuracy and robustness of the method for various types of image tampering.

[0097] In summary, this invention provides a highly robust and precise image tampering detection method that does not rely on pixel-level labels. Through ingenious feature fusion, memory unit framework, and loss function design, it can effectively identify and locate tampered regions in images. Compared with existing methods that can only identify whether an entire image is tampered with, this invention achieves higher precision in image tampering detection, providing a powerful tool and method for the field of image security. This method is expected to play a positive role in a wide range of applications, including digital forensics and image verification.

[0098] Example 2

[0099] An image tampering detection method includes:

[0100] The image to be tested is input into the image tampering detection model constructed by the weakly supervised image tampering detection model construction method described above, and the result of whether the image to be tested is a tampered image is obtained. If it is a tampered image, the tampered area is output.

[0101] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0102] Example 3

[0103] A computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute a method for constructing an image tampering detection model based on weak supervision as described in Embodiment 1 above.

[0104] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0105] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing an image tampering detection model based on weak supervision, characterized in that, include: The sample images are input into an image tampering detection network. This network includes: a region feature construction unit, used to extract RGB features and noise-sensitive features from each image and fuse them to obtain fused features; semantic segmentation of each image to obtain multiple binary mask images covering different regions of the corresponding image; and region features of different regions of the corresponding image obtained based on the product of the fused features and each binary mask image; a dual-memory unit framework, including a normal feature memory unit and a tampering feature memory unit, used to output the matching results between the features stored in each unit and each region feature, as well as the enhanced features of each region feature, through query and read operations of each unit; and to add the two enhanced features output by the two units and concatenate them with the corresponding region feature to obtain the concatenated features; and a classifier used to perform tampering image recognition based on the concatenated feature set of each sample image. The detection loss is calculated, and the parameters of the image tampering detection network are iteratively trained to complete the model construction. The detection loss includes a memory unit loss based on constraints and the matching results. The constraints include: the features stored in the tampered feature memory unit do not match any region features of the real sample image, but match at least one region feature of the tampered sample image; the features stored in the normal feature memory unit match all region features of the real sample image, but do not match at least one region feature of the tampered sample image; the features of the real sample image are close to the region features of the real region in the tampered sample image, while the region features of the tampered region are far away. The query and read operations are respectively represented as follows: ； ； In the formula, Indicates regional characteristics, C represents the features stored in the memory unit, and C represents the feature dimension extracted by the region feature construction unit. Refers to the sigmoid function. This indicates the matching result. This refers to the enhanced feature; The matching result between the features stored in each memory cell and the features of each region is a three-dimensional vector with the length and width dimensions matching the features of that region and the height dimension matching the number of feature blocks stored in that memory cell. The calculation of the memory unit loss is implemented as follows: The matching result between the features stored in each memory cell and the features of each region is converted into a matching score between the features of that region and the features stored in that memory cell; The memory cell loss is calculated based on the matching score between the features of each region and the features stored in each memory cell, as well as the constraints.

2. The image tampering detection model construction method according to claim 1, characterized in that, The loss condition, which is based on the constraint that the features of the real sample image are close to the regional features of the real region in the tampered sample image, while the regional features of the tampered region are far away, is: The ternary loss is calculated based on the region features in the tampered sample image that have the highest matching score with the tampered feature memory unit, the region features in the tampered sample image that have the highest matching score with the normal feature memory unit, and the region features of any region in the real sample image. .

3. The image tampering detection model construction method according to claim 1, characterized in that, The conversion is implemented as follows: Based on the matching results between the features stored in each memory unit and the features of each region, and the binary mask image corresponding to the features of that region, according to the formula... Calculate the similarity score vector between the features of this region and the features stored in this memory unit. In the formula, Represents the pixels in the feature of this region. The number of elements in the corresponding matching result vector is the same as the number of feature blocks stored in that memory unit; This represents the pixel value corresponding to the feature in the binary mask of the region. This represents the total number of pixels in the corresponding image region within the region feature; the similarity score vector between each region feature and the features stored in each memory unit. The average of the top K similarities is taken as the matching score between the features of the region and the features stored in the memory unit.

4. The image tampering detection model construction method according to claim 2, characterized in that, A matching score of 0 indicates that the features stored in the memory unit do not match the features of the region at all, while a matching score of 1 indicates that the features stored in the memory unit match the features of the region at all. The constraints include: the matching scores between the features stored in the tampered feature memory unit and the features of each region of the real sample image should all be 0, and the maximum matching score among the features of each region of the tampered sample image should be 1; the matching scores between the features stored in the normal feature memory unit and the features of each region of the real sample image should all be 1, and the minimum matching score among the features of each region of the tampered sample image should be 0; the features of the real sample image are close to the regional features of the real region in the tampered sample image, while the regional features of the tampered region are far away, with 0 indicating a complete mismatch and 1 indicating a complete match; The memory cell loss includes: In the formula, This represents the matching loss between the features stored in the tampered feature memory unit and the real sample image. This represents the matching loss between the features stored in the real feature memory unit and the real sample image. This represents the matching loss between the features stored in the tampered feature memory unit and the tampered sample image. This represents the matching loss between the features stored in the true feature memory unit and the tampered sample image. This represents a vector composed of the matching scores of the features of each region of the real sample image and the features stored in the tampered feature memory unit. Indicates length same A vector in which all values ​​are 0. This represents a vector composed of the matching scores between the features of each region of the real sample image and the features stored in the real feature memory unit. Indicates length same A vector in which all values ​​are 1, and BCE represents the binary cross-entropy loss; This represents the largest matching score in the vector composed of the matching scores of the features of each region of the tampered sample image and the features stored in the tampered feature memory unit. The value is 1. This represents the smallest matching score in the vector composed of matching scores between the features of each region of the tampered sample image and the features stored in the memory units of the true features. The value is 0.

5. The image tampering detection model construction method according to claim 4, characterized in that, The detection loss is: In the formula, Indicates detection loss, , This represents the image-level label-based classification loss of the classifier. , All of these are hyperparameters.

6. The method for constructing an image tampering detection model according to any one of claims 1 to 5, characterized in that, A noisy view of the sample image is extracted to extract the noise-sensitive features from the noisy view; wherein, the noisy view is extracted using BayarConv convolution, and the RGB features are extracted from the sample image and the noise-sensitive features are extracted from the noisy view using a ResNet-50 backbone network; The specific method for obtaining the multiple binary mask images is as follows: SAM is used to perform semantic segmentation on each sample image, resulting in a set of binary mask images; By performing region cross-removal and small region deletion on the binary mask image set, multiple binary mask images covering different regions of the corresponding image and covering the sample image are obtained.

7. The image tampering detection model construction method according to any one of claims 1 to 5, characterized in that, The calculation method for the regional features is as follows: ； In the formula, The fusion features represent the characteristics of each sample image. This represents any binary mask image corresponding to the sample image; This indicates that in the sample image The regional characteristics of the corresponding area. This indicates a max pooling operation. This represents hyperparameters.

8. A method for detecting image tampering, characterized in that, include: The image to be tested is input into the image tampering detection model constructed by the image tampering detection model construction method based on weak supervision as described in any one of claims 1 to 7, and the result of whether the image to be tested is a tampered image is obtained. If it is a tampered image, the tampered area is output.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute a method for constructing an image tampering detection model based on weak supervision as described in any one of claims 1 to 7.

Images