Image tampering detection model construction method based on weak supervision

Abstract

The invention belongs to the field of computer multimedia security, and particularly relates to a weak supervision-based image tampering detection model construction method, which comprises the following steps of: extracting RGB (Red, Green, Blue) features and noise sensitive features of each image through a region feature construction unit, fusing the RGB features and the noise sensitive features, carrying out semantic segmentation on each image to obtain a plurality of binary mask patterns of the corresponding image, and constructing a weak supervision-based image tampering detection model; based on the product of the fusion feature and each binary mask image, obtaining region features of different regions of the corresponding image; a normal feature memory unit and a tampering feature memory unit are introduced, a matching result of features stored in each unit and each regional feature and an enhanced feature of each regional feature are output through query and reading operations of each unit, and the two enhanced features output by the two units are added and then spliced with the corresponding regional features. Based on splicing feature detection classification, constraint-based memory unit loss is creatively introduced in loss calculation, and high-fineness image tampering detection can be realized under the condition of no pixel-level label information.

Description

Technical Field

[0001] The present invention belongs to the field of computer multimedia security, and more specifically, relates to a method for constructing an image tampering detection model based on weak supervision. Background Art

[0002] With the rapid development of science and technology, digital media technology has made great progress, and digital products have entered the homes of ordinary people. Ordinary people can also record the moments around them with their mobile phones. Massive media data is generated in life and transmitted to the Internet. With the development and maturity of digital media technology, a large number of image modification tools have gradually appeared in the public's field of vision, such as open source editing software such as Meitu XiuXiu and PhotoShop, and even mobile phone cameras have built-in beauty filter functions. These software are very simple and easy to understand. People can modify digital images at any time according to their own ideas, and it is difficult to distinguish them with the naked eye.

[0003] However, some people with ulterior motives will maliciously tamper with images and post them on the Internet, which will not only affect people's judgment of the authenticity of information, but may even cause adverse effects on society. Over the years, malicious incidents caused by image tampering have occurred frequently, causing huge hidden dangers to multimedia information security, mainly in terms of pirated information, rumor release, and false information. Therefore, image forensics research has become a very important part of multimedia security. Now the field of online public opinion analysis attaches great importance to the authenticity of images, and there are also professionals in the news media and judicial departments who are specifically responsible for identifying the authenticity of images. The authenticity of images in the process of online information dissemination affects people's daily lives and is related to social stability. Ensuring that the image information obtained is authentic has become a very important need in today's life.

[0004] Since image tampering can cause great security risks, it is particularly important to be able to collect evidence from image data and detect the authenticity of images. Existing image forensics technologies are mainly divided into active forensics and passive forensics according to different detection methods. The active image forensics method refers to embedding relevant verification information such as digital signatures and digital watermarks in the image, and then extracting and verifying the information. However, the authenticity requirements of the embedded information and the high cost of embedding have led to the inability of active forensics technology to be widely used.

[0005] The blind forensics method of images is passive forensics, which directly judges the source and integrity through images, making the method more widely used. With the development of information technology, the efficiency of deep learning technology in image processing has gradually emerged. In recent years, the blind forensics technology of image tampering has also mainly used methods based on deep neural networks to detect whether images have been tampered with. The existing methods mainly perform tampering detection at the image level. When detecting based on image-level labels, the detection can only detect whether the image has been tampered with, and cannot detect specific tampered areas. If you want to detect specific tampered areas, you need to rely on pixel-level labels, but this requires a lot of manpower investment and increases the complexity of the model application in actual environments. Summary of the invention

[0006] In view of the defects and improvement needs of the prior art, the present invention provides a method for constructing an image tampering detection model based on weak supervision, which aims to achieve high-precision tampering image detection based on image-level labels.

[0007] To achieve the above objective, according to one aspect of the present invention, a method for constructing an image tampering detection model based on weak supervision is provided, comprising:

[0008] The sample image is input into the image tampering detection network, which includes: a regional feature construction unit, which is used to extract the RGB features and noise-sensitive features of each image and fuse the two to obtain fused features, perform semantic segmentation on each image, obtain multiple binary mask images corresponding to different regions of the image and covering the entire image, and obtain regional features of different regions of the corresponding image based on the product of the fused features and each binary mask image; a dual memory unit framework, which includes a normal feature memory unit and a tampering feature memory unit, which is used to output the matching results of the features stored in the unit and each regional feature and the enhanced features of each regional feature through query and read operations of each unit, and add the two enhanced features output by the two units and splice them with the corresponding regional features to obtain spliced ​​features; a classifier, which is used to identify tampered images based on the spliced ​​feature set of each sample image;

[0009] Calculate the detection loss to iteratively train the parameters of the image tampering detection network and complete the model construction;

[0010] Among them, the detection loss includes memory unit loss based on constraints and the matching results, and the constraints include: the features stored in the tampered feature memory unit do not match the regional features of the real sample image, and match at least one regional feature of the tampered sample image; the features stored in the normal feature memory unit match the regional features of the real sample image, and do not match at least one regional feature of the tampered sample image; the features of the real sample image are close to the regional features of the real area in the tampered sample image, and the regional features of the tampered area are farther.

[0011] Furthermore, the matching result of the feature stored in each memory unit and each regional feature is a three-dimensional vector with the same length and width as the regional feature and the same height as the number of feature blocks stored in the memory unit;

[0012] The calculation method of the memory unit loss is as follows:

[0013] Converting the matching result between the feature stored in each memory unit and each regional feature into a matching score between the regional feature and the feature stored in the memory unit;

[0014] The memory unit loss is calculated based on the matching score between each region feature and the feature stored in each memory unit and the constraint.

[0015] Furthermore, the loss based on the constraint that the features of the real sample image are close to the regional features of the real area in the tampered sample image, and the regional features of the tampered area are far away is:

[0016] The ternary loss L is calculated based on the regional features with the highest matching score with the tampered feature memory unit in the tampered sample image, the regional features with the highest matching score with the normal feature memory unit in the tampered sample image, and the regional features of any region in the real sample image. trip .

[0017] Furthermore, the conversion is implemented as follows:

[0018] Based on the matching results of the features stored in each memory unit and the features of each region and the binary mask map corresponding to the regional features, according to the formula Calculate the similarity score vector S between the regional feature and the feature stored in the memory unit, where T(i, j) represents the matching result vector corresponding to the pixel point (i, j) in the regional feature in the corresponding matching result, and the number of elements is the same as the number of feature blocks stored in the memory unit; M(i, j) represents the pixel value corresponding to the pixel point in the binary mask image corresponding to the regional feature; N m Represents the total number of pixels in the image area corresponding to the regional feature; takes the average of the top K similarities from the similarity score vector S between each regional feature and the feature stored in each memory unit as the matching score between the regional feature and the feature stored in the memory unit.

[0019] Furthermore, a matching score value of 0 indicates that the features stored in the memory unit do not match the regional features at all, and a matching score value of 1 indicates that the features stored in the memory unit match the regional features completely;

[0020] The constraints include: the matching scores of the features stored in the tampered feature memory unit and the regional features of the real sample image should all be 0, and the maximum matching score of the features of the regional features of the tampered sample image should be 1; the matching scores of the features stored in the normal feature memory unit and the regional features of the real sample image should all be 1, and the minimum matching score of the features of the regional features of the tampered sample image should be 0; the features of the real sample image are close to the regional features of the real area in the tampered sample image, and the regional features of the tampered area are farther, 0 indicates a complete mismatch, and 1 indicates a complete match;

[0021] The memory unit loss includes:

[0022]

[0023]

[0024] In the formula, represents the matching loss between the features stored in the tampered feature memory unit and the real sample image, Represents the matching loss between the features stored in the real feature memory unit and the real sample image, Represents the matching loss between the features stored in the tampered feature memory unit and the tampered sample image, Represents the matching loss between the features stored in the real feature memory unit and the tampered sample image, Represents a vector consisting of the matching scores of the features of each region of the real sample image and the features stored in the tampered feature memory unit. Indicates length The value of is 0. Represents a vector consisting of the matching scores between the features of each region of the real sample image and the features stored in the real feature memory unit. Indicates length The values ​​of are all 1, BCE represents the binary cross entropy loss; It represents the maximum matching score in the vector composed of the matching scores of each region feature of the tampered sample image and the features stored in the tampered feature memory unit. The value is 1. It represents the smallest matching score in the vector composed of the matching scores of the features of each region of the tampered sample image and the features stored in the real feature memory unit. The value is 0.

[0025] Furthermore, the detection loss is:

[0026] L=L cls +λ 1 L m +λ 2 L trip

[0027] Where L represents the detection loss, L cls represents the classification loss of the classifier based on image-level labels, λ 1 , 2 are all hyperparameters.

[0028] Further, extracting a noise view of the sample image to extract the noise-sensitive feature from the noise view; wherein the noise view is extracted using a BayarConv convolution, and a ResNet-50 backbone network is used to extract the RGB feature from the sample image and the noise-sensitive feature from the noise view;

[0029] The method for obtaining the multiple binary mask images is specifically as follows:

[0030] Using SAM, semantic segmentation is performed on each sample image to obtain a set of binary mask images;

[0031] The binary mask map set is processed by region cross removal and small region deletion to obtain multiple binary mask maps corresponding to different regions of the image and covering the sample image.

[0032] Furthermore, the regional features are calculated as follows:

[0033] G m =G f ×(p+Maxpooling(M));

[0034] In the formula, G f represents the fusion feature of each sample image, M represents any binary mask image corresponding to the sample image; G m represents the regional features of the region corresponding to M in the sample image, Maxpooling(·) represents the maximum pooling operation, and p represents the hyperparameter;

[0035] The query and the read operations are respectively expressed as:

[0036]

[0037] G aug =TM b ;

[0038] In the formula, G m Represents regional characteristics, M b represents the features stored in the memory unit, C represents the dimension of the output features of the regional feature construction unit, σ(x) refers to the sigmoid function, T represents the matching result, G aug Represents the enhanced feature.

[0039] The present invention also provides an image tampering detection method, comprising:

[0040] The image to be tested is input into the image tampering detection model constructed by the image tampering detection model construction method based on weak supervision as described above, and the result of whether the image to be tested is a tampered image is obtained. If it is a tampered image, the tampered area continues to be output.

[0041] The present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the above-mentioned method for constructing an image tampering detection model based on weak supervision.

[0042] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0043] (1) The present invention extracts the RGB features and noise-sensitive features of each image through a regional feature construction unit and fuses the two to obtain a fused feature, performs semantic segmentation on each image, obtains multiple binary mask images corresponding to different regions of the image and covering the entire image, and obtains regional features of different regions of the corresponding image based on the product of the fused feature and each binary mask image; through a dual memory unit framework, it includes a normal feature memory unit and a tampered feature memory unit, outputs the matching result of the feature stored in the unit and each regional feature and the enhanced feature of each regional feature through query and read operations of each unit, and adds the two enhanced features output by the two units and splices them with the corresponding regional features to obtain a spliced ​​feature, performs detection and classification based on the spliced ​​feature, and creatively introduces a constraint-based memory unit loss in the loss calculation. The entire process only uses image-level supervision information to complete the recognition and detection of the tampered image, and can also complete the extraction and division of the specific tampered area in the tampered image. That is, the present invention can achieve high-precision image tampering detection without pixel-level label information, and is therefore a high-precision image tampering detection under weak supervision. The method of the present invention can improve the deployability and detection accuracy of tampered image detection and ensure the authenticity of forensic images.

[0044] (2) After obtaining the scores and enhanced features, since some interfering pixels of the regional features may be introduced when calculating the regional features based on the mask map, affecting the scores of subsequent memory units, another problem that needs to be solved is that the pixels in other tampered region proposals may have a greater interference with the scores of the current tampered region proposal in the two memory units. In order to solve this problem, the present method further preferably proposes a matching result based on the features stored in each memory unit and each regional feature and the binary mask map corresponding to the regional feature, according to the formula Calculate the similarity score vector S between the regional feature and the feature stored in the memory unit. Each memory unit contains multiple feature blocks. In order to improve the accuracy, take the average of the top K similarities from the similarity score vector S between each regional feature and the feature stored in each memory unit as the matching score between the regional feature and the feature stored in the memory unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A framework diagram of a method for constructing an image tampering detection model based on weak supervision provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of SAM binary mask generation provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of memory unit training provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Embodiment 1

[0050] A method for constructing an image tampering detection model based on weak supervision, such as Figure 1 As shown, including:

[0051] The sample image is input into the image tampering detection network, which includes: a regional feature construction unit, which is used to extract the RGB features and noise-sensitive features of each image and fuse the two to obtain fused features, perform semantic segmentation on each image, obtain multiple binary mask images corresponding to different regions of the image and covering the entire image, and obtain regional features of different regions of the corresponding image based on the product of the above fused features and each binary mask image; a dual memory unit framework, which includes a normal feature memory unit and a tampering feature memory unit, which is used to output the matching results of the features stored in the unit and each regional feature and the enhanced features of each regional feature through query and read operations of each unit, and add the two enhanced features output by the two units and splice them with the corresponding regional features to obtain spliced ​​features; a classifier, which is used to identify tampered images based on the spliced ​​feature set of each sample image;

[0052] Calculate the detection loss to iteratively train the parameters of the image tampering detection network and complete the model construction;

[0053] Among them, the above-mentioned detection loss includes memory unit loss based on constraints and the above-mentioned matching results, and the constraints include: the features stored in the tampered feature memory unit do not match the regional features of the real sample image, and match at least one regional feature of the tampered sample image; the features stored in the normal feature memory unit match the regional features of the real sample image, and do not match at least one regional feature of the tampered sample image; the features of the real sample image are close to the regional features of the real area in the tampered sample image, and the regional features of the tampered area are farther.

[0054] With more and more post-processing techniques aimed at concealing traces of forgery in tampered images, the task of detecting weak signs of tampering in RGB space has become increasingly complex. Today's image tampering detection techniques often rely on time-consuming manual annotations to establish authenticity standards, which are reflected in well-known datasets such as CASIA and IMD. Some methods even require pixel-level annotation for supervision and training, which requires a lot of manpower and increases the complexity of model application in practical environments. To address these challenges, the present invention introduces a weakly supervised image tampering detection method (abbreviated as: WSISD). This method only needs to understand the authenticity status of the image, that is, whether the image is forged or real. The core idea of ​​the model is that the forged area in the image will show features that are significantly different from the normal area. If this information can be captured through the memory unit, the tampered image area can be identified. This method can reduce the dependence of the blind forensics scheme based on deep learning on pixel-level labels through regional feature construction, the introduction of a dual memory unit framework, and the constraint design of memory unit loss.

[0055] In the specific scheme, for the feature extraction part, this method uses the features of the noise domain as supplementary information to enhance the detection effect. The noise domain can better capture the texture information of the image tampering area, thereby enhancing the tampering detection ability of the model.

[0056] For the input image X, as a preferred implementation, the regional feature construction unit can use the ResNet-50 network to extract relevant RGB features, represented as G r For the noise stream, the regional feature construction unit can first use BayarConv convolution to extract the noise view of the input image and extract the noise-sensitive features, and then use the ResNet-50 network to extract the noise-sensitive features in the noise view, which can be expressed as:

[0057] G n =ResNet(BayarConv(X));

[0058] In order to combine multi-view information, the present invention converts G r and G n Connect them to generate fusion features Gf .

[0059] However, since the weak supervision method does not have pixel-level tampering labels, if the fusion feature G of the entire image is directly f Inputting into the memory unit for detection will result in a mixture of real and tampered features from the tampered image, making the detection purpose unattainable.

[0060] This method uses the inherent uniqueness that the tampered region in the image is often a region with semantic information, that is, the tampered region in the image can be regarded as an independent entity, which forms the basis of the present invention. Therefore, the present method proposes to perform semantic segmentation on each image and generate fine tampered region suggestions. As a preferred implementation, SAM can be used for semantic segmentation. SAM is a powerful semantic segmentation model. In this method, it is used to segment the input image into multiple precise tampered region suggestions, and the above-mentioned extracted overall image features (i.e., fusion features G f ) is divided into regional feature sets in different areas. Since some of the different areas are real areas and some are tampered areas, the tampered features in the image can be separated from the real features, which is convenient for the model to perform subsequent identification.

[0061] Semantic segmentation can generate a set of binary mask maps, which are represented by M, such as Figure 2 As shown, each image to be tested can generate multiple mask images of different areas (which include background information), and the obtained set of binary mask images are preprocessed. During the preprocessing process, smaller areas and areas with high repetition are deleted. For each input image, multiple binary mask images corresponding to different areas of the image and covering the entire image are implemented.

[0062] By combining these multiple binary mask images with the fusion features G extracted above f By performing product calculation, a set of regional features can be generated for each input image. The set contains the fused feature information of all possible tampered regions in the image (i.e., all tampered region suggestions). That is, multiple binary mask images of each input image cover all semantic regions of the input image. These masks and the aforementioned fused features are multiplied element by element to obtain a set of refined fused features, highlighting the vivid features of each region and focusing the features of the image in each region. As a preferred implementation, in order to ensure that the feature information in each region is sufficient, this method proposes formula G m =G f ×(p+Maxpooling(M)) to obtain the feature information in each region, that is, to obtain the regional features of each region, where G mrepresents the feature set optimized by SAM, that is, the regional feature set composed of the regional features of each region in the input image. p represents a hyperparameter, which is used to control the influence of non-semantic information features on the fusion features to prevent excessive semantic information from affecting the judgment of tampering. The reason for using maximum pooling for the mask map M is to retain the edge information of the region. The features at the edge of each tampered region will be greatly different from the real features, and at the same time, the information loss can be minimized. In this process, the fusion features of noise and RGB and the mask generated by semantic segmentation are Figure 1 It is used to emphasize multiple specific areas in the image. The feature screening mechanism ensures the separation of tampered areas and real areas in the tampered image, that is, the tampered features and real features, thereby alleviating the potential confusion introduced by the non-tampered areas in the above-mentioned fusion features.

[0063] The feature fusion method based on semantic segmentation can flexibly obtain key visual clues in the tampered area of ​​the image and separate the tampered area from the real area, so that the image features can be accurately parsed into different areas, avoiding the difficulty of subsequent classification due to feature confusion. This method not only reduces the detection method's demand for pixel-level label information, but also significantly improves the overall effectiveness and accuracy of the tampering detection mechanism.

[0064] The above uses semantic segmentation to divide the input image into multiple regions, and obtains the regional features of each region that integrate noise features and RGB features. On this basis, in order to enhance the distinction ability without pixel-level supervision and realize image tampering detection and positioning, a dual memory unit framework is introduced. The framework includes a tampering feature based unit and a real feature memory unit, one for storing common tampering features, and the other for storing common real features. By allowing different memory units to memorize the corresponding feature information and compare and match it with the regional features of each region, the method can accurately distinguish the fine region suggestions obtained by semantic segmentation and identify the tampered region and the real region.

[0065] In order to achieve high-precision tampering detection, the method of this embodiment converts the tampering detection task at the pixel level into recognition and identification of the tampered area and the real area.

[0066] Specifically, during training, the two memory units memorize the features of the tampered area and the real area of ​​the image respectively, thereby distinguishing the two. Figure 3As shown in the figure, in order to promote the memory unit to accurately memorize different feature information, the framework uses the query and read operations of the memory unit for training. This operation is similar to the attention mechanism in deep learning. The query operation refers to activating the result of multiplying the regional feature with the feature stored in the memory unit, and using the activation result as the matching result. The read operation refers to multiplying the matching result with the feature stored in the memory unit to obtain the feature information enhanced by the memory unit. This allows the dual memory unit framework to evaluate the similarity between the input feature and the information stored in the memory unit by querying the memory unit during testing.

[0067] As a preferred implementation, the above query and read operations are respectively expressed as:

[0068]

[0069] G aug =TM b ;

[0070] In the formula, G m Represents regional characteristics, M b represents the features stored in the memory unit, C represents the dimension of the output features of the backbone network, σ(x) refers to the sigmoid function, which is a monotonically increasing activation function with a value range of (0,1), T represents the matching result, and G aug Represents the enhanced feature.

[0071] It should be noted that the matching result between the feature stored in each memory unit and each regional feature is a three-dimensional vector whose length and width dimensions are the same as the regional feature, and whose height dimension is the same as the number of feature blocks stored in the memory unit; as a preferred implementation method, the calculation method of memory unit loss is: first, the matching result between the feature stored in each memory unit and each regional feature is converted into a matching score between the regional feature and the feature stored in the memory unit; then, based on the matching score between each regional feature and the feature stored in each memory unit and the above constraints, the memory unit loss is calculated.

[0072] As a further preferred implementation, the loss based on the constraint that the feature of the real sample image is close to the regional feature of the real area in the tampered sample image, and the regional feature of the tampered area is far is:

[0073] The ternary loss L is calculated based on the regional features with the highest matching score with the tampered feature memory unit in the tampered sample image, the regional features with the highest matching score with the normal feature memory unit in the tampered sample image, and the regional features of any region in the real sample image. trip .

[0074] After obtaining the score and enhancing the features, since some interfering pixels may be introduced when calculating the regional features based on the mask image, affecting the scores of subsequent memory units, another problem that needs to be solved is that the pixels in other tampered region proposals may have a greater interference with the scores of the current tampered region proposal in the two memory units. In order to solve this problem, this method preferably proposes the implementation of the above conversion as follows:

[0075] Based on the matching results of the features stored in each memory unit and the features of each region and the binary mask map corresponding to the regional features, according to the formula Calculate the similarity score vector S between the regional feature and the feature stored in the memory unit, where T(i, j) represents the matching result vector corresponding to the pixel point (i, j) in the regional feature in the corresponding matching result, and the number of elements is the same as the number of feature blocks stored in the memory unit; M(i, j) represents the pixel value corresponding to the pixel point in the binary mask image corresponding to the regional feature; N m Represents the total number of pixels in the image area corresponding to the regional feature; each memory unit contains multiple feature blocks. In order to improve the accuracy, the average of the top K similarities is taken from the similarity score vector S between each regional feature and the feature stored in each memory unit as the matching score between the regional feature and the feature stored in the memory unit.

[0076] During the training process, the features of both the real and tampered images need to be processed by the dual memory unit architecture to enhance the ability to distinguish. Since the weakly supervised task does not have detailed pixel-level labels, it is necessary to give the memory unit a stronger constraint. The loss function is designed as follows: When the regional features of the real image are compared with the features stored in the real feature memory unit, the matching score expected is will be unified to 1 (the value of 1 is exemplary, other values ​​are also possible), because all areas in the real image contain real features, so it should be completely consistent with the information in the real memory unit, where Subscript a represents the score of the real image, and superscript a represents that the score is the result of matching with the real memory unit. Similarly, when these images are input into the tampered feature memory unit, the matching score obtained is All should be displayed as 0, because their features should be completely inconsistent with the information in the tampered feature memory unit. Through the constraints of these two loss functions, it can be ensured that the two memory units learn the corresponding features during training.

[0077] Similarly, unlike the features of the real image, since the tampered area in the tampered image only appears in one position in the image, when the feature is matched with the tampered feature memory unit, the maximum value of the matching score is required to be 1, that is, to ensure that at least one tampered regional feature in the regional feature set of multiple regions of the tampered image matches the tampered feature memory unit. Accordingly, in the process of matching with the real feature memory unit, the minimum value of the matching score obtained should be 0, that is, at least one region in the image is tampered and does not match the real memory unit.

[0078] Therefore, memory cell losses may include:

[0079]

[0080]

[0081] In the formula, represents the matching loss between the features stored in the tampered feature memory unit and the real sample image, Represents the matching loss between the features stored in the real feature memory unit and the real sample image, Represents the matching loss between the features stored in the tampered feature memory unit and the tampered sample image, Represents the matching loss between the features stored in the real feature memory unit and the tampered sample image, Represents a vector consisting of the matching scores of the features of each region of the real sample image and the features stored in the tampered feature memory unit. Indicates length The value of is 0. Represents a vector consisting of the matching scores between the features of each region of the real sample image and the features stored in the real feature memory unit. Indicates length The values ​​of are all 1, BCE represents the binary cross entropy loss; It represents the maximum matching score in the vector composed of the matching scores of each region feature of the tampered sample image and the features stored in the tampered feature memory unit. The value is 1. It represents the smallest matching score in the vector composed of the matching scores of the features of each region of the tampered sample image and the features stored in the real feature memory unit. The value is 0.

[0082] This method is a method for performing image tampering detection in a weakly supervised form. In this scheme, the only supervisory information that the network can obtain is the image-level label of whether the image has been tampered with, rather than the tampered area information specific to the pixel level. Therefore, in order to achieve the task of detecting and locating image tampering that cannot be traced back to the source on the Internet, this method needs to set a sophisticated loss function as a constraint. The loss function is mainly divided into two parts, one is the memory unit loss function, and the other is the image-level supervised tampering classifier loss function.

[0083] The dual memory unit framework uses dual memory units to complete the weakly supervised tampering detection task. In order to ensure that the tampered features and the real features can be effectively distinguished, this embodiment designs a loss function composed of four memory unit loss functions, which are used to train two memory units respectively. Each memory unit loss function is determined by calculating the similarity between the input image feature set and the memory unit entry. The goal of this loss function is to increase the similarity score of the matching area with the memory unit and reduce the similarity score of the non-matching area to enhance the accuracy of memory unit retrieval.

[0084]

[0085] The first two terms on the right side of the formula are intended to constrain the learning of tampered features of the tampered region of the image by the tampered feature memory unit, while the last two terms are intended to promote the learning of real region features of the image in the real feature memory unit.

[0086] In order to ensure that the tampered feature memory unit and the real feature memory unit can learn effective features during the training process and enhance the distinction between the contents of the two memory units, this embodiment also introduces a ternary loss function. The loss function can encourage the matching features to show obvious differences between the tampered feature memory unit and the real feature memory unit, thereby better identifying the tampered area.

[0087]

[0088] In the formula, L trip represents the ternary loss, Indicates the regional feature in the tampered sample image that has the highest matching score with the tampered feature memory unit, and the feature can be regarded as a representative of the tampered regional feature in the tampered image; It indicates the regional feature in the tampered sample image that has the highest matching score with the normal feature memory unit. This feature can be regarded as the representative of the real regional feature in the tampered image. The regional features of any area in the real sample image can be regarded as the representative of the real image features. These three features participate in the calculation of the ternary loss, making the features of the real image close to the real regional features in the tampered image, but far from the tampered regional features of the tampered image, so that the two memory units show a large difference during the training process.

[0089] During the test, the test image is passed through the regional feature construction unit to obtain the regional feature set, and the regional feature set is input into the dual memory unit framework for query reading operation. The enhanced features of each regional feature are combined with the regional features and input into the trained classifier for score calculation. In the final assessment, if the features of any region in the image are identified as tampered, the entire image will be marked as tampered, otherwise the image is judged to be authentic. If the image is considered to be authentic, there is no need to output the pixel-level tampered area. However, if the image is identified as a tampered image, when outputting the pixel-level tampered position, the model will use the area that best matches the tampered memory unit as the pixel-level positioning result of the image tampering. This method can effectively identify and locate the tampered area in the image, providing valuable information for subsequent analysis and investigation.

[0090] This method adds the enhanced features of each region after being enhanced by two memory units to obtain the feature G aug Connect with the regional features to form a unified feature representation. This step not only ensures the integrity of the image feature information, but also improves the expressiveness of the information in the memory unit.

[0091] After feature connection, this method uses a linear classifier to complete the image-level classification task. The loss function of this classifier is designed to calculate the binary cross entropy loss between the image classification score of the linear classifier and the corresponding label. The design of this loss function is the specific implementation strategy of this method for training through image-level supervision information.

[0092] L cls =BCE(linear(G m ⊕G aug ),y)

[0093] In the formula, y represents the image-level label information of the training image, and linear represents the linear classifier.

[0094] In order to make the network learning effect better, the present invention preferably adopts a composite loss function design scheme, which combines memory unit loss, triple loss and CLS loss. Among them, the two losses of the memory unit are mainly used to ensure that the memory unit can effectively store and retain the corresponding image feature information during the learning process, so as to promote the self-learning and optimization of the dual memory unit; and the loss of the classifier is mainly to guide the network to use the label information to have a better classification effect in the image tampering classification task, so as to achieve more accurate image tampering detection.

[0095] L=L cls +λ 1 L m +λ 2 L trip

[0096] In the formula, λ 1 and λ 2 It is a balanced hyperparameter. The method of mixing multiple loss functions in the design process of the total loss function can optimize the tampering detection performance of WSITD from multiple angles and improve the accuracy and robustness of the method for various image tampering situations.

[0097] In general, the present invention provides a highly robust and high-precision image tampering detection method that does not rely on pixel-level labels. Through clever feature fusion, memory unit framework and loss function design, it can effectively identify and locate the tampered area in the image. Compared with the existing method that can only identify whether the entire image is a tampered image, the present invention achieves higher-precision image tampering detection and provides powerful tools and methods for the field of image security. This method is expected to play a positive role in a wide range of applications, including digital forensics, image verification and other fields.

[0098] Embodiment 2

[0099] A method for detecting image tampering, comprising:

[0100] The image to be tested is input into the image tampering detection model constructed by the image tampering detection model construction method based on weak supervision as described above, and the result of whether the image to be tested is a tampered image is obtained. If it is a tampered image, the tampered area continues to be output.

[0101] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.

[0102] Embodiment 3

[0103] A computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a method for constructing an image tampering detection model based on weak supervision as described in the first embodiment above.

[0104] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.

[0105] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing an image tampering detection model based on weak supervision, characterized in that: include: The sample image is input into the image tampering detection network, which includes: a regional feature construction unit, which is used to extract the RGB features and noise-sensitive features of each image and fuse the two to obtain fused features, perform semantic segmentation on each image, obtain multiple binary mask images corresponding to different regions of the image and covering the entire image, and obtain regional features of different regions of the corresponding image based on the product of the fused features and each binary mask image; a dual memory unit framework, which includes a normal feature memory unit and a tampering feature memory unit, which is used to output the matching results of the features stored in the unit and each regional feature and the enhanced features of each regional feature through query and read operations of each unit, and add the two enhanced features output by the two units and splice them with the corresponding regional features to obtain spliced ​​features; a classifier, which is used to identify tampered images based on the spliced ​​feature set of each sample image; Calculate the detection loss to iteratively train the parameters of the image tampering detection network and complete the model construction; Among them, the detection loss includes memory unit loss based on constraints and the matching results, and the constraints include: the features stored in the tampered feature memory unit do not match the regional features of the real sample image, and match at least one regional feature of the tampered sample image; the features stored in the normal feature memory unit match the regional features of the real sample image, and do not match at least one regional feature of the tampered sample image; the features of the real sample image are close to the regional features of the real area in the tampered sample image, and the regional features of the tampered area are farther.

2. The method for constructing an image tampering detection model according to claim 1, characterized in that: The matching result of the feature stored in each memory unit and each regional feature is a three-dimensional vector with the same length and width as the regional feature and the same height as the number of feature blocks stored in the memory unit; The calculation method of the memory unit loss is as follows: Converting the matching result between the feature stored in each memory unit and each regional feature into a matching score between the regional feature and the feature stored in the memory unit; The memory unit loss is calculated based on the matching score between each region feature and the feature stored in each memory unit and the constraint.

3. The method for constructing an image tampering detection model according to claim 2, characterized in that: The loss based on the constraint that the features of the real sample image are close to the regional features of the real area in the tampered sample image, and the regional features of the tampered area are far away is: The ternary loss L is calculated based on the regional features with the highest matching score with the tampered feature memory unit in the tampered sample image, the regional features with the highest matching score with the normal feature memory unit in the tampered sample image, and the regional features of any region in the real sample image. trip .

4. The method for constructing an image tampering detection model according to claim 2, characterized in that: The conversion is implemented as follows: Based on the matching results of the features stored in each memory unit and the features of each region and the binary mask map corresponding to the regional features, according to the formula Calculate the similarity score vector S between the regional feature and the feature stored in the memory unit, where T(i, j) represents the matching result vector corresponding to the pixel point (i, j) in the regional feature in the corresponding matching result, and the number of elements is the same as the number of feature blocks stored in the memory unit; M(i,j) represents the pixel value corresponding to the binary mask image corresponding to the feature of the region; N m Represents the total number of pixels in the image area corresponding to the regional feature; takes the average of the top K similarities from the similarity score vector S between each regional feature and the feature stored in each memory unit as the matching score between the regional feature and the feature stored in the memory unit.

5. The method for constructing an image tampering detection model according to claim 3, characterized in that: A match score value of 0 indicates that the features stored in the memory unit do not match the regional features at all, and a match score value of 1 indicates that the features stored in the memory unit match the regional features completely; The constraints include: the matching scores of the features stored in the tampered feature memory unit and the regional features of the real sample image should all be 0, and the maximum matching score of the features of the regional features of the tampered sample image should be 1; the matching scores of the features stored in the normal feature memory unit and the regional features of the real sample image should all be 1, and the minimum matching score of the features of the regional features of the tampered sample image should be 0; the features of the real sample image are close to the regional features of the real area in the tampered sample image, and the regional features of the tampered area are farther, 0 indicates a complete mismatch, and 1 indicates a complete match; The memory unit loss includes: In the formula, Represents the matching loss between the features stored in the tampered feature memory unit and the real sample image, Represents the matching loss between the features stored in the real feature memory unit and the real sample image, Represents the matching loss between the features stored in the tampered feature memory unit and the tampered sample image, Represents the matching loss between the features stored in the real feature memory unit and the tampered sample image, Represents a vector consisting of the matching scores of the features of each region of the real sample image and the features stored in the tampered feature memory unit. Indicates length The value of is 0. Represents a vector consisting of the matching scores between the features of each region of the real sample image and the features stored in the real feature memory unit. Indicates length The values ​​of are all 1, BCE represents the binary cross entropy loss; It represents the maximum matching score in the vector composed of the matching scores of each region feature of the tampered sample image and the features stored in the tampered feature memory unit. The value is 1. It represents the smallest matching score in the vector composed of the matching scores of the regional features of the tampered sample image and the features stored in the real feature memory unit. The value is 0.

6. The method for constructing an image tampering detection model according to claim 5, characterized in that: The detection loss is: L=L cls +λ1L m +λ2L trip Where L represents the detection loss, L cls represents the classification loss of the classifier based on image-level labels, and λ1 and λ2 are both hyperparameters.

7. The method for constructing an image tampering detection model according to any one of claims 1 to 6, characterized in that: Extracting a noise view of a sample image to extract the noise-sensitive feature from the noise view; wherein the noise view is extracted using a BayarConv convolution, and a ResNet-50 backbone network is used to extract the RGB feature from the sample image and the noise-sensitive feature from the noise view; The method for obtaining the multiple binary mask images is specifically as follows: Using SAM, semantic segmentation is performed on each sample image to obtain a set of binary mask images; The binary mask map set is processed by region cross removal and small region deletion to obtain multiple binary mask maps corresponding to different regions of the image and covering the sample image.

8. The method for constructing an image tampering detection model according to any one of claims 1 to 6, characterized in that: The regional characteristics are calculated as follows: G m =G f ×(p+Maxpooling(M)); In the formula, G f represents the fusion feature of each sample image, M represents any binary mask image corresponding to the sample image; G m represents the regional features of the region corresponding to M in the sample image, Maxpooling(·) represents the maximum pooling operation, and p represents the hyperparameter; The query and the read operations are respectively expressed as: G aug =TM b ; In the formula, G m Represents regional characteristics, M b represents the features stored in the memory unit, C represents the feature dimension extracted by the regional feature construction unit, σ(x) refers to the sigmoid function, T represents the matching result, G aug Represents the enhanced feature.

9. A method for detecting image tampering, characterized in that: include: The image to be tested is input into the image tampering detection model constructed by the method for constructing an image tampering detection model based on weak supervision as described in any one of claims 1 to 8, and a result of whether the image to be tested is a tampered image is obtained. If it is a tampered image, the tampered area continues to be output.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a method for constructing an image tampering detection model based on weak supervision as described in any one of claims 1 to 8.

Images