Document image tampering positioning and desensitization positioning method and terminal based on contrast learning

Abstract

The application discloses a kind of based on contrast learning's document image tampering positioning and desensitization positioning method, terminal and storage medium, method includes: obtaining document image dataset, and extracting the tampering feature and desensitization feature of the document image dataset;According to the tampering feature and desensitization feature extracted to the document image dataset, the contrast training of tampering feature enhancement is carried out, and the tampering feature enhancement model after training is obtained;According to the tampering feature enhancement model after training, tampering positioning and desensitization positioning detection are carried out to input document image, and tampering positioning and desensitization positioning detection result is output;The application proposes new based on contrast learning's document image tampering positioning and desensitization positioning method, more effectively realizes the excavation to tampering trace, improves the positioning ability to tampering and desensitization, and tampering area and desensitization area are distinguished.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document image tampering positioning, in particular to a document image tampering positioning and desensitization positioning method based on contrast learning, a terminal and a storage medium. BACKGROUND

[0002] With the popularization of online business procedures, some important document materials such as contracts, proof documents and application forms are often stored and spread in electronic image format. However, due to the inclusion of key text information, such images are often easily tampered with illegally. If used by lawbreakers to seek undue benefits, or uploaded to social media to mislead public opinion, it will have a serious impact on individuals and society. At the same time, when the demand side has a document image to be checked and needs a third-party agency to help identify authenticity and tampering, in order to avoid exposing sensitive information, desensitization processing is often performed first, resulting in such images containing desensitization traces, which affects the detection of tampering traces. Therefore, the forensic research on document images has become a key topic, and it is necessary to develop an effective document image tampering positioning algorithm and distinguish between tampering and desensitization areas.

[0003] However, the existing technology has low positioning accuracy for document image tampering positioning, insufficient ability to explore tampering traces, poor ability to transfer document scene images, and cannot distinguish between tampering and desensitization, making it difficult to accurately identify tampering of document images and difficult to protect user privacy while positioning tampering of files.

[0004] Therefore, the prior art still needs to be improved. SUMMARY

[0005] The technical problem to be solved by the present application is that, in view of the defects of the prior art, the present application provides a document image tampering positioning and desensitization positioning method based on contrast learning, a terminal and a storage medium, to solve the technical problem of low generalizability of traditional deep learning fall detection methods.

[0006] The technical solution adopted by the present application to solve the technical problem is as follows:

[0007] In a first aspect, the present application provides a document image tampering positioning and desensitization positioning method based on contrast learning, comprising:

[0008] obtaining a document image dataset and extracting tampering features and desensitization features of the document image dataset;

[0009] performing contrast training of tampering feature enhancement according to the extracted tampering features and desensitization features of the document image dataset, to obtain a trained tampering feature enhancement model;

[0010] According to the trained tampering feature enhancement model, tampering positioning and desensitization positioning detection are performed on an input document image, and a tampering positioning and desensitization positioning detection result is output.

[0011] In an implementation manner, the document image dataset is acquired, and tampering features and desensitization features of the document image dataset are extracted, including:

[0012] The document image dataset is acquired.

[0013] Based on a feature pyramid network, the document image dataset is down-sampled at each level from bottom to top in an encoder through a ConvNeXt-Tiny backbone network, and the tampering features and the desensitization features of the document image dataset are extracted.

[0014] In an output stage of the encoder, a pyramid pooling module is connected, different scales are used to pool the extracted feature maps, a plurality of feature maps of different sizes are obtained, and the plurality of feature maps of different sizes are spliced in a channel dimension.

[0015] In an implementation manner, the contrast training of tampering feature enhancement is performed according to the extracted tampering features and the desensitization features of the document image dataset, including:

[0016] The contrast training of tampering feature enhancement is performed according to the tampering features and the desensitization features of the document image dataset, and the feature maps output by the pyramid pooling module are optimized and learned through a contrast learning module.

[0017] Features between tampering feature pixel points are pulled closer, and features between the tampering feature pixel points and non-tampering feature pixel points are pulled away, to perform the contrast training of tampering feature enhancement.

[0018] In an implementation manner, the features between the tampering feature pixel points are pulled closer, and the features between the tampering feature pixel points and the non-tampering feature pixel points are pulled away, including:

[0019] The pixel points with an image pixel-level label marked as 0 are set as real-class pixel points.

[0020] The pixel points with an image pixel-level label marked as 1 are set as tampering-class pixel points.

[0021] The pixel points with an image pixel-level label marked as 2 are set as desensitization-class pixel points.

[0022] The real-class pixel points and the desensitization-class pixel points are set as non-tampering feature pixel points, and corresponding labels are set as 0; the tampering-class pixel points are set as tampering feature pixel points, and corresponding labels are set as 1.

[0023] According to the label information, supervised contrast learning is performed on different mapping features inside the feature map: the features between the tampered feature pixels are pulled closer, and the features between the tampered feature pixels and the non-tampered feature pixels are pushed away.

[0024] In an implementation manner, the supervised contrast learning on the different mapping features inside the feature map according to the label information comprises:

[0025] A mapping feature of a feature point in the feature map is selected as an index feature, and the supervised contrast learning is performed on the different mapping features inside the feature map;

[0026] According to the label information, when the index feature is a tampered feature, a corresponding positive sample is set as the tampered feature, and a corresponding negative sample is set as a real feature and a desensitization feature;

[0027] According to the label information, when the index feature is the real feature or the desensitization feature, a corresponding positive sample is set as the real feature and the desensitization feature, and a corresponding negative sample is set as the tampered feature.

[0028] In an implementation manner, the tampering positioning and desensitization positioning detection on the input document image are performed according to the trained tampered feature enhancement model, and a tampering positioning and desensitization positioning detection result is outputted, which comprises:

[0029] The tampering positioning and desensitization positioning detection on the input document image are performed according to the trained tampered feature enhancement model;

[0030] After the feature pyramid module outputs a feature map and the feature map is merged in a channel dimension, the feature map is integrated by a global context attention mechanism module;

[0031] The feature map is decoded by a decoder, and the tampering positioning and desensitization positioning detection result is outputted.

[0032] In an implementation manner, the processing process of the global context attention mechanism module comprises a context modeling stage, a feature conversion stage and a feature fusion stage.

[0033] The integration processing of the feature map by the global context attention mechanism module after the feature pyramid module outputs the feature map and the feature map is merged in the channel dimension comprises:

[0034] In the context modeling stage, the information of all positions of the feature map is aggregated by dimension transformation, convolution operation and Softmax operation to obtain a global context feature;

[0035] In the dimension transformation stage, the inter-channel dependency is captured through convolution operation, layer normalization and ReLU activation function;

[0036] In the feature fusion stage, the global context features are aggregated to the features of each position by element-wise addition of the original feature map and the transformed feature map, and the feature pyramid module output feature map is integrated.

[0037] In an implementation manner, based on the feature pyramid network, the feature map is enlarged from top to bottom in the decoder through a bilinear interpolation up-sampling layer, and the image space is recovered step by step.

[0038] Between the encoder and the decoder, the first semantic information feature map and the second semantic information feature map are fused through lateral connection.

[0039] According to the decoding result, the tampering positioning and desensitization positioning detection result is output.

[0040] In a second aspect, the present application further provides a terminal, comprising a processor and a memory, the memory stores a document image tampering positioning and desensitization positioning program based on contrast learning, and the document image tampering positioning and desensitization positioning program based on contrast learning is used to implement the operation of the document image tampering positioning and desensitization positioning method based on contrast learning as described in the first aspect when executed by the processor.

[0041] In a third aspect, the present application further provides a computer readable storage medium, the storage medium stores a document image tampering positioning and desensitization positioning program based on contrast learning, and the document image tampering positioning and desensitization positioning program based on contrast learning is used to implement the operation of the document image tampering positioning and desensitization positioning method based on contrast learning as described in the first aspect when executed by the processor.

[0042] The technical scheme of the present application has the following effects:

[0043] The present application obtains a document image data set, extracts tampering features and desensitization features of the document image data set, performs contrast training of tampering feature enhancement according to the extracted tampering features and desensitization features of the document image data set, obtains a trained tampering feature enhancement model, performs tampering positioning and desensitization positioning detection on an input document image according to the trained tampering feature enhancement model, and finally outputs the tampering positioning and desensitization positioning detection result. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in the drawings without creative labor.

[0045] Figure 1 is a flowchart of a document image tampering positioning and desensitization positioning method based on contrast learning in an implementation manner of the present application.

[0046] Figure 2 is a schematic diagram of a document image tampering desensitization positioning algorithm framework based on contrast learning in an implementation manner of the present application.

[0047] Figure 3 is a schematic diagram of a tampering feature enhancement contrast learning module in an implementation manner of the present application.

[0048] Figure 4 is a schematic diagram of a global context attention mechanism module in an implementation manner of the present application.

[0049] Figure 5 is a schematic diagram of pixel-level three-class positioning results of various algorithms on DocTD in an implementation manner of the present application.

[0050] Figure 6 is a schematic diagram of tampering positioning and desensitization positioning effect and F1 index of different model schemes in an implementation manner of the present application.

[0051] Figure 7 is a fine-tuning effect diagram on the FindIt mixed dataset in an implementation manner of the present application.

[0052] Figure 8 is a fine-tuning effect diagram on the Real-World 2022 mixed dataset in an implementation manner of the present application.

[0053] Figure 9 is a functional schematic diagram of a terminal in an implementation manner of the present application.

[0054] The implementation of the present application, functional characteristics and advantages will be further described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] The application provides a document image tampering positioning and desensitization positioning method and terminal based on contrast learning. In order to make the purpose, technical scheme and advantages of the application more clear and explicit, the application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application.

[0056] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms unless specifically stated otherwise. It should be further understood that the use of the phrase "comprising" in the specification of the application means that the features, integers, steps, operations, elements and / or components are present, but not excluding the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0057] Those skilled in the art can understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0058] Exemplary method

[0059] The research object of the application is an electronic document image, including contracts, applications, notification letters, certification documents and other common document types in real life. Such documents often contain important text information or private content. When the original text content of such document images is tampered with, it will cause serious social impact and harm.

[0060] Because the positioning accuracy of the prior art for document image tampering positioning is low, the ability to explore tampering traces is not strong enough, the migration ability for document scene images is not good enough, and it is also impossible to distinguish between tampering and desensitization, it is difficult to accurately identify the tampering of document images, and it is also difficult to protect user privacy while positioning the tampering of the document. Therefore, it is necessary to develop an effective document image tampering positioning algorithm and distinguish between tampered areas and desensitized areas.

[0061] In order to solve the above technical problems, the embodiment of the present application provides a document image tampering positioning and desensitization positioning method based on contrast learning, which aims to provide a new document image tampering positioning and desensitization positioning method based on contrast learning to more effectively realize the exploration of tampering traces and improve the positioning ability of tampering and desensitization, and distinguish tampering areas and desensitization areas.

[0062] As shown in Figure 1 The embodiment of the present application provides a document image tampering positioning and desensitization positioning method based on contrast learning, which includes the following steps:

[0063] Step S100, acquiring a document image data set and extracting tampering features and desensitization features of the document image data set.

[0064] In the embodiment, the document image tampering positioning and desensitization positioning method based on contrast learning is applied to a terminal, which includes but is not limited to a computer, a mobile terminal and other devices; and the terminal is provided with a training migration platform of the document image tampering positioning and desensitization positioning model based on contrast learning.

[0065] In the embodiment, in order to solve the problem of low positioning accuracy of the existing method for document images, an end-to-end pixel-level positioning network based on a convolutional neural network is constructed to predict the tampering areas and desensitization areas of the document image pixel by pixel.

[0066] Specifically, the tampering of the document image is the change of the text content, specifically including splicing, copying and moving, erasing, text adding and text replacing, etc., and the desensitization is to cover the personal private information appearing in the image, including taking the methods of mosaicking and smearing, etc. The tampering positioning and desensitization positioning are to predict the tampering areas and desensitization areas of the document image and distinguish them.

[0067] As shown in Figure 2 In the embodiment, the convolutional encoder-decoder structure based on the feature pyramid network is used to train and identify the document image to be detected. The feature pyramid network is a method for detecting and identifying targets at different scales, which includes a bottom-up image feature extraction structure, a top-down decoding structure and a lateral connection feature fusion structure, and obtains more accurate feature representation on targets of different scales and sizes, so as to realize more accurate positioning.

[0068] In the embodiment, a document image data set is acquired, and the tampering features and desensitization features of the document image data set are extracted through a convolutional layer for subsequent contrast training and positioning detection.

[0069] Specifically, in one implementation manner of the embodiment, step S100 includes the following steps:

[0070] Step S110, obtain the document image dataset.

[0071] Step S120, based on the feature pyramid network, perform bottom-up multi-level down-sampling on the document image dataset in the encoder through the ConvNeXt-Tiny backbone network, and extract tampering features and desensitization features of the document image dataset.

[0072] In this embodiment, in the encoding stage, ConvNeXt-Tiny is used as the backbone network to perform multi-level down-sampling on the input image from bottom to top, and extract image tampering features and desensitization features.

[0073] Step S130, in the output stage of the encoder, connect the pyramid pooling module, pool the extracted feature maps using different scales to obtain a plurality of feature maps of different sizes, and concatenate the plurality of feature maps of different sizes in the channel dimension.

[0074] In this embodiment, in the last output stage of the encoder, the pyramid pooling module is connected, and different scales of pooling are used on the feature maps to obtain a plurality of feature maps of different sizes, and then these feature maps are concatenated in the channel dimension, thereby improving the ability of the network to obtain global information.

[0075] In this embodiment, the basic structure of the algorithm is the encoder-decoder structure based on the feature pyramid network, which can obtain more accurate and rich feature representations on targets of different scales and sizes, more effectively realize the excavation of tampering traces, and improve the positioning ability of tampering and desensitization.

[0076] As shown in Figure 1 In an implementation manner of the embodiment of the present application, the document image tampering positioning and desensitization positioning method based on contrast learning further includes the following steps:

[0077] Step S200, perform contrast training of tampering feature enhancement according to the extracted tampering features and desensitization features of the document image dataset, to obtain a trained tampering feature enhancement model.

[0078] In this embodiment, in order to solve the problem that the existing method has insufficient ability to excavate image tampering traces, a tampering trace enhancement contrast learning module is designed, which effectively realizes the excavation of tampering traces by making the distance between tampering features close to each other and the distance between tampering features and non-tampering features far from each other.

[0079] In this embodiment, the tampering feature enhancement contrast learning module is as shown in Figure 3As shown. Contrastive learning is a feature representation method in deep learning, aiming to solve the problem of insufficient labeled data in supervised learning. The core idea is to make the distance between samples of the same class in the feature space closer, and the distance between samples of different classes in the feature space farther, so as to learn more robust and more discriminative feature representation.

[0080] In the embodiment, the tampering feature enhancement contrast training is performed according to the tampering features and the desensitization features of the extracted document image data set, and a trained tampering feature enhancement model is obtained, so as to realize the technical effect of enhancing the attention of the enhancement network to the tampering region and enhancing the ability to discover tampering traces.

[0081] Specifically, in an implementation manner of the embodiment, step S200 includes the following steps:

[0082] Step S210, performing tampering feature enhancement contrast training according to the tampering features and the desensitization features of the document image data set, and optimizing and learning the feature map output by the pyramid pooling module through a contrast learning module.

[0083] In the embodiment, a tampering feature enhancement contrast learning module is designed to perform enhancement learning on the tampering features of the intermediate layer of the network, further discover weak tampering traces, and improve the positioning ability to the tampering region

[0084] Specifically, the input image size is HxWx3, H and W are the height and width of the image, and the output feature after the pyramid pooling module is denoted as Where h=H / d, w=W / d, h and w are the height and width of the feature map, and d is the reduction ratio.

[0085] Step S220, the features between tampering feature pixels are pulled closer, and the features between the tampering feature pixels and the non-tampering feature pixels are pulled away, to perform tampering feature enhancement contrast training.

[0086] In the embodiment, in the supervised contrast learning process, the tampering information is focused on, and the purpose is to pull the tampering features closer to each other and pull the tampering features away from the non-tampering features, so as to discover weaker tampering traces of the document image.

[0087] Specifically, in an implementation manner of the embodiment, step S220 includes the following steps:

[0088] Step S221, setting the pixel points with image pixel-level label marked as 0 as real class pixel points.

[0089] Step S222, setting the pixel points with image pixel-level label marked as 1 as tampering class pixel points.

[0090] Step S223, set the pixel points with image pixel-level label marked as 2 as desensitization class pixel points.

[0091] Step S224, set the real class pixel points and the desensitization class pixel points as non-tampering feature pixel points, and set the corresponding labels as 0; set the tampering class pixel points as tampering feature pixel points, and set the corresponding labels as 1.

[0092] In this embodiment, since the image pixel-level label M contains 0, 1, and 2, i.e., corresponding to three classes of real, tampering, and desensitization, the labels of real and desensitization pixel points are both regarded as 0, and the tampering is still 1, i.e., the real and desensitization are regarded as one class, and the tampering is regarded as another class. By adopting the three-channel output and pixel-level three-class label supervised training mode, the tampering positioning and desensitization positioning of the document image are simultaneously realized, and good performance is achieved.

[0093] Step S225, according to the label information, supervised contrast learning is performed on different mapping features inside the feature map: the features between the tampering feature pixel points are pulled closer to each other, and the features between the tampering feature pixel points and the non-tampering feature pixel points are pulled away from each other.

[0094] In this embodiment, the label M is reduced to the size of the feature map F through average pooling, i.e., Therefore, the corresponding label of each point on the feature map is known, and according to the label information, supervised contrast learning is performed on different mapping features inside the feature map: the features between the tampering feature pixel points are pulled closer to each other, and the features between the tampering feature pixel points and the non-tampering feature pixel points are pulled away from each other, to obtain the trained tampering feature enhancement model.

[0095] Specifically, in one implementation manner of this embodiment, step S225 includes the following steps:

[0096] Step S225a, a mapping feature of a feature point in the feature map is selected as an index feature, and the supervised contrast learning is performed on different mapping features inside the feature map.

[0097] Step S225b, according to the label information, when the index feature is a tampering feature, the corresponding positive sample is set as the tampering feature, and the corresponding negative sample is set as a real feature and a desensitization feature.

[0098] Step S225c, according to the label information, when the index feature is the real feature or the desensitization feature, the corresponding positive sample is set as the real feature and the desensitization feature, and the corresponding negative sample is set as the tampering feature.

[0099] A feature point in the feature map is selected as a query feature, and according to the label information m, when the query is a tampered feature, the positive sample key corresponding to the query is a tampered feature, and the negative sample key is a real feature and a desensitized feature; when the query is a real feature or a desensitized feature, the positive sample key is a real feature and a desensitized feature, and the negative sample key is a tampered feature.

[0100] For each query feature q i , the supervised contrastive learning loss is:

[0101]

[0102] Where A i represents the set of all positive sample mapping features k i with the same label as q + , and k - represents the negative sample mapping feature in F that is different from q i in label, and the above features are subjected to L2 regularization processing. τ is a hyperparameter, representing a temperature coefficient, and the smaller it is, the higher the attention degree of the contrastive loss function to difficult negative samples. By calculating the average of the contrastive learning loss of all mapping vectors on the feature map, the expression of the final contrastive loss can be obtained as:

[0103]

[0104] Because the tampered region and the desensitized region are small, too many negative samples will reduce the learning ability and efficiency of the network, in order to balance the importance of positive and negative samples, in the tampering positioning and desensitization positioning stages of the network, the present application adopts the Focal loss function, the formula is as follows:

[0105]

[0106] Where C represents the class set, including real, tampered and desensitized, y i (c)∈{0,1,2} represents the class label of pixel i, p i (c)∈[0,1] is the probability of pixel i being predicted as class c, and γ is the focusing parameter, which is taken as 2 in the following experiment.

[0107] The final overall loss function L is the weighted sum of the positioning loss function and the contrastive loss function, and the formula is as follows:

[0108] L=L focal +λL contra ,

[0109] Since the contrastive loss is always about 10 times that of the Focal loss during training, the weight λ is set to 0.1 in order to balance the supervisory role of the two losses on the network.

[0110] In this embodiment, to address the problem that existing methods cannot distinguish between tampered and de-identified regions, the localization network output is modified to have three channels. During training, three types of labels are used for supervised learning, and during prediction, each pixel in the document image is classified as real, tampered, or de-identified. The encoder features are further optimized through a contrastive learning module, bringing the features of tampered pixels closer together and distancing the features of tampered and untampered pixels, thereby uncovering subtler tampering traces in the document image. This makes the network more attentive to tampered regions, improves the accuracy of tampered region localization, and enhances its ability to detect tampering traces.

[0111] like Figure 1 As shown, in one implementation of this invention, the document image tampering localization and de-identification localization method based on contrastive learning further includes the following steps:

[0112] Step S300: Based on the trained tamper feature enhancement model, perform tamper location and desensitization location detection on the input document image, and output the tamper location and desensitization location detection results.

[0113] In this embodiment, the input document image information is acquired, and the tampering localization and de-identification localization detection of the input document image are performed according to the trained tampering feature enhancement model. Based on the feature pyramid network, a convolutional encoder-decoder network is constructed and a three-channel output and pixel-level three-class label supervised training method is adopted. At the same time, a context attention mechanism module is introduced into the network to effectively suppress image content and enhance the integration processing of the feature pyramid network fused features, thereby realizing the tampering localization and de-identification localization of the document image, and outputting the tampering localization and de-identification localization detection results.

[0114] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0115] Step S310: Perform tamper location and desensitization location detection on the input document image based on the trained tamper feature enhancement model.

[0116] Step S320: The feature map is output from the feature pyramid module, merged along the channel dimension, and then integrated through the global context attention mechanism module.

[0117] Step S330: Decode the feature map using a decoder and output the tamper location and desensitization location detection results.

[0118] In the embodiment, the feature pyramid network outputs feature maps of different scales, all the feature maps are upsampled to 1 / 4 of the width and height of the input image by a bilinear interpolation algorithm, and are merged in the channel dimension. Since the merged feature map contains rich decoding information, in order to effectively suppress the image content and enhance the tampering and desensitization traces, the global context attention mechanism module is connected to the merged feature. By aggregating and arranging the multi-scale information fused by the decoder, the image content is suppressed, so that the network pays more attention to the subtle tampering traces and desensitization traces of the image, and outputs the tampering positioning and desensitization positioning detection results.

[0119] Specifically, in an implementation manner of the embodiment, step S320 includes the following steps:

[0120] Step S321, the processing procedure of the global context attention mechanism module includes a context modeling stage, a feature conversion stage, and a feature fusion stage.

[0121] Step S322, in the context modeling stage, the information of all positions of the feature map is aggregated by dimension transformation, convolution operation, and Softmax operation to obtain global context features.

[0122] Step S323, in the dimension transformation stage, the dependency between channels is captured by convolution operation, layer normalization, and ReLU activation function.

[0123] Step S324, in the feature fusion stage, the global context features are aggregated to the features of each position by element-wise addition of the original feature map and the transformed feature map, and the output feature map of the feature pyramid module is integrated and processed.

[0124] In the embodiment, the global context attention mechanism module includes three parts, namely context modeling, feature conversion, and feature fusion, as shown in Figure 4

[0125] Specifically, in the context modeling stage, the information of all positions of the feature map is aggregated by dimension transformation, 1x1 convolution operation, and Softmax operation to obtain global context features; in the dimension transformation stage, the dependency between channels is captured by 1x1 convolution operation, layer normalization (LayerNorm, LN), and ReLU activation function; in the feature fusion stage, the global context features are aggregated to the features of each position by element-wise addition of the original feature map and the transformed feature map.

[0126] ​In the embodiment, the global context attention mechanism module can establish effective long-distance dependence and has the characteristics of light weight. By aggregating and arranging the multi-scale information fused by the decoder, the image content is suppressed, and the network pays more attention to the subtle tampering traces and desensitization traces of the image.

[0127] Specifically, in an implementation manner of the embodiment, step S330 includes the following steps:

[0128] In step S331, based on the feature pyramid network, the feature map is enlarged from top to bottom in the decoder by a bilinear interpolation upsampling layer, and the image space is recovered step by step.

[0129] In step S332, the first semantic information feature map and the second semantic information feature map are fused by lateral connection between the encoder and the decoder.

[0130] In step S333, the tampering positioning and desensitization positioning detection results are output according to the decoding results.

[0131] In the embodiment, in the decoding stage, the feature map is enlarged from top to bottom by a bilinear interpolation upsampling layer, and the image space is recovered step by step. The first semantic information feature map and the second semantic information feature map are fused by lateral connection between the encoder and the decoder. The first semantic information feature map is a feature map with strong low-resolution semantic information, and the second semantic information feature map is a feature map with weak high-resolution semantic information but rich spatial information. Specifically, the feature map with strong low-resolution semantic information and the feature map with weak high-resolution semantic information but rich spatial information are fused by lateral connection between the encoder and the decoder, and the learning of the tampering traces and the desensitization traces is realized in multiple scales, so that the positioning results are more fine. The decoding network outputs multi-level fusion features, and the context attention mechanism module integrates and processes the multi-level fusion features to suppress the image content and reduce the false alarm of the final output results of the network.

[0132] Specifically, the detection results output by the embodiment are as follows: the black area represents the real area, the white area represents the tampered area, and the gray area represents the desensitized area.

[0133] In the embodiment, by introducing the context attention mechanism module in the network, the image content is effectively suppressed, the integration processing of the fusion features of the feature pyramid network is enhanced, the network pays more attention to the tampering traces and the desensitization traces, and the prediction results of the tampered area and the desensitized area are more fine and more comprehensive, and the false alarm rate is also lower.

[0134] The above embodiment is described below by way of experimental verification:

[0135] (1) Experimental setup

[0136] The algorithm of the application is firstly trained and tested on a pre-established document image tampering and desensitization dataset DocTD, the performance of the proposed model is evaluated, including comparative experiments with existing tampering positioning algorithms, network ablation experiments and robustness experiments. Then in order to verify the cross-dataset performance, the Real-World 2022 dataset and the FindIt dataset are subjected to desensitization operation to generate a mixed dataset containing tampering and desensitization operation, and the two mixed datasets are tested and fine-tuned. The methods compared in the experiment are: ADQ1, Noiseprint, RRU-Net, MVSS-Net, DenseFCN and DIDNet.

[0137] The evaluation indexes used in the experiment are pixel-level F1 score, IOU and Matthews Correlation Coefficient (MCC), the formulas of which are as follows. In addition, the area under the Receiver Operating Characteristic (ROC) curve (Area Under Curve, AUC) is also used to evaluate the performance.

[0138]

[0139]

[0140]

[0141] Among them, the four parameters respectively represent True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN).

[0142] The model of the application is based on the PyTorch framework. In the training stage, the batch size is set to 40, the AdamW optimizer is used, the initial learning rate is 0.0001, the weight decay is 0.0005, and all experiments are run on 4 Nvidia Tesla P100 GPU graphics cards. The label reduction ratio d is 32; the temperature coefficient τ is 0.07; the reduction rate r in the global context attention module is 8.

[0143] (2) Comparative experiment with existing algorithms

[0144] The algorithm ConTDNet of the present application is firstly trained and tested on the pre-established document image tampering and desensitization dataset DocTD to evaluate the tampering and desensitization positioning performance, the number of images is 2000, and the training, verification and test set is divided as 8:1:1. During training, the images are uniformly cropped into non-overlapping image blocks of 512x512 size, and the image blocks containing tampering or desensitization are selected as training data; during testing, the to-be-tested image is cropped with overlap, and the cropped image block size is also 512x512, with a step of 256, the prediction result of each image block is fused to obtain the probability map of the whole test image, and then the channel index where the maximum probability value in the probability map is taken as the final test result, and each pixel point is classified as real, tampering or desensitization. The network structure of the present application outputs a three-classification result at the pixel level, while other algorithms can only make a two-classification decision. When comparing the performance with the comparison algorithm, the present application adopts two experimental methods.

[0145] ①Pixel-level two-classification—overall tampering and desensitization positioning performance evaluation

[0146] Since tampering and desensitization both introduce external traces to the image, this experiment takes tampering and desensitization as a class (positive class) and real as another class (negative class) for supervised learning of other algorithms, calculates the overall performance of tampering and desensitization positioning of the present application method and the comparison algorithm, and calculates the performance of the comparison algorithm under the optimal threshold value, and the experimental results are shown in Table 1.

[0147] As shown in Table 1, ConTDNet performs best, followed by DenseFCN and RRU-Net. The algorithm of the present application outputs three channels, and the prediction result of each pixel point takes the channel index where the maximum probability value is located, so there is no optimal threshold value. When the prediction result of the comparison algorithm under the optimal threshold value is taken, the algorithm proposed by the present application still performs best, and the overall F1 and IOU of tampering and desensitization positioning are 0.035 and 0.051 higher than DenseFCN, respectively. In addition, in actual scene, images often have no label, so the optimal threshold value cannot be calculated, and therefore the algorithm of the present application is more practical.

[0148] Table 1 Overall performance of document image tampering and desensitization positioning (pixel-level two-classification)

[0149]

[0150] ②Pixel-level three-classification—tampering positioning performance and desensitization positioning performance are evaluated respectively

[0151] To ensure the consistency of the supervision information, the output structure of the comparison algorithm is changed to three-channel output, and real, tampering and desensitization are taken as three kinds of pixel-level labels for supervised learning, and the performance of tampering positioning and the performance of desensitization positioning are calculated respectively, and the experimental results are shown in Table 2. The comparison algorithm selected in this experiment is the supervised tampering positioning method RRU-Net, MVSS-Net and DenseFCN, and the number of output channels of the network at the end of the present application is changed to three channels, and the other parameter settings are unchanged.

[0152] Table 2 Document image tampering positioning / desensitization positioning performance (pixel-level three classification)

[0153]

[0154] It can be seen that the effects of several comparison algorithms and the algorithm of the present application on positioning the desensitization area are generally good, and basically the desensitization area can be identified. In terms of desensitization positioning, the F1 index of the algorithm of the present application is 0.941, which is better than 0.919 of RRU-Net. In terms of tampering positioning, the F1 and AUC indexes of the algorithm of the present application are 0.826 and 0.978, which have better positioning ability than other algorithms, and the improved algorithm can more effectively explore the tampering traces. And from the visualization effect of Figure 5 It can be seen that the prediction results of ConTDNet for tampering areas and desensitization areas are more detailed and more comprehensive, and the false alarm rate is also lower.

[0155] (3) Ablation experiment

[0156] Table 3 F1 of tampering and desensitization positioning of different variable settings

[0157]

[0158] The algorithm of the present application further optimizes and learns the encoder features through the comparison learning module, so that the features between the tampering pixel points are close to each other, and the features between the tampering and non-tampering are far from each other, thereby exploring the weaker tampering traces of the document image. The decoding network outputs multi-level fusion features, and the context attention mechanism module integrates and processes them to suppress the image content and reduce the false alarm of the final output result of the network. In this experiment, the effectiveness of the two modules is verified, and the experimental results are shown in Table 3.

[0159] From the comparison of the experimental results of the first and second rows, it can be seen that after introducing the contrast learning module, the index of desensitization positioning does not change, but the F1 index of tampering positioning increases by 0.034, and the model is more effective in capturing tampering traces. From the comparison of the experimental results of the first and third rows, it can be found that after introducing the context attention mechanism, the positioning performance of tampering and desensitization is improved. From the last row, it can be seen that the model finally adopted by the present application achieves the best performance in tampering and positioning, and the F1 index is improved compared with the second and third rows, indicating that the contrast learning module and the context attention mechanism module have a positive effect on the network performance, and both modules are indispensable. As shown in Figure 6 from the positioning effect, after introducing the contrast learning module, the network can enhance the attention to the tampering area, and the positioning accuracy of the tampering area of schemes 2 and 4 is higher, and the ability to discover tampering traces is stronger; after introducing the context attention module, the image content is effectively suppressed, and the positioning ability of tampering and desensitization is improved.

[0160] (4) Cross-dataset experiment

[0161] The data of the experiment comes from the Real-World 2022 dataset and the FindIt dataset, and since both datasets only contain tampering operations and no desensitization operations, a Python script is used to call the Photoshop interface to add desensitization operations to part of the images to generate a tampering and desensitization mixed dataset. The desensitization types include various mosaics and smearing operations mentioned in the second chapter. Specifically, there are 4000 tampering images in the Real-World 2022 dataset, and there are no real images. 2000 of them are randomly selected for desensitization processing, and the other 2000 are not processed. The final Real-World 2022 mixed dataset includes two types of images: tampering images and tampering and desensitization images, each with 2000 images. In the FindIt dataset, 360 real images are randomly selected, of which 180 are not operated, and the other 180 are used to generate data containing only desensitization operations; in addition, 180 tampering images in the dataset are desensitized to generate data containing both tampering and desensitization. The final FindIt mixed dataset includes four types of images: real images, desensitization images, tampering images, and tampering and desensitization images, each with 180 images. The above datasets generate corresponding pixel-level three-class labels, and when desensitization operations occur in the original tampering area, the area is marked as a desensitization area.

[0162] To evaluate the performance of the network model of the present application on the public dataset, the present application directly tests and fine-tunes the two mixed datasets respectively. When fine-tuning the Real-World 2022 mixed dataset, the number of training, validation, and testing is divided as 1000, 200, 2800, and each set contains 2 image types, and the number of each type is equal. When fine-tuning the FindIt mixed dataset, the number of training, validation, and testing is divided as 100, 20, 600, and each set contains 4 image types, and the number of each type is equal. Due to the different tampering and desensitization conditions of different image types, in order to more objectively evaluate the positioning performance, the present application calculates different indicators for different images. For real images in the FindIt mixed dataset, the present application uses the False Positive Rate (FPR) index to evaluate the performance, and the formula is as follows:

[0163]

[0164] Wherein, FP represents the total number of pixels classified as positive samples (tampering, desensitization) by mistake, and TN represents the total number of pixels correctly classified as negative samples (real). For tampered images and tampered and desensitized images, the present application calculates the F1 and AUC positioning indicators of the tampered region, and for desensitized images and tampered and desensitized images, the present application calculates the F1 and AUC positioning indicators of the desensitized region. The experimental results are shown in Tables 4 and 5.

[0165] Table 4 Positioning performance on Real-World 2022 mixed dataset

[0166]

[0167] Table 5 Positioning performance on FindIt mixed dataset

[0168]

[0169]

[0170] From the experimental results, it can be seen that the performance of the algorithm of the application is not high when directly testing all images on the Real-World 2022 mixed dataset, but after fine-tuning with a small amount of data, it shows good tampering positioning and desensitization positioning capabilities. When directly tested on the FindIt mixed dataset, the false alarm rate of the algorithm of the application on real images is relatively high, and the positioning capability for tampered regions and desensitized regions is insufficient, only slightly better than some comparative algorithms; and after fine-tuning with a small amount of samples, the false alarm rate on real images is significantly reduced, and the tampering positioning and desensitization positioning accuracy is also significantly increased, and the F1 index is increased by 0.176 and 0.091 respectively, which is better than other algorithms, proving that the network is more effective in discovering tampering traces during the training process, and at the same time enhances the ability to distinguish tampered regions and desensitized regions.

[0171] Figure 7 and Figure 8 The positioning effect of each type of image on the two datasets after fine-tuning is shown. It can be seen that each algorithm has good detection effect on the desensitized region, and the misjudgment of tampered regions or real regions as desensitized is less. In the detection of real images, the false detection rate of ConTDNet is lower. At the same time, compared with other algorithms, ConTDNet has higher detection accuracy for text image tampered regions and more comprehensive detection range, and can effectively discover tampering traces on tampered images and tampered desensitized images.

[0172] (5) Robustness experiment

[0173] In this section, the robust performance of the algorithm ConTDNet of the application is evaluated. In order to simultaneously view the tampering positioning performance and desensitization positioning performance, the comparative algorithms use RRU-Net, MVSS-Net and DenseFCN, and the output of each algorithm is changed to three channels, and pixel-level three-class labels are applied for supervision.

[0174] The same data enhancement method is used for all algorithms during training. Specifically, it mainly includes the following: random cropping of pictures, random horizontal and vertical flipping, random addition of zero-mean Gaussian noise with different standard deviations, random selection of median filter with different filter kernel sizes to blur the image, and in addition, different JPEG factor compression of document images is also used.

[0175] During testing, several different post-processing operations were applied to the DocTD dataset images, including median filter blur, adding Gaussian noise, scaling, and JPEG compression. Among them, the median filter blur kernel size was 3, 5, 7, 9, and 11; the Gaussian noise mean was 0, and the standard deviation was 5, 10, and 15, 20, and 25; the scaling factor size was selected as 0.3, 0.5, 0.7, 0.9, 1.25, and 1.5, and the compression quality factor was 60, 70, 80, 90, and 100. The performance of image tamper localization and desensitization localization under each post-processing operation was tested, and the images without any post-processing were also tested, and the experimental results are shown in Table 6.

[0176] Table 6 F1 of DocTD dataset image tamper localization and desensitization localization under different post-processing

[0177]

[0178] It can be seen that ConTDNet performs best compared to other algorithms, and performs well in tamper localization performance and desensitization localization performance. When the post-processing operation is intensified, it has less effect on desensitization localization, but has greater effect on tamper localization. Specifically, for different post-processing attacks, the F1 of desensitization localization is always stable at about 0.9. In terms of tamper localization, ConTDNet has poor anti-attack ability for median filter and Gaussian noise, and when the median filter blur kernel increases from 3 to 11, the F1 index decreases by 0.254, and when the standard deviation of Gaussian noise increases from 5 to 25, the F1 decreases by 0.132. For scaling and JPEG compression, the tamper localization performance is relatively stable, and when the scaling factor is 0.7 or more, the F1 performance is stable at about 0.5, and only when the scaling factor is 0.5 does it perform slightly worse; when the picture compression quality is low, even when the JPEG compression factor is 60, the F1 index of tamper localization can still reach 0.58.

[0179] The technical solutions of the above embodiments achieve the following technical effects:

[0180] The embodiment achieves tamper localization and desensitization localization of a document image and achieves good performance by constructing a convolutional decoding network based on a feature pyramid network and adopting three-channel output and pixel-level three-class label supervision training. By introducing a context attention mechanism module in the network, the image content is effectively suppressed, the integration processing of the fused features of the feature pyramid network is enhanced, and the network more effectively focuses on tamper traces and desensitization traces. By designing a tamper feature enhanced contrast learning module, the tamper features of the intermediate layers of the network are enhanced and learned, further weak tamper traces are explored, and the localization ability of tamper regions is improved.

[0181] Exemplary apparatus

[0182] Based on the above-mentioned embodiments, the application further provides a terminal, comprising: a processor, a memory, an interface, a display screen and a communication module connected through a system bus; wherein the processor is used to provide computing and control capabilities; the memory comprises a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operating system and the computer program in the storage medium to run; the interface is used to connect external devices, such as mobile terminals and computer devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0183] The computer program is executed by the processor to implement the operations of the document image tampering positioning and desensitization positioning method based on contrast learning.

[0184] Those skilled in the art can understand that, Figure 9 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0185] In one embodiment, a terminal is provided, comprising: a processor and a memory, wherein the memory stores a document image tampering positioning and desensitization positioning program based on contrast learning, and the document image tampering positioning and desensitization positioning program based on contrast learning is executed by the processor to implement the operations of the document image tampering positioning and desensitization positioning method based on contrast learning as described above.

[0186] In one embodiment, a computer readable storage medium is provided, wherein the storage medium stores a document image tampering positioning and desensitization positioning program based on contrast learning, and the document image tampering positioning and desensitization positioning program based on contrast learning is executed by the processor to implement the operations of the document image tampering positioning and desensitization positioning method based on contrast learning as described above.

[0187] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the application can include non-volatile and / or volatile memory.

[0188] In conclusion, the application provides a document image tampering positioning and desensitization positioning method based on contrast learning, a terminal and a storage medium, the method comprising: obtaining a document image data set, and extracting tampering features and desensitization features of the document image data set; performing contrast training of tampering feature enhancement according to the extracted tampering features and desensitization features of the document image data set, to obtain a trained tampering feature enhancement model; performing tampering positioning and desensitization positioning detection on an input document image according to the trained tampering feature enhancement model, and outputting a tampering positioning and desensitization positioning detection result; the application proposes a new document image tampering positioning and desensitization positioning method based on contrast learning, which more effectively realizes the excavation of tampering traces, improves the positioning ability of tampering and desensitization, and distinguishes tampering areas and desensitization areas.

[0189] It should be understood that the application is not limited to the above examples, and can be improved or changed by those of ordinary skill in the art according to the above description, and all such improvements and changes shall fall within the protection scope of the appended claims of the application.

Claims

1. A document image tampering localization and desensitization localization method based on contrastive learning, characterized in that, The method comprises the following steps: obtaining a document image data set and extracting tampering features and desensitization features of the document image data set; performing contrast training of tampering feature enhancement according to the extracted tampering features and desensitization features of the document image data set, to obtain a trained tampering feature enhancement model; performing tampering positioning and desensitization positioning detection on an input document image according to the trained tampering feature enhancement model, and outputting tampering positioning and desensitization positioning detection results; the contrast training of tampering feature enhancement according to the extracted tampering features and desensitization features of the document image data set comprises: performing contrast training of tampering feature enhancement according to the tampering features and desensitization features of the document image data set, and optimizing and learning the feature map output by the pyramid pooling module through a contrast learning module; the features between tampering feature pixel points are pulled closer, and the features between the tampering feature pixel points and the non-tampering feature pixel points are pulled away, to perform contrast training of tampering feature enhancement.

2. The contrastive learning based document image tampering localization and sanitization localization method of claim 1, wherein, the method comprises the following steps: obtaining a document image data set; based on a feature pyramid network, performing downsampling on the document image data set from bottom to top in an encoder through a ConvNeXt-Tiny backbone network, to extract tampering features and desensitization features of the document image data set; in the output stage of the encoder, a pyramid pooling module is connected, different scales are used to pool the extracted feature map, a plurality of feature maps of different sizes are obtained, and the plurality of feature maps of different sizes are spliced in the channel dimension.

3. The contrastive learning based document image tampering localization and sanitization localization method of claim 2, wherein, the features between tampering feature pixel points are pulled closer, and the features between the tampering feature pixel points and the non-tampering feature pixel points are pulled away, which comprises: setting the pixel points with image pixel-level label marked as 0 as real class pixel points; setting the pixel points with image pixel-level label marked as 1 as tampering class pixel points; setting the pixel points with image pixel-level label marked as 2 as desensitization class pixel points; setting the real class pixel points and the desensitization class pixel points as non-tampering feature pixel points, and setting the corresponding labels as 0; setting the tampering class pixel points as tampering feature pixel points, and setting the corresponding labels as 1; according to the label information, performing supervised contrast learning on different mapping features inside the feature map: pulling the features between the tampering feature pixel points closer, and pulling the features between the tampering feature pixel points and the non-tampering feature pixel points away.

4. The contrastive learning based document image tampering localization and sanitization localization method of claim 3, wherein, the supervised contrast learning on different mapping features inside the feature map according to the label information comprises: selecting a feature point mapping feature in the feature map as an index feature, and performing the supervised contrast learning on different mapping features inside the feature map; according to the label information, when the index feature is a tampering feature, setting the corresponding positive sample as the tampering feature, and setting the corresponding negative sample as a real feature and a desensitization feature; According to the label information, when the index feature is the real feature or the desensitization feature, the corresponding positive sample is set as the real feature and the desensitization feature, and the corresponding negative sample is set as the tampered feature.

5. The contrastive learning based document image tampering localization and sanitization localization method of claim 1, wherein, The tamper positioning and desensitization positioning detection of the input document image is performed according to the trained tamper feature enhancement model, and a tamper positioning and desensitization positioning detection result is output. The tamper positioning and desensitization positioning detection of the input document image is performed according to the trained tamper feature enhancement model; After the feature pyramid module outputs the feature map and the feature map is merged in the channel dimension, the feature map is integrated by a global context attention mechanism module; The feature map is decoded by a decoder, and the tamper positioning and desensitization positioning detection result is output.

6. The contrastive learning based document image tampering localization and sanitization localization method of claim 5, wherein, The processing process of the global context attention mechanism module includes a context modeling stage, a feature conversion stage, and a feature fusion stage; After the feature pyramid module outputs the feature map and the feature map is merged in the channel dimension, the feature map is integrated by a global context attention mechanism module; In the context modeling stage, the information of all positions of the feature map is aggregated by dimension transformation, convolution operation and Softmax operation to obtain global context features; In the dimension transformation stage, the dependence between channels is captured by convolution operation, layer normalization and ReLU activation function; In the feature fusion stage, the global context features are aggregated to the features of each position by element-wise addition of the original feature map and the transformed feature map, and the feature pyramid module output feature map is integrated.

7. The contrastive learning based document image tampering localization and sanitization localization method of claim 5, wherein, The feature map is decoded by a decoder, and the tamper positioning and desensitization positioning detection result is output. Based on the feature pyramid network, the feature map is enlarged from top to bottom by a bilinear interpolation upsampling layer in the decoder, and the image space is recovered level by level; Between the encoder and the decoder, the first semantic information feature map and the second semantic information feature map are fused by lateral connection; According to the decoding result, the tamper positioning and desensitization positioning detection result is output.

8. A terminal, characterized by comprising: It includes: A processor and a memory, the memory stores a document image tamper positioning and desensitization positioning program based on contrast learning, and the document image tamper positioning and desensitization positioning program based on contrast learning is used to implement the operation of the document image tamper positioning and desensitization positioning method based on contrast learning in any one of claims 1-7 when the processor executes.

9. A computer-readable storage medium, characterized in that, The storage medium stores a document image tamper positioning and desensitization positioning program based on contrast learning, and the document image tamper positioning and desensitization positioning program based on contrast learning is used to implement the operation of the document image tamper positioning and desensitization positioning method based on contrast learning in any one of claims 1-7 when the processor executes.

Images