A qualification certificate image tamper detection method based on a multi-scale suspicious prediction frame

Abstract

The application provides a qualification certificate image tampering detection method based on a multi-scale suspicious prediction frame, and relates to the field of image passive forensics tampering detection. The method comprises the following steps: inputting a to-be-detected image into a first branch, optimizing a predicted suspicious frame and a Mask frame to obtain a prediction frame; inputting the to-be-detected image into a second branch, detecting a tampered area of the image, and outputting a tampered area prediction map; and fusing the obtained prediction frame and the tampered area prediction map to obtain a final mask output. The first branch is a suspicious area detection branch, and the second branch is a tampered area detection branch. The qualification certificate image tampering detection method based on the multi-scale suspicious prediction frame. By combining the multi-scale suspicious prediction frame with the multi-scale feature extraction and positioning of the tampered area, the problems of different sizes of the tampered area, low detection precision and low detection efficiency in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of passive image forensics tampering detection technology, specifically a method for detecting tampering of qualification certificate images based on multi-scale suspicious prediction boxes. Background Technology

[0002] Image tampering detection technology is an forensic technique that analyzes the features of digital images to verify their authenticity and integrity. This technology plays a crucial role in information security, data communication, and many other areas, and is of great significance for ensuring data security and maintaining social stability. Current academic research on image tampering detection primarily focuses on detecting tampering in natural images. However, fake images that commonly pose risks and losses are often qualification certificates and electronic documents. In recent years, many fraudsters have introduced techniques to forge various documents and transfer records to commit fraud, significantly increasing the difficulty of identification. When existing tampering detection methods are directly applied to these new types of images, their accuracy drops drastically, and their detection efficiency is also very low compared to natural images. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] To address the shortcomings of existing technologies, this invention provides a method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes. By combining the use of multi-scale suspicious prediction boxes to define the tampered area with multi-scale feature extraction to locate the tampered area, this method solves the problems of inconsistent tampered area sizes, low detection accuracy, and low detection efficiency in existing technologies.

[0005] (II) Technical Solution

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes, comprising:

[0007] The image to be tested is input into the first branch, and the predicted suspicious boxes and mask boxes are optimized to obtain the predicted boxes;

[0008] The image to be tested is input into the second branch, the tampered area of ​​the image is detected, and the tampered area prediction map is output.

[0009] The obtained prediction bounding box and the tampered area prediction map are fused together to obtain the final mask output;

[0010] The first branch is the suspicious area detection branch;

[0011] The second branch is the tampering area detection branch.

[0012] Preferably, the suspicious region detection branch specifically includes using the Focus module to take values ​​at pixel intervals to obtain four independent feature layers, stacking them to expand the input channel by four times, performing feature extraction through a cross-stage partial network module, and adding a spatial pyramid network structure for feature extraction using max pooling with different pooling kernel sizes; the resulting feature map introduces a cross-stage partial network structure into the feature pyramid structure of the YOLOv5 model to extract features from multiple feature layers, extracting three feature layers: the middle layer, the lower middle layer, and the bottom layer; using these three feature layers to construct the feature pyramid layer, the feature pyramid can fuse features from feature layers of different resolutions; a new transformer prediction head is set to replace the previous prediction head part of the YOLOv5 model, the transformer prediction head can capture global information and rich contextual information to obtain suspicious region boxes.

[0013] Preferably, the step of inputting the image to be tested into the first branch and optimizing the predicted suspicious boxes and mask boxes to obtain the predicted boxes specifically includes:

[0014] The image to be tested is input into the suspicious region detection branch based on the YOLOv5 model for processing. The image to be tested is sliced ​​by the Focus module, and the residual features are learned by the Conv basic convolution module and the cross-stage partial network module. The cross-stage partial network module increases the number of channels through cancat, and the high-level features are extracted and fused by the spatial pyramid pooling module.

[0015] The Neck structure design follows the feature pyramid structure, constructing high-level semantic feature maps at all scales from top to bottom. A new convolutional attention module is added to the Neck structure, which sequentially shortens the attention map along two independent dimensions: channel and space. The attention map is then multiplied with the input feature map to obtain adaptive feature refinement. The main body of the prediction head consists of three transformer-based detectors. The first layer is a multi-head attention layer, and the second layer is a fully connected layer. Using the transformer prediction head increases the ability to capture different information in the image, obtains suspicious prediction boxes, and performs intersection-union (IUU) optimization calculations on these boxes with the mask boxes to obtain the final prediction boxes.

[0016] Preferably, the tampered area detection branch specifically includes:

[0017] The model based on MobileNetV3 uses the SENet attention mechanism to recalibrate the network features of the input image to be tested, selectively emphasizing informative features and suppressing useless features, and using a new activation function to improve detection accuracy while reducing time consumption; a long short-term memory network is used to help identify frequency domain edge features, and the features obtained after calibrating the channel weights are fused with the edge features, and the tampering region prediction map of this branch is output through the mask prediction module.

[0018] Preferably, the step of inputting the image to be tested into the second branch, detecting the tampered areas of the image, and outputting a tampered area prediction map specifically includes:

[0019] The image to be tested is input into the tampering region detection branch based on MobileNetv3. A coarse deconstruction search is performed using a neural architecture search method to select an optimal configuration. SENet is used to automatically obtain the importance of each feature channel, and multi-scale features are extracted while maintaining lightweight operation. The added long short-term memory network learns the correlation between image patches and learns the spatial difference information on the tampering boundary. In the frequency domain, it captures the difference features on the boundary between the tampered region and the real region. Finally, the final tampering region prediction map is obtained through a mask output module composed of convolutional layers.

[0020] Preferably, the step of fusing the obtained prediction bounding box and the tampered region prediction map to obtain the final mask output specifically includes:

[0021] The suspicious prediction boxes obtained from the suspicious region detection branch and the tampered region prediction maps obtained from the tampered region detection branch are continuously regressed under the optimized calculation of the CIoU loss function to obtain the most consistent and accurate tampered detection map, which is then output as the final mask result.

[0022] Preferably, the loss function used in the suspicious region detection branch is the DIoU loss function, and the calculation formula is as follows:

[0023]

[0024]

[0025] Among them, B and B gt Let represent the detection box and the ground truth box, respectively, and c represent the diagonal distance of the smallest rectangle that can simultaneously cover the prediction box and the ground truth box;

[0026] The loss function used in the tampered region detection branch is the cross-entropy loss function, and the calculation formula is as follows:

[0027]

[0028] Among them, G k This represents the true value label, P. k The label represents the predicted output.

[0029] Preferably, the loss function used in the fusion process of the results of the suspicious region detection branch and the tampered region detection branch is the CIoU loss function, and the calculation formula is as follows:

[0030]

[0031] Where α is a parameter used to balance the proportions, and v is a parameter used to describe the consistency of the aspect ratios between the predicted bounding box and the ground truth.

[0032] The overall loss function of the model is:

[0033] L = L DIoU +L CIoU +L CR .

[0034] In a second aspect, a computer-readable storage medium is provided for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the methods described.

[0035] Thirdly, a computing device is provided, comprising:

[0036] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described.

[0037] (III) Beneficial Effects

[0038] This invention presents a method for detecting tampering in qualification certificate images based on multi-scale suspicious prediction boxes. The suspicious region detection branch utilizes its CSP structure and transformer prediction head to extract features more flexibly, adapting to feature scale and being more sensitive to small-region tampering detection. With the help of loss functions and mask boxes, it optimizes and calculates more accurate prediction boxes. The tampering region detection branch, from a lightweight and efficient perspective, uses SENet and LSTM to learn spatial difference information on tampering boundaries, better identifying inconsistencies between real and tampered regions, obtaining a prediction map of the tampered region. The prediction map and prediction box are combined for optimization to obtain the final accurate location of the tampered region. Compared with existing technologies, the method proposed in this invention significantly improves the accuracy of locating tampered regions in qualification certificate images. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the process for detecting image tampering on the qualification certificate of this invention;

[0040] Figure 2 This is a model diagram of the method for detecting image tampering of qualification certificates according to the present invention;

[0041] Figure 3 This is a schematic diagram of the CSP cross-stage network structure in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the SPP spatial pyramid network structure in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the transformer prediction head structure in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0045] Example

[0046] like Figure 1 As shown, this embodiment of the invention provides a method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes, including the following steps:

[0047] S10, input the image to be tested into the suspicious region detection branch and the tampered region detection branch, and process them respectively.

[0048] The first branch is a suspicious region detection branch based on the YOLOv5 model. It uses the Focus module to extract values ​​at pixel intervals, obtaining four independent feature layers. These layers are then stacked to expand the input channels by four times. Feature extraction is then performed through a Cross-Stage Partial Network (CSP) module. To further increase the network's receptive field, a Spatial Pyramid Pooling network (SPP) structure is added, using max pooling with different kernel sizes for feature extraction. (See [link to relevant documentation]). Figure 4The obtained feature maps incorporate a CSP structure within the YOLOv5 Feature Pyramid Networks (FPN) architecture to extract features from multiple feature layers, resulting in three feature layers: an intermediate layer, a lower-middle layer, and a bottom layer. These three effective feature layers are then used to construct the FPN layers, allowing the feature pyramid to fuse features from layers of different resolutions. (See also...) Figure 5 A new Transformer Head (TFE) is set up to replace the previous prediction head part of the YOLO model. The Transformer Head can capture global information and rich contextual information to obtain suspicious region boxes. The suspicious region boxes are further optimized with the pre-processed mask boxes through the DIoU loss function to obtain the final prediction boxes of the suspicious region detection module.

[0049] The second branch is the tampering region detection branch. The input image to be tested is used in a MobileNetV3-based model, where the SENet attention mechanism is employed to recalibrate network features, selectively emphasizing informative features and suppressing less useful ones. A new activation function is used to improve detection accuracy while reducing time consumption. Then, a Long Short-Term Memory (LSTM) network is used to help identify frequency domain edge features. The features obtained after calibrating the channel weights are fused with the edge features to better identify inter-class differences. Finally, the tampering region prediction map of this branch is output through a mask prediction module. The predicted bounding boxes and prediction maps obtained from the two branches are then optimized using the CIoU loss function to obtain the final mask output.

[0050] S20, the image to be tested is input into the suspicious region detection branch. The suspicious region detection branch based on YOLO outputs the suspicious prediction box, and then optimizes it with the pre-processed mask box through the DIoU loss function to obtain the prediction box result of this branch.

[0051] S30, the image to be tested is input into the tampered region detection branch. For the tampered region, multi-scale features are extracted using Mobilenetv3, and LSTM is used to identify the boundary features of the tampered region. Then, the tampered region prediction map is obtained by comparing the inconsistency between the tampered region features and the real region features.

[0052] S40, the suspicious prediction box obtained from the suspicious region detection branch and the tampered region prediction map obtained from the tampered region detection branch are continuously regressed under the optimization calculation of the CIoU loss function to obtain the most consistent and accurate tampered detection map, which is then output as the final mask result.

[0053] In step S10, the image to be tested is input into the suspicious region detection branch and the tampering detection branch for processing, as shown in the schematic diagram below. Figure 2 As shown.

[0054] In step S20, the image to be tested is first input into the suspicious region detection branch. The image is then sliced ​​using the Focus block, obtaining a value for every single pixel in each image, similar to neighbor-based downsampling to obtain four images without information loss. The W and H information are then concentrated into the channel space to obtain a double-downsampled feature map with no information loss. (See [link to relevant documentation]). Figure 3 After CSP structure processing, finer-grained features are extracted. The residual structure in the CSP structure can increase the gradient value of backpropagation between layers, avoiding gradient vanishing due to deepening. Its structure diagram is shown below. Figure 3 As shown. After feature extraction by the preceding structure, the Neck FPN structure further extracts information from the image, enhancing the network's feature fusion capability. A new CBAM attention module is added to the Neck structure, inferring attention maps sequentially along two independent dimensions: channel and space. Then, the attention maps are multiplied with the input feature maps to obtain adaptive feature refinement. Finally, the prediction head consists of three transformer-based detectors. The first layer is a multi-head attention layer, and the second layer is a fully connected layer. Using the transformer prediction head increases the ability to capture different information from the image, obtaining suspicious prediction boxes. These boxes are then compared with the mask boxes using the intersection-union ratio (IoU) optimization calculation to obtain the final prediction box for this branch.

[0055] In step S30, the image to be tested is input into the tampering region detection branch based on MobileNetv3. A coarse deconstruction search is performed using a neural architecture search method to select an optimal configuration. SENet is used to automatically obtain the importance of each feature channel, and multi-scale features are extracted while maintaining lightweight operation. The added long short-term memory network learns the correlation between image patches and learns the spatial difference information on the tampering boundary based on the log-likelihood distance. In the frequency domain, the difference features on the boundary between the tampered region and the real region are captured. Finally, the final tampering region prediction map of this branch is obtained through a mask output module composed of convolutional layers.

[0056] In step S40, the suspected bounding boxes obtained from the suspected region detection branch and the tampered region prediction maps obtained from the tampered region detection branch are continuously regressed under the optimized calculation of the CIoU loss function to obtain the most consistent and accurate tampered detection map, which is then output as the final mask result. Different loss functions are used for different branches and when fusing the results of two branches. The loss function used in the suspected region detection branch is the DIoU loss function, which is scale-insensitive, can directly minimize the distance between the two target boxes, and has a particularly fast regression speed. The calculation formula is as follows:

[0057]

[0058]

[0059] Among them B and B gt represents the detection box and the ground truth box, respectively, and c represents the diagonal distance of the smallest rectangle that can simultaneously cover the prediction box and the ground truth box.

[0060] The loss function used in the tamper region detection branch is the cross-entropy loss function, calculated as follows:

[0061]

[0062] Among them G k This represents the true value label, P. k The label represents the predicted output.

[0063] The loss function used in the fusion of the results from the two branches is the CIoU loss function, which can accelerate convergence while better describing the overlap information. Its calculation formula is as follows:

[0064]

[0065] Where α is a parameter used to balance the proportions, and v is a parameter used to describe the consistency of the aspect ratio between the predicted bounding box and the ground truth.

[0066] The overall loss function of the model is:

[0067] L = L DIoU +L CIoU +L CR .

[0068] Embodiments of this application may be provided as methods or computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application may be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0069] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes, characterized in that, include: The image to be tested is input into the first branch, and the predicted suspicious boxes and mask boxes are optimized to obtain the suspicious prediction boxes; The image to be tested is input into the second branch, the tampered area of ​​the image is detected, and the tampered area prediction map is output. The obtained suspicious prediction boxes and tampered area prediction maps are fused together to obtain the final mask output; The first branch is the suspicious area detection branch; The second branch is the tampered area detection branch; The suspicious region detection branch specifically includes using the Focus module to take values ​​at pixel intervals to obtain four independent feature layers, stacking them to expand the input channel by four times, performing feature extraction through a cross-stage partial network module, and adding a spatial pyramid network structure for feature extraction using max pooling with different pooling kernel sizes; the resulting feature map introduces a cross-stage partial network structure into the feature pyramid structure of the YOLOv5 model to extract features from multiple feature layers, extracting three feature layers: the middle layer, the lower middle layer, and the bottom layer; using these three feature layers to construct the feature pyramid layer, the feature pyramid can fuse features from feature layers of different resolutions; a new transformer prediction head is set to replace the previous prediction head part of the YOLOv5 model. The transformer prediction head can capture global information and rich contextual information to obtain suspicious boxes; The tampered area detection branch specifically includes: The model based on MobileNetV3 uses the SENet attention mechanism to recalibrate the network features of the input image to be tested, selectively emphasizing informative features and suppressing useless features, and using a new activation function to improve detection accuracy while reducing time consumption; a long short-term memory network is used to help identify frequency domain edge features, and the features obtained after calibrating the channel weights are fused with the edge features, and the tampering region prediction map of this branch is output through the mask prediction module.

2. The method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes according to claim 1, characterized in that: The step of inputting the image to be tested into the first branch and optimizing the predicted suspicious bounding boxes and mask boxes to obtain suspicious predicted bounding boxes specifically includes: The image to be tested is input into the suspicious region detection branch based on the YOLOv5 model for processing. The image to be tested is sliced ​​by the Focus module, and the residual features are learned by the Conv basic convolution module and the cross-stage partial network module. The cross-stage partial network module increases the number of channels through cancat, and the high-level features are extracted and fused by the spatial pyramid pooling module. The Neck structure design follows the feature pyramid structure, constructing high-level semantic feature maps at all scales from top to bottom. A new convolutional attention module is added to the Neck structure, which infers attention maps sequentially along two independent dimensions: channel and space. The attention maps are then multiplied with the input feature maps to obtain adaptive feature refinement. The main body of the prediction head consists of three transformer-based detectors. The first layer is a multi-head attention layer, and the second layer is a fully connected layer. Using the transformer prediction head increases the ability to capture different information in the image, obtains suspicious boxes, and performs intersection-union (IUU) calculations with the mask boxes to obtain the final suspicious prediction boxes.

3. The method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes according to claim 2, characterized in that: The step of inputting the image to be tested into the second branch, detecting the tampered areas of the image, and outputting a prediction map of the tampered areas specifically includes: The image to be tested is input into the tampering region detection branch based on MobileNetv3. A coarse deconstruction search is performed using a neural architecture search method to select an optimal configuration. SENet is used to automatically obtain the importance of each feature channel, and multi-scale features are extracted while maintaining lightweight operation. The added long short-term memory network learns the correlation between image patches and learns the spatial difference information on the tampering boundary. In the frequency domain, it captures the difference features on the boundary between the tampered region and the real region. Finally, the final tampering region prediction map is obtained through a mask output module composed of convolutional layers.

4. The method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes according to claim 1, characterized in that: The process of fusing the obtained suspicious prediction boxes and tampered area prediction maps to obtain the final mask output specifically includes: The suspicious prediction boxes obtained from the suspicious region detection branch and the tampered region prediction maps obtained from the tampered region detection branch are continuously regressed under the optimized calculation of the CIoU loss function to obtain the most consistent and accurate tampered detection map, which is then output as the final mask result.

5. The method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes according to claim 1, characterized in that: The loss function used in the suspicious region detection branch is the DIoU loss function, and the calculation formula is as follows: in, and These represent the predicted bounding box and the ground truth bounding box, respectively. This represents the diagonal distance of the smallest rectangle that can simultaneously cover both the predicted bounding box and the ground truth bounding box. ρ 2 This indicates the calculation of the Euclidean distance between two center points; The loss function used in the tampered region detection branch is the cross-entropy loss function, and the calculation formula is as follows: in, This represents the true value label. The label representing the predicted output. k Represents the number of categories.

6. The method for detecting image tampering of qualification certificates based on multi-scale suspicious prediction boxes according to claim 5, characterized in that: The loss function used in the fusion process of the results of the suspicious region detection branch and the tampered region detection branch is the CIoU loss function, and the calculation formula is as follows: in, It is a parameter used to balance the proportions. This is a parameter used to describe the consistency of the aspect ratio between the predicted bounding box and the ground truth bounding box. ρ 2 This indicates the calculation of the Euclidean distance between two center points; The overall loss function of the model is: 。 7. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods according to claims 1-6.

8. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods according to claims 1-6.

Images