Certificate image tampering detection method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a certificate image tampering detection method and device, a storage medium and electronic equipment. According to the method, certificate image tampering detection is realized by utilizing a certificate image tampering detection model. According to the model, a multi-modal fusion image is used as a sample image, the feature performance of a tampered area in different modals is learned, in the training process, based on the characteristic that the edge style of the tampered area is different from that of an original certificate image, cross fusion learning is carried out on local features and edge features of the tampered area and global features of the certificate image, and the tampered area is obtained. Therefore, the model has a precise recognition capability on certificate image tampering attacks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a document image tampering detection method, device, storage medium and electronic equipment. Background Art

[0002] In the field of identity authentication based on ID card image recognition, attackers usually attack by tampering with real ID card images. With the continuous development of tampering technology, tampered ID card images are becoming more and more sophisticated. The current tampered ID card image recognition scheme is difficult to cope with attacks of sophisticated tampered ID card images. Summary of the invention

[0003] One or more embodiments of the present specification provide a document image tampering detection method, device, storage medium and electronic device, which can accurately and effectively identify refined tampered document images.

[0004] In a first aspect, a training method for a document image tampering detection model is provided, wherein the document image tampering detection model comprises: an edge detection branch, an element style detection branch, a global detection branch, a feature fusion module and a prediction head; the method comprises:

[0005] Selecting multiple target certificates, and obtaining multimodal fusion images of the target certificates as sample images, wherein the sample images include tampered images and non-tampered images, and labels of the sample images are used to indicate whether the multimodal fusion images are tampered images;

[0006] Inputting the sample image into the edge detection branch to obtain a tampered edge feature image;

[0007] Inputting the sample image into the element style detection branch to obtain a tampered element feature image;

[0008] Inputting the sample image into the global detection branch to obtain a global feature image;

[0009] Using the feature fusion module to perform feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image;

[0010] The fused feature image is input into the prediction head, and a loss function is constructed based on the difference between the prediction result of the prediction head and the label of the sample image to update the parameters of the document image tampering detection model.

[0011] As an optional implementation manner of the method of the first aspect, the method further includes:

[0012] Constructing a first training sample set according to the sample image, wherein the first training sample set includes a first document image sample that has been tampered with, and a first label of the first document image sample is an edge mask image of the tampered area;

[0013] The first document image sample is input into the edge detection branch, and a first loss function is constructed based on the difference between the detection result of the edge detection branch on the first document image sample and the first label to train the edge detection branch to recognize and locate the edge of the tampered area.

[0014] As an optional implementation manner of the method of the first aspect, the element style detection branch includes an element extraction branch and a style detection branch; inputting the sample image into the element style detection branch to obtain a tampered element feature image specifically includes:

[0015] Input the sample image into the element extraction branch to obtain the document element;

[0016] For each element of the sample image, the style detection branch is used to detect whether the element and other elements belong to the sample image; if so, the element is determined to be a non-tampered element, otherwise, the element is determined to be a tampered element, and the tampered element feature image of the element is output.

[0017] Specifically, the method further includes:

[0018] constructing a second training sample set according to the sample image, wherein the second training sample set includes a second certificate image sample that has been tampered with, and a second label of the second certificate image sample includes an element region mask label and an element category label, and the element category label is used to indicate whether the element is a tampered element;

[0019] Inputting the second certificate image sample into the element extraction branch to obtain extracted certificate elements;

[0020] Determining a second loss function according to the document element and the element region mask label;

[0021] Based on the document element of the second document image sample and the element category label of the document element, construct a positive and negative sample pair, wherein the positive sample pair includes two non-tampered elements, and the negative sample pair includes one non-tampered element and one tampered element;

[0022] Input the positive and negative sample pairs into the style detection branch, adopt a contrastive learning strategy, and construct a third loss function;

[0023] The second loss function and the third loss function are used to train the element style detection branch to be able to identify and locate tampered elements.

[0024] As an optional implementation manner of the method of the first aspect, using the feature fusion module to perform feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image specifically includes:

[0025] The feature fusion module is used to perform pairwise cross feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain the fused feature image.

[0026] Specifically, the prediction head is used to predict the probability score of the sample image being a tampered image based on each set of fused feature images of two-by-two cross-feature fusions; and a loss function is constructed based on the difference between the prediction result of the prediction head and the label of the sample image, which specifically includes:

[0027] Performing a weighted summation on all probability scores of the sample image to obtain a total probability score of the sample image;

[0028] The loss function is constructed according to the difference between the total probability score and the label of the sample image.

[0029] As an optional implementation manner of the method of the first aspect, obtaining the multimodal fusion image of the target document specifically includes:

[0030] Acquire a normal frame image, a flash frame image, and a difference image between the normal frame image and the flash frame image of the target document;

[0031] The normal frame image, the flash frame image and the difference image are fused to obtain a multimodal fused image of the target certificate.

[0032] In a second aspect, a document image tampering detection method is provided, comprising:

[0033] Obtain the document image to be detected;

[0034] The certificate image to be detected is input into the certificate image tampering detection model pre-trained according to the above-mentioned certificate image tampering detection model training method, and the tampering detection result of the certificate image to be detected is determined according to the output result of the certificate image tampering detection model.

[0035] In a third aspect, a training device for a document image tampering detection model is provided, wherein the document image tampering detection model comprises: an edge detection branch, an element style detection branch, a global detection branch, a feature fusion module and a prediction head; the training device comprises:

[0036] A first data acquisition module is configured to acquire a multimodal fusion image of a target document as a sample image, wherein the sample image includes a tampered image and a non-tampered image, and a label of the sample image is used to indicate whether the multimodal fusion image is a tampered image;

[0037] The data processing module is configured to input the sample image into the edge detection branch to obtain a tampered edge feature image; input the sample image into the element style detection branch to obtain a tampered element feature image; input the sample image into the global detection branch to obtain a global feature image; use the feature fusion module to perform feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image; input the fused feature image into the prediction head, and construct a loss function based on the difference between the prediction result of the prediction head and the label of the sample image to update the parameters of the document image tampering detection model.

[0038] As an optional implementation manner of the device according to the third aspect, the data processing module is further used for:

[0039] Constructing a first training sample set according to the sample image, wherein the first training sample set includes a first document image sample that has been tampered with, and a first label of the first document image sample is an edge mask image of the tampered area;

[0040] The first document image sample is input into the edge detection branch, and a first loss function is constructed based on the difference between the detection result of the edge detection branch on the first document image sample and the first label to train the edge detection branch to recognize and locate the edge of the tampered area.

[0041] As an optional implementation manner of the device according to the third aspect, the element style detection branch includes an element extraction branch and a style detection branch; and the data processing module is specifically used for:

[0042] Input the sample image into the element extraction branch to obtain the document element;

[0043] For each element of the sample image, the style detection branch is used to detect whether the element and other elements belong to the sample image; if so, the element is determined to be a non-tampered element, otherwise, the element is determined to be a tampered element, and the tampered element feature image of the element is output.

[0044] Specifically, the data processing module is also used for:

[0045] constructing a second training sample set according to the sample image, wherein the second training sample set includes a second certificate image sample that has been tampered with, and a second label of the second certificate image sample includes an element region mask label and an element category label, and the element category label is used to indicate whether the element is a tampered element;

[0046] Inputting the second certificate image sample into the element extraction branch to obtain extracted certificate elements;

[0047] Determining a second loss function according to the document element and the element region mask label;

[0048] Based on the document element of the second document image sample and the element category label of the document element, construct a positive and negative sample pair, wherein the positive sample pair includes two non-tampered elements, and the negative sample pair includes one non-tampered element and one tampered element;

[0049] Input the positive and negative sample pairs into the style detection branch, adopt a contrastive learning strategy, and construct a third loss function;

[0050] The second loss function and the third loss function are used to train the element style detection branch to be able to identify and locate tampered elements.

[0051] As an optional implementation manner of the device described in the third aspect, the data processing module is specifically used to use the feature fusion module to perform pairwise cross-feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain the fused feature image.

[0052] Specifically, the prediction head of the data processing module is specifically used to predict the probability score of the sample image being a tampered image based on the fused feature image of each group of pairwise cross-feature fusions; the first data processing module is specifically used to perform weighted summation on all probability scores of the sample image to obtain the total probability score of the sample image; and construct the loss function based on the difference between the total probability score and the label of the sample image.

[0053] As an optional implementation manner of the device described in the third aspect, the first data acquisition module is specifically used to obtain a normal frame image, a flash frame image, and a difference image between the normal frame image and the flash frame image of the target document; and perform image fusion on the normal frame image, the flash frame image, and the difference image to obtain a multimodal fusion image of the target document.

[0054] In a fourth aspect, a document image tampering detection device is provided, the detection device comprising:

[0055] A second data acquisition module is configured to acquire an image of a document to be detected;

[0056] The detection module is configured to input the certificate image to be detected into the certificate image tampering detection model pre-trained according to the above-mentioned certificate image tampering model training method, and determine the tampering detection result of the certificate image to be detected according to the output result of the certificate image tampering detection model.

[0057] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is run on an electronic device, the electronic device executes the above-mentioned document image tampering model training method, or executes the above-mentioned document image tampering detection method.

[0058] In a sixth aspect, an electronic device is provided, including:

[0059] at least one memory for storing a program;

[0060] At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above-mentioned document image tampering model training method, or execute the above-mentioned document image tampering detection method.

[0061] The beneficial effect of the training method of the document image tampering model described in the embodiments of this specification is that the method uses multimodal fusion images as sample images to enable the model to learn the characteristic performance of the tampered area in different modes. During the training process, based on the different characteristics of the edge style of the tampered area and the original document image, the local features, edge features of the tampered area and the global features of the document image are cross-fused and learned, thereby improving the model's recognition ability for tampered document image attacks.

[0062] The training device for the certificate image tampering detection model, the certificate image tampering detection method and the device described in the embodiments of this specification also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 The structural diagram of a certificate image tampering detection model described in the embodiments of this specification is exemplarily shown.

[0065] Figure 2 The schematic diagram of the structure of an edge detection branch described in the embodiment of this specification is exemplarily shown.

[0066] Figure 3 The structural diagram of an element style detection branch described in the embodiments of this specification is exemplarily shown.

[0067] Figure 4 The structural diagram of the document image tampering detection model described in the embodiments of this specification in an implementation scenario is exemplarily shown.

[0068] Figure 5 The structural diagram of a cross-feature fusion module described in the embodiment of this specification is exemplarily shown.

[0069] Figure 6 The flowchart of a training method for a document image tampering detection model described in an embodiment of this specification is exemplarily shown.

[0070] Figure 7 The structural diagram of a training device for a document image tampering detection model described in an embodiment of this specification is exemplarily shown.

[0071] Figure 8 The flowchart of a method for detecting tampering of a certificate image described in an embodiment of this specification is exemplarily shown.

[0072] Fig. 9 The structure diagram of a device for detecting tampering of a certificate image described in an embodiment of this specification is exemplarily shown.

[0073] Fig.10 The structural diagram of an electronic device described in an embodiment of this specification is exemplarily shown. DETAILED DESCRIPTION

[0074] First of all, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise.

[0075] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0076] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0077] In the field of identity authentication based on document image recognition, attackers usually attack by tampering with the real document image. Due to the particularity of the document, the tampered location is usually an element with information and identity information, such as face image, document number, address, date, name, etc. Most of the current tampering recognition schemes are based on the difference between the local features of the tampered area and the overall features of the document image. However, with the development of tampering technology, the local features of the tampered area have become closer and closer to the overall features of the document image, which makes the current recognition model insensitive to such refined tampering attacks.

[0078] In view of this, one or more embodiments of the present specification propose a document image tampering detection method, device, storage medium and electronic device, which can effectively improve the accuracy of document image tampering detection.

[0079] The document image tampering detection method, device, storage medium and electronic device described in one or more embodiments of this specification will be further described in detail below in conjunction with the drawings and specific embodiments of the specification, but this detailed description does not constitute a limitation on the embodiments of this specification.

[0080] Please refer to Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of a document image tampering detection model proposed in one or more embodiments of this specification. The input of the document image tampering detection model is a multimodal fusion image of a target document, that is, a sample image, and the output of the document image tampering detection model is the probability distribution of the sample image being a tampered image.

[0081] like Figure 1 As shown in the figure, the document image tampering detection model mainly includes: edge detection branch, element style detection branch, global detection branch, feature fusion module and prediction head. Among them, the edge detection branch is used to extract the edge features of the tampered area from the input sample image. The element style detection branch is used to extract the tampered element area features from the input sample image. The global detection branch is used to extract the global features of the input sample image. The feature fusion module is used to fuse the above-mentioned edge features, tampered element area features and global features. The prediction head predicts the input fusion features to obtain the predicted probability distribution results.

[0082] In some embodiments, the edge detection branch can be implemented using an edge detection model based on Hough transform or a target detection model with edge detection function. The following takes the boundary-guided camouflage target detection model (Boundaryguided Network, BGNet) as an example to illustrate the implementation principle of the edge detection branch.

[0083] like Figure 2 As shown, BGNet consists of a backbone network, an edge-aware module (EAM), an edge-guided feature module (EFM), and a context aggregation module (CAM).

[0084] The backbone network adopts the Res2Net-50 structure to extract features of the input image step by step. The obtained features are arranged from low level to high level, namely f1, f2, f3, f4 and f5.

[0085] The boundary-aware module EAM mines object-related edge semantic information from low-level features f2 containing local edge details and high-level features f5 containing global location information under object boundary supervision. e .

[0086] The edge-guided feature module (EFM) integrates the edge semantic information from the edge-aware module (EAM) with the multi-level features (f2, f3, f4, f5) extracted by the backbone network to guide feature learning and thus enhance boundary representation.

[0087] The context aggregation module CAM gradually aggregates multi-level fusion features in a top-down manner and obtains a prediction result based on each aggregated feature.

[0088] In this embodiment, the edge semantic information f output by the edge perception module EAM is selected e As the edge features mentioned above.

[0089] In the case of document tampering, the tampering is usually done on text or face images, rather than random tampering. Therefore, the object to be detected by the element style detection branch is the face element and text element of the sample image. The element style detection branch needs to locate the face element and text element of the sample image before detection, and then detect whether these elements are tampered elements. For this purpose, in some embodiments, the above-mentioned element style detection branch can be used as follows Figure 3 The network structure described is implemented. Please refer to Figure 3 , the element style detection branch includes an element extraction branch and a style detection branch. Among them, the element extraction branch is used to extract elements from the sample image. These elements can be text elements or image elements. Correspondingly, the element extraction method can adopt face detection or text line detection. The style detection branch is used to compare each element of the sample image with other elements to determine whether the element and the compared element belong to the same sample image. If so, the element is determined to be a non-tampered element, otherwise, the element is determined to be a tampered element, and the tampered element feature image of the element is output.

[0090] In some implementations, the feature fusion module may directly concatenate the features to be fused to obtain the fused features.

[0091] In some embodiments, the feature fusion module may also adopt a cross-feature fusion method to obtain a fused feature image. Figure 4 , Figure 4 FIG. 4 is a schematic diagram showing the structure of the above-mentioned document image tampering detection model in an implementation scenario. Figure 4 As shown, the feature fusion module may include a first cross-feature fusion module, a second cross-feature fusion module and a third cross-feature fusion module. The first cross-feature fusion module is used to perform cross-feature fusion on the tampered edge feature image and the tampered element feature image to obtain a first fused feature image. The second cross-feature fusion module is used to perform cross-feature fusion on the tampered edge feature image and the global feature image to obtain a second fused feature image. The third cross-feature fusion module is used to perform cross-feature fusion on the tampered element feature image and the global feature image to obtain a third fused feature image.

[0092] The first cross feature fusion module, the second cross feature fusion module and the third cross feature fusion module have the same structure. The structure of each cross feature fusion module is as follows: Figure 5 As shown, it includes a spatial feature shrinkage module SFS, an iterative cross-modal feature enhancement module ICFE and a bimodal feature fusion module Fusion.

[0093] The spatial feature shrinkage module SFS is used to shrink the two input modal features F to be fused. Rand F T Compress in the spatial dimension to obtain the modal feature F R The feature map T R and modal characteristics F T The feature map T T .

[0094] Feature map T R and feature map T T The CFE module in the iterative cross-modal feature enhancement module ICFE is input to perform cross-modal feature enhancement. R and feature map T T Before entering the CFE module, each feature map needs to be flattened into a set of tags and a learnable position embedding is added. The position embedding is a trainable parameter used to encode the spatial information between different tags. After that, we can get a set of feature maps T with position embedding R and T T As the input of the CFE module.

[0095] The CFE module takes the input feature map T with position embedding R and T T , one of the feature maps T with position embedding R Projected to matrices VT and KT respectively, another feature map T with position embedding T Projected onto the matrix QR, the specific process is as follows:

[0096] V T =T T W V , K T =T T W K , Q R =T R W Q

[0097] Among them, W V , W K and W Q represents the weight matrix.

[0098] Then, the CFE module builds the correlation matrix through dot product operation, and then uses the softmax function to normalize the correlation matrix to represent the feature map T R and T T The similarity between:

[0099]

[0100] Among them, D K is a constant value used for normalization, Z TRepresents the normalized result.

[0101] Next, the CFE module transforms the vector Z T Reprojected back to the original space through nonlinear transformation and added to the input sequence through residual connection to obtain vector T T ′:

[0102] T T ′=α·Z T W O +β·T T

[0103] Among them, α and β represent learnable parameters, W O represents the weight matrix.

[0104] Finally, the CFE module uses a two-layer fully connected feedforward network (FFN) to further refine the global information, improve the robustness and accuracy of the model, and output enhanced features

[0105]

[0106] Among them, γ and δ represent learnable parameters.

[0107] Since the feature map T R and feature map T T Usually not completely aligned, therefore, a dual CFE module structure is adopted here to obtain complementary information to enhance the feature map T R and feature map T T , no parameters are shared between the two CFE modules.

[0108] The bimodal feature fusion module Fusion fuses the two enhanced features output by the dual CFE module to obtain the above-mentioned fused feature image (F fused ).

[0109] Please refer to Figure 4, corresponding to the above-mentioned feature fusion module, the above-mentioned prediction head is specifically used to predict the probability score of the sample image being a tampered image based on the fused feature image of each group of two-by-two cross-feature fusions. Specifically, the above-mentioned prediction head includes a first prediction head, a second prediction head, a third prediction head and a weighting module. The first prediction head is used to predict the first fused feature image output by the first cross-feature fusion module to obtain the first probability score of the sample image being a tampered image. The second prediction head is used to predict the second fused feature image output by the second cross-feature fusion module to obtain the second probability score of the sample image being a tampered image. The third prediction head is used to predict the third fused feature image output by the third cross-feature fusion module to obtain the third probability score of the sample image being a tampered image. The weighting module is used to perform a weighted summation of the first probability score, the second probability score and the third probability score to obtain the total probability score of the sample image being a tampered image.

[0110] Based on the structure of the above-mentioned document image tampering detection model, it can be seen that the model can cross-fuse the local features, edge features of the tampered area and the global features of the document image, thereby improving the model's recognition ability for tampered document image attacks.

[0111] Based on the above-mentioned document image tampering detection model, this embodiment proposes a training method for the document image tampering detection model. Figure 6 , Figure 6 A flowchart of a method for training a document image tampering detection model is exemplarily shown.

[0112] like Figure 6 As shown, the training method includes steps S600 to S610.

[0113] S600: Select multiple target certificates and obtain multimodal fusion images of the target certificates as sample images.

[0114] The target certificate mentioned above is a certificate that can be used for identity authentication, such as an ID card, a driver's license, etc.

[0115] The above-mentioned multimodal fusion image of the target document refers to a fusion image of the document images of the target document under different acquisition states. The document image collected under each acquisition state is a document modal image. These document modal images can be selected according to needs, and this embodiment does not impose any restrictions on this.

[0116] For example, in a scenario where a document is tampered with by mapping, the mapping area may not be obvious in the normal frame image, but in the flash frame, since the material of the mapping is different from that of the document, the reflective characteristics of the mapping area are different from the reflective characteristics of the document itself. In this case, the mapping area can show a more obvious distinguishing feature in the flash frame. In this scenario, the normal frame image and the flash frame image of the target document can be collected for image fusion to obtain a multimodal fusion image of the target document. In addition, the difference image after the normal frame image and the flash frame image are aligned can be added to highlight the tampered area, that is, the normal frame image and the flash frame image of the target document and the difference image after the normal frame image and the flash frame image are aligned are fused to obtain a multimodal fusion image of the target document.

[0117] There are many ways to obtain the multimodal fusion image through image fusion, such as a multimodal image fusion method based on a deep multi-scale multimodal convolutional dictionary learning network Deep MCDL (Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network), a multimodal image fusion method based on dynamic feature enhancement Fusion Mamba (Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba), a multimodal image fusion method DDFM (Delousing Diffusion Model for Multi-Modality Image Fusion), etc. This embodiment does not limit this.

[0118] The above-mentioned sample images include tampered images and non-tampered images, and the labels of the sample images are used to indicate whether the multimodal fusion image is a tampered image.

[0119] S602: Input the sample image into the edge detection branch to obtain a tampered edge feature image.

[0120] The above-mentioned edge detection branch is used to identify and locate the edge of the tampered area. In order to train the edge detection branch's ability to identify and locate the edge of the tampered area, the edge detection branch can be trained by constructing a first training sample set and a first loss function.

[0121] It should be noted that the edge detection branch can be trained in advance using the above training method, and then fine-tuned during the training of the entire document image tampering recognition model. The edge detection branch can also be trained using the above first loss function during the training of the entire document image tampering recognition model.

[0122] The training process of the edge detection branch can be done in the following way:

[0123] First, a tampered first document image sample may be selected from the sample image to form a first training sample set, and the first label of the first document image sample is an edge mask image of the tampered area.

[0124] Next, the first document image sample is input into the edge detection branch to obtain a detection result of the edge detection branch on the first document image sample.

[0125] Finally, according to the difference between the detection result of the edge detection branch on the first document image sample and the first label, a first loss function is constructed to train the edge detection branch to recognize and locate the edge of the tampered area.

[0126] S604: Input the sample image into the element style detection branch to obtain a tampered element feature image.

[0127] In some embodiments, the element style detection branch includes an element extraction branch and a style detection branch, wherein the element extraction branch is used to extract the document element, and the style detection branch is used to determine whether the extracted element is a tampered element. Specifically, the style detection branch needs to detect for each element of the input sample image whether the element and other elements belong to the sample image; if so, the element is determined to be a non-tampered element, otherwise, the element is determined to be a tampered element, and the tampered element feature image of the element is output.

[0128] In order to train the element style detection branch so that it has the above capabilities, the contrastive learning strategy can be used to train the element style detection branch. Specifically, it is necessary to construct positive and negative sample pairs, where the positive sample pairs are elements of the same source (i.e., belonging to the same document image), and the negative sample pairs are elements of different sources (i.e., belonging to different document images). The element style detection branch is trained using positive and negative sample pairs and contrastive learning strategies, so that the element style detection branch can identify elements of different sources from other elements, that is, tampered elements.

[0129] It should be noted that the element style detection branch can be pre-trained using the above training method, and then the parameters of the element style detection branch can be fine-tuned during the training of the entire document image tampering recognition model. The element style detection branch can also be trained using the above contrast learning strategy during the training of the entire document image tampering recognition model.

[0130] The training process of the element style detection branch can be done in the following way:

[0131] A second training sample set is constructed according to the sample image, the second training sample set includes tampered second document image samples, the second labels of the second document image samples include element region mask labels and element category labels, and the element category labels are used to characterize whether the element is a tampered element.

[0132] The second document image sample is input into the element extraction branch to obtain the extracted document element, and the second loss function is determined according to the document element and the element area mask label.

[0133] Based on the document elements of the second document image sample and the element category labels of the document elements, positive and negative sample pairs are constructed, wherein the positive sample pair includes two non-tampered elements and the negative sample pair includes one non-tampered element and one tampered element.

[0134] The positive and negative samples are input into the style detection branch, and a contrastive learning strategy is adopted to construct the third loss function.

[0135] The second loss function and the third loss function are used to train the element style detection branch to identify and locate tampered elements.

[0136] S606: Input the sample image into the global detection branch to obtain a global feature image.

[0137] The above-mentioned global detection branch can be implemented by using an image feature encoder. Specifically, the image feature encoder can be implemented by using a convolutional neural network.

[0138] S608: Using a feature fusion module to fuse the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image.

[0139] In some implementations, the feature fusion module may directly concatenate the tampered edge feature image, the tampered element feature image, and the global feature image to obtain a fused feature.

[0140] In some embodiments, the feature fusion module may also adopt a cross-feature fusion method to obtain a fused feature image. Figure 4 The feature fusion module may include a first cross-feature fusion module, a second cross-feature fusion module and a third cross-feature fusion module. The first cross-feature fusion module is used to perform cross-feature fusion on the tampered edge feature image and the tampered element feature image to obtain a first fused feature image. The second cross-feature fusion module is used to perform cross-feature fusion on the tampered edge feature image and the global feature image to obtain a second fused feature image. The third cross-feature fusion module is used to perform cross-feature fusion on the tampered element feature image and the global feature image to obtain a third fused feature image.

[0141] S610: Input the fused feature image into the prediction head, and construct a loss function based on the difference between the prediction result of the prediction head and the label of the sample image to update the parameters of the document image tampering detection model.

[0142] Please refer to Figure 4 , corresponding to the above-mentioned feature fusion module, the above-mentioned prediction head may specifically include a first prediction head, a second prediction head, a third prediction head and a weighting module. The first prediction head is used to predict the first fused feature image output by the first cross-feature fusion module to obtain a first probability score that the sample image is a tampered image. The second prediction head is used to predict the second fused feature image output by the second cross-feature fusion module to obtain a second probability score that the sample image is a tampered image. The third prediction head is used to predict the third fused feature image output by the third cross-feature fusion module to obtain a third probability score that the sample image is a tampered image. The weighting module is used to perform a weighted summation of the first probability score, the second probability score and the third probability score to obtain a total probability score that the sample image is a tampered image, that is, S=w1·S1+w2·S2+w3·S3, where w1, w2, and w3 are all weight coefficients, S1 is the first probability score, S2 is the second probability score, and S3 is the third probability score.

[0143] Corresponding to the training method of the above-mentioned document image tampering detection model, this embodiment also provides a training device for the document image tampering detection model. Figure 7 , Figure 7 The schematic diagram of the structure of a training device for a document image tampering detection model is shown as an example. It should be noted that: Figure 6 The training method of the document image tampering detection model shown can rely on Figure 7 The training device shown implements, but is not limited to, this training device.

[0144] like Figure 7 As shown, the training device of the document image tampering detection model includes:

[0145] The first data acquisition module 701 is configured to acquire a multimodal fusion image of a target document as a sample image. The sample image includes a tampered image and a non-tampered image, and the label of the sample image is used to indicate whether the multimodal fusion image is a tampered image.

[0146] The data processing module 702 is configured to input the sample image into the edge detection branch to obtain a tampered edge feature image; input the sample image into the element style detection branch to obtain a tampered element feature image; input the sample image into the global detection branch to obtain a global feature image; use the feature fusion module to perform feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image; input the fused feature image into the prediction head, and construct a loss function based on the difference between the prediction result of the prediction head and the label of the sample image to update the parameters of the document image tampering detection model.

[0147] Regarding the above-mentioned first data acquisition module 701, the first data acquisition module 701 is specifically used to obtain a normal frame image, a flash frame image, and a difference image between the normal frame image and the flash frame image of the target document, and then fuse the normal frame image, the flash frame image, and the difference image to obtain a multimodal fusion image of the target document.

[0148] With respect to the first data acquisition module 701, the first data acquisition module 701 is also used to train the edge detection branch to recognize and locate the edge of the tampered area. The training process is as follows:

[0149] Constructing a first training sample set according to the sample image, the first training sample set including a tampered first document image sample, wherein a first label of the first document image sample is an edge mask image of the tampered area;

[0150] The first document image sample is input into the edge detection branch, and a first loss function is constructed according to the difference between the detection result of the edge detection branch on the first document image sample and the first label to train the edge detection branch to recognize and locate the edge of the tampered area.

[0151] With respect to the first data acquisition module 701, the first data processing module 701 is also used to train the element style detection branch to have the ability to identify and locate tampered elements. The training process is as follows:

[0152] Constructing a second training sample set according to the sample image, the second training sample set includes a tampered second document image sample, the second label of the second document image sample includes an element region mask label and an element category label, and the element category label is used to indicate whether the element is a tampered element;

[0153] Inputting the second document image sample into the element extraction branch to obtain extracted document elements;

[0154] Determining a second loss function according to the document element and the element region mask label;

[0155] Based on the document elements of the second document image sample and the element category labels of the document elements, constructing positive and negative sample pairs, wherein the positive sample pair includes two non-tampered elements and the negative sample pair includes one non-tampered element and one tampered element;

[0156] The positive and negative samples are input into the style detection branch, and the contrastive learning strategy is adopted to construct the third loss function;

[0157] The second loss function and the third loss function are used to train the element style detection branch to identify and locate tampered elements.

[0158] With respect to the above-mentioned data processing module 702, the feature fusion module deployed in the data processing module 702 is specifically used to perform cross-feature fusion of the tampered edge feature image, the tampered element feature image and the global feature image, to obtain a fused feature image. The prediction head deployed in the data processing module 702 is specifically used to predict the probability score of the sample image being a tampered image based on each group of cross-feature fused fused feature images, and to perform weighted summation of all probability scores of the sample image to obtain the total probability score of the sample image; finally, a loss function is constructed based on the difference between the total probability score and the label of the sample image.

[0159] For the above-mentioned training device of the document image tampering detection model, taking the module as an example of a software functional unit, the first data acquisition module 701 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first data acquisition module 701 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0160] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0161] As an example of a hardware functional unit, the first data acquisition module 701 may include at least one computing device, such as a server, etc. Alternatively, the first data acquisition module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0162] The multiple computing devices included in the first data acquisition module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the first data acquisition module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first data acquisition module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0163] In other embodiments, the first data acquisition module 701 can be used to execute any step in the training method of the above-mentioned document image tampering detection model, and the data processing module 702 can be used to execute any step in the training method of the above-mentioned document image tampering detection model. The steps that the first data acquisition module 701 and the data processing module 702 are responsible for implementing can be specified as needed, and the first data acquisition module 701 and the data processing module 702 respectively implement different steps in the training method of the above-mentioned document image tampering detection model to achieve all the functions of the training device of the above-mentioned document image tampering detection model.

[0164] In this implementation, the training device can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to realize the training function of the training device of the document image tampering detection model.

[0165] Based on the certificate image tampering detection model trained by the above-mentioned certificate image tampering detection model training method, this embodiment also proposes a certificate image tampering detection method.

[0166] like Figure 8 As shown, in some embodiments, the document image tampering detection method may include steps S800 to S802:

[0167] S800: Acquire the document image to be detected.

[0168] The document image to be detected may be a document image of a single modality or a multi-modal fusion image of the same document, which is not limited in this embodiment.

[0169] S802: Inputting the certificate image to be detected into the certificate image tampering detection model, and determining the tampering detection result of the certificate image to be detected according to the output result of the certificate image tampering detection model.

[0170] Specifically, after the document image to be detected is input into the document image tampering detection model, the document image tampering detection model will output the probability score of the document image to be detected being a tampered image. If the probability score is greater than a preset threshold, it can be determined that the document image to be detected is a tampered image, otherwise, it is determined that the document image to be detected is a non-tampered image.

[0171] Corresponding to the above-mentioned document image tampering detection method, this embodiment also provides a document image tampering detection device. Fig. 9 , Fig. 9 An example of a document image tampering detection device is shown, which can be used to implement the document image tampering detection method described above. It should be noted that the document image tampering detection method described in one or more embodiments of the present application can rely on Fig. 9 The detection device shown implements, but is not limited to, this device.

[0172] like Fig. 9 As shown, the detection device comprises:

[0173] The second data acquisition module 901 is configured to acquire the document image to be detected.

[0174] The detection module 902 is configured to input the certificate image to be detected into the certificate image tampering detection model, and determine the tampering detection result of the certificate image to be detected according to the output result of the certificate image tampering detection model.

[0175] With respect to the second data acquisition module 901 , the document image to be detected acquired by the second data acquisition module 901 may be a document image of a single modality or a multi-modality fusion image of the same document, and this embodiment does not impose any limitation on this.

[0176] For the above-mentioned detection module 902, after the certificate image tampering detection model outputs the probability score that the certificate image to be detected is a tampered image, the detection module 902 compares the probability score with a preset threshold. If the probability score is greater than the preset threshold, it can be determined that the certificate image to be detected is a tampered image; otherwise, it is determined that the certificate image to be detected is a non-tampered image.

[0177] For the above-mentioned document image tampering detection device, taking a module as an example of a software functional unit, the second data acquisition module 901 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the second data acquisition module 901 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0178] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0179] As an example of a hardware functional unit, the second data acquisition module 901 may include at least one computing device, such as a server, etc. Alternatively, the second data acquisition module 901 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0180] The multiple computing devices included in the second data acquisition module 901 can be distributed in the same region or in different regions. The multiple computing devices included in the second data acquisition module 901 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the second data acquisition module 901 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0181] In other embodiments, the second data acquisition module 901 can be used to execute any step in the above-mentioned document image tampering detection method, and the detection module 902 can be used to execute any step in the above-mentioned document image tampering detection method. The steps that the second data acquisition module 901 and the detection module 902 are responsible for implementing can be specified as needed, and the second data acquisition module 901 and the detection module 902 respectively implement different steps in the above-mentioned document image tampering detection method to achieve all the functions of the above-mentioned document image tampering detection device.

[0182] In this implementation, the document image tampering detection device can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to implement the document image tampering detection function.

[0183] In some embodiments, an electronic device is also provided. Fig.10, the electronic device includes: a bus 1001, a processor 1002, a memory 1003 and a communication interface 1004. The processor 1002, the memory 1003 and the communication interface 1004 communicate with each other through the bus 1001. The electronic device can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the electronic device.

[0184] The bus 1001 may be a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The bus 1001 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 1001 may include a path for transmitting information between various components of the electronic device (for example, the processor 1002, the memory 1003, and the communication interface 1004).

[0185] The processor 1002 may include any one or more processors such as a CPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0186] The memory 1003 may include a volatile memory, such as a random access memory (RAM). The memory 1003 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0187] The memory 1003 stores executable program codes, and the processor 1002 executes the executable program codes to respectively implement the functions of the aforementioned first data acquisition module 701 and the data processing module 702, that is, to implement the functions of the aforementioned training device for the document image tampering detection model, thereby implementing the aforementioned training method for the document image tampering detection model. Alternatively, the processor 1002 executes the executable program codes to respectively implement the functions of the aforementioned second data acquisition module 901 and the detection module 902, that is, to implement the functions of the aforementioned document image tampering detection device, thereby implementing the aforementioned document image tampering detection method.

[0188] The communication interface 1004 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the electronic device and other devices or a communication network.

[0189] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned training method of the document image tampering detection model, or implements the above-mentioned document image tampering detection method.

[0190] The computer-readable storage medium may be any available medium that can be stored in the electronic device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the electronic device to execute the above-mentioned training method of the document image tampering detection model, or to execute the above-mentioned document image tampering detection method.

[0191] It is to be understood that the structure illustrated in the embodiments of this specification does not constitute a specific limitation on the system of the embodiments of this specification. In other embodiments of the specification, the above system may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0192] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0193] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0194] It should be noted that the above examples are only specific embodiments of the present invention, and the present invention is obviously not limited to the above examples, and there are many similar variations. All variations directly derived or associated from the contents disclosed by the technicians in this field should fall within the protection scope of the present invention.

Claims

1. A training method for a document image tampering detection model, the document image tampering detection model comprising: Edge detection branch, element style detection branch, global detection branch, feature fusion module and prediction head; The method comprises: Selecting multiple target certificates, obtaining multimodal fusion images of the target certificates as sample images, wherein the sample images include tampered images and non-tampered images, and labels of the sample images are used to indicate whether the multimodal fusion images are tampered images; Inputting the sample image into the edge detection branch to obtain a tampered edge feature image; Inputting the sample image into the element style detection branch to obtain a tampered element feature image; Inputting the sample image into the global detection branch to obtain a global feature image; Using the feature fusion module to perform feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image; The fused feature image is input into the prediction head, and a loss function is constructed based on the difference between the prediction result of the prediction head and the label of the sample image to update the parameters of the document image tampering detection model.

2. The method of claim 1, further comprising: Constructing a first training sample set according to the sample image, wherein the first training sample set includes a first document image sample that has been tampered with, and a first label of the first document image sample is an edge mask image of the tampered area; The first document image sample is input into the edge detection branch, and a first loss function is constructed based on the difference between the detection result of the edge detection branch on the first document image sample and the first label to train the edge detection branch to recognize and locate the edge of the tampered area.

3. The method according to claim 1, wherein the element style detection branch comprises an element extraction branch and a style detection branch; Inputting the sample image into the element style detection branch to obtain a tampered element feature image specifically includes: Input the sample image into the element extraction branch to obtain the document element; For each element of the sample image, the style detection branch is used to detect whether the element and other elements belong to the sample image; if so, the element is determined to be a non-tampered element, otherwise, the element is determined to be a tampered element, and the tampered element feature image of the element is output.

4. The method of claim 3, further comprising: constructing a second training sample set according to the sample image, wherein the second training sample set includes a second certificate image sample that has been tampered with, and a second label of the second certificate image sample includes an element region mask label and an element category label, and the element category label is used to indicate whether the element is a tampered element; Inputting the second certificate image sample into the element extraction branch to obtain extracted certificate elements; Determining a second loss function according to the document element and the element region mask label; Based on the document element of the second document image sample and the element category label of the document element, construct a positive and negative sample pair, wherein the positive sample pair includes two non-tampered elements, and the negative sample pair includes one non-tampered element and one tampered element; Input the positive and negative sample pairs into the style detection branch, adopt a contrastive learning strategy, and construct a third loss function; The second loss function and the third loss function are used to train the element style detection branch to be able to identify and locate tampered elements.

5. The method according to claim 1, wherein the feature fusion module is used to perform feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image, specifically comprising: The feature fusion module is used to perform pairwise cross feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain the fused feature image.

6. The method according to claim 5, wherein the prediction head is specifically used to predict the probability score of the sample image being a tampered image based on each set of fused feature images of two-by-two cross-feature fusions; and constructing a loss function based on the difference between the prediction result of the prediction head and the label of the sample image, specifically comprising: Performing a weighted summation on all probability scores of the sample image to obtain a total probability score of the sample image; The loss function is constructed according to the difference between the total probability score and the label of the sample image.

7. The method according to claim 1, obtaining the multimodal fusion image of the target document, specifically comprising: Acquire a normal frame image, a flash frame image, and a difference image between the normal frame image and the flash frame image of the target document; The normal frame image, the flash frame image and the difference image are fused to obtain a multimodal fused image of the target certificate.

8. A method for detecting tampering of a certificate image, comprising: Obtain the document image to be detected; The document image to be detected is input into a document image tampering detection model pre-trained according to the method according to any one of claims 1 to 7, and a tampering detection result of the document image to be detected is determined according to an output result of the document image tampering detection model.

9. A training device for a document image tampering detection model, the document image tampering detection model comprising: Edge detection branch, element style detection branch, global detection branch, feature fusion module and prediction head; The device comprises: A first data acquisition module is configured to acquire a multimodal fusion image of a target document as a sample image, wherein the sample image includes a tampered image and a non-tampered image, and a label of the sample image is used to indicate whether the multimodal fusion image is a tampered image; The data processing module is configured to input the sample image into the edge detection branch to obtain a tampered edge feature image; input the sample image into the element style detection branch to obtain a tampered element feature image; input the sample image into the global detection branch to obtain a global feature image; use the feature fusion module to perform feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain a fused feature image; input the fused feature image into the prediction head, and construct a loss function based on the difference between the prediction result of the prediction head and the label of the sample image to update the parameters of the document image tampering detection model.

10. The device according to claim 9, wherein the first data processing module is further configured to: Constructing a first training sample set according to the sample image, wherein the first training sample set includes a first document image sample that has been tampered with, and a first label of the first document image sample is an edge mask image of the tampered area; The first document image sample is input into the edge detection branch, and a first loss function is constructed based on the difference between the detection result of the edge detection branch on the first document image sample and the first label to train the edge detection branch to recognize and locate the edge of the tampered area.

11. The device according to claim 9, wherein the element style detection branch comprises an element extraction branch and a style detection branch; and the first data processing module is specifically configured to: Input the sample image into the element extraction branch to obtain the document element; For each element of the sample image, the style detection branch is used to detect whether the element and other elements belong to the sample image; if so, the element is determined to be a non-tampered element, otherwise, the element is determined to be a tampered element, and the tampered element feature image of the element is output.

12. The device according to claim 11, wherein the data processing module is further used for: constructing a second training sample set according to the sample image, wherein the second training sample set includes a second certificate image sample that has been tampered with, and a second label of the second certificate image sample includes an element region mask label and an element category label, and the element category label is used to indicate whether the element is a tampered element; Inputting the second certificate image sample into the element extraction branch to obtain extracted certificate elements; Determining a second loss function according to the document element and the element region mask label; Based on the document element of the second document image sample and the element category label of the document element, construct a positive and negative sample pair, wherein the positive sample pair includes two non-tampered elements, and the negative sample pair includes one non-tampered element and one tampered element; Input the positive and negative sample pairs into the style detection branch, adopt a contrastive learning strategy, and construct a third loss function; The second loss function and the third loss function are used to train the element style detection branch to be able to identify and locate tampered elements.

13. The device as described in claim 9, wherein the data processing module is specifically used to use the feature fusion module to perform pairwise cross-feature fusion on the tampered edge feature image, the tampered element feature image and the global feature image to obtain the fused feature image.

14. In the device as described in claim 13, the prediction head of the data processing module is specifically used to predict the probability score of the sample image being a tampered image based on the fused feature image of each group of pairwise cross-feature fusions; the first data processing module is specifically used to perform weighted summation on all probability scores of the sample image to obtain the total probability score of the sample image; and construct the loss function based on the difference between the total probability score and the label of the sample image.

15. In the device as described in claim 9, the first data acquisition module is specifically used to obtain a normal frame image, a flash frame image, and a difference image between the normal frame image and the flash frame image of the target document; and perform image fusion on the normal frame image, the flash frame image, and the difference image to obtain a multimodal fusion image of the target document.

16. A document image tampering detection device, comprising: A second data acquisition module is configured to acquire an image of a document to be detected; The detection module is configured to input the certificate image to be detected into the certificate image tampering detection model pre-trained according to the method described in any one of claims 1 to 7, and determine the tampering detection result of the certificate image to be detected according to the output result of the certificate image tampering detection model.

17. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7, or the method according to claim 8.

18. An electronic device, comprising: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7, or execute the method according to claim 8.

Images