Method for identifying forged certificate images

Abstract

The present invention discloses a method for identifying forged document images, which belongs to the field of computer vision technology and includes steps such as image preprocessing, extracting image spatial features, sequence classification, and calculating the probability value of true or false documents. The present invention does not need to manually design and extract image features, but directly learns features from data. By cleverly designing a network structure, it can better learn subtle features with authenticity identification, has stronger robustness and generalization ability, does not need to maintain an additional template image library, and converts the image to be detected into a model input format after preprocessing and directly inputs it into a pre-trained model for prediction, that is, to obtain the confidence of the true or false document, and compares the confidence with a set threshold based on the confidence, so as to obtain the prediction result of the real or forged document, and can realize end-to-end training and reasoning, and directly obtain the final result without extracting features first and then predicting.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method for identifying forged document images. Background Art

[0002] For the identification of forged documents that are photographed, printed or copied, the traditional method uses image processing. For example, the prior art patent 1 (patent number: CN111311550A, patent name: forged document identification method, device and electronic equipment) determines the edge thickness value of the shadow area formed by the edge of the document to be identified based on the image of the document to be identified; determines the authenticity of the document to be identified based on the edge thickness value of the shadow area; and identifies the real document and the forged documents such as copied, printed, etc. by judging the consistency of the shooting direction and the edge thickness value;

[0003] Prior art patent 2 (patent number: CN106485639A, patent name: Method and device for identifying forged ID images) extracts image features of the image to be detected; based on the image features, searches for similar images in the image library query system; calculates the similarity between the image to be detected and the similar images; when the similarity exceeds a set threshold, determines that the image to be detected is a forged ID image;

[0004] Prior art patent 3 (patent number: CN111275042A, patent name: Method, device and electronic device for identifying forged documents) relies on the laser marking area; the authenticity of the document to be identified is determined based on the average chromaticity change of the laser marking area; wherein the laser marking area is the area where the document to be identified has a dispersion reflection phenomenon under light. Thus, the identification of forged documents based on the laser film of the document is realized;

[0005] In the algorithm-based method, the existing paper (Y. Rao and J. Ni, "A deep learning approach to detection of splicing and copy-move forgeries in images") was collected in 2016 IEEE International Workshop on Information Forensics and Security (WIFS), Abu Dhabi, United Arab Emirates, December 2016, pp. 1-6. doi: 10.1109 / WIFS.2016.7823911.) firstly crops the image into small images, then uses a lightweight convolutional neural network (CNN) to extract image features, then fuses the extracted features, and finally uses SVM for classification. In the deep learning-based algorithm, perjury identification can be regarded as an image classification task. The usual image classification model only uses CNN networks, such as resnet, vgg, etc. In the perjury identification task, the CNN model cannot capture the more discriminative local areas well.

[0006] The above-mentioned technical patents have the following defects: (1) Patents 1 and 2 require manual image processing to extract image features, and their generalization ability is weak. Patent 1 relies heavily on edge shadow areas and has high requirements on light sources and shooting directions, which is difficult to apply in actual complex scenes. Patent 2 also needs to maintain an image library to determine whether it is perjury by image matching similarity. The disadvantage is that it is necessary to maintain an image library. If the image library is large, the calculation amount for similarity matching is also large, and it cannot be guaranteed that the image library can cover all forged document features. (2) Existing papers crop the original image, use CNN to extract the cropped features, and then use SVM to learn these features. The process is too cumbersome and cannot achieve end-to-end model training and reasoning. In addition, only the features of each slice can be learned. The slices of the same image cannot interact with each other, and important distinguishing features are inevitably missing, and the importance of each slice cannot be distinguished. Therefore, a method for identifying forged document images is proposed to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to solve the problems in the prior art and to propose a method for identifying forged document images.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] The method for identifying forged document images comprises the following steps:

[0010] S1. Image preprocessing: Based on deep learning technology, for a document image to be detected as input, the preprocessing stage converts the image into a floating-point tensor and normalizes the pixel values ​​to between 0 and 1;

[0011] S2. Extract image spatial features: First, use CNN to extract image spatial features, then use AvgPool3D to reduce the spatial features, and then slice and splice the reduced spatial features into sequence features;

[0012] S3, sequence classification: Output the sequence features to the bidirectional LSTM model, simulate scanning the image from left to right and from right to left, and finally use the local attention mechanism to calculate the feature vector of the bidirectional LSTM output sequence;

[0013] S4. Calculate the probability value of the authenticity of the certificate: Input the feature vector into the fully connected network, and the final output obtained from the fully connected network is the probability value of the authenticity of the certificate.

[0014] Preferably, the image size required for spatial feature extraction in S2 is 640x640, and the CNN in S2 is a lightweight fully convolutional network model based on CNN. The spatial feature map obtained after CNN extracts spatial features is Where C0 is the number of channels, H0 and W0 are the height and width of the input image respectively.

[0015] Preferably, AvgPool3D in S2 reduces the channel, height and width of the spatial feature map, and the final feature map obtained after AvgPool3D reduction is X∈R C×H×W .

[0016] Preferably, in S2, the final feature space graph X∈R C×H×W , divided into k blocks in height, let H c is the height of each block, k is the number of blocks, then the blocks Use the Reshape operation to convert it into a sequence feature That is, the length of the sequence feature F (Sequence Length) is W×k, and the feature dimension (Embedding Size) is H c ×C.

[0017] Preferably, for the simulation of scanning the image from left to right and from right to left in S3, an example is given, assuming that the length of the converted sequence F is 5, that is, x i ∈R d ,d=H c ×C, forward features output by bidirectional LSTM and backward features

[0018] Preferably, in S3, for the forward output sequence and the backward output sequence Calculate the attention weights and attention features respectively, and output the sequence F forward f For example, the attention weight score∈R 5 ,params∈R d , where params are the parameters that can be learned by the model, express The transpose of , the attention-based vector feature is obtained by calculating the forward output sequence and the attention weight Similarly, the same calculation is performed on the backward output sequence to obtain an attention-based vector feature

[0019] Preferably, in S4, and Concatenate to produce a vector feature V out ∈R 2d After converting the output sequence of the bidirectional LSTM into a vector, it can be used as the input of the fully connected layer. The output of the fully connected layer is a vector pred∈R 2 , which represents the probability of a real document and a forged document.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. The present invention does not require artificial design to extract image features, but directly learns features from data. By cleverly designing the network structure, it can better learn subtle features with authenticity identification, and has stronger robustness and generalization ability.

[0022] 2. The present invention does not need to maintain an additional template image library. The image to be detected is converted into a model input format after preprocessing and directly input into a pre-trained model for prediction, that is, the confidence of the authenticity of the certificate is obtained, and the confidence is compared with the set threshold to obtain the prediction result of the real or forged certificate.

[0023] 3. The model designed in the present invention can realize end-to-end training and reasoning, and directly obtain the final result without extracting features first and then making predictions. Through clever model design, the network structure can efficiently extract local features, which is more helpful for identifying authentic documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of the method for identifying forged certificate images proposed by the present invention;

[0025] Figure 2 A schematic diagram of the network structure of the method for identifying forged certificate images proposed by the present invention;

[0026] Figure 3 A schematic diagram of the structure of the method for identifying forged certificate images proposed by the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of the method for identifying forged certificate images proposed by the present invention. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] Reference Figure 1-4 , a method for identifying forged document images, comprising the following steps:

[0030] S1. Image preprocessing: Based on deep learning technology, for a document image to be detected as input, the preprocessing stage converts the image into a floating-point tensor and normalizes the pixel values ​​to between 0 and 1;

[0031] S2. Extract image spatial features: First, use CNN to extract image spatial features, then use AvgPool3D to reduce the spatial features, and then slice and splice the reduced spatial features into sequence features;

[0032] S3, sequence classification: Output the sequence features to the bidirectional LSTM model, simulate scanning the image from left to right and from right to left, and finally use the local attention mechanism to calculate the feature vector of the bidirectional LSTM output sequence;

[0033] S4. Calculate the probability value of the authenticity of the certificate: Input the feature vector into the fully connected network, and the final output obtained from the fully connected network is the probability value of the authenticity of the certificate.

[0034] The image size required for spatial feature extraction in S2 is 640x640. The CNN in S2 is a lightweight fully convolutional network model based on CNN. The spatial feature map obtained after CNN extracts spatial features is Where C0 is the number of channels, H0 and W0 are the height and width of the input image respectively.

[0035] It should be noted that in the stage of CNN extracting image spatial features, for an input image, in order to ensure that the dimension of the sequence features obtained by subsequent conversion remains unchanged, the input image should be cropped or scaled to a fixed size (640x640). The design of the lightweight fully convolutional network model based on CNN refers to the Skip-Connection structure of ResNet, which only contains 13 layers.

[0036] In S2, AvgPool3D reduces the channel, height and width of the spatial feature map. At the same time, in order to ensure that the dimension of the generated sequence feature is not too high, the final feature map obtained after AvgPool3D reduction is X∈R C×H×W .

[0037] In S2, the final feature space graph X∈R C×H×W , divided into k blocks in height, let H c is the height of each block, k is the number of blocks, then the blocks Use the Reshape operation to convert it into a sequence feature That is, the length of the sequence feature F (Sequence Length) is W×k, and the feature dimension (Embedding Size) is H c ×C.

[0038] It should be noted that: in order to simulate scanning the image from top to bottom and from left to right, the spatial features extracted by CNN need to be converted into a sequence, and then bidirectional LSTM is applied to the sequence features. The method of converting the bidirectional LSTM output sequence to a vector is used. After the spatial features extracted by the CNN network are converted into sequence features, in order to establish the connection between the front and back slices, bidirectional LSTM is used for forward and backward scanning.

[0039] For the simulation of scanning the image from left to right and from right to left in S3, an example is given, assuming that the length of the converted sequence F is 5, that is, x i ∈R d ,d=H c ×C, forward features output by bidirectional LSTM and backward features

[0040] It should be noted that after the spatial features extracted by the CNN network are converted into sequence features, in order to establish the connection between the front and back slices, a bidirectional LSTM is used for forward and backward scanning.

[0041] For the forward output sequence in S3 and the backward output sequence Calculate the attention weights and attention features respectively, and output the sequence F forward f For example, the attention weight score∈R 5 ,params∈R d , where params are the parameters that can be learned by the model, express The transpose of , the attention-based vector feature is obtained by calculating the forward output sequence and the attention weight Similarly, the same calculation is performed on the backward output sequence to obtain an attention-based vector feature

[0042] In S4, you need to and Concatenate to produce a vector feature V out ∈R 2d After converting the output sequence of the bidirectional LSTM into a vector, it can be used as the input of the fully connected layer. The output of the fully connected layer is a vector pred∈R 2 , which represents the probability of a real document and a forged document.

[0043] It should be noted that: since the final input to the fully connected layer requires a vector, the output sequence of the bidirectional LSTM needs to be converted into a vector. Using the local attention method, not every slice on a graph contributes equally, and simple averaging cannot reflect those important and more discernible areas. Therefore, the attention mechanism is introduced to dynamically calculate the weight of each slice based on the input sequence, thereby improving the contribution of important local areas.

[0044] in Figure 3 Represents the bidirectional LSTM sequence processing process

[0045] a. Bidirectional LSTM performs forward and backward operations. Assume that the sequence length is 5, f0-f4 are forward outputs, and b0-b4 are backward outputs;

[0046] b. Merge the forward features and backward features horizontally respectively;

[0047] c. Output sequence attention weights and feature vectors obtained based on the attention mechanism (taking forward features as an example, backward features are similar);

[0048] d. Forward and backward features are calculated based on attention.

[0049] Figure 4It means that the forward feature vector and the backward feature vector are concatenated vertically as the final feature vector input to the fully connected layer.

[0050] In the specific operation of the present invention, the model invented above is first trained on the training data set to obtain a model weight file (implemented using pytorch, and the obtained weight file is in PTH format);

[0051] Secondly, convert the model weight file PTH format to Open Neural Network Exchange (ONNX);

[0052] Again, use ONNX to implement inference, based on the onnxruntime library. This includes preprocessing of logarithmic image normalization and resizing, and postprocessing of the inference output using softmax to obtain probability values;

[0053] Finally, take the probability of the output representing a real document and set a threshold (for example, 0.9). When the probability is greater than this value, it is considered a real document, otherwise it is a forged document.

[0054] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for identifying forged document images, characterized in that: The steps include: S1. Image preprocessing: Based on deep learning technology, for a document image to be detected as input, the preprocessing stage converts the image into a floating-point tensor and normalizes the pixel values ​​to between 0 and 1; S2. Extract image spatial features: First, use CNN to extract image spatial features, then use AvgPool3D to reduce the spatial features, and then slice and splice the reduced spatial features into sequence features; AvgPool3D reduces the channel, height and width of the spatial feature map. At the same time, in order to ensure that the dimension of the generated sequence feature is not too high, the final feature map obtained after AvgPool3D reduction is , the final feature space map , divided into k blocks in height, let is the height of each block, k is the number of blocks, then the blocks ,use The operation converts it into a sequence feature , that is, the length of the sequence feature F is , the feature dimension is ; S3, sequence classification: Output the sequence features to the bidirectional LSTM model, simulate scanning the image from left to right and from right to left, and finally use the local attention mechanism to calculate the feature vector of the bidirectional LSTM output sequence; simulate scanning the image from left to right and from right to left, and assume that the length of the converted sequence F is 5, that is, , the forward feature output by the bidirectional LSTM and backward features ; For the forward output sequence and the backward output sequence , respectively calculate its attention weight and attention feature ,in are parameters that can be learned by the model, express The transpose of , the attention-based vector feature is obtained by calculating the forward output sequence and the attention weight Similarly, the same calculation is performed on the backward output sequence to obtain an attention-based vector feature ; S4. Calculate the probability value of the authenticity of the certificate: Input the feature vector into the fully connected network, and the final output obtained from the fully connected network is the probability value of the authenticity of the certificate.

2. The method for identifying forged document images according to claim 1, characterized in that: The image size required for spatial feature extraction in S2 is 640x640. The CNN in S2 is a lightweight fully convolutional network model based on CNN. The spatial feature map obtained after CNN extracts spatial features is , where C0 is the number of channels, H0 and W0 are the height and width of the input image respectively.

3. The method for identifying forged document images according to claim 1, characterized in that: In S4, it is necessary to and This produces a vector feature After converting the output sequence of the bidirectional LSTM into a vector, it can be used as the input of the fully connected layer. The output of the fully connected layer is a vector , which represents the probability of a real document and a forged document.

Images