Method for offline handwritten signature verification based on artificial intelligence
By extracting signature features based on artificial intelligence, combined with the idea of inverse identification network, the problem that the existing technology cannot effectively deal with offline handwritten signature authentication is solved, and signature verification with high accuracy and applicability is achieved.
Patent Information
- Application Number
- CN202210028558.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-01-11
AI Technical Summary
The existing signature verification technology cannot effectively handle offline handwritten signature authentication, and the identity verification of the writer is not accurate and real-time enough, and the application scenarios are limited.
Using an artificial intelligence-based method, the texture and depth features of the signature are extracted through SVM classifiers and twin neural networks, combined with the inverse identification network idea, the input image is flipped and fused multiple times, and the feature pyramid network and attention mechanism are extracted. Depth features are finally calculated and verified by multiple judges.
It realizes reliable and accurate handwritten signature verification in offline scenarios, improves the reliability and accuracy of the method, and is suitable for signature authenticity verification of any person.
Smart Images

Figure CN114360071B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for realizing offline handwritten signature verification based on artificial intelligence, and belongs to the field of information technology, in particular to the field of handwritten signature verification technology. Background Art
[0002] With the development of network technology, people are paying more and more attention to their privacy and security. Identity verification is an indispensable part in many work fields. Signature verification, as a kind of identity verification, can often assist in verifying personal identity, and has the advantages of being simple and fast. Compared with biometric features such as fingerprints, signatures are easier to obtain.
[0003] Although there are many signature verification schemes, there are still many problems. For example, the Chinese patent application number CN201680028530.4 implements a signature authentication method, terminal, stylus and system. The method includes when a user signs on the touch panel of the terminal with a stylus, the terminal detects the first signature data, the first signature data includes the coordinate information of the contact point between the stylus and the touch panel and the pressure information of the stylus on the touch panel at the contact point; and the terminal receives the second signature data sent by the stylus, the second signature data includes the motion trajectory information of the stylus detected by the stylus when the user signs on the touch panel with the stylus and the pressure information of the user on the stylus. The first signature data and the second signature data are matched and compared with the pre-stored signature template data to obtain a matching rate. When the matching rate is not lower than the preset matching rate threshold, the signature authentication is passed. The embodiment of the invention extracts more user signature feature data through the stylus, increases the difficulty for attackers to imitate user signature information, and improves the security of the signature authentication system. However, this patent has the following shortcomings: 1. It can only process online handwritten signatures, but not offline handwritten signatures; 2. It only collects pressure information, coordinate information, motion trajectory and other information on the touch panel, but does not perform feature extraction on the shape of the handwriting, which is not accurate enough; 3. It is impossible to collect a large amount of data first and then post-process it, and the application scenarios are not rich enough.
[0004] Another example: Chinese patent application number CN202011436507.3 discloses a handwritten signature comparison method and system based on image recognition, the method is applied to a handwritten signature comparison system, and the system has a basic database, a graph database, a background management module and a display interface, wherein the method includes: firstly, the handwritten signature data of all account managers and customers are collected separately and imported into the graph database through the background management system for preprocessing. If it is an electronic signature, it is directly connected to the background management system to enter the graph database; the customer signature features are extracted by the existing image recognition technology, and compared with the signature features of the account manager in the database, and the similarity ranking is obtained. For the result data exceeding the threshold, an early warning reminder is issued, and the result data is transferred to the background for manual processing. However, the patent has the following disadvantages: 1. The main purpose is to identify the signature that may be the leader, so as to further process it, and the application scenario is too limited; 2. The accuracy is not high enough, and the signature of the leader can only be found within a certain error range, and it cannot be specifically corresponding to a certain person; 3. The signature data of new users cannot be processed in real time.
[0005] It can be seen that the existing signature verification technology solutions do not solve the problem of offline handwritten signature authentication well, and do not process and solve tasks independent of the writer, and cannot be applied to the authenticity verification of signatures of any person. Therefore, how to reliably and accurately realize offline handwritten signature authentication has become a technical problem that needs to be solved urgently in the field of handwritten signature verification technology. Summary of the invention
[0006] In view of this, an object of the present invention is to invent a method to achieve reliable and accurate offline handwritten signature authentication.
[0007] In order to achieve the above object, the present invention proposes a method for offline handwritten signature verification based on artificial intelligence, which comprises the following steps:
[0008] (1) Preprocessing the signature image M1 to be verified according to the first feature extraction method to obtain the texture feature F1 of the signature image M1 to be verified; Preprocessing the real signature image M2 according to the first feature extraction method to obtain the texture feature F2 of the real signature image M2; The specific content of the first feature extraction method is: graying, background removal, histogram shift, size adjustment, interpolation and quantization, gray level co-occurrence matrix calculation and texture feature calculation are performed on the signature image in sequence;
[0009] (2) Inputting the texture features F1 and F2 into a first decision device to obtain a first decision result D0; the first decision result D0 can only be 0 or 1, 0 indicates that the signatures are not of the same person, and 1 indicates that the signatures are of the same person; the first decision device is a SVM classifier;
[0010] (3) Preprocessing the signature image M1 to be verified according to the second feature extraction method to obtain the depth feature P1 of the signature image M1 to be verified; Preprocessing the flipped image IM1 of the signature image M1 to be verified according to the second feature extraction method to obtain the depth feature IP1 of the flipped image IM1 of the signature image M1 to be verified;
[0011] Preprocess the real signature image M2 according to the second feature extraction method to obtain the depth feature P2 of the real signature image M2; preprocess the flipped image IM2 of the real signature image M2 according to the second feature extraction method to obtain the depth feature IP2 of the flipped image IM2 of the real signature image M2;
[0012] The second feature extraction method is a method of extracting deep features of an image through a backbone network and an attention network;
[0013] (4) The deep features P1 and P2 are combined into a fused feature FP1, which is input into a second decision device to obtain a second decision result D1; the second decision result D1 can only have a value of 0 or 1, where 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0014] The deep features P1 and IP2 are combined into a fusion feature FP2, which is input into a third decision device to obtain a third decision result D2; the third decision result D2 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0015] The deep features IP1 and P2 are combined into a fusion feature FP3, which is input into a fourth decision device to obtain a fourth decision result D3; the fourth decision result D3 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0016] The deep features IP1 and IP2 are combined into a fusion feature FP4, which is input into the fifth decision device to obtain a fifth decision result D4; the fifth decision result D4 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0017] The second, third, fourth and fifth decision devices are composed of twin neural networks with exactly the same structure and parameters;
[0018] (5) Calculate the final verification result D according to the following formula:
[0019] D=D0+D1+D2+D3+D4
[0020] If the D value is greater than or equal to 3, the signature is true, otherwise it is false.
[0021] The specific content of the second feature extraction method includes the following steps:
[0022] (31) First adjust the size of the signature image to the set size;
[0023] (32) Input the resized signature image into the feature pyramid network, and pass through a series of 3*3 convolutional layers,
[0024] After 3*3 pooling layer and activation layer processing, three feature maps of size 110*77, 55*38, and 27*19 are obtained, named fe1, fe2, and fe3 respectively;
[0025] (33) Upsampling the feature map fe3 once to adjust its size to 55*38, named fe4, concatenating the feature maps fe2 and fe4 to obtain cfe2, upsampling fe4 once again to adjust its size to 110*77, named fe5, concatenating fe1 and fe5 to obtain cfe1, and finally concatenating cfe1, cfe2, and fe3 to obtain a combined feature fe;
[0026] (34) performing contrast enhancement on the signature image and then downsampling the image to obtain a feature Rfe;
[0027] (35) The feature fe obtained previously is used as the Q feature of the attention network, the feature Rfe obtained previously is used as the K feature of the attention network, and the feature Rfe obtained previously is used as the V feature of the attention network, and is input into the attention network;
[0028] (36) The attention network outputs the deep features of the signature image.
[0029] The specific working process of the twin neural network is: use a 3*3 convolution layer, a 3*3 pooling layer and a ReLU activation function to obtain two 110*77*128 feature vectors, connect the two 110*77*128 feature vectors into a 110*77*256 feature vector, and obtain a 110*77*32 feature after convolution, pooling and other operations. Use a flattening layer to flatten the 110*77*32 feature to obtain a one-dimensional feature vector of length 271040, and input the one-dimensional vector into a 271040*1 fully connected layer to obtain a judgment result.
[0030] The twin neural network is pre-trained using a binary cross entropy loss function.
[0031] The specific content of the histogram shift operation is: the histogram of the grayscale values of the signature image pixels is shifted to zero and the white color of the signature image background is maintained, which is specifically achieved by subtracting the minimum grayscale value in the signature image from the signature image pixels.
[0032] The beneficial effects of the present invention are as follows: for the problem of handwritten signature verification in offline scenarios, the present invention uses frameworks such as SVM and twin neural networks to perform feature extraction and result classification, and uses the idea of an inverse identification network to invert the pixels of the input image to obtain multiple groups of data for simultaneous verification; the present invention not only uses deep learning methods, but also combines machine learning methods, so that the method of the present invention has higher reliability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is a flow chart of the method for offline handwritten signature verification based on artificial intelligence proposed by the present invention. DETAILED DESCRIPTION
[0034] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.
[0035] See also Figure 1 , introduces the method for offline handwritten signature verification based on artificial intelligence proposed by the present invention, the method comprising the following steps:
[0036] (1) Preprocessing the signature image M1 to be verified according to the first feature extraction method to obtain the texture feature F1 of the signature image M1 to be verified; Preprocessing the real signature image M2 according to the first feature extraction method to obtain the texture feature F2 of the real signature image M2; The specific content of the first feature extraction method is: graying, background removal, histogram shift, size adjustment, interpolation and quantization, gray level co-occurrence matrix calculation and texture feature calculation are performed on the signature image in sequence;
[0037] In the embodiment, the signature image is resized to a standard size of 220*155; the specific content of the interpolation and quantization operation is that after the resizing, the signature image needs to be quantized using some preset grayscale values, that is, to ensure that the grayscale pixel values in the signature image are converted to some specific values.
[0038] In an embodiment, the grayscale values at fixed intervals are set to 0, 10, 20, ..., 150, ..., 250. For the grayscale value of each pixel of the signature image, it is converted into the nearest specified grayscale value, for example, 7.3 needs to be converted to 10, and 147 needs to be converted to 150.
[0039] The grayscale co-occurrence matrix is a list of the frequencies of various pixel grayscale mixtures in an image. The grayscale co-occurrence matrix analyzes the relationship between two pixels at a time, called the reference pixel and the adjacent pixel, and then records the occurrence of each pixel combination in the matrix. The co-occurrence matrix is defined by the joint probability density of pixels at two locations. It not only reflects the distribution characteristics of brightness, but also reflects the position distribution characteristics between pixels with the same brightness or close brightness. It is a second-order statistical feature of image brightness changes. It is the basis for defining a set of texture features. The grayscale co-occurrence matrix of an image can reflect the comprehensive information of the image grayscale about direction, adjacent intervals, and change amplitude. It is the basis for analyzing local patterns of images and their arrangement rules.
[0040] The specific content of the texture feature calculation operation is: using the gray level co-occurrence matrix (GLCM) to calculate the homogeneity, contrast, correlation and other detailed texture features of the signature image, and then calculating the average value of these detailed texture features. The obtained average value is the texture feature of the signature image.
[0041] (2) Inputting the texture features F1 and F2 into a first decision device to obtain a first decision result D0; the first decision result D0 can only be 0 or 1, 0 indicates that the signatures are not of the same person, and 1 indicates that the signatures are of the same person; the first decision device is a SVM classifier;
[0042] (3) Preprocessing the signature image M1 to be verified according to the second feature extraction method to obtain the depth feature P1 of the signature image M1 to be verified; Preprocessing the flipped image IM1 of the signature image M1 to be verified according to the second feature extraction method to obtain the depth feature IP1 of the flipped image IM1 of the signature image M1 to be verified;
[0043] According to the idea of the inverse authentication network, the color change of the image will not affect the attribution of the signature. Therefore, for the two input images M1 (the signature image to be verified) and M2 (the real signature image), pixel inversion is performed to obtain images IM1 (the signature image to be verified after pixel inversion) and IM2 (the real signature image after pixel inversion). In principle, the comparison result of M1 and M2 should be consistent with the comparison results of the three pairs of data: M1 and IM2, IM1 and M2, and IM1 and IM2.
[0044] Image inversion refers to inverting the value of each pixel in the image. In the embodiment, the value range of the pixel is 0 to 255.0. Assuming that the value of a certain pixel is a, pixel inversion is to convert the value a into 255.0-a. After the values of all pixels of the entire image are converted, the pixel inversion is completed.
[0045] Preprocess the real signature image M2 according to the second feature extraction method to obtain the depth feature P2 of the real signature image M2; preprocess the flipped image IM2 of the real signature image M2 according to the second feature extraction method to obtain the depth feature IP2 of the flipped image IM2 of the real signature image M2;
[0046] The second feature extraction method is a method for extracting deep features of an image through a backbone network and an attention network; in an embodiment, a ReLU layer is used as an activation function after each convolution layer of the backbone network, and a BatchNormlization layer is connected after each convolution layer for normalization operation;
[0047] The backbone network uses FPN (Feature Pyramid Network). FPN network is a classic network that uses multi-scale features of images. After several downsampling operations, feature maps of various sizes can be obtained respectively. Then upsampling and splicing can adjust the feature maps of various scales to the same size and fuse them.
[0048] For detailed information on the Attention network, see the literature: Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need[J].arXiv, 2017.
[0049] (4) The deep features P1 and P2 are combined into a fused feature FP1, which is input into a second decision device to obtain a second decision result D1; the second decision result D1 can only have a value of 0 or 1, where 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0050] The deep features P1 and IP2 are combined into a fusion feature FP2, which is input into a third decision device to obtain a third decision result D2; the third decision result D2 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0051] The deep features IP1 and P2 are combined into a fusion feature FP3, which is input into a fourth decision device to obtain a fourth decision result D3; the fourth decision result D3 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0052] The deep features IP1 and IP2 are combined into a fusion feature FP4, which is input into the fifth decision device to obtain a fifth decision result D4; the fifth decision result D4 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person;
[0053] The second, third, fourth and fifth decision devices are composed of twin neural networks with exactly the same structure and parameters;
[0054] (6) Calculate the final verification result D according to the following formula:
[0055] D=D0+D1+D2+D3+D4
[0056] If the D value is greater than or equal to 3, the signature is true, otherwise it is false.
[0057] The specific content of the second feature extraction method includes the following steps:
[0058] (31) First, adjust the size of the signature image to a set size. In the embodiment, the size is adjusted to 220*155;
[0059] (32) Input the resized signature image into the feature pyramid network, and after a series of 3*3 convolutional layers, 3*3 pooling layers, and activation layers, obtain three feature maps of sizes 110*77, 55*38, and 27*19, which are named fe1, fe2, and fe3 respectively;
[0060] (33) Upsampling the feature map fe3 once to adjust its size to 55*38, named fe4, connecting the feature maps fe2 and fe4 to obtain cfe2, then upsampling fe4 once to adjust its size to 110*77, named fe5, connecting fe1 and fe5 to obtain cfe1, and finally connecting cfe1, cfe2, and fe3 to obtain a combined feature fe. In the embodiment, the size of the combined feature fe is 110*77*128 (width*height*number of channels);
[0061] (34) The most important part of a signature image is the part with strokes, while the background part is not important, so the signature image is contrast enhanced and then downsampled to obtain the feature Rfe; in the embodiment, the signature image after contrast enhancement is downsampled to 110*77*3, and then input into the convolution layer to adjust the number of channels to 128, that is, the size of the combined feature Rfe is 110*77*128 (width*height*number of channels);
[0062] (35) The attention network uses the self-attention method, which requires defining three sets of features: Q, K, and V. The feature fe obtained previously is used as the Q feature of the attention network, the feature Rfe obtained previously is used as the K feature of the attention network, and the feature Rfe obtained previously is used as the V feature of the attention network, and is input into the attention network; this allows the attention network to pay more attention to the stroke information, thereby ignoring the background information as much as possible (the background is mostly white / black and accounts for a large proportion), and obtaining more discriminative features.
[0063] (36) The attention network outputs the deep features of the signature image.
[0064] The specific working process of the twin neural network is: use a 3*3 convolution layer, a 3*3 pooling layer and a ReLU activation function to obtain two 110*77*128 feature vectors, connect the two 110*77*128 feature vectors into a 110*77*256 feature vector, and obtain a 110*77*32 feature after convolution, pooling and other operations. Use a flattening layer to flatten the 110*77*32 feature to obtain a one-dimensional feature vector of length 271040, and input the one-dimensional vector into a 271040*1 fully connected layer to obtain a judgment result.
[0065] The twin neural network is trained in advance using a binary cross entropy loss function. In an embodiment, the loss functions of the four twin neural networks are added together and used as the total loss function for network training.
[0066] The specific content of the histogram shift operation is: the histogram of the grayscale values of the signature image pixels is moved to zero and the white color in the signature image background is maintained (the white value is 255 in the embodiment), which is specifically achieved by subtracting the minimum grayscale value in the signature image from the signature image pixels.
[0067] The inventors have conducted a large number of experiments on the method of the present invention and obtained good experimental results, which shows that the method of the present invention is effective and feasible.
Claims
1. A method for offline handwritten signature verification based on artificial intelligence, characterized in that: The method comprises the following steps: (1) Preprocessing the signature image M1 to be verified according to the first feature extraction method to obtain the texture feature F1 of the signature image M1 to be verified; Preprocessing the real signature image M2 according to the first feature extraction method to obtain the texture feature F2 of the real signature image M2; The specific content of the first feature extraction method is: graying, background removal, histogram shift, size adjustment, interpolation and quantization, gray level co-occurrence matrix calculation and texture feature calculation are performed on the signature image in sequence; (2) Inputting the texture features F1 and F2 into a first decision device to obtain a first decision result D0; the first decision result D0 can only be 0 or 1, 0 indicates that the signatures are not of the same person, and 1 indicates that the signatures are of the same person; the first decision device is a SVM classifier; (3) Preprocessing the signature image M1 to be verified according to the second feature extraction method to obtain the depth feature P1 of the signature image M1 to be verified; Preprocess the flipped image IM1 of the signature image M1 to be verified according to the second feature extraction method to obtain the deep feature IP1 of the flipped image IM1 of the signature image M1 to be verified; Preprocess the real signature image M2 according to the second feature extraction method to obtain the depth feature P2 of the real signature image M2; Preprocess the flipped image IM2 of the real signature image M2 according to the second feature extraction method to obtain the deep feature IP2 of the flipped image IM2 of the real signature image M2; The second feature extraction method is a method of extracting deep features of an image through a backbone network and an attention network; (4) The deep features P1 and P2 are combined into a fused feature FP1, which is input into a second decision device to obtain a second decision result D1; the second decision result D1 can only have a value of 0 or 1, where 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person; The deep features P1 and IP2 are combined into a fusion feature FP2, which is input into a third decision device to obtain a third decision result D2; the third decision result D2 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person; The deep features IP1 and P2 are combined into a fusion feature FP3, which is input into a fourth decision device to obtain a fourth decision result D3; the fourth decision result D3 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person; The deep features IP1 and IP2 are combined into a fusion feature FP4, which is input into the fifth decision device to obtain a fifth decision result D4; the fifth decision result D4 can only have a value of 0 or 1, 0 indicates that the signature is not the same person, and 1 indicates that the signature is the same person; The second, third, fourth and fifth decision devices are composed of twin neural networks with exactly the same structure and parameters; (5) Calculate the final verification result D according to the following formula: D=D0+D1+D2+D3+D4 If the D value is greater than or equal to 3, the signature is true, otherwise it is false.
2. The method for offline handwritten signature verification based on artificial intelligence according to claim 1 is characterized in that: The specific content of the second feature extraction method includes the following steps: (31) First adjust the size of the signature image to the set size; (32) Input the resized signature image into the feature pyramid network, and after a series of 3*3 convolutional layers, 3*3 pooling layers, and activation layers, obtain three feature maps of sizes 110*77, 55*38, and 27*19, which are named fe1, fe2, and fe3 respectively; (33) Upsampling the feature map fe3 once to adjust its size to 55*38, named fe4, concatenating the feature maps fe2 and fe4 to obtain cfe2, upsampling fe4 once again to adjust its size to 110*77, named fe5, concatenating fe1 and fe5 to obtain cfe1, and finally concatenating cfe1, cfe2, and fe3 to obtain a combined feature fe; (34) performing contrast enhancement on the signature image and then downsampling the image to obtain a feature Rfe; (35) The feature fe obtained previously is used as the Q feature of the attention network, the feature Rfe obtained previously is used as the K feature of the attention network, and the feature Rfe obtained previously is used as the V feature of the attention network, and is input into the attention network; (36) The attention network outputs the deep features of the signature image.
3. The method for offline handwritten signature verification based on artificial intelligence according to claim 1, characterized in that: The specific working process of the twin neural network is: use a 3*3 convolution layer, a 3*3 pooling layer and a ReLU activation function to obtain two 110*77*128 feature vectors, connect the two 110*77*128 feature vectors into a 110*77*256 feature vector, and obtain a 110*77*32 feature after convolution, pooling and other operations. Use a flattening layer to flatten the 110*77*32 feature to obtain a one-dimensional feature vector of length 271040, and input the one-dimensional vector into a 271040*1 fully connected layer to obtain a judgment result.
4. The method for offline handwritten signature verification based on artificial intelligence according to claim 1, characterized in that: The twin neural network is pre-trained using a binary cross entropy loss function.
5. The method for offline handwritten signature verification based on artificial intelligence according to claim 1 is characterized in that: The specific content of the histogram shift operation is: the histogram of the grayscale values of the signature image pixels is shifted to zero and the white color of the signature image background is maintained, which is specifically achieved by subtracting the minimum grayscale value in the signature image from the signature image pixels.
