A handwritten signature identification method and device based on deep learning

Abstract

The present invention discloses a method and device for identifying handwritten signatures based on deep learning, the device comprising a user interaction component, a database component, and a signature identification reasoning component; the present invention identifies the regional position of the handwritten signature in the image based on the signature image to be identified written by the user, identifies the character information corresponding to the handwritten word, compares the currently identified signature image with the historical signature sample, and displays the recognition result and the comparison result to the user; the handwritten signature image is compared in an efficient and intelligent way, which not only reduces the harmful risk of signature imitation to production things, but also saves the manual review cost with high performance accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a handwritten signature identification method and device based on deep learning. Background Art

[0002] Handwritten signatures are widely used in business, finance, and judicial scenarios. The purpose of using handwritten signatures in paper documents is to clarify the identity of the signer and to indicate that the signer agrees with the content specified in the signed document or is responsible for the authenticity of the signed document, and it has certain legal effect. However, in real life, it is common for people to replace their own signatures or imitate their signatures, which ultimately leads to administrative losses or financial fraud. Therefore, a high-performance signature identification method and device is urgently needed to avoid negative impacts or economic losses in people's production and life.

[0003] In actual business, there are generally two ways to obtain handwritten signatures. One is to use electronic devices to obtain comprehensive information such as the writer's status during the writing process and the final image of the signature; the other is to use external electronic devices, such as scanning equipment or handheld camera equipment, to obtain handwritten signature images after writing on paper. The first method usually uses handwritten signatures as biometrics for identity recognition. This method of using external input devices to identify signatures is usually called online signature authentication; the second data acquisition method focuses on analyzing the semantic information presented by the signature imaging during authentication. This method is usually called offline signature authentication. Online signature authentication is somewhat invasive to users and has high business environment requirements for input devices; while offline signature authentication does not require special data input devices, and application deployment is more flexible, and is widely used in actual business.

[0004] The existing technologies in offline signature identification are mainly divided into traditional image technology analysis methods and neural network feature extraction methods based on deep learning. Traditional image analysis methods mainly process the binary information of the image, smooth the image by filtering and other methods, and then identify the authenticity of the signature by focal features or edge information features. The method based on deep learning is to use a neural network to extract the feature vector of the signature image, and then use the cosine measurement method to calculate whether the angle between the signature images is within the threshold range, and judge the authenticity of the signature by the threshold. Both methods have poor robustness and large errors in actual operation effects. Traditional methods are difficult to handle signature images with complex background information, and extracting image feature matrices for cosine distance calculation cannot cope with imitations with high similarity of notes. Summary of the invention

[0005] In order to solve the above technical problems mentioned in the background technology, the present invention provides a handwritten signature identification method and device based on deep learning, which aims to efficiently and accurately identify data signatures through a neural network model. The calculation method is different from the existing cosine similarity calculation, and an end-to-end classification neural network is used to directly output the results, and the problem of low accuracy of semantic information caused by image feature extraction is improved.

[0006] A handwritten signature identification method based on deep learning provided by the present invention comprises the following steps:

[0007] Step S1, performing training data preprocessing operations on signature samples, combining samples of multiple categories of features of the signature samples, combining the samples into information tuple data, and using multiple signature sample information tuple data as training data sets;

[0008] Step S2, using the processed signature information tuple data as input data, sending the signature information tuple data into the signature identification classification neural network for model training, training the signature identification classification neural network through the training program loop parameter setting, stopping the training when the number of training loop iterations reaches the pre-collected threshold, and outputting the trained weighted signature recognition model;

[0009] The trained weighted signature recognition model is used to predict the classification result of the signature sample, and whether the signature is a real signature is determined according to the classification prediction result;

[0010] The signature authentication classification neural network is a multi-stream inverse feature convolution classification neural network with an attention mechanism, and the model type of the signature authentication classification neural network belongs to an end-to-end structure model;

[0011] Step S3: receiving a signature image to be authenticated, and inputting the signature image to be authenticated into the weighted signature recognition model, so that the weighted signature recognition model outputs an authentication and classification result of the signature image.

[0012] Preferably, step S1 includes: a single signature sample contains multiple categories of features, sample tuples are organized by category, a Cartesian product is established between multiple categories of features of the same signature, repeated combinations are filtered out, and the remaining combinations are used as signature object information tuples. Multiple signature object information tuples constitute a signature information tuple data set.

[0013] Preferably, the step of performing training data preprocessing on the signature sample in step S1 further includes: unifying the image size of the signature sample, graying the image, and performing image grayscale inversion after copying a copy of the grayed sample image;

[0014] Accordingly, the step S3 specifically includes:

[0015] Receive a signature image to be identified, pre-process the signature image to a uniform size through data, grayscale the image and copy the grayscale image to perform grayscale inversion; input the grayscale image and the grayscale inversion image into the weighted signature recognition model, obtain the recognition result from the weighted signature recognition model, and determine whether the signature image to be identified is a real signature based on the classification result.

[0016] Preferably, the single signature sample information tuple is composed of true value sample data and forged sample data, and the combination relationship includes the person and the person's information group, and the person and the imitation information group.

[0017] On the other hand, the present invention provides a handwritten signature identification device based on deep learning, the device comprising:

[0018] The user interaction component 100 is used to receive the signature image to be authenticated;

[0019] Database component 200, used to store signature samples that have undergone training data preprocessing operations and provide historical sample information of the signature samples;

[0020] The signature authentication reasoning component 300 is used to construct a reasoning service using a signature authentication model with training weights, the service receives the signature image to be authenticated, predicts the classification result of the signature sample, determines the classification result according to a threshold set by the system, and outputs the final authentication result of the signature image to be authenticated;

[0021] The signature authentication model with training weights is obtained by inputting tuple data of historical sample information of the signature sample into a signature authentication classification neural network and then performing model training; the signature authentication classification neural network is a multi-stream inverse feature convolution classification neural network with an attention mechanism, and the model type of the signature authentication classification neural network belongs to an end-to-end structural model;

[0022] The user interaction component 100 is also used to display the identification and classification result of the signature image to be identified output by the weighted signature recognition model.

[0023] Preferably, a single signature sample in the signature samples contains multi-category features, sample tuples are organized by category, a Cartesian product is established between multi-category features of the same signature, repeated combinations are filtered out, and the remaining combinations are used as signature object information tuples, and multiple signature object information tuples constitute a signature information tuple data set.

[0024] Preferably, the signature authentication reasoning component is specifically used to establish a signature authentication reasoning service, the data input model performs reasoning data pre-processing, and the post-processing of the reasoning result also includes:

[0025] For the signature sample and the signature image to be identified, the image data is preprocessed into a uniform size, the image is grayscaled and copied for grayscale inversion, the grayscale image and the grayscale inversion image are input into the signature identification model, the model outputs the classification recognition result, and determines whether the image to be signed is a real signature based on the classification result.

[0026] The beneficial effects of the present invention are as follows: the present invention identifies the regional position of the handwritten signature in the image based on the handwritten signature image (signature image to be identified), identifies the character information corresponding to the handwritten word, compares the currently identified signature image with the historical signature image (signature sample), and displays the recognition result and the comparison result to the user; the handwritten signature image is compared in an efficient and intelligent way, which not only reduces the harmful risks of signature imitation to production things, but also saves the manual review cost with high performance accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the structure of a handwritten signature identification device based on deep learning in an embodiment of the present invention;

[0028] Figure 2 It is a schematic diagram of the main embodiment flow of the handwritten signature identification method based on deep learning in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of sample data of signature after user interaction in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a signature sample information tuple in an embodiment of the present invention;

[0031] Figure 5 The present invention is a flowchart showing how to obtain recognition results by imitating the category tuple input model written by others in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0033] In order to make the technical solution of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] Example 1

[0035] Figure 1This is a flow chart of a handwritten signature identification device based on deep learning proposed in the present invention, which includes the following steps:

[0036] The user interaction component 100 is used to receive the user's signature sample and the signature image to be authenticated, such as Figure 3 The user interactive cropped image shown is a cropped image containing only the signature information, and the basic meta information of the signature sample is input;

[0037] It should be noted that the user interaction component 100 provides a visual user interface and provides user interaction logic, such as submitting a signature image, filling in signature information, applying for signature authentication, viewing inference recognition results, etc.;

[0038] It can be understood that the user interaction component mainly functions to receive signature images, receive user signature authentication requests, and present authentication results;

[0039] The database component 200 is used to store the signature samples that have been preprocessed with the training data and provide the historical sample information of the signature samples; the information of the stored signature samples includes meta information, signature images, and identification reasoning results;

[0040] It is understandable that the database component stores signature sample information to determine whether the current signature information and signature image already exist in the database. If there is no signature record in the database, a new signature sample record is created. If there is a signature record in the database, the current information is updated to the corresponding table.

[0041] The signature authentication reasoning component 300 is used to construct a reasoning service using a signature authentication model with training weights, the service receives the signature image to be authenticated, predicts the classification result of the signature sample, determines the classification result according to a threshold set by the system, and outputs the final authentication result of the signature image to be authenticated;

[0042] The signature authentication model with training weights is obtained by inputting tuple data of historical sample information of the signature sample into a signature authentication classification neural network and then performing model training; the signature authentication classification neural network is a multi-stream inverse feature convolution classification neural network with an attention mechanism, and the model type of the signature authentication classification neural network belongs to an end-to-end structural model;

[0043] It is understandable that the signature authentication reasoning component performs signature authentication reasoning on the user's specified signature sample and the signature image to be authenticated, outputs the signature authentication model reasoning result, and determines whether the handwritten signature in the image to be signed is written by the user based on the reasoning result.

[0044] It is understandable that the signature authentication reasoning component 300 interacts with the data storage component 200, and the signature authentication component 300 outputs the signature authentication result and stores it in the data storage component 200, and the result is also used to display it to the user interaction component 100 for the user to view the result;

[0045] In a specific implementation, a user applies for an authentication request in a visual interface, and the signature authentication reasoning component outputs a signature authentication result and displays it to the visual user interface.

[0046] Specifically, in the implementation method, signature authentication is performed for the first time, the user interaction component submits the signature information and initiates a signature authentication request; the database component 200 stores the data metadata when the user sends an inference request; the signature authentication component 300 receives the inference request, and after the inference is completed, the authentication result is sent to the database component 200; finally, the authentication result is displayed by the visual user interface.

[0047] Example 2

[0048] The embodiment of the present invention is used for a handwritten signature authentication method, such as Figure 2 As shown, the following steps are included:

[0049] Step S1, preprocess the (offline) signature sample data for training data; the main content of the preprocessing is to select high-quality samples for model training, and the handwritten signature image in the handwritten signature sample image data is clear, the font is correct, and the human eye can distinguish it; the multi-category features of the signature sample are combined into information tuple data, and multiple signature sample information tuple data are used as training data sets. The offline signature sample is a signature image written on a paper document, and the image can be taken by a camera or input into a computer through a scanning instrument. This embodiment uses customized offline data, 20 signers, and each signature sample has 14 real data and 38 counterfeit data.

[0050] Step S2, using the signature information tuple data after data processing as input data, sending the signature information tuple data into the signature identification classification neural network for model training, training the signature identification classification neural network through the training program loop parameter setting, stopping the training when the number of training loop iterations reaches the pre-collected threshold, and outputting the trained weighted signature recognition model; wherein the trained weighted signature recognition model is used to predict the classification result of the signature sample, and judging whether the signature is a real signature according to the classification prediction result; wherein the signature identification classification neural network is a multi-stream inverse feature convolution classification neural network with an attention mechanism, and the model type of the signature identification classification neural network belongs to an end-to-end structural model;

[0051] Step S3: receiving a signature image to be authenticated, and inputting the signature image to be authenticated into the weighted signature recognition model, so that the weighted signature recognition model outputs an authentication and classification result of the signature image.

[0052] In a specific implementation, this embodiment obtains the signature image of the person himself (i.e., the signature sample) and the signature image to be identified, processes the data into a model input format and then inputs it into a signature identification model. The model outputs a signature classification result, which is divided into two categories, "personal signature" and "others' imitation" in this embodiment. The model outputs probability values ​​of different categories and determines whether the signature image to be identified is written by the person himself based on the classification probability.

[0053] It should be noted that in the above step S1, the training data preprocessing of the offline samples also includes combining the "personal signature" sample and the "other people's imitation" sample of a single signature sample, the combination relationship is the Cartesian product of "personal" and "personal" and the Cartesian product of "personal" and "imitation", filtering the repeated arrangements in the combination, and keeping the meta-information group as the relationship between "personal" samples and "personal" samples, and the relationship between "personal" samples and "imitation" samples. In this embodiment, each signature sample is written 14 times by the person himself and imitated 2 times by each of the other 19 people. The signature samples with the constructed combination relationship are image processed, and the same image size, in this embodiment, the same image size is 160x160 pixels. Specifically, each sample image is grayed, and then a copy of the grayscale image is copied to invert the image grayscale. The tuple data of each sample finally constitutes two types of data such as Figure 4 As shown, type one includes "my signature" Figure A grayscale image, "my signature" Figure A grayscale transposed image, "my signature" Figure B grayscale image, and "my signature" Figure B grayscale transposed image; type two includes "my signature" Figure A grayscale image, "my signature" Figure A grayscale transposed image, "others' imitation" Figure A grayscale image, and "others' imitation" Figure A grayscale transposed image; in this embodiment, my signature Figure A and Figure B are the image numbers of 14 signatures written by myself, and the image numbers of others' imitations are similarly the imitation signature image numbers of the current signature; in this embodiment, the type one-tuple data is the "my signature" category, and the type two-tuple data is the "others' imitation" category.

[0054] It should be noted that in the above step S2, the multi-stream inverse feature convolutional neural network with attention mechanism in the training mode is input with 4 images, and the tuple data after data preprocessing in step S1 is input into the neural network for training. The convolutional neural network is added with non-local attention mechanism to enhance the convolution kernel's ability to process the global receptive field information of the image, and the convolution module with attention for extracting image feature information is used to process multiple images in a targeted manner, that is, each convolution module processes one image, and then the multi-stream module features are fused through cross-feature fusion to improve the neural network's perception of the changes in the feature information of the signature information after grayscale and grayscale inversion. The feature fusion is passed through the fully connected layer and then classified and output the probability of each category through the activation function.

[0055] It should be noted that in the above step S3, the input of the trained signature identification model is 4 images. The data preprocessing makes the signature image of the person and the signature image to be identified the same size, grayscales the images respectively, copies the grayscale images and performs grayscale inversion, and then converts the grayscale image of the person's signature, the grayscale inversion image of the person's signature, the grayscale image of the signature to be identified, and the grayscale inversion image of the signature to be identified into the grayscale image of the person's signature. Figure 4 An image is input into the model, and the model outputs the probability that the signature to be identified is "my own signature" and the probability of "someone else's signature". The sum of the probabilities of "my own signature" and "someone else's signature" is 1, so the classification result with the highest predicted probability value is taken as the identification result, that is, the probability of "my own signature" is 0.6 and the probability of "someone else's signature" is 0.4. The model predicts that the signature image to be identified is "my own signature".

[0056] According to step S3, Figure 5 Taking as an example, the process of imitating the category tuple input model and obtaining the recognition result is demonstrated.

[0057] The present invention is based on the user's handwritten signature image to be identified, identifies the regional position of the handwritten signature in the image, identifies the character information corresponding to the handwritten word, compares the currently identified signature image with the historical signature sample, and displays the recognition result and the comparison result to the user; the handwritten signature image comparison is performed in an efficient and intelligent way, which not only reduces the harmful risk of signature imitation to production things, but also saves the manual review cost with high performance accuracy.

[0058] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A handwritten signature identification method based on deep learning, characterized in that: The method comprises the following steps: Step S1, performing training data preprocessing operations on signature samples, combining samples of multiple categories of features of the signature samples, combining the samples into information tuple data, and using multiple signature sample information tuple data as training data sets; Step S2, using the processed signature information tuple data as input data, sending the signature information tuple data into the signature identification classification neural network for model training, training the signature identification classification neural network through the training program loop parameter setting, stopping the training when the training loop iteration number reaches the pre-collected threshold, and outputting the trained weighted signature recognition model; The trained weighted signature recognition model is used to predict the classification result of the signature sample, and whether the signature is a real signature is determined according to the classification prediction result; The signature authentication classification neural network is a multi-stream inverse feature convolution classification neural network with an attention mechanism, and the model type of the signature authentication classification neural network belongs to an end-to-end structural model; Step S3, receiving a signature image to be authenticated, and inputting the signature image to be authenticated into the weighted signature recognition model, so that the weighted signature recognition model outputs an authentication and classification result of the signature image; Among them, the signature identification and classification neural network is configured as a convolutional neural network plus a non-local attention mechanism to enhance the convolution kernel's ability to process image global receptive field information, and the convolution module with attention that extracts image feature information performs single processing on multiple images in a targeted manner, that is, each convolution module processes an image, and then the multi-stream module features are fused through cross-feature fusion to improve the neural network's perception of changes in feature information of the signature after grayscale and grayscale inversion. The feature fusion passes through the fully connected layer and then outputs the probability of each category through activation function classification.

2. The method according to claim 1, characterized in that The step S1 includes: a single signature sample contains multiple categories of features, sample tuples are organized by category, a Cartesian product is established between multiple categories of features of the same signature, repeated combinations are filtered out, and the remaining combinations are used as signature object information tuples. Multiple signature object information tuples constitute a signature information tuple data set.

3. The method according to claim 1, characterized in that The step of performing training data preprocessing on the signature sample in step S1 further includes: unifying the image size of the signature sample, graying the image, and performing image grayscale inversion after copying a grayed sample image; Accordingly, the step S3 specifically includes: Receive a signature image to be identified, pre-process the signature image to a uniform size through data, grayscale the image and copy the grayscale image to perform grayscale inversion; input the grayscale image and the grayscale inversion image into the weighted signature recognition model, obtain the recognition result from the weighted signature recognition model, and determine whether the signature image to be identified is a real signature based on the classification result.

4. The method according to claim 2, characterized in that The single signature sample information tuple is composed of true sample data and forged sample data, and the combination relationship includes the person and his information group, and the person and the imitation information group.

5. A handwritten signature identification device based on deep learning, characterized in that: The device comprises: A user interaction component, used for receiving a signature image to be authenticated; A database component, used to store signature samples that have undergone training data preprocessing operations and provide historical sample information of the signature samples; A signature authentication reasoning component is used to construct a reasoning service using a signature authentication model with training weights, the service receives the signature image to be authenticated, predicts the classification result of the signature sample, determines the classification result according to a threshold set by the system, and outputs the final authentication result of the signature image to be authenticated; The signature authentication model with training weights is obtained by inputting tuple data of historical sample information of the signature sample into a signature authentication classification neural network and then performing model training; the signature authentication classification neural network is a multi-stream inverse feature convolution classification neural network with an attention mechanism, and the model type of the signature authentication classification neural network belongs to an end-to-end structural model; The user interaction component is further used to display the identification and classification result of the signature image to be identified output by the signature recognition model with training weights; Among them, the signature identification and classification neural network is configured as a convolutional neural network plus a non-local attention mechanism to enhance the convolution kernel's ability to process image global receptive field information, and the convolution module with attention that extracts image feature information performs single processing on multiple images in a targeted manner, that is, each convolution module processes an image, and then the multi-stream module features are fused through cross-feature fusion to improve the neural network's perception of changes in feature information of the signature after grayscale and grayscale inversion. The feature fusion passes through the fully connected layer and then outputs the probability of each category through activation function classification.

6. The identification device according to claim 5, characterized in that: A single signature sample in the signature sample contains multi-category features, sample tuples are organized by category, a Cartesian product is established between multi-category features of the same signature, repeated combinations are filtered, and the remaining combinations are used as signature object information tuples. Multiple signature object information tuples constitute a signature information tuple data set.

7. The identification device according to claim 5, characterized in that: The signature authentication reasoning component is specifically used to establish a signature authentication reasoning service, the data input model performs reasoning data pre-processing, and the post-processing of the reasoning result also includes: For the signature sample and the signature image to be identified, the image data is preprocessed into a uniform size, the image is grayscaled and copied for grayscale inversion, the grayscale image and the grayscale inversion image are input into the signature identification model, the model outputs the classification recognition result, and determines whether the image to be signed is a real signature based on the classification result.

Images