Multi-style handwritten electronic signature recognition system and method and storage medium

Abstract

The invention discloses a handwritten electronic information recognition model construction method based on multiple styles, which comprises the following steps of: collecting a plurality of reserved sample signatures of different styles and a test signature to construct a sample pair, and respectively extracting multi-dimensional handwriting features to construct a user representation vector; abstracting the multi-dimensional signature writing style features through weight sharing according to the representation vector to obtain features with highly consistent handwriting styles of the handwritten electronic information; calculating attention weights of the test signature and the plurality of reserved sample signatures, and establishing correlation between the test signature and the reserved sample signatures of different styles on each feature component; and performing true and false dichotomy judgment training through forward reasoning to obtain handwriting feature information recognition module construction parameters, and constructing a multi-style reserved sample signature recognition model. According to the method, the model better learns the feature information and the difference degree between the multi-style reserved sample signature and the verification signature, and the recognition and verification accuracy of the multi-style reserved sample signature is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-style handwritten electronic signature recognition model and a method thereof. Background Art

[0002] With the development of science and technology, electronic signature recognition technology has been widely used in finance, law, e-commerce and other fields. As an important part of it, handwritten electronic signature recognition technology aims to verify the user's identity by comparing the user's written signature with the pre-stored signature sample through a computer algorithm. In this process, it is necessary to collect the user's written signature trajectory, pressure and dynamic information, process it through the data processing module, and then compare it with the stored signature sample to achieve signature recognition and verification.

[0003] Existing handwritten electronic signature recognition technology usually relies on a single style of sample signatures for verification. This method has a low recognition accuracy when the same user generates multiple styles of signatures due to different situations or time changes. For example, the signatures written by users at different times and on different devices may have obvious differences, and existing recognition systems often cannot accurately identify these differences. The handwritten signatures of the same user often fail to pass identity authentication due to differences in writing styles, thus affecting the accuracy of signature recognition and user experience.

[0004] The current handwritten electronic signature recognition technology has the following major problems when faced with multi-style signatures: 1. The recognition system can only process signature samples of a single style and cannot adapt to the diversity of signature styles; 2. The existing data processing methods have limited processing capabilities for multi-style signatures and cannot effectively extract and utilize the signature's feature information; 3. When faced with multi-style signatures, the existing signature recognition and verification methods are often unable to accurately determine whether the signatures match, which affects the accuracy and efficiency of signature recognition.

[0005] For example, the publication number "CN115620312A" and the name "Cross-modal character handwriting verification method, system, device and storage medium" discloses a cross-modal recognizable single-word character handwriting verification method, which involves the field of electronic signature technology. The online electronic writing character data and paper writing character data of the signatory are obtained, and the writer's identity identification is associated, and a database including a training set, a verification set and a test set is constructed; the deep learning neural network is trained and verified, and the network model is optimized using the training set data. The character space attention mechanism is used to refer to the standard sample, and the unique writing method or important writing method of the standard sample is learned to capture the key differences and unique commonalities of the characters; the verified handwriting recognition model collects the online handwritten character data set, obtains the characters to be verified and the handwriting features for similarity calculation, and comprehensively judges and verifies the identity of the signatory.

[0006] This application mainly solves the problem of cross-modal single-word handwriting verification, but does not take into account the differences in handwriting styles of the same writer at different times and places. The use of the character space attention mechanism as a reference standard sample is intended to focus on the data differences of different modalities, but does not consider the impact of data differences of different styles and the corresponding processing. This will bring challenges to the recognition and verification of neural network models and the accuracy of signatory identity verification.

[0007] Publication number CN115984975A, name of electronic signature verification method, system, equipment and medium based on graph convolutional neural network, collects handwritten electronic signature handwriting authentic and imitation data; trains the backbone network constructed by the convolutional neural network to obtain a feature extractor, and extracts the feature vectors of all samples in the sample database; calculates the sample similarity based on the feature vector; constructs a graph structure of similar samples, and uses the graph structure to train the graph convolutional neural network model to obtain a handwritten signature handwriting graph convolution classifier; based on the multi-dimensional attribute information of the handwritten electronic signature to be verified, uses the feature vector extracted by the feature extractor to construct a test graph structure, and verifies the authenticity of the electronic signature based on the graph convolution classifier.

[0008] This application is mainly based on the verification and comparison of electronic signatures based on graph convolutional neural networks. The style differences caused by different scenarios and writing methods of writers cannot be eliminated and accurately identified through graph convolutional neural networks.

[0009] Publication No. CN114220178A, name “Signature Identification System and Method Based on Channel Attention Mechanism”, the collected electronic signature handwriting data is combined with coordinates, pressure and time information to convert into a signature image, the orthographed and imitation signature data in the data set are dynamically sampled and paired into image pairs, and then the image pairs are further enhanced and spliced ​​and combined as the final input image, and the comprehensive feature vector representation of the orthographed signature and imitation signature pair or the orthographed signature and the orthographed signature pair is obtained through the interaction of the ordinary convolution layer, the channel attention convolution layer, and the pooling layer, and then the comprehensive feature vector is weighted in different dimensions in combination with the channel attention mechanism, and finally the weighted feature vector is subjected to binary classification authenticity identification using a fully connected classifier.

[0010] This method uses channel attention to better focus on the feature differences between the authentic signature pair and the imitation signature pair, so that the model can learn better and improve the model's recognition and verification capabilities for authentic and imitation data. However, it still does not solve the problem of differences in writing styles among writers. Different styles in authentic writing and similar styles in imitation writing will still affect the accuracy of the model. Summary of the invention

[0011] In order to overcome the above-mentioned deficiencies in the prior art, the present invention provides a handwritten electronic signature recognition method based on multi-style samples.

[0012] Specifically, it includes collecting multi-style electronic signature data through a handwritten signature collection device; filtering, denoising and feature extraction of the electronic signature data; using a deep learning algorithm and multi-style electronic signature data to train a deep neural network model. In order to make the model pay more attention to the differences between multi-style signatures, an attention mechanism is introduced into the deep neural network to improve the learning ability of the model. After the deep neural network model training is completed, the model is used to identify and verify multi-style electronic signatures. The model outputs the verification signature and the sample signature and the similarity to determine whether the signatures are written by the same person.

[0013] Based on the first aspect of the present application, a multi-style handwritten electronic signature recognition system is proposed, including an acquisition module, a convolutional neural network, an attention module, and a feedforward neural network. The acquisition module acquires multiple multi-style sample signatures and verification signatures, and pre-processes them to obtain a multi-dimensional style representation vector; the convolutional neural network extracts the style feature vector of each signature according to the style representation vector, and constructs a multi-style feature vector matrix through the attention module to obtain the attention weight of each style sample signature feature vector to the verification signature feature vector, and obtains the attention feature vector of each style feature according to the correlation between different sample style features and the verification signature style features, and calculates the multi-style weighted feature vector; the feedforward neural network determines whether the verification signature and the sample signature match through binary classification prediction based on the multi-style weighted feature vector and the verification signature style feature vector.

[0014] Further preferably, the attention module calls a deep learning algorithm to calculate attention weights according to the feature vectors of various style signatures, and then performs weighted summation based on the attention weights and the corresponding feature vectors to obtain the attention feature vector.

[0015] Further preferably, the attention module verifies the signature feature vector V test、 The feature vector of the retained signature is subjected to softmax operation, specifically, the formula is called:

[0016]

[0017] Get the attention weight Atten corresponding to the retained signature and call the formula:

[0018]

[0019] Multiply the attention weight by the feature vector of the corresponding retained signature to obtain the attention feature vector of the retained signature Among them, d k Represents the dimension of the feature vector.

[0020] Further preferably, the verification signature feature vector is concatenated with the retained signature attention feature vector after weighted summation of attention weights, and input into a feedforward neural network. Multiple linear layers and activation layers in the feedforward neural network map it into a two-dimensional vector, corresponding to the probability output of the true or false binary classification.

[0021] Further preferably, the convolutional layer and the linear layer in the neural network model extract the multi-dimensional signature writing features according to the user representation vector to obtain the deep style feature vector of the signature writing handwriting, wherein the multi-dimensional signature writing features include: geometric features, dynamic features, frequency domain features, time domain features, pressure features based on multiple styles.

[0022] Based on the second aspect of the present application, a multi-style handwritten electronic signature recognition method is proposed, including collecting multiple multi-style sample signatures and verification signatures, and pre-processing to obtain a multi-dimensional style representation vector; extracting the style feature vector of each signature according to the style representation vector, constructing a multi-style feature vector matrix through an attention module, obtaining the attention weight of each style sample signature feature vector to the verification signature feature vector, obtaining the attention feature vector of each style feature according to the correlation between different sample style features and the verification signature style features, and calculating the multi-style weighted feature vector; the multi-style weighted feature vector and the verification signature style feature vector are input into a feedforward neural network for binary classification prediction to determine whether the verification signature and the sample signature match.

[0023] Further preferably, a deep learning algorithm is called to calculate the attention weights according to the feature vectors of various style signatures, and then a weighted sum is performed according to the attention weights and the corresponding feature vectors to obtain the attention feature vector.

[0024] Further preferably, according to the verification signature feature vector V test、 The feature vector of the retained signature is subjected to softmax operation, specifically, the formula is called:

[0025]

[0026] Get the attention weight Atten corresponding to the retained signature and call the formula:

[0027]

[0028] Multiply the attention weight by the feature vector of the corresponding retained signature to obtain the attention feature vector of the retained signature Among them, d k Represents the dimension of the feature vector.

[0029] Further preferably, the verification signature feature vector and the attention feature vector of the retained signature after weighted summation of the attention weights are concatenated and input into a feedforward neural network. Multiple linear layers and activation layers in the feedforward neural network map it into a two-dimensional vector, corresponding to the probability output of the true or false binary classification.

[0030] Based on the third aspect of the present application, a method for constructing a handwritten electronic information recognition model based on multiple styles is proposed, including collecting a plurality of sample signatures and test signatures of different styles to construct sample pairs, extracting multi-dimensional features of the handwriting of each signature to construct a representation vector; inputting the representation vector into a deep neural network to extract a multi-dimensional signature writing style feature vector through weight sharing, adding an attention module to the deep neural network model, calculating the attention weights according to the writing style feature vectors of each signature, obtaining the correlation between each feature component of the test signature and each feature component of each sample signature, and obtaining a multi-style attention feature vector; inputting the multi-style attention feature vector feature vector and the test signature feature vector feature vector into a feedforward neural network, performing judgment training through forward reasoning, obtaining the construction parameters of the handwriting feature information recognition module, and constructing a multi-style sample signature recognition model.

[0031] Further preferably, obtaining the multi-style attention feature vector includes: calling a deep learning algorithm to calculate attention weights according to the feature vectors of various style signatures respectively, and obtaining the multi-style attention feature vector by weighted summation according to the attention weights and the corresponding feature vectors.

[0032] Further preferably, the calculation of the multi-style attention feature vector includes constructing a matrix of multi-style retained signature feature vectors and a test signature feature vector, and test、 The signature feature vector of the retained sample is subjected to softmax operation. Specifically, the formula is called:

[0033]

[0034] Get the sample signature attention weight Atten and call the formula:

[0035]

[0036] Multiply the attention weight by the feature vector of the corresponding retained signature to obtain the attention feature vector of the retained signature Among them, d k Represents the dimension of the feature vector.

[0037] Further preferably, a cross entropy loss function is used during the training process to calculate the loss value of the prediction result and the true value of the test signature sample until the loss value is reduced to a preset threshold and the model training is completed, wherein, according to the formula:

[0038]

[0039] Calculate the loss value.

[0040] Further preferably, during the training process, all parameters in the neural network model are updated by a gradient descent algorithm, including: the number of convolution kernels of the convolution layer, the parameter weights of the linear layer, and the parameter weights of the regularization layer, thereby constructing a multi-style sample retention signature recognition model.

[0041] Based on the fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute the method described above.

[0042] The present invention utilizes multi-style electronic writing information sequence data, calls a deep learning algorithm to train a deep neural network model, extracts multi-style feature vectors of multiple sample signatures and test signatures through the deep neural network model, adds an attention module to the deep neural network, calculates the attention weight, obtains the importance of the current style sample signature and the test signature in each style feature component, and can better learn the feature information and difference between multi-style signatures. It has strong adaptability and can meet the needs of diversified signature styles. No matter what situation or time the user writes the signature, it can be accurately recognized, which improves the adaptability and flexibility of the system; the present invention has a high recognition accuracy rate, by extracting the most representative features that can best show the high consistency of the handwriting of electronic handwriting information, obtaining the features representing the user's own characteristics in the handwriting information, training the neural network, and improving the recognition accuracy and accuracy of the recognition model. And when verifying the signature, the current signature data of the new user is compared with the multi-style sample signature sample data, thereby improving the accuracy of signature recognition. Compared with the prior art that only processes and compares signature samples of a single style, the recognition accuracy rate of the present invention is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a handwritten electronic signature function module structure in an exemplary embodiment of the present application;

[0044] Figure 2 Flowchart of attention weight calculation for multi-style retention signature and verification signature in an exemplary embodiment of the present application;

[0045] Figure 3 Schematic diagram of a multi-style sample signature verification and recognition model in an exemplary embodiment of the present application;

[0046] Figure 4 Examples of hardware devices that can be applied to various aspects of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0048] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0049] like Figure 1 What is shown is a schematic diagram of the structure of a handwritten electronic signature function module based on the present exemplary embodiment, including a handwritten electronic signature collection part that obtains multi-style handwritten electronic signatures and performs electronic signature data processing, extracts signature feature data, and identifies and verifies the electronic signature based on the signature feature data through a deep learning model.

[0050] Firstly, a handwritten signature collection device is used to collect multi-style electronic signature data of different devices, different scenarios, and different writing methods; after filtering and denoising the multi-style electronic signature data, signature feature data is extracted to obtain handwritten electronic signature sequence data; using the multi-style handwritten electronic signature sequence data, a deep learning algorithm is called and a deep neural network model is trained, and an attention module is added to the deep neural network. The attention module obtains the attention weights of the test signature and each retained sample signature based on various styles, constructs an attention weight matrix, and extracts multi-style feature vectors of multiple retained sample signatures and test signatures through the deep neural network model; after the deep neural network model training is completed, the model parameters and network settings are obtained, a multi-style handwritten electronic signature recognition model is constructed, and the multi-style handwritten electronic signature is recognized and verified through the electronic signature recognition model.

[0051] The present invention uses an attention weight matrix to focus on the difference and correlation between the test signature and multiple multi-style sample signatures, thereby improving the learning ability of the model. Specifically, it includes:

[0052] The collection device collects multi-style handwritten electronic writing information. The electronic signature collection device uses a touch screen, pressure sensor and acceleration sensor to collect signature data. The collection module collects signatures of the same user in different scenes (such as different time, different place, different equipment, different writing method, etc.) in different styles to obtain the user's signature samples in multiple styles.

[0053] Electronic signature data processing. The data preprocessing unit filters and removes noise from the collected original data of the multi-style signature samples, and extracts multi-dimensional signature features, including geometric features of the signature handwriting (such as pen tip coordinates and shape), dynamic features (such as pressure changes, dynamic time regularization, etc.), frequency domain features (such as rhythm features), time domain features (such as pen tip movement speed, acceleration) and pressure features (such as maximum pressure, average pressure). By extracting the feature vector data of different writing styles in the multi-dimensional signature features, the multi-style feature vector data is used to construct a feature sequence or feature matrix as the only representation feature that reflects the user's writing signature style information. Further, the feature data can be compressed to reduce storage space.

[0054] The neural network model learns the user's characterization features and learns to distinguish various writing styles based on the inherent rules of writing according to the characterization features of each signature data.

[0055] Specifically, the neural network model further extracts and compresses the multi-dimensional signature writing style features (including: geometric features, dynamic features, frequency domain features, time domain features, and pressure features) according to the user representation vector. The convolutional layer and linear layer in the neural network model extract the deeper style features of the signature handwriting. These style features fully reflect the handwriting features with specific characteristics brought about by the long-term habits formed in the writing process of the same signatory, can maximize the writing style of the writer, and have obvious differences from the writing styles of others, thereby distinguishing different writers.

[0056] The deep learning algorithm and multi-style signature samples are used to train the deep neural network model and build a multi-style handwritten electronic information recognition model.

[0057] Due to the different environments, writing situations, writing devices or time changes of the signer, the signature of the same content will produce writing information of various styles, that is, the writing styles of multiple sample signatures of the same signer are quite different. However, due to the long-term habit formation and the formation of specific arm movements of the same writer, it is necessary to optimize the process of calculating the similarity of multiple sample signature feature vectors and test signature sample feature vectors. By adding an attention module to the deep neural network model, the correlation between the feature components of the multi-style sample signatures and the training sample test signature is calculated, and the importance of each style feature of the current test signature and the sample signature is obtained through the normalized exponential function (softmax function).

[0058] Specifically, it includes constructing training sample pairs, such as a sample pair containing multiple signatures of different styles signed by the same user multiple times and test signatures of any style with the same content written by any other person, thereby forming several sample pairs. Among them, the signature of the same person is used as the positive sample, and the signatures of different people are used as the imitation samples. Thus, a many-to-one training sample pair is formed. The unique representation features of the sample signatures and test signatures of different styles obtained after preprocessing are input into the neural network, and the feature vectors of the sample signatures and verification signatures of different styles are obtained through the Transformer module (such as a feedforward neural network, a multi-head attention layer, a layer normalization layer, etc.) or the CNN module (such as a one-dimensional convolutional layer, a two-dimensional convolutional layer, a linear layer, an activation layer, etc.). Usually, the Transformer module includes a feedforward neural network, a multi-head attention layer, a layer normalization layer, etc., and the multi-head attention layer uses a self-attention mechanism and is multi-headed.

[0059] The attention module in the present invention uses a cross-attention mechanism and is single-headed.

[0060] like Figure 3 As shown, an attention module is added to the neural network, and the attention module is between the backbone neural network (i.e., the Transformer module or CNN module mentioned above) and the feedforward neural network.

[0061] The attention module calls the deep learning algorithm to obtain the feature vectors of various style signatures, and then multiplies the attention weight by the feature vector to obtain the weighted sum of the feature vectors. Figure 3 As shown in the figure, the feature vector after weighted summation is concatenated with the feature vector of the verified signature. The concatenation is performed along the dimension of the sequence length L and then input into a feedforward neural network. The output of the feedforward neural network is converted into a 2D vector through a linear layer, an activation layer, and a pooling layer, and the result of the signature authenticity classification is obtained.

[0062] The deep neural network can select a convolutional neural network, input the multi-style sample electronic signature and test signature data into the convolutional neural network, and perform true and false binary classification training until the loss of the neural network is reduced to the preset loss value, that is, the training of the deep convolutional neural network is completed.

[0063] The deep convolutional neural network is trained using signature samples. The deep convolutional neural network is a weight-sharing neural network model. For multiple sample signatures and test signatures of different styles, such as: the number of multi-style sample electronic signatures is Refs, the number of test electronic signatures is 1, and the number of signatures is Refs+1. The multi-dimensional signature style features are extracted through the preprocessing module to obtain Refs+1 representation vectors, such as the representation vector V of the sample signature. ref, the characterization vector V of the test signature test。 The dimension of each signature representation vector is d.

[0064] The attention module in the deep convolutional neural network calculates the attention weights according to the representation vectors of each signature, that is, the attention weights of each retained signature and the test signature are calculated respectively, and the correlation between each feature component of the test signature and each feature component of each retained signature is obtained. Each feature component includes the elements of the electronic signature feature vector in each writing style feature dimension. For example, a feature vector is 128-dimensional, and the elements of each dimension represent a signature style feature component.

[0065] If the number of multi-style signature samples is Refs, then attention weight calculations need to be performed Refs times. Calculate the attention weights of each multi-style sample signature and test signature respectively, and sum up all the obtained attention weights as the total attention weight. Calculate the weighted feature vector of the sample signature according to the total attention weight.

[0066] By training the deep convolutional neural network for true and false binary classification, the loss of the predicted results and true values ​​of the test samples is calculated during the training process. The training of the deep convolutional neural network is completed until the loss of the neural network is reduced to the preset loss value, and a multi-style handwritten information recognition model is obtained.

[0067] For example, cross entropy loss is used as the loss function. The loss function is called according to the formula:

[0068]

[0069] Among them, y i represents the label of the i-th sample, Indicates the predicted value of the first sample, and b indicates the number of samples during training (i.e., the number of pairs of retained signatures and test signatures). The loss is calculated during training and training is stopped until the cross entropy loss is reduced to a preset value.

[0070] During the training process, all parameters in the neural network model are updated through the gradient descent algorithm, including the number of convolution kernels in the convolution layer, the parameter weights of the linear layer, the parameter weights of the regularization layer, etc. When the loss value reaches the predetermined threshold, the model training is completed and all parameters of the neural network model are obtained. After the model training is completed, the parameters of the neural network model are determined, and a multi-style sample signature recognition model is constructed to verify the verification signature. The model is constructed using the parameters to recognize and verify the verification signature and the multi-style sample signature.

[0071] like Figure 2 Shown is a flowchart of one of the exemplary embodiments of calculating the attention weights of multi-style retention signatures and test signatures.

[0072] Multiple styles of retained sample signature feature data of multiple retained sample signature feature vectors (including retained sample 1 feature vector, retained sample 2 feature vector, retained sample 3 feature vector...) and test signature feature data (test feature vector) are input into the deep neural network model.

[0073] The feature vectors of the sample signatures of each style are multiplied with the feature vectors corresponding to the test signatures, and a softmax operation is performed to obtain a weight matrix. Further, the feature vectors of the sample signatures are multiplied with the weight matrix to obtain the weight feature vectors of the sample signatures of various styles.

[0074] The difference between the signature feature data of the retained style and the test signature feature data is obtained through softmax operation. It is expected that the model can pay attention to these differences during training and better learn the inherent rules of the comparison of different style signatures.

[0075] The following example illustrates that according to the feature vector V of the test signature sample test , the feature vector V of any signature among multiple retained signatures ref , first, for the eigenvector V ref Transpose and multiply with the test signature feature vector (expressed as ), calling the formula:

[0076] Perform softmax operation, calculate the attention weight Atten of the retained signature and the test signature sample, and construct the attention weight matrix.

[0077] The softmax operation is used to calculate the attention weight for each retained signature, and the attention weights of all style features of all style retained signatures and the corresponding features of the test signature are obtained.

[0078] in, Indicates V ref The transpose of k Represents the dimension of the feature vector. For example, the feature vector V of the retained signature and the test signature test and V ref is an L*D matrix, where L represents the sequence length of the electronic signature and D represents the characteristic length of the electronic signature. After multiplication, we get an L*L matrix. After the softmax operation, each row is subjected to a normalized exponential function operation (softmax operation), and the result is still an L*L matrix.

[0079] Furthermore, according to the attention weight obtained from a certain retained signature and a certain feature vector of the retained signature, the importance of the current style retained signature and the test signature in each style feature component is obtained.

[0080] For example, the attention weight can be multiplied by the feature vector of the retained sample signature. This multiplication is a matrix multiplication (i.e., the attention weight and the retained sample signature vector are weighted summed), and the importance of the current style retained sample signature and the test signature in each style feature component is obtained.

[0081] Specifically:

[0082] Calculate the importance of style feature components (i.e. weighted eigenvector).

[0083] By calculating the attention weights of each feature of the test signature and each style retained sample signature, the attention weight matrix is ​​obtained, and the feature correlation between the test signature and the retained sample signatures of different styles is established. The attention weight matrix is ​​further associated with the corresponding retained sample signature (such as multiplication), and the attention weight obtained can represent the importance of the current style retained sample signature and the test signature in each style feature component, so as to better learn the feature information and difference between the multi-style retained sample signatures and the test signature.

[0084] By the above method, the attention weight of each pair of retained sample-test signature is calculated. For example, the dimension of the feature vector is set to 512. For example, in this embodiment, it is assumed that the test signature and the retained sample signature are 1vs3. The acquisition unit obtains a multi-style retained sample signature sample signed by the same signer three times, collects and obtains a test signature sample, and uses the feature vector of the test signature sample to perform softmax operation with the feature components of the three multi-style retained sample signature samples to obtain an attention matrix, and obtains the weights of the test signature and the three style retained sample signatures in each style feature, respectively, and obtains the difference and similarity between the signature style features. The obtained three attention weights are weighted summed, and the result is used as the attention weight of the multi-style retained sample signature. Then, the attention weight is multiplied with the feature vector of the retained sample signature. The multiplication is a matrix multiplication (that is, the attention weight is weighted summed with the retained sample signature vector), that is, the weighted sum feature vector of the current style retained sample signature is obtained, and then the weighted sum feature vectors of different styles and the test signature feature vector are input into the feedforward neural network.

[0085] In electronic signature recognition, the verification signature is the central object that needs attention, and multiple sample signatures have different styles, which brings great difficulty to the recognition task. Therefore, the attention module is used to evaluate the correlation between various features of the verification signature and various features of sample signatures of different styles. Sample signatures of different styles are the content that the verification signature needs to match. The present invention calculates the attention weights of the verification signature and the sample signature through softmax operation, and constructs the attention weight Atten matrix. The matrix essentially establishes the correlation between various style features of the verification signature and various style features of sample signatures of different styles. Further, the attention module multiplies the attention weight matrix with the feature vector of the current style sample signature to obtain the importance of the current style sample signature and the verification signature in each feature.

[0086] After the attention module, the signature feature vector to be verified and the weighted feature vector after weighted summation are added. The concatenated vector is obtained by concatenating the vectors (along the dimension of the row vectors). The concatenated vectors are input into a feedforward neural network (including multiple linear layers and activation layers) to map the input vectors into two-dimensional vectors, corresponding to the probability output of the true or false binary classification.

[0087] After the neural network model with the attention module is trained, a multi-style electronic signature verification model is obtained, which performs signature recognition and verification on electronic signatures of any scene, any signature style, font, etc. When the user verifies the signature, the multi-style signature recognition model compares the verification signature with the multi-style sample signature, specifically including extracting multi-dimensional features from the multi-style sample and the verification signature, and obtaining multi-dimensional feature vectors of the multi-style sample and the verification signature (e.g., the dimension is d k ), if the feature vector of the retained signature is recorded as V ref , the feature vector of the verified signature is recorded as V test The attention module calculates the attention weight of each pair of feature vectors of the retained signature and the verification signature, and obtains the correlation between each feature component of the verification signature and each feature component of the retained signature. Further, the softmax operation and weighted operation are called to obtain the weighted feature vectors of various style signatures.

[0088] For multiple signatures with different styles, multiple style weighted feature vectors will be obtained (for example, 3 Represents the features of signatures of different retention styles). After the attention module, The input is fed into a feedforward neural network and mapped into a two-dimensional vector through multiple linear layers and activation layers in the feedforward neural network.

[0089] like Figure 3Shown is an example diagram of a multi-style sample signature verification and recognition model in this exemplary embodiment, based on the probability output of the true and false binary classification corresponding to the multi-layer perceptron network.

[0090] As further explained by taking three styles of sample signatures as an example in this embodiment, sample signatures 1, 2, 3, and test signatures of various styles are preprocessed to obtain multi-dimensional style representation vectors reflecting their own styles, which are input into a weight-sharing backbone neural network (a deep convolutional neural network CNN is used as an example in this example), and the style feature vectors V corresponding to each sample signature and verification signature are obtained through the convolutional layer, linear layer, and activation layer in the network. ref1， V ref2， V ref3， V test ;

[0091] Through the attention module, the style feature vectors of multi-style retained signatures and the style feature vector matrix of the verification signature are constructed, and the softmax operation is performed to calculate the attention weights of the signature feature vectors of each style retained and the verification signature feature vectors, respectively, and the attention weight matrix is ​​constructed to obtain the correlation between different retained style features and the corresponding style features of the verification signature, and further associate the weights in the weight matrix with the corresponding signature style feature vector to obtain the style feature attention vector;

[0092] The weighted feature vectors of the three style signatures obtained are The signature feature vector to be verified is input into the multi-layer perceptron network; the multi-layer perceptron network determines whether the verification signature and the retained signature match by predicting the probability of true or false binary classification. If the predicted similarity exceeds the preset threshold, the verification passes, otherwise the verification fails.

[0093] like Figure 3 As shown, the 3 weighted feature vectors obtained by the attention module are Its dimension is the same as the feature vector dimension of the verification signature, so we further transform the three The feature vector of the verified signature is concatenated along the sequence length L, and the concatenated result is input into a feedforward neural network. The output of the feedforward neural network is converted into a 2D vector through a linear layer, an activation layer, and a pooling layer, and the result of the signature authenticity classification is obtained.

[0094] The present invention can be widely used in application fields such as finance, law, and e-commerce. In the financial field, electronic signature recognition technology can be used for important operations such as logging into a bank account and transferring money to improve the security of transactions. In the legal field, electronic signature recognition technology can be used for the signing and verification of contracts to ensure the validity and traceability of contracts. In the field of e-commerce, electronic signature recognition technology can be used for payment confirmation for online shopping to improve the convenience and reliability of transactions. In addition, with the development of science and technology, the demand for electronic signature recognition technology in emerging fields such as remote office and online education is also increasing. Therefore, the handwritten electronic signature recognition method of the present invention has broad market prospects and application needs.

[0095] Furthermore, the exemplary embodiments of the present application also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0096] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0097] refer to Figure 4 , the structural block diagram of the electronic device 300 that can be used as the server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0098] like Figure 4 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0099] A plurality of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, an output unit 307, a storage unit 308, and a communication unit 309. The input unit 306 may be any type of device capable of inputting information to the electronic device 300, and the input unit 306 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 307 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 308 may include, but is not limited to, a disk, an optical disk. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0100] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above. For example, in some embodiments, the reconstruction and decomposition of the muscle movement trajectory redrawn according to the original trajectory of the signature stroke, and the decomposition of its logarithmic velocity curve, etc. can be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. In some embodiments, the computing unit 301 can be configured to execute the signature handwriting dynamic acquisition implementation method by any other appropriate means (e.g., by means of firmware).

[0101] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0102] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0106] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0107] The applicant of the present invention has made a detailed illustration and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the detailed description of the above-mentioned implementation is not intended to limit the scope of the invention claimed for protection, but only represents the selected implementation methods of the present invention. Therefore, changes in the example numerical values ​​or replacement of functional modules should still fall within the scope of the present invention.

Claims

1. A multi-style handwritten electronic signature recognition system, characterized in that: It includes an acquisition module, a convolutional neural network, an attention module, and a feedforward neural network. The acquisition module acquires multiple multi-style sample signatures and verification signatures, and pre-processes them to obtain a multi-dimensional style representation vector; the convolutional neural network extracts the style feature vector of each signature according to the style representation vector, and constructs a multi-style feature vector matrix through the attention module to obtain the attention weight of each style sample signature feature vector to the verification signature feature vector, and obtains the attention feature vector of each style feature according to the correlation between different sample style features and the verification signature style features, and calculates the multi-style weighted feature vector; the feedforward neural network determines whether the verification signature and the sample signature match through binary classification prediction based on the multi-style weighted feature vector and the verification signature style feature vector.

2. The system according to claim 1, characterized in that The attention module is single-headed and uses a cross-attention mechanism. It is located between the backbone neural network and the feedforward neural network. The attention module calls a deep learning algorithm to calculate the attention weights according to the feature vectors of various style signatures, and then performs weighted summation based on the attention weights and corresponding feature vectors to obtain the attention feature vector.

3. The system according to claim 2, characterized in that The attention module verifies the signature feature vector V test , perform softmax operation on the feature vector of the retained signature, specifically, call the formula: Get the attention weight Atten corresponding to the retained signature and call the formula: Multiply the attention weight by the feature vector of the corresponding retained signature to obtain the attention feature vector of the retained signature Among them, d k Represents the dimension of the feature vector.

4. The system according to any one of claims 1 to 3, characterized in that: The verification signature feature vector and the retained signature attention feature vector after weighted summation of the attention weights are concatenated along the sequence length and input into the feedforward neural network. Multiple linear layers and activation layers in the feedforward neural network map it into a two-dimensional vector, corresponding to the probability output of the true or false binary classification.

5. The system according to any one of claims 1 to 3, characterized in that: The convolutional layer and linear layer in the neural network model extract the multi-dimensional signature writing features according to the user representation vector to obtain the deep style feature vector of the signature handwriting, where the multi-dimensional signature writing features include: geometric features, dynamic features, frequency domain features, time domain features, and pressure features.

6. A multi-style handwritten electronic signature recognition method, characterized in that: The method includes collecting multiple multi-style sample signatures and verification signatures, and pre-processing them to obtain a multi-dimensional style representation vector; extracting the style feature vector of each signature according to the style representation vector, constructing a multi-style feature vector matrix through an attention module, obtaining the attention weight of each style sample signature feature vector to the verification signature feature vector, obtaining the attention feature vector of each style feature according to the correlation between different sample style features and the verification signature style features, and calculating the multi-style weighted feature vector; the multi-style weighted feature vector and the verification signature style feature vector are input into a feedforward neural network for binary classification prediction to determine whether the verification signature and the sample signature match.

7. The method according to claim 6, characterized in that The deep learning algorithm is called to calculate the attention weights according to the feature vectors of various style signatures, and then the attention feature vector is obtained by weighted summation based on the attention weights and the corresponding feature vectors.

8. The method according to claim 7, characterized in that According to the verification signature feature vector V test , perform softmax operation on the feature vector of the retained signature, specifically, call the formula: Get the attention weight Atten corresponding to the retained signature and call the formula: Multiply the attention weight by the feature vector of the corresponding retained signature to obtain the attention feature vector of the retained signature Among them, d k Represents the dimension of the feature vector.

9. The method according to any one of claims 6 to 8, characterized in that: The verification signature feature vector and the attention feature vector of the retained signature after weighted summation of the attention weights are concatenated along the sequence length and input into the feedforward neural network. Multiple linear layers and activation layers in the feedforward neural network map it into a two-dimensional vector, corresponding to the probability output of the true or false binary classification.

10. A method for constructing a handwritten electronic information recognition model based on multiple styles, characterized in that: Collect multiple sample signatures and test signatures of different styles to construct sample pairs, extract multi-dimensional features of the handwriting of each signature to construct a representation vector; input the representation vector into a deep neural network to extract the multi-dimensional signature writing style feature vector through weight sharing, add an attention module to the deep neural network model, calculate the attention weight according to the writing style feature vector of each signature, obtain the correlation between each feature component of the test signature and each feature component of each sample signature, and obtain a multi-style attention feature vector; input the multi-style attention feature vector and the test signature feature vector into a feedforward neural network, perform judgment training through forward reasoning, obtain the construction parameters of the handwriting feature information recognition module, and construct a multi-style sample signature recognition model.

11. The method according to claim 10, characterized in that The method of obtaining a multi-style attention feature vector includes: calling a deep learning algorithm to calculate attention weights according to feature vectors of various style signatures, and performing weighted summation according to the attention weights and corresponding feature vectors to obtain the multi-style attention feature vector.

12. The method according to claim 11, characterized in that The calculation of the multi-style attention feature vector includes constructing a multi-style sample signature feature vector and a test signature feature vector matrix, and test , perform softmax operation on the retained signature feature vector, specifically, call the formula: Get the sample signature attention weight Atten and call the formula: Multiply the attention weight by the feature vector of the corresponding retained signature to obtain the attention feature vector of the retained signature Among them, d k Represents the dimension of the feature vector.

13. The method according to any one of claims 10 to 12, characterized in that: During the training process, the cross entropy loss function is used to calculate the loss value of the predicted result and the true value of the test signature sample until the loss value is reduced to the preset threshold, and the model training is completed. According to the formula: Calculate the loss value.

14. The method according to any one of claims 10 to 12, characterized in that: During the training process, all parameters in the neural network model are updated through the gradient descent algorithm, including: the number of convolution kernels in the convolution layer, the parameter weights of the linear layer, and the parameter weights of the regularization layer, thereby constructing a multi-style sample retention signature recognition model.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to cause the computer to execute the method according to any one of claims 6-14.

Images