An online handwritten signature handwriting retrieval method and system based on faiss

Abstract

The application claims a faiss-based online handwritten signature handwriting retrieval method and system, and relates to the field of electronic signature handwriting recognition. By using big data and deep learning technology to train and extract handwriting features with identity recognition, the electronic signature data features are vectorized, and then a large-scale handwriting feature library is established. For the handwriting data to be identified, the handwriting feature vector is extracted, and then compared with the feature vector in the feature library. The vector similarity is calculated, and the final topN handwriting retrieval result is obtained according to the similarity ranking.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of handwritten electronic signature handwriting recognition, and particularly relates to a one-to-many large-scale electronic handwriting data retrieval method and system. BACKGROUND

[0002] Handwriting recognition is divided into 1:1 handwriting recognition and 1:N handwriting recognition. The former is also called handwriting authentication, that is, to identify whether the test sample and the sample are the signature handwriting of the same person. The latter is to determine whether the test sample belongs to a certain person in the library, that is, to identify the identity information of the signer according to the handwriting data. The handwriting feature vector with identity discrimination needs to be extracted from the test sample, and then compared with the handwriting features in the library. The sample with the highest similarity is taken, and if the similarity score is greater than the threshold, it is determined that the test sample is written by the identity of the sample. Otherwise, there is no matching identity. The existing handwriting recognition is mostly for offline static signature images, and the handwriting features are extracted from the images for feature comparison and recognition.

[0003] Publication No. CN109446905A, entitled "Electronic signature auditing method, device, computer equipment and storage medium" provides an electronic signature auditing method, device, computer equipment and storage medium. The terminal sends an electronic signature corresponding to the user identifier; the electronic signature is input into a pre-trained feature extraction model for prediction to obtain the corresponding target handwriting feature; the pre-set handwriting feature corresponding to the user identifier is queried; the target handwriting feature and the pre-set handwriting feature are input into a pre-trained feature matching model for prediction to obtain the corresponding target feature similarity; and the electronic signature is authenticated according to the target feature similarity. The method can improve the security of electronic signature.

[0004] Publication No. CN111950333A, entitled "A method for recognizing electronic handwritten signature based on neural network" discloses a method for recognizing electronic handwritten signature based on neural network, which includes signature registration and authentication of users. When registering, the user forms a signature source file on the signature terminal multiple times, and the source file records the horizontal and vertical coordinates of the signature trajectory and the pressure value when signing. The signature source file is submitted and saved to the signature database and bound with the user. The users in the signature library are labeled and bound according to the user. The normalized feature value and handwriting graph are input into the deep neural network for mapping data, and the training model is obtained after multiple layers of calculation. After training, the user's to-be-identified data is input into the network for calculation to obtain the result value. The user identity of the to-be-identified user is determined according to the result. The present application uses neural network for training, and identifies the user through the user's feature value. The efficiency is high, and the accuracy is also good.

[0005] The aforementioned existing technologies are all aimed at 1:1 handwriting recognition, i.e. handwriting authentication. They perform a 1:1 similarity matching between the target handwriting features and preset handwriting features. Using this similarity matching technology can only solve the problem of 'is it you' using the signature handwriting, but cannot solve the problem of 'who you are'. Summary of the Invention

[0006] The technical problem this invention aims to solve is to provide a method and system for efficient electronic signature handwriting retrieval based on Faiss for 1:N online electronic signature handwriting recognition tasks, thereby improving the speed and accuracy of large-scale electronic signature handwriting recognition and enhancing its efficiency and precision. It achieves efficient and accurate large-scale retrieval of handwritten electronic signature data; 1:N online handwritten electronic signature handwriting recognition enables the function of "searching for people by their handwriting and identifying people by their handwriting." This application, through one-to-many handwriting recognition, efficiently and quickly identifies "who you are" in a handwriting sample database of tens of millions, achieving millisecond-level response and efficient retrieval of the signature most similar to the handwriting to be identified.

[0007] The technical solution of this invention to solve the above-mentioned problems is to propose an efficient and accurate method and system for large-scale retrieval of handwritten electronic signatures based on the AI ​​similarity search framework Faiss and by combining deep learning to extract feature vectors of online electronic signatures.

[0008] To achieve large-scale online electronic signature handwriting retrieval and realize the goal of "searching for people by their handwriting and recognizing people by their handwriting", this invention provides an efficient electronic signature handwriting retrieval method and system based on Faiss. It utilizes big data and deep learning technologies to train and extract handwriting features with identity-distinguishing capabilities. By vectorizing the features of electronic signature data, a large-scale handwriting feature library is established. For the handwriting data to be identified, handwriting feature vectors are extracted and compared with feature vectors in the feature library to calculate similarity. The final top N handwriting retrieval results are obtained by ranking the similarity scores.

[0009] An efficient electronic signature handwriting retrieval method based on Faiss is proposed. This method involves training and extracting handwriting feature vectors based on supervised user signature data, training and optimizing deep learning convolutional neural network model parameters to establish a handwriting feature extraction model, extracting electronic signature handwriting features from a handwriting database online, adding feature data vectors for large-scale retrieval to establish a Faiss feature vector index library and determining the index type, extracting the handwriting feature vector to be retrieved, and comparing the handwriting feature vector to be retrieved with the signature feature data in the Faiss feature vector index library according to the index type to obtain the retrieval results.

[0010] Further preferably, the training and extraction of the handwriting feature vector comprises obtaining electronic signature data of each user ID's own signature in different ways on multiple different devices and enhanced handwriting sequence data of the above data, extracting coordinate information of each touch point on the trajectory sequence, pressure value and timestamp T information in the signature data, extracting user handwriting behavior features, style features, layout features, writing features, and pen features, and converting them into signature sequence data saved in a csv format file.

[0011] Further preferably, a large amount of positive writing training data for each user is constructed using sigma lognormal or self-encoding technology for handwriting sequence data enhancement, comprising: input handwriting sequence data preprocessing, sigma lognormal function fitting or word self-encoding model fitting, reconstruction of handwriting sequence through the fitted mathematical model, and generation of sequence data through parameter jitter control.

[0012] Further preferably, the establishment of the handwriting feature extraction model specifically comprises: calculating the classification loss function value through forward calculation of the user behavior features of different signature data under each user ID, deriving and updating the variable parameters in the network model through error back propagation, continuously reducing the loss function value in each iteration, obtaining the optimized network model and parameters, and obtaining the handwriting feature extraction model. The user ID is classified and the classification loss is calculated using the cross-entropy loss function, and the formula is called:

[0013] The classification loss function value L of the predicted value and the user ID category is calculated, where M represents the number of user IDs, y ic represents the category, and y ic is 1, otherwise 0, pi c represents the probability of belonging to the c category.

[0014] Further preferably, each convolution kernel of the convolution layer of the deep learning convolutional neural network model performs sliding filtering on the signature behavior features to extract the behavior features of each sliding window, the behavior features output by the previous layer are input to the next layer, the features are abstracted through depth, the dimension of the input feature vector is reduced through the pooling layer, and the input dimension is fixed to a specific length through the fully connected layer.

[0015] Further preferably, the signature data feature vector is retrieved, and the formula is called: The Euclidean distance d 12 between the signature data to be detected and the signature feature data in the index library is calculated, the similarity is determined according to the distance, the signature data corresponding to the top predetermined number of signature feature vectors with high similarity is taken as the retrieved signature data, and the retrieval result is obtained. Wherein, x 1k represents the kth element value of the reference vector in the feature library, x2k represents the value of the kth element of the handwriting feature vector to be retrieved, and n represents the dimension of the feature vector.

[0016] The application also provides a system for efficient electronic signature handwriting retrieval based on faiss, which comprises a data acquisition module, a model training module, a handwriting feature extraction module, and a signature feature retrieval and recognition module.

[0017] The application realizes one-to-many online electronic signature handwriting recognition, iteratively trains a deep learning model on a supervised large data set, ensures personalized representation of handwriting features and accuracy of handwriting retrieval, and improves handwriting feature similarity calculation and handwriting retrieval speed based on a faiss vector retrieval framework. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 Fig. 1 is a schematic diagram of the handwriting retrieval process of the application;

[0019] Figure 2 Fig. 2 is a structure diagram of a short-term variational autoencoder;

[0020] Figure 3 Fig. 3 is a hierarchical structure diagram of a convolutional neural network model;

[0021] Figure 4 Fig. 4 is a retrieval result page diagram. DETAILED DESCRIPTION

[0022] The embodiments of the application are described in detail below with reference to the accompanying drawings and specific examples, so as to facilitate understanding of the technical solutions of the application.

[0023] Figure 1The handwriting retrieval process is shown. The data acquisition module extracts the training handwriting feature vector based on the supervised user signature data, the model training module trains and optimizes the deep learning convolutional neural network model parameters, establishes the handwriting feature extraction model, the feature extraction module extracts the electronic signature handwriting features in the handwriting library for online retrieval, adds the feature data vector for large-scale retrieval to establish the faiss feature vector index library and determines the index type; the handwriting feature vector to be retrieved is extracted; the signature feature recognition module compares the handwriting feature vector to be retrieved with the signature feature data in the faiss feature vector index library according to the index type, and obtains the retrieval result. Specifically, the signature data in the handwriting library to be retrieved is traversed, input into the handwriting feature extraction model, and forward convolution layer, pooling layer and full connection layer inference operation is performed, and the fixed dimension feature vector (such as 1024 dimensions, the dimension is determined by the pooling layer) of each signature data is extracted through the full connection layer, and the extracted large-scale handwriting feature vector is added to faiss and the feature vector index is established.

[0024] The data acquisition module extracts the signature sequence data of the training handwriting feature model. A large number of user account ID electronic signature data is used, each user ID has multiple self-signatures on different devices and in different ways, the entire electronic signature data and the generated handwriting sequence data obtained by the above data enhancement are obtained, the X, Y coordinate information, pressure value P and timestamp T information of each touch point on the trajectory sequence are extracted, and the user signature behavior features are extracted from these original data, including: signature overall style features, stroke layout features, signature writing features, pen operation features, etc. The sequence data of each signature is saved in a data exchange format csv file, which is used for training and optimization of the feature extraction model. The deep learning convolutional neural network is trained and the handwriting feature extraction model is established. The user signature behavior features of different signature data of a large number of user IDs are used to forward calculate the predicted classification, and the cross-entropy loss function value is calculated with the real user ID category. The derivative is obtained by error back propagation, and the variable parameters (weights) in the handwriting feature extraction model are updated and optimized. Then in each iteration, the loss function value is continuously reduced according to the weight, and the optimized handwriting feature extraction model and parameters are obtained, and the handwriting feature extraction model is established.

[0025] Sigma lognormal (lognormal distribution accumulation) or auto-encoding technology is used to perform handwriting sequence data enhancement to construct a large number of positive writing training data under each user, including: input handwriting sequence data preprocessing, sigmalognormal function fitting or auto-encoding model fitting, handwriting sequence reconstruction through the fitted mathematical model, and sequence data generation through parameter jitter control. In this embodiment, a short-term variational auto-encoding model is used for data enhancement, Figure 2The short-term variational autoencoder structure diagram includes an encoder and a decoder. The encoder can adopt a bidirectional recurrent network, and the decoder adopts a Gaussian mixture model and a recurrent neural network. τ represents a Gaussian mixture model factor, which is used to adjust random disturbance of a data sequence, σ represents a speed mean value, μ represents a speed standard deviation, h represents data of the bidirectional recurrent neural network, and arrows represent two directions.

[0026] First, the handwriting sequence is split into short time sequences according to strokes; second, the handwriting sequence data is encoded and decoded by the short-term variational autoencoder to realize sequence-to-sequence conversion. The encoder adopts a bidirectional recurrent neural network to encode the input short time handwriting sequence into a latent feature vector z. Specifically, the bidirectional recurrent neural network encodes a forward sequence S and a reverse handwriting sequence into an h vector.

[0027] h → =encode → (S), h ← =encode ← (Sreverse), h=[h → ; h ← ]

[0028] The h vector is projected to a mean vector μ and a variance vector and then the standard deviation of the normal distribution is calculated. Finally, a distribution transformation is performed, that is, Z is sampled from , which is equivalent to sampling a sample from , so that the latent feature vector z is equal to the mean and standard deviation and the standard normal distribution transformation.

[0029] The decoder uses an autoregressive recurrent neural network to convert the latent feature vector into a handwriting sequence output. The autoregressive recurrent neural network predicts the probability distribution parameters of the next point at each time, and the output of the recurrent neural network is input into the Gaussian mixture model of M normal distributions to predict the final point information. Importantly, the generated handwriting sequence is not just a copy of the input sequence, but new handwriting data with similar features to the input handwriting sequence.

[0030] In order to make the model more robust to signature handwriting of different numbers of characters, the algorithm also uses the subsequence of the name as a means of enhancing this type of sample (such as “Wang Jiahao”). The subsequence can be a single character sequence, a multi-character sequence, a multi-character splicing sequence, etc. In this way, the retrieval effect of non-identical Chinese characters can be improved.

[0031] The faiss feature vector index is established, the index type is set according to the faiss, and the index type is determined according to the accuracy requirement, speed, memory size and the like. Faiss (Facebook AI Similarity Search, Facebook AI Similarity Search Library) defines multiple index types according to the search method, and the index types are mainly defined according to the balance of accuracy, time and memory, such as: types based on accurate search, types based on memory limit and index types based on data size. The index type of accurate search is used in this embodiment, and the extracted feature vector used for large-scale retrieval is added.

[0032] The feature extraction module obtains the feature vector of the handwriting data to be searched. The user signs an electronic signature online on a page, and uses the loaded handwriting feature extraction model (which has the ability of feature extraction through the above training, and extracts the features of the handwriting to be searched) to extract the handwriting feature vector of the uploaded handwritten electronic signature data, to obtain the fixed-dimension feature vector of the handwriting data to be searched.

[0033] The search and recognition module searches the feature vector, calculates the distance between the feature vector of the handwriting data to be searched and the feature vector in the library, and can use the Euclidean distance in this embodiment, and calls the formula: The Euclidean distance d between the reference vector in the feature library and the feature vector of the handwriting data to be searched is calculated 12 , wherein x 1k represents the kth element value in the reference vector in the feature library, x 2k represents the kth element value in the handwriting feature reference vector to be searched, k represents the kth element value in the vector, and n represents the dimension of the feature vector. Obtain the Euclidean distance between all handwriting feature vectors to be searched and the corresponding feature vectors in the feature library, and sort them from small to large according to the Euclidean distance. The smaller the distance, the higher the similarity of the handwriting features, and the closer to the front in the order. On the contrary, the lower the similarity, the farther to the back in the order; the signature data corresponding to the handwriting feature vectors after sorting is taken as the searched signature data, and the handwriting feature vectors with a predetermined number of high similarity are obtained as the searched signature data, to obtain the search result, and draw the corresponding signature image to visually display the searched handwriting data.

[0034] The model for extracting the electronic signature handwriting features adopts a deep learning convolutional neural network, such as Figure 3The convolutional neural network model hierarchy diagram is shown, including: convolutional layer, pooling layer, fully connected layer and classification layer, the convolutional layer is composed of a plurality of one-dimensional convolution kernels, each convolution kernel of the convolutional layer performs sliding filtering on the signature behavior features to extract the behavior features of each sliding window, the behavior features output by the previous layer are input to the next layer, and the features are abstracted through the depth of the network, the first, second, third and fourth parameters of the convolutional layer in the figure represent the number of convolution kernels, the size of the convolution kernel, the step size and the number of zeros added at the beginning and end of the data; the pooling layer reduces the dimension of the input and reduces the calculation amount, only the size and step size of the pooling kernel are two parameters; the fully connected layer fixes the input dimension of the feature vector to a specific length, and the network structure finally outputs 1024 dimensions. Facilitate the use of subsequent retrieval system features and user category classification.

[0035] The loss function is used to classify the user ID and calculate the loss value, and the classification loss can use the cross-entropy loss function. (Other functions known to those skilled in the art can also be used for classification and loss value calculation) The following will be further described by taking the cross-entropy loss function as an example: call formula:

[0036]

[0037] The predicted value and the classification loss function value L of the user ID category are calculated, wherein N represents the number of training samples, M represents the number of user ID categories, y ic represents the user category predicted by the feature extraction model, y ic is 1, otherwise 0, p ic represents the probability of predicting belonging to the c category. i represents the i-th sample, Li represents the loss value of the i-th sample, and c is the category number (which can be 1 to the number of users, and can specifically identify which person writes the signature), y ic represents the category, y ic is 1, otherwise 0, p ic represents the probability of predicting belonging to the c category.

[0038] As Figure 4 is a search result page diagram. If the electronic signature is "Li Lin", the electronic signature data corresponding to the handwriting feature vector with the closest distance or the largest distance is retrieved and drawn on the result page, and the signature corresponding to the ID with the closest distance has the highest similarity with the original signature. According to the distance threshold, it can be determined whether the most similar signature is signed by the person himself.

Claims

1. A method for efficient online handwritten electronic signature handwriting retrieval based on faiss, characterized in that, Based on supervised user signature data, a handwriting feature vector is trained and extracted, including: obtaining electronic signature data of each user ID signed in different ways on multiple different devices, extracting coordinate information, pressure value and timestamp information of each touch point on the trajectory sequence in the signature data, extracting user handwriting behavior features, converting them into signature sequence data saved in a csv format file, using a lognormal distribution accumulation or a short-term variational autoencoder to encode and decode the handwriting sequence data, the short-term variational autoencoder comprising: an encoder using a bidirectional recurrent network to encode short-time sequences split by the handwriting sequence into latent feature vectors, a decoder using a Gaussian mixture model and an autoregressive recurrent neural network to convert the latent feature vectors into new handwriting sequence outputs with similar features to the input handwriting sequence, and generating enhanced handwriting sequence data through parameter jittering; training and optimizing the parameters of a deep learning convolutional neural network model to establish a handwriting feature extraction model, wherein the convolutional layer uses a one-dimensional convolution kernel, the behavior features output by the previous layer are input into the next layer, the features are abstracted continuously, the input dimension is reduced through a pooling layer, and a fixed-length handwriting feature vector is output through a fully connected layer; online extraction of electronic signature handwriting features in a handwriting library, addition of feature data vectors to establish a faiss feature vector index library and use of an accurate search index type; extraction of a handwriting feature vector to be searched; the search and recognition module compares the handwriting feature vector to be searched with the signature feature data in the faiss feature vector index library according to the index type, calculates the Euclidean distance of the feature vectors in the index library based on the faiss index type, obtains the handwriting feature similarity according to the Euclidean distance, and obtains the signature data corresponding to the top N handwriting feature vectors with high similarity as the search result.

2. The method of claim 1, wherein, The handwriting feature extraction model specifically comprises: forward calculation of the user behavior feature of different signature data under each user ID to obtain a classification loss function value, derivation and update of variable parameters in the network model through error backpropagation, continuous reduction of the loss function value in each iteration, and obtaining of an optimized network model and parameters to obtain the handwriting feature extraction model.

3. The method of claim 2, wherein, The generation of enhanced handwriting sequence data through parameter jittering specifically comprises: encoding and decoding of the handwriting sequence data through the short-term variational autoencoder, reconstruction of the handwriting sequence through a fitting model, and generation of handwriting sequence data through parameter jittering control to realize sequence-to-sequence conversion.

4. The method of claim 3, wherein, When training the deep learning convolutional neural network model, a cross-entropy loss function is used to classify user IDs and calculate the classification loss, and the formula is called: The classification loss function value L of the predicted value and the user ID category is calculated, where M represents the number of user IDs, y ic represents the category, which belongs to the c category y ic is 1, otherwise 0, p ic represents the probability of predicting that the category belongs to the c category.

5. A system for efficient electronic signature handwriting retrieval based on faiss, characterized in that, The system comprises: a data acquisition module based on supervised user signature data, obtaining electronic signature data of each user ID on multiple different devices and generating enhanced handwriting sequence data according to the above electronic signature data, training and extracting handwriting feature vectors, comprising: obtaining electronic signature data of each user ID signing in different ways on multiple different devices, extracting coordinate information, pressure value and timestamp information of each touch point on the trajectory sequence in the signature data, extracting user handwriting behavior features, converting them into signature sequence data saved in a csv format file, using a logarithmic normal distribution accumulation or a short-term variational autoencoder to encode and decode the handwriting sequence data, the short-term variational autoencoder comprising: an encoder using a bidirectional recurrent network to encode the short time sequence split by the handwriting sequence into a latent feature vector, a decoder using a Gaussian mixture model and an autoregressive recurrent neural network to convert the latent feature vector into a new handwriting sequence output with similar features to the input handwriting sequence, and generating enhanced handwriting sequence data by parameter jittering; a model training module trains a deep learning convolutional neural network model according to the training handwriting feature vectors to obtain handwriting feature extraction parameters and establish a handwriting feature extraction model, wherein the convolutional layer uses a one-dimensional convolution kernel, the behavior features output by the previous layer are input to the next layer, the features are abstracted continuously, the input dimension is reduced through the pooling layer, and a fixed-length handwriting feature vector is output through the fully connected layer; the handwriting feature extraction model extracts electronic signature handwriting features in the handwriting library to be searched, adds feature data vectors to establish a faiss feature vector index library and uses an accurate search index type to extract the handwriting feature vector to be searched; a signature feature retrieval and recognition module compares the handwriting feature vector to be searched with the signature feature data in the faiss feature vector index library according to the index type, calculates the Euclidean distance between the handwriting feature vector and the feature vector in the index library based on the faiss index type, obtains the handwriting feature similarity according to the Euclidean distance, and obtains the signature data corresponding to the top N handwriting feature vectors with high similarity as the search result.

6. The system of claim 5, wherein, The handwriting feature extraction model is established by: using the user signature behavior features of different signature data of each user ID to forward calculate a classification loss function value, deriving and updating the variable parameters in the optimization network model through error back propagation, continuously reducing the loss function value in each iteration, obtaining the optimized network model and parameters, and obtaining the handwriting feature extraction model.

7. The system of claim 5 or 6, wherein, The data acquisition module further comprises an encoder and a decoder, the encoder uses a bidirectional recurrent network to encode the handwriting sequence into a latent feature vector, the decoder uses a Gaussian mixture model and a recurrent neural network to convert the latent feature vector into a handwriting sequence output, and the handwriting sequence is split into several short time sequences according to strokes.

Images