Cross-modal Character Handwriting Verification Method, System, Device and Storage Medium

Abstract

The present invention discloses a handwriting verification method for cross-modal recognizable single-character characters, which relates to the technical field of electronic signatures. The online electronic writing character data and paper writing character data of the signer are obtained and associated with the writer identity identifier to construct a database including a training set, a validation set and a test set; the deep learning neural network is trained and verified. The training set data is used to optimize the network model. The character space attention mechanism is adopted to refer to the standard sample, learn the unique or important writing methods of the standard sample, capture the key differences and unique commonalities of the characters, and obtain the writer's handwriting feature vector of the multi-modal character image. The handwriting recognition model is obtained by optimizing the loss function of the handwriting feature vector; the verified handwriting recognition model collects the online handwritten character data set, obtains the characters to be verified and the handwriting features for similarity calculation, and comprehensively judges and verifies the identity of the signer.

Description

Technical Field

[0001] The present invention relates to the technical field of online handwritten electronic handwriting verification, and particularly relates to a cross-modal character handwriting verification method, system, device, and storage medium. Background Art

[0002] With the advancement of the paperless process and the popularity of touch mobile devices and electronic writing devices, online handwriting appears in all aspects of people's lives, and verification and recognition based on online handwriting have also received extensive attention from professionals. Especially with the development of artificial intelligence and deep learning technologies, by building a convolutional neural network to model online handwriting images or by using a recurrent neural network to model online time-series data, on the one hand, personalized handwriting representations unique to each user can be learned through representation learning methods, and then handwriting verification can be performed using the similarity of the representations. On the other hand, contrastive learning can be carried out by constructing comparison sample pairs to learn the differences between positive and negative sample pairs for handwriting verification. However, basically all the above-mentioned solutions take the entire handwriting data as the input. For example, when performing handwritten electronic signature verification, the entire handwritten electronic signature is used as the modeling object. This often has the problem that it is difficult to align characters due to signature differences. At the same time, in actual business applications, only the similarity of the handwriting is restricted, and no recognizable restrictions are imposed on each character. There are often various situations such as artistic fonts and scribbles that cause problems of unrecognizability, affecting the quality and effectiveness of the signed content.

[0003] The Chinese patent application with the publication number CN201310405207.2 and the title "A method for template expansion of online handwriting authentication based on characters" discloses an online test handwriting and registered handwriting comparison and template automatic expansion mechanism. When the comparison between the registered handwriting and the test handwriting passes, the unregistered characters written are automatically registered. It completely relies on the algorithm comparison result and does not perform content verification on the unregistered characters written, which poses a risk of contaminating the registered data. The Chinese patent application with the publication number CN202111540184.7 and the title "Signature authentication system and method based on channel attention mechanism" directly reverses the pixels of the entire signature image and splices them as the input of a multi-channel network, and uses the channel attention mechanism model combined with the cross-entropy loss function for binary classification. Since the signature handwriting image is too sparse and the background occupies a very large proportion, it is very difficult to align the multi-channel handwriting, and it is difficult for the convolution operation to learn the fine-grained information of the corresponding strokes. The Chinese patent application with the publication number CN201611122474.9 and the title "Offline handwritten signature authentication method and system" discloses an offline handwritten signature authentication method and system. Through preprocessing such as binarization, shear boundary, size normalization, tilt correction, and distance reduction of the offline signature, then extracting pulse-coupled neural network features from the grayscale image of the shear boundary, extracting texture features including local binary pattern features and gray-level co-occurrence matrix features from the normalized grayscale image, and extracting low-order moment features from the normalized binary image. Then, these features of the entire signature image are dimensionally reduced. After obtaining the feature vector with a lower feature dimension, calculate the distance between them, estimate their similarity, and judge its authenticity through the threshold method, or directly use a classifier to train and predict it. The invention often has problems such as poor generalization and low accuracy in handwriting comparison through traditional image processing and manual features. Summary of the Invention

[0004] In view of the problems in the prior art that usually take the entire signature as the modeling object, it is very difficult to align the handwriting, the signature data is easily contaminated, etc., resulting in low accuracy in signature recognition and verification, the present invention proposes an online handwriting verification technology based on distinguishable single-character characters, aiming to standardize the distinguishability of online handwritten Chinese characters, and on this basis, perform handwriting verification on individual characters, and finally obtain the final handwriting identity authentication result through comprehensive judgment of multiple characters.

[0005] The technical solution of the present invention to solve the above technical problems is to adopt single-character collection, which can effectively avoid situations such as connected writing, abbreviations, and artistic signatures during the signing process. By introducing a character recognition module to standardize online handwritten content, the signed character content can be made recognizable. Thus, a cross-modal recognizable single-character handwriting verification method is provided. Obtain the electronic signature handwriting image, the digitized paper signature image, and the online writing character handwriting, and associate the signed user identity identifier and store it in the database for handwriting registration; preprocess the data in the database, clean abnormal data, eliminate the differences between paper data and electronic data, and the preprocessed data is echoed as a multi-dimensional character trajectory image, or directly combine the sequence features into a two-dimensional sequence; Echo the character trajectory image, two-dimensional sequence, and paper signature image as training set samples and input them into the deep learning neural model for training. Adopt the character space attention mechanism to refer to the standard sample, learn the unique or important writing methods of the standard sample, capture the key differences and unique commonalities of the characters, and obtain the writer's handwriting feature vector of the multi-modal character image. Optimize the loss function of the handwriting feature vector to obtain the handwriting recognition model; use the verification set to statistically verify the handwriting recognition model to determine the final single-character recognition model. The single-character recognition model calculates the similarity according to the online handwritten character data set and comprehensively judges and verifies the signer's identity.

[0006] Further preferably, divide the registered data into training set, verification set, and test set according to different classifications of signers, signing devices, or media, and determine positive and negative samples. The characters with the same content signed by the same person in the training set are positive samples, and the characters with different content signed by the same person and the characters with the same content signed by other people are used as negative samples.

[0007] Further preferably, the preprocessing includes removing data with out-of-bounds trajectory coordinates, reversed timestamps, too short point lengths, too short signing times, and abnormal signing directions, removing duplicate points, outliers, and isolated points, repairing the stroke state and pressure values, unifying the handwriting sampling rate, and performing denoising, binarization, and thinning processing on the digitized paper handwritten character data; The echoing into a character trajectory image includes echoing into a multi-dimensional character image jointly according to the character trajectory information and the generated handwriting features, and uniformly scaling to a fixed size determined by the feature dimension. The two-dimensional sequence is [sequence length * feature dimension].

[0008] Further preferably, the backbone feature extraction network includes: two multi-level spatial attention modules are cascaded through transition layers and dense connection modules respectively. One path takes the character samples in the training set as input, and the other path takes the standard sample corresponding to the content of the character sample as input. Obtain the feature difference weights at each level according to the character sample and the standard sample, and merge the features of the feature difference weights at each level and input them into a 1*1 convolutional layer to obtain the writer's handwriting feature vector of the multi-modal character image.

[0009] Further preferably, the backbone feature extraction network learns the handwriting characteristics of Chinese characters. For the same character written by different people, the backbone feature extraction network focuses on the differences between the standard sample and the online handwritten character, outputs the probabilities of different writers according to the multi-branch classification function, trains the writer classifier through the classification function, creates a writer classifier for each Chinese character, or multiple Chinese characters share one writer classifier.

[0010] Further preferably, perform one or more layers of feature vector operations on the feature map of the representation network by inputting the character samples in the training set. Train the standard writing of different Chinese characters through a representation network, so that the representation network can extract the feature vectors of the standard Chinese character writing. Input the feature vectors of the standard writing and the stylized training set samples into the backbone feature extraction network, and obtain the writer handwriting feature vectors of different writers' multi-modal character images according to the differential information between the handwritten characters of different users and the standard Chinese character writing.

[0011] Further preferably, optimize the backbone feature extraction network through the loss function of the writer handwriting feature vector, reduce the within-class vector angle and increase the between-class vector angle to obtain the character handwriting recognition model. According to the formula:

[0012]

[0013] Calculate the loss function loss. Among them, N represents the number of training samples, m represents the angular margin parameter, θ j represents the angle between the j-th sample feature vectors, and s represents the feature scaling factor.

[0014] Further preferably, use the validation set to statistically verify the validation effect of the handwriting recognition model. Construct comparison sample pairs using the positive and negative samples in the validation set, assign different weights to each single-character according to the AUC index, calculate the feature similarity between the sample pairs, statistically analyze and fit the corresponding accuracy rates under different similarities of each multi-modal character. The handwriting recognition model with an accuracy rate reaching the threshold is the final single-character recognition model. Among them, according to the formula:

[0015]

[0016] Calculate the similarity P of the character pair (A i , B i ). Among them, AUC i represents the AUC evaluation index corresponding to the single-character i in the test set, N represents the number of registered signatures or text lines, and prob(A i , B i ) represents the probability value corresponding to the similarity of the i-character pair (A i , B i ) obtained by fitting.

[0017] In the second aspect, the present invention proposes a cross-modal handwriting verification system that can identify single-word characters, including: a handwriting registration module, a data preprocessing module, a model training module, a handwriting recognition module, a data verification module, a feature extraction module, a single-word character recognition model, a handwriting registration module, which is used to obtain the signer's online signature page writing characters and handwriting data, and paper writing character data and associate the writer's identity identification to perform handwriting registration; a preprocessing module, which is used to preprocess the data in the database, clean abnormal data, eliminate the difference between paper data and electronic data, and echo the preprocessed data into a multi-dimensional character trajectory image, or directly combine the sequence features into a two-dimensional sequence; a model The training module uses echoed character trajectory images, two-dimensional sequences, and paper signature images as training set samples to input into the deep learning neural model for training. It adopts the character space attention mechanism to refer to the standard samples and learn the unique or important writing methods of the standard samples. The feature extraction module captures the key differences and unique commonalities of the characters, obtains the handwriting feature vector of the writer of the multimodal character image, and obtains the handwriting recognition model through the loss function optimization of the handwriting feature vector; the data verification module uses the verification set to statistically verify the handwriting recognition model to determine the final single-word character recognition model. The single-word character recognition model calculates the similarity based on the online handwritten character dataset, and comprehensively judges and verifies the identity of the signer.

[0018] Further preferably, the single-word character recognition model includes: a convolution layer, a maximum pooling layer, a multi-level spatial attention module, a feature merging module, and a 1*1 convolution layer. The spatial attention modules at each level are connected through a transition layer and a dense connection module. The single-word character samples and the standard character samples are input into the multi-level spatial attention module through the convolution layer and the maximum pooling layer. The feature difference weights at each level are obtained in turn through the spatial attention module at each level. The feature difference weights at each level are input into the 1*1 convolution layer output feature vector through feature merging. The feature vector selection is then connected to the multi-layer perception machine optimization to optimize the category loss function.

[0019] In a third aspect, the present invention proposes an electronic device comprising: one or more processors, a memory, and one or more applications, which are stored in the memory and configured to be loaded and run by the one or more processors to execute the cross-modal recognizable single-word character handwriting verification method described above.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal handwriting verification method for recognizing single-word characters described in the above steps.

[0021] In order to achieve the legibility of handwritten content, compare and verify handwritten notes, and then conduct identity verification. The present invention provides a handwriting verification method based on distinguishable single-character characters, which ensures the legibility of characters by limiting the online signing content and handwriting collection form and the recognizable constraints on the collected content. Electronic handwritten characters, character trajectories, and electronic images of paper characters are collected, and combined with the character OCR algorithm. Based on multi-modal characters, a character spatial attention mechanism is used to refer to standard samples, learn the unique or important writing methods of standard samples, capture the key differences and unique commonalities of characters, and obtain the handwriting feature vector of the writer of the multi-modal character image. The model adopts dense connections and has the ability to possess high-level semantic information and low-level detail attention, obtaining better representation ability, eliminating the interference of the background and the problem of sparse handwriting, solving the problem that the entire signature or multiple signatures cannot be centered and are easily interfered by the background of the space between characters. Through comprehensive comparison of single-character handwriting, the accuracy and effect of handwriting comparison of the overall signature or signed content can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The figure shows a schematic flow diagram of the single-character handwriting verification method of the present invention;

[0023] Figure 2 It is a schematic flow diagram of the character handwriting registration of the present invention;

[0024] Figure 3 It is a schematic diagram of the backbone network structure of one of the embodiments;

[0025] Figure 4 Schematic diagram of the page for collecting characters in regions. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to facilitate a clear understanding of the present invention, make the technical problems to be solved, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. In the following description, providing specific details such as specific configurations and components is only to help a comprehensive understanding of the embodiments of the present invention. Therefore, those skilled in the art should clearly understand that various changes and modifications can be made to the embodiments described here without departing from the scope and spirit of the present invention. In addition, for the sake of clarity and conciseness, the description of known functions and structures is omitted. It should be understood that the embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0027] The online handwritten signature page of the signing device can collect single handwritten characters or characters in a form that can be split, including but not limited to single-screen single-character signing, regional signing or character segmentation algorithm acquisition, and the system knows the content of the handwritten characters. Use the character recognition algorithm to verify the handwritten input characters to determine whether the signature content is the preset characters. Train the image-based single-word handwriting classification or representation network model, optimize the classification loss, update the network weights, so that samples of the same person and the same word are aggregated in the vector space, and other samples are pulled away; use the trained network to extract character representations, compare the similarity of the representation vectors of the retained characters and the verification characters, and perform verification to determine whether the characters are written by the same person; combine the comparison results of each character to obtain the handwriting verification result of the handwritten text line.

[0028] like Figure 1 The figure shows a flow chart of the single-word handwriting verification method of the present invention, which includes a handwriting registration module, a handwriting recognition module, a data verification module, a data preprocessing module, a content verification module, a feature extraction module, and an online handwritten character data set acquisition module. The handwriting registration module obtains online written characters and handwriting-associated users for registration; the handwriting recognition module extracts characters to be verified and handwriting, performs data verification, preprocessing, and content verification on the registered characters and handwriting, and the characters to be verified and handwriting, and then uses a trained and optimized handwriting feature extraction model to extract user handwriting features; obtains a large number of online handwritten character data sets, constructs a single-word character handwriting recognition model through model training and model verification, and calculates similarity based on the registered characters and handwriting features, and the characters to be verified and handwriting features, and comprehensively judges whether the verified signature or text line is written by the same person.

[0029] Handwriting registration: After the characters written on the signature page pass the content verification, the text characters and the signature user's unique identification are submitted to the database for handwriting registration. Figure 2 This is a diagram of the character handwriting registration process. It includes: character collection, handwriting registration, determining whether the collected data is valid, if so, pre-processing the data and identifying the characters, determining whether it is a preset character, and if so, registering successfully.

[0030] Specifically, the data acquisition module supports the system to collect a single handwritten character, obtain the electronic signature handwriting image, the digitized paper signature image, or the online character handwriting information, associate the unique identifier of the signing user and store it in the database for subsequent handwriting comparison. The online character handwriting information includes character handwriting information such as stroke trajectory coordinates (x, y), pen touch state s, pressure P, and timestamp T. The electronic signature signing page on the signing device includes, but is not limited to, signing forms that can collect complete single characters, such as single-screen single-character writing, partitioned single-character writing, or complete collection and then segmentation. According to the preset text content, the handwriting registration module writes characters on the signature page according to the standard character prompts. After passing the content verification, the text characters and the unique identifier of the signing user are submitted to the database for handwriting registration.

[0031] For content verification, a character recognition model can be used to recognize the character content of the collected single character, or the content is set to pass the content verification if it exceeds the specified similarity. The Chinese character corresponding to the maximum recognition probability is used as the recognition result, and it is compared whether the recognized content is the preset character content. If so, the verification is passed and the submission is completed; if not, the writer is prompted to rewrite the content.

[0032] A database is constructed, and the same and different characters written by different writers are used as different categories to construct a dataset. In order to be compatible with the cross-modal comparison of paper signature data and electronic signature data, the online electronic writing character data and paper writing character data of the same content of the same signer are obtained, and the identity identifier of the writer is associated to construct a dataset. In addition to the collected online handwritten character data, digitized paper handwritten character data is further incorporated for construction, and the training set, validation set, and test set are divided by category. The training set, validation set, and test set can be divided according to different signers and signing devices or media. In the training set, the characters of the same content signed by the same person are positive samples, and the characters of other content signed by the same person and the characters of the same content signed by other people are negative samples.

[0033] The preprocessing module preprocesses the data in the database, performs effective cleaning, and cleans abnormal data, including: abnormal data such as out-of-bounds trajectory coordinates, reversed timestamp, too short point length, too short signing time, abnormal signing direction, etc.; removing duplicate points, repairing the pen touch state, repairing the pressure value, removing wild points, outliers, and unifying the handwriting sampling rate; performing preprocessing such as denoising, binarization, and thinning on the digitized paper handwritten character data to eliminate the differences between paper data and electronic data.

[0034] The preprocessed data is echoed as a multi-dimensional character trajectory image, or is not echoed as a character trajectory image, and the sequence features are directly combined into a two-dimensional sequence. Character image features include: can be directly applied to the corresponding stroke trajectory; another method is not to echo as a trajectory image, forming a two-dimensional sequence (i.e., image) such as [sequence length * feature dimension]. For example, the preprocessed data is jointly echoed into a multi-dimensional character image based on the handwritten character trajectory information and the handwriting features generated by other paper electronic images, and uniformly scaled to a fixed size determined by the feature dimension (such as 128x128x n, where n is the feature dimension including: velocity, acceleration, angular velocity, pressure, or other high-order features).

[0035] A deep learning neural network consisting of a backbone feature extraction network and a loss function network was established. The network model was optimized using multimodal training data. The training set consisted of samples containing the same Chinese characters from the same person as each other, with each other serving as positive samples and the rest as negative samples. The preprocessed multimodal data was then fed into the backbone feature extraction network model using the echo image and preprocessed paper images as training samples. The goal of model training was to reduce the distance between positive samples and increase the distance between negative samples.

[0036] The backbone feature extraction network can be a convolutional network structure such as VGGS, ResNet, DenseNet, or a custom network. The backbone feature extraction network learns the handwriting characteristics of handwritten Chinese characters. For the same character written by different people, the backbone feature extraction network's attention module focuses on the differences between the standard character and the handwritten character. The fully connected layer and multi-branch classification function output the probability of different writers. The classification function constructs a network to train a writer classifier for classification. A network can be built to train a writer classifier for each Chinese character, or a single network can be used to train a writer classifier for multiple characters. Due to the large number of Chinese characters, to enhance the ability to distinguish between different characters within a class, the character content can be used as a spatial attention mechanism.

[0037] The backbone feature extraction network structure of this embodiment adopts a character space attention mechanism, which allows the network to refer to standard samples, learn the unique or important writing methods of the samples, capture the key differences and unique commonalities of the characters, and the dense connection module merge function concat connects multiple arrays to aggregate multi-layer features. It has the ability to pay attention to high-level semantic information and low-level details, and can obtain better representation capabilities.

[0038] like Figure 3This is a schematic diagram of the backbone feature extraction network structure in the embodiments of the present invention. Two multi-level spatial attention mechanism modules are cascaded through transition layers and dense connection modules respectively. One takes the character samples in the training set as input, and the other takes the standard samples corresponding to the content of the character samples in the training set as input, and processes the character samples and standard samples to obtain the feature difference weights at each level. As shown in the figure, the single-character input on the left is the training set sample, and the standard sample input on the right is the standard sample corresponding to the content of the training set sample. The standard sample can be obtained by means such as averaging or specifying.

[0039] Specifically, the backbone feature extraction network includes: a convolutional layer, a max pooling layer, a multi-level spatial attention module, a feature merging module, a 1*1 convolutional layer. The spatial attention modules at all levels are connected through transition layers and dense connection modules. The positive and negative samples in the training set and the standard characters are input into the convolutional layer, processed by the max pooling layer, and then the feature difference weights at each level are obtained through the multi-level spatial attention module, transition layer, and dense connection module. The features output by each transition layer are merged and input into the 1*1 convolutional layer to obtain the writer's handwriting feature vector of the multi-modal character image.

[0040] Another implementation of the character spatial attention mechanism is: directly implement one or more layers of feature vector operations on the feature map of the characterization network with the character samples in the training set as input. The feature vector is learned for different Chinese character networks. Different standard Chinese character writings can be trained through a separate sub-network, so that the sub-network can extract the feature vectors of the standard Chinese character writings, extract the standard writing character vectors, and apply the extracted standard writing vectors to the input stylized training set samples. According to the differential information between the handwritten characters of different users and the standard Chinese character writings, the writer's handwriting feature vector of the multi-modal character image of different writers is obtained.

[0041] Based on the features output by the backbone feature extraction network, they are represented by digital vectors of a specific dimension to characterize the handwriting information of the writer. Optionally, a multi-layer perceptron can be connected for optimization, and iterative training and verification are performed to optimize the class loss function. The loss function can adopt methods such as contrastive loss, softmax loss, and triplet loss.

[0042] The embodiments of the present invention adopt an 18-layer residual network with an Additive Angular Margin (ArcFace) loss function as the loss function for learning the handwriting feature vector to optimize the backbone feature extraction network to reduce the angle of the in-class vectors and increase the angle of the inter-class vectors. Specifically, according to the formula:

[0043]

[0044] Calculate it as the loss function loss. Among them, N represents the number of training samples, m represents the angular interval parameter, θ j represents the angle of the j-th category, s represents the feature scaling factor, y i represents the category output by sample i, and n represents the number of categories.

[0045] Use the training set samples and validation set sample data to iteratively train the backbone feature extraction network, calculate the error between the model prediction and the official label through the loss function, optimize the network model parameters, obtain the character handwriting recognition model, extract the handwriting feature vectors, make the angles between the feature vectors of the same category of handwritten characters smaller, and the included angles of the feature vectors extracted between different categories larger.

[0046] Obtain the character feature vector representation by the weights of the multi-modal sample character classification model trained by the backbone feature extraction network. Since the categories of the character handwriting recognition model are set by people and characters, the output multi-modal character feature vectors have the habit information of the writer, and the similarity can be calculated by Euclidean, cosine or other vector measurement methods. According to the constraint of the loss function, the smaller the vector included angle between the same categories, and the larger the vector included angle between different categories.

[0047] In this embodiment, the cosine similarity is used to compare the similarity degree of two character vectors, and it is determined whether two characters with the same content are written by the same person according to the pre-set similarity threshold.

[0048] According to the formula:

[0049]

[0050] Calculate the cosine similarity of two characters, where A and B represent the feature vectors extracted by the two characters through the handwriting recognition model, A i , B i represents the i-th element value of the corresponding feature vectors (A, B). n represents the length of the vector, and ||A|| represents the modulus of vector A.

[0051] Use the validation set to statistically analyze the test effects of the trained handwriting recognition model on Chinese character characters obtained in different writing styles and on different media, such as accuracy or equal error rate (EER), as well as the accuracy and AUC metrics of each Chinese character under each threshold segment. Use the positive and negative samples in the validation set to construct comparison sample pairs, calculate the feature similarity between the sample pairs, and statistically analyze and fit the corresponding accuracy rates of each multimodal character at different similarity levels. For signatures or signed text lines, through the comparison results of each character and by synthesizing the similarities of each character, obtain the final comparison result of the signature or text line. The statistical methods include, but are not limited to, voting method, averaging method, weighted discrimination method, etc. In this embodiment, the comprehensive discrimination method is taken as an example to compare characters. According to the formula:

[0052]

[0053] Calculate the similarity P of the character pair (A i , B i ), where AUC i represents the AUC evaluation index corresponding to the single-character i in the test set, N represents the number of registered signatures or text lines, and prob(A i , B i ) represents the probability value corresponding to the similarity of the i-character pair (A i , B i ) obtained by fitting.

[0054] Assign different weights to each single-character through the AUC metric, fully considering the different impacts of the complexity of single-character on the overall comparison of signatures or text lines. Among them, prob(A i , B i ) can be statistically obtained from the precision at different thresholds in the validation set or can be obtained through sigmoid function conversion. If sigmoid function conversion is adopted, it can be calculated according to the formula:

[0055]

[0056] Calculate the probability value prob(A i , B i ) obtained by fitting, where similarity represents the similarity of the character pair (A i , B i ), and threshold represents the distance threshold corresponding to the single-character i.

[0057] As shown in Figure 4 is a schematic diagram of the regional character acquisition page. For handwriting comparison based on the above recognizable characters, when the system collects handwritten character data, the signing page must be able to collect single handwritten characters or the form in which the collected characters can be split, including but not limited to single-screen single-character signing, regional signing, or obtained through character cutting algorithms, etc.Figure 4 As one of the ways of signing by sub-regions, the writing area is restricted by a cross grid, that is, each character can only be written within the cross grid area, and individual characters within each cross grid are extracted for subsequent content verification and individual character handwriting comparison.

Claims

1. A handwriting verification method for cross-modal recognizable single-character words, characterized in that, Obtain electronic signature handwriting images, digitized paper signature images, and online writing character handwritings, associate the signed user identity identifiers and store them in the database for handwriting registration; preprocess the data in the database, clean abnormal data, eliminate the differences between paper data and electronic data, and the preprocessed data is echoed as a multi-dimensional character trajectory image, or directly combine the sequence features into a two-dimensional sequence; echo the character trajectory image, two-dimensional sequence, and paper signature image as training set samples and input them into a deep learning neural model for training. Adopt a character space attention mechanism to refer to the standard samples, learn the unique or important writing methods of the standard samples, capture the key differences and unique commonalities of the characters, obtain the writer's handwriting feature vectors of the multi-modal character images, and optimize through the loss function of the handwriting feature vectors to obtain a handwriting recognition model; use the validation set to statistically verify the handwriting recognition model to determine the final single-character handwriting recognition model. The single-character handwriting recognition model calculates the similarity based on the online handwritten character dataset and comprehensively judges and verifies the signer's identity. During training, perform one or more layers of feature vector operations on the feature map of the characterization network by inputting the character samples in the training set. Train the standard writing methods of different Chinese characters through a characterization network, so that the characterization network can extract the feature vectors of the standard Chinese character writing methods. Input the feature vectors of the standard writing methods and the stylized training set samples into the backbone feature extraction network. According to the differential information between the handwritten characters of different users and the standard Chinese character writing methods, obtain the writer's handwriting feature vectors of the multi-modal character images of different writers; the backbone feature extraction network learns the handwriting features of handwritten Chinese characters. For the same character written by different people, the backbone feature extraction network focuses on the differences between the standard samples and the online handwritten characters, and outputs the probabilities of different writers according to the multi-branch classification function. Train the writer classifier through the classification function. Create a writer classifier for each Chinese character, or multiple Chinese characters share one writer classifier.

2. The method according to claim 1, wherein Divide the registered data into training set, validation set, and test set according to different classifications of signers, signing devices, or media, determine positive and negative samples. The characters with the same content signed by the same person in the training set are positive samples, and the characters with different content signed by the same person and the characters with the same content signed by other people are used as negative samples.

3. The method according to claim 1, characterized in that, The preprocessing includes eliminating data with out-of-bounds trajectory coordinates, reversed timestamps, too short point lengths, too short signing times, and abnormal signing directions, removing duplicate points, wild points, and outliers, repairing the stroke state and pressure values, unifying the handwriting sampling rate, and performing denoising, binarization, and thinning processing on the digitized paper handwritten character data; the echoed character trajectory image includes echoing into a multi-dimensional character image jointly according to the character trajectory information and the generated handwriting features, and uniformly scaling to a fixed size determined by the feature dimension. The two-dimensional sequence is: sequence length * feature dimension.

4. The method according to any one of claims 1 to 3, characterized in that The backbone feature extraction network includes: two multi-level spatial attention modules cascaded through transition layers and dense connection modules respectively. One takes the character samples in the training set as input, and the other takes the standard samples corresponding to the content of the character samples as input. The feature difference weights at each level are obtained by processing the character samples and the standard samples, and the features of the feature difference weights at each level are merged and input into a 1×1 convolutional layer to obtain the writer's handwriting feature vector of the multi-modal character image.

5. The method according to claim 1, characterized in that The backbone feature extraction network is optimized through the loss function of the writer's handwriting feature vector to reduce the angle of the intra-class vectors and increase the angle of the inter-class vectors, resulting in a character handwriting recognition model. The loss function loss is calculated according to the formula: Among them, N represents the number of training samples, m represents the angular interval parameter, and θ j represents the angle between the feature vectors of the j-th sample, s represents the scaling factor, and y i represents the category output by sample i, and n represents the number of categories.

6. The method according to claim 5, characterized in that, The verification effect of the handwriting recognition model is statistically verified using the verification set. The positive and negative samples in the verification set are used to construct comparison sample pairs. Different weights are assigned to each single-character according to the Auc index, the feature similarity between the sample pairs is calculated, and the accuracy corresponding to different similarities of each multi-modal character is statistically calculated and fitted. The handwriting recognition model with an accuracy reaching the threshold is the final single-character recognition model. Among them, according to the formula: Calculate the similarity P of the character pair (A i , B i ), where AUC i represents the AUC evaluation index corresponding to the single-character i in the test set, N represents the number of registered signatures or text lines, and prob(A i , B i ) represents the probability value corresponding to the similarity of the i character pair (A i , B i ).

7. A handwriting verification system for cross-modal recognizable single-character words, characterized in that, It includes a handwriting registration module, a data preprocessing module, a model training module, a handwriting recognition module, a data verification module, a feature extraction module, and a single-character recognition model; the handwriting registration module is used to obtain the written characters and handwriting data of the signer's online signature page, and the paper-written character data is associated with the writer's identity identifier for handwriting registration; The preprocessing module is used to preprocess the data in the database, clean abnormal data, eliminate the differences between paper data and electronic data. The preprocessed data is echoed as a multi-dimensional character trajectory image, or the sequence features are directly combined into a two-dimensional sequence; the model training module uses the echoed character trajectory image, two-dimensional sequence, and paper signature image as training set samples to input into a deep learning neural model for training. The character spatial attention mechanism is adopted to refer to the standard samples to learn the unique or important writing methods of the standard samples. The feature extraction module captures the key differences and unique commonalities of the characters to obtain the writer's handwriting feature vector of the multi-modal character image. The handwriting recognition model is obtained through the optimization of the loss function of the handwriting feature vector; The data verification module uses the verification set to statistically verify the handwriting recognition model to determine the final single-character recognition model. The single-character recognition model calculates the similarity according to the online handwritten character dataset to comprehensively judge and verify the identity of the signer; During training, one or more layers of feature vector operations are implemented on the feature map of the characterization network with the character samples in the training set as input. The standard writing of different Chinese characters is trained through a characterization network, enabling the characterization network to extract the feature vectors of the standard Chinese character writing. The feature vectors of the standard writing and the stylized training set samples are input into the backbone feature extraction network. According to the differential information between the handwritten characters of different users and the standard Chinese character writing, the writer's handwriting feature vectors of multi-modal character images of different writers are obtained. The backbone feature extraction network learns the handwriting features of handwritten Chinese characters. For the same character written by different people, the backbone feature extraction network focuses on the differences between the standard samples and the online handwritten characters, and outputs the probabilities of different writers according to the multi-branch classification function. The writer classifier is trained through the classification function. A writer classifier is built for each Chinese character, or multiple Chinese characters share one writer classifier.

8. The system according to claim 7, wherein The single-character recognition model includes: a convolutional layer, a max pooling layer, a multi-level spatial attention module, a feature merging module, and a 1*1 convolutional layer. The multi-level spatial attention modules are connected through a transition layer and a dense connection module. The dense connection module merges the features of multiple layers through the concat function of the merging function. The single-character samples and the standard character samples are input into the multi-level spatial attention module through the convolutional layer and the max pooling layer. The feature difference weights at each level are obtained through each level of the spatial attention module in turn. The feature difference weights at each level are input into the 1*1 convolutional layer through feature merging to output the feature vectors. The feature vectors are then selected and optimized through a multi-layer perceptron to optimize the category loss function.

9. An electronic device, characterized in that, Including: One or more processors, a memory, and one or more applications, which are stored in the memory and configured to be loaded and run by the one or more processors to execute the cross-modal distinguishable single-character handwriting verification method according to any one of claims 1-6.

10. A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the cross-modal distinguishable single-character handwriting verification method according to any one of claims 1-6.

Images