Electronic signature handwriting identification method based on VILT multi-mode model

Abstract

The invention discloses an electronic signature handwriting identification method based on a VILT multi-mode model. The method comprises the following steps: 1) collecting electronic signature information of a party; 2) preprocessing the collected information; 3) taking the preprocessed information as a sample; 4) constructing a multi-modal learning model based on a Transform network, namely, a VILT multi-modal model; 5) respectively sending the electronic data information and the electronic image information in the sample into a VILT multi-modal model, and carrying out model training; 6) performing optimization and parameter adjustment on the model by relying on the verification set; 7) taking the electronic signature handwriting of the party needing to be identified as a detected material, and sending the detected material into the trained and optimized VILT multi-modal model to obtain a similarity value; and 9) outputting a detection result. According to the invention, the mapping capability of the electronic image information to the electronic data information is improved, and the accuracy of electronic signature handwriting identification is improved.

Description

Technical Field

[0001] The invention proposes an electronic signature handwriting identification method based on a VILT multimodal model, and particularly relates to the technical field of handwriting identification. Background Art

[0002] An electronic signature is an electronic signature formed by writing with a stylus on an electronic writing device. Its handwriting information consists of two parts: electronic signature data information and electronic signature image information. Electronic signature data information refers to a data set recorded by an electronic device that is related to the entire writing process and is composed of a series of variable quantities; electronic signature image information refers to image information that records the outline of the electronic signature handwriting by saving screenshots and exporting them to PDF, jpg, etc., thereby reflecting the morphological characteristics of the electronic signature handwriting.

[0003] When the stylus touches the touch screen of the writing device, the acquisition device will collect the position coordinates of the pen tip on the screen at fixed time intervals and record them as a single number or an X-axis and Y-axis coordinate data pair. Through these data, the stroke trajectory, writing order, etc. can be determined, including the number of data points, position coordinates, the starting position of each stroke, the acquired data and the writing time. The acquisition device can also record the force value of the acquisition point to reflect the strength of the writing. Therefore, the electronic signature data information includes the position coordinates and force information of the writing trajectory, which can be used to determine the action characteristics such as writing speed, acceleration, and pen strength. It plays an important role in analyzing the personal attributes of the writer and the specific state when writing, and provides a basis for the inspection of electronic signature handwriting.

[0004] At present, the traditional signature handwriting identification method relies on a single data modality, which is difficult to effectively deal with the risk of signature forgery and fraud. With the development of artificial intelligence technology, deep learning methods based on multimodal data have made significant progress. Therefore, the present invention is an electronic signature handwriting identification method based on the VILT multimodal model, which has important practical significance and application value. Summary of the invention

[0005] In view of the technical problems existing in the prior art, the present invention proposes an electronic signature handwriting identification method based on the VILT multimodal model, which realizes the multimodal feature learning and fusion of the collected electronic signature data information and electronic signature image information with a multimodal learning model based on the Transformer network, thereby improving the mapping ability of electronic image information to electronic data information and the accuracy of electronic signature authenticity handwriting identification.

[0006] The technical solution adopted by the present invention is as follows: a method for electronic signature handwriting identification based on the VILT multimodal model, the steps of which are:

[0007] S1: The electronic signature handwriting information formed by the party using a stylus is collected through an electronic writing device, including electronic signature data information and electronic signature image information; the electronic signature data information includes the X-axis and Y-axis data of the handwriting position information and the Z-axis data of the handwriting pressure information, and the electronic signature image information refers to the electronic signature image.

[0008] S2: Preprocess the collected electronic signature data information and electronic signature image information respectively; organize and count the electronic data information, convert the format, and perform data correction preprocessing, and perform noise elimination, image binarization, character normalization, signature area extraction, and image size normalization preprocessing on the electronic image information.

[0009] S21: Based on the data of the X-axis and Y-axis, the trajectory of the writing movement is restored, the distance between the two data points is calculated, and a schematic diagram of the distance changing over time is generated. The speed and its changes can be clearly seen through the diagram; based on the data of the Z-axis, a schematic diagram of the pen force changing over time is generated; finally, the above electronic signature data information is corrected and sorted using the Excel spreadsheet analysis method.

[0010] S22: Use bilateral filtering to eliminate noise from the electronic signature image; use a variable threshold method to process the electronic signature image to ensure that the signature stroke information is not lost; use a bilinear interpolation algorithm to reduce the image and adjust all signature images to a uniform size.

[0011] S3: Use the pre-processed electronic signature handwriting information of the parties as samples, use K-Fold cross-validation to increase the amount of training data, establish a data set based on the samples, and divide the data set into a training set and a validation set;

[0012] S4: Construct a multimodal learning model based on the Transformer network, namely the VILT multimodal model, and input the electronic signature data information as the text feature part of the model, and the electronic signature image information as the image feature part of the model;

[0013] S41: The VILT multimodal model is a model that combines vision and language. The model structure includes an input representation module, a joint encoder, and an image position encoding module.

[0014] S42: Input representation module: It includes two parts: image information embedding layer and data information embedding layer. The image information embedding layer processes the electronic signature image into blocks, each block is called a patch, and each patch is linearly embedded into a vector of fixed length. The data information embedding layer is processed by word segmentation, and the text is converted into vector representation through the embedding layer;

[0015] S43: Joint Encoder: The VILT multimodal model uses a single Transformer, which consists of multiple self-attention layers, including a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections;

[0016] S44: Position encoding module: Since the Transformer itself is insensitive to the order of input, position encoding is added to distinguish the order of input elements. In VILT, both visual and language inputs need to be added with their respective position information: Image block position encoding: Add the position information of each image patch in the original image; Text word position information: Add position information to the text input to identify the order of words in the sentence;

[0017] S45: The input electronic signature image will be divided into several patches at the embedding layer and input into the Transformer encoder as several groups of vectors. The electronic signature images are all of uniform pixel size, that is, the image is divided into several grayscale images of blocks 1; the multi-pixel patch is mapped to a one-dimensional vector through linear transformation, and finally a matrix with a compressed specification is obtained to achieve dimensionality reduction processing of the image patch; after obtaining the matrix containing all the image data, it is flattened again, and the obtained vector is convenient for subsequent processing by the subsequent Transformer encoder;

[0018] S46: For the input electronic signature data information, the sequence length is divided according to the corresponding number in the electronic data embedding layer. Each sequence dimension corresponds to a single type of electronic data information. Since the electronic signature data itself is one-dimensional information, there is no need for vectorization operation. All electronic signature data can be flattened once and sent to the subsequent operation process. After adding position coding and modal coding to the input, the learnable classification coding is added and sent to the Transformer encoder for feature extraction. The multi-layer perceptron part establishes a nonlinear mapping relationship between the multi-modal input information and the generated electronic signature image.

[0019] S5: The electronic data information and electronic image information in the sample are respectively sent to the VILT multimodal model to train the model; the electronic signature data information uses word embedding to perform data feature conversion, and adds position encoding to pass it to the transformer encoder for feature extraction; the electronic signature image information uses patch embedding, adds position encoding, and finally passes it to the transformer encoder for attention mechanism calculation; hyperparameters need to be set before training the VILT multimodal model, including the number of training times, batch size, learning rate, and regularization parameters.

[0020] S6: Optimize the model and adjust parameters by relying on the validation set;

[0021] S7: The electronic signature handwriting of the party to be identified is used as the sample, the electronic signature data information and electronic signature image information of the sample are obtained and sent to the trained and optimized VILT multimodal model to obtain the detection result, that is, the electronic signature handwriting similarity value;

[0022] S8: Obtain corresponding test results according to the national standard of handwriting identification.

[0023] According to the national standard for handwriting identification, the test results are divided into three categories and nine types: definitive opinion, indeterminate opinion and unable to judge. The specific evaluation criteria are as follows.

[0024] ① When the model's test results for the sample are >90%, it is judged as a definitive opinion and a positive and identical appraisal opinion is made;

[0025] ② When 85% < the test results of the model on the sample ≤ 90%, it is judged as an indeterminate opinion and a highly likely identical appraisal opinion is made;

[0026] ③ When 80% < the model's test results on the sample ≤ 85%, it is judged as an indeterminate opinion and an appraisal opinion is made that is likely to be identical, that is, tends to affirm the identity;

[0027] ④ When 70% < the test result of the model on the sample ≤ 80%, it is judged as an inconclusive opinion and a possible identical appraisal opinion is made;

[0028] ⑤ When 50% < the test result of the model on the sample ≤ 70%, the appraisal opinion is made that it is impossible to judge;

[0029] ⑥ When 40% < the model's test results on the sample ≤ 50%, it is judged as an inconclusive opinion and a possibly inconsistent appraisal opinion is made;

[0030] ⑦ When 35% < the test results of the model on the sample ≤ 40%, it is judged as an indeterminate opinion and an appraisal opinion is made that is likely to be different, that is, tends to deny the same;

[0031] ⑧ When 30% < the test result of the model on the sample ≤ 35%, it is judged as an inconclusive opinion and a highly likely non-identical appraisal opinion is made;

[0032] ⑨ When the model’s test results for the sample are ≤30%, it is judged as a definitive opinion and a negative identification opinion is made.

[0033] The beneficial effects of the present invention are:

[0034] This patent proposes a method for electronic signature handwriting identification based on the VILT multimodal model. The sample data set is constructed using the collected electronic signature handwriting information of the parties to train the VILT multimodal model. The validity of the training data is increased by the K-Fold cross-validation method, and the electronic signature data and signature image of the sample are combined and input into the optimized VILT multimodal model to generate the detection result - the similarity value of the electronic signature handwriting. According to the national handwriting identification standards, a specific detection conclusion is finally obtained. This patent provides a scientific reference and evaluation basis for the industry to study the inspection and identification of such signature handwriting. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 :Flowchart of electronic signature handwriting identification method;

[0036] Figure 2 : VILT model structure diagram;

[0037] Figure 3 : K-Fold cross validation flow chart;

[0038] Figure 4 : A simple diagram of the self-attention mechanism;

[0039] Figure 5 : Transformer encoder module structure diagram. DETAILED DESCRIPTION

[0040] Embodiments of the present invention are described in detail below, and the specific implementation examples described herein are only used to explain the present invention, rather than to limit the present invention. In order to simplify the disclosure of the embodiments of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, embodiments of the present invention can repeat reference numbers or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, and itself does not indicate the relationship between the various embodiments or settings discussed.

[0041] The flowchart of the electronic signature handwriting identification method based on the VILT multimodal model is as follows: Figure 1 As shown, the following steps are included:

[0042] S1: Collect the electronic signature handwriting information formed by the party using a stylus through an electronic writing device, including electronic signature data information and electronic signature image information;

[0043] S2: Pre-processing the collected electronic signature data information and electronic signature image information of the parties respectively;

[0044] S3: Use the pre-processed electronic signature handwriting information of the parties as samples, use K-Fold cross-validation to increase the amount of training data, establish a data set based on the samples, and divide the data set into a training set and a validation set;

[0045] S4: Construct a multimodal learning model based on the Transformer network, namely the VILT multimodal model, and input the electronic signature data information as the text feature part of the model, and the electronic signature image information as the image feature part of the model;

[0046] S5: The electronic data information and electronic image information in the sample are respectively sent to the VILT multimodal model to train the model; the electronic signature data information uses word embedding to perform data feature conversion, and adds position encoding to pass it to the transformer encoder for feature extraction; the electronic signature image information uses patch embedding, adds position encoding, and finally passes it to the transformer encoder for attention mechanism calculation;

[0047] S6: Optimize the model and adjust parameters by relying on the validation set;

[0048] S7: The electronic signature handwriting of the party to be identified is used as the sample, the electronic signature data information and electronic signature image information of the sample are obtained and sent to the trained and optimized VILT multimodal model to obtain the detection result, that is, the electronic signature handwriting similarity value;

[0049] S8: Obtain corresponding test results according to the national standard of handwriting identification.

[0050] Step S1 specifically includes:

[0051] The electronic signature information of the user is collected through the electronic signature handwriting input device. Specifically, in this embodiment, the electronic signature handwriting board includes an intelligent writing book, an electronic stylus pen, and a Bluetooth transmission module. The electronic signature information is transmitted to the data processing platform through the Bluetooth transmission module (HC-05 Bluetooth).

[0052] This paper uses a PC with Ubuntu 20.04 LTS 64-bit operating system as a data processing platform, and uses an NVIDIA GeForce RTX 3090 graphics accelerator card to build a deep learning framework based on PyTorch-GPU, CUDA11.7 and other commonly used Python libraries for accelerated training.

[0053] Step S2, preprocessing the collected electronic signature data information and electronic signature image information of the parties respectively, wherein the electronic data information is preprocessed by organizing statistics, converting formats, correcting data, etc., and the electronic image information is preprocessed by noise elimination, image binarization, character normalization, signature area extraction, and image size normalization.

[0054] Step S3, taking the pre-processed electronic signature handwriting information of the party as a sample, establishing a data set based on the sample, and dividing the data set into a training set and a verification set.

[0055] The Vision-and-Language Transformer (VILT) neural network is a multimodal learning extension model based on the basic structure of Transformer, which aims to solve the problem of joint understanding between text and images. The model uses the multi-head attention mechanism of the Transformer network to obtain sequence information between different modal information, simplifies image information into a non-convolution processing method, and enables the model to process image and text information in an approximate way, balancing the expression strength of the two information, and greatly improving the model operation speed.

[0056] The VILT network uses the idea of ​​weighting the local domain features of the query sequence in the Vision Transformer (VIT) network by multi-head attention, thereby reducing the computational cost while ensuring the quality of feature extraction. The main structure of the VILT network can be divided into an embedding layer, multi-head attention, Transformer encoder, multilayer perceptron (MLP), and cross-modal attention.

[0057] Similar to the VIT network, the embedding layer is responsible for converting the input text and image information into an embedded representation that the model can process. It can be divided into a language embedding layer that processes text input and a visual embedding layer that processes image input. In the entire VILT embedding layer, the outputs of the language embedding layer and the visual embedding layer will be connected into a unified output sequence for subsequent Transformer encoder training, allowing the VILT network to process text and image information at the same time and realize multimodal learning of text and images.

[0058] The self-attention mechanism is the core of VILT and other derivative algorithms of Transformer. It enhances the observation of key areas, so that the model can pay attention to information related to the current position, quickly extract relevant features, obtain the internal pixels of the input image and the overall correlation, and finally improve the speed and accuracy of feature extraction. In the field of vision, self-attention aims to make each feature contain the feature information map of all images according to the weights between the feature pixels in each input image. Compared with the traditional convolutional network, which has strict temporal characteristics at the time of input, the self-attention mechanism can use parallel computing and has a better global vision while ensuring the training accuracy through the attention weight mechanism.

[0059] The self-attention mechanism can be regarded as similar to the pooling layer in the convolutional network, whose weight is determined by the sequence and the position related to the sequence. The self-attention mechanism is shown in the following figure: Figure 4 As shown in Figure 2. Self-attention is similar to the attention mechanism, and is also composed of a query vector (Query), a key vector (Key), and a value vector (Value), which are derived from the same image block X. Assume that the input image block x 1 ~x 4 , which is converted into a vector a in the encoding 1 ~a 4 After it is sent to the Self-attention layer, the input vector is respectively combined with three different trainable transfer matrices (Transformation matrix) W q , W k , W v Multiply it to get the query vector q of its input feature vector i , key-value vector k i With the value vector v i .

[0060] Since the VILT multimodal model integrates information from different modalities, the system needs to make certain selections for different modal information during training, analyze the content under different modalities and identify their relevance, so as to optimize the information processing capabilities of different modalities and optimize the learning speed of the model.

[0061] The VILT network calculates the similarity between images and texts. It focuses on the [class] token part of the vector output, performs binary classification tasks, and removes unmatched text-image pairs. At the same time, it uses the Optimal Transport algorithm to calculate the distance between the text and image distributions, and uses this as a benchmark to weight the input text-image information and filter out more relevant information.

[0062] The Transformer encoder is the core component of the network. Its structure is composed of multiple Transformer encoder modules stacked together. Figure 5 As shown in the figure, the Transformer encoder module consists of a normalization layer, a multi-head attention mechanism, a Dropout layer, and an MLP Block layer. A normalization layer is deployed at the input of each Transformer encoder to perform layer normalization on the input information. The mean variance of each encoded input information is normalized to standardize the output form of each sub-layer, which can effectively alleviate the problems of gradient vanishing and gradient exploding during network training.

[0063] The multi-layer perceptron module (MLP block) is used to learn the nonlinear mapping of input information and the feature representation of vector sequences. The added learnable classification vector [class] token is further processed to generate a global feature representation vector, and finally the image generation task is completed. The residual connection method is used in the structure to deepen the number of network layers while allowing the model to directly learn the transformation of input data, avoiding the problem of gradient disappearance or gradient explosion. Finally, a part of the network neurons are randomly set to zero through the DropOut layer to avoid the model's over-reliance on a specific part of the network, enhance the generalization ability of the network, and effectively alleviate the risk of overfitting during training.

[0064] Step S4 specifically includes:

[0065] A multimodal learning model based on Transformer network, namely VILT multimodal model, is constructed. The electronic signature data information is input as the text feature part of the model, and the electronic signature image information is input as the image feature part of the model.

[0066] The input electronic signature image will be divided into 9 patches in the embedding layer and input into the Transformer encoder as 9 groups of vectors. The electronic signature images are all 51*51 pixels in size, that is, the image is divided into 9 grayscale images of 17*17 in size. The data size of each image patch is approximately [17*17*1]. The multi-pixel patch is mapped to a one-dimensional vector through linear transformation, and finally a matrix of size [3, 3, 289] is obtained to achieve dimensionality reduction processing of the image patch. After obtaining the matrix containing all the image data, it is flattened again to obtain a vector of size

[2601] to facilitate subsequent processing by the subsequent Transformer encoder.

[0067] The original text embedding layer is used to convert the electronic data information into an embedded representation and is renamed the electronic data embedding layer. For the input electronic data information, the sequence length is divided according to the corresponding number in the electronic data embedding layer, and each sequence dimension corresponds to a single type of electronic data information. Since the electronic signature data itself is one-dimensional information, there is no need for vectorization operations. All electronic signature data can be flattened once and sent to the subsequent operation process. After adding position coding and modal coding to the input, the learnable classification coding is added and sent to the Transformer encoder for feature extraction, and the multi-layer perceptron part establishes a nonlinear mapping relationship between the multimodal input information and the generated electronic signature image.

[0068] During the training process, the cross-modal attention mechanism is used to identify the correspondence between electronic data information and electronic image information, and to filter out information about signatures with poor quality and handwriting deviation in the electronic image information, which can effectively avoid the difficulty in model convergence due to errors in preliminary identification.

[0069] Since the ultimate training target of the model should be electronic signature data, and there is a high correlation between different electronic signature data images, in order to make the model network more suitable for the electronic signature data handwriting identification task, the present invention introduces the Arcface loss function into the model training.

[0070] The Arcface loss function is based on the Soft max loss function, which pays more attention to the difference part of the vector correlation. The specific loss function is as follows: Among them, x i for y i The feature vector of the i-th sample in, N is the batch size Batchsize, W j is the weight parameter for model training in column j, b j is the loss function bias term, n represents the number of categories. During the training process, the loss function bias term b yi Increasing from 0, the parameter That is, the weight W j With the eigenvector x i The angle between ||x i || is the radius of the hypersphere. At this time, L2 is as follows. In order to highlight the similarity of the same handwriting distribution and the difference of different handwriting distribution in the handwriting identification process, an angle parameter m is added to reduce the angle between them, reduce the increase of the loss function caused by the deepening of the training depth, optimize the category weight and feature input in the loss function, reduce the distance of the same handwriting distribution, and expand the external distance of the non-distribution. The loss function loss after adding is as follows.

[0071]

[0072] The Arcface loss function is used to improve the problem that the cross entropy loss function of the previous image loss function does not consider the class spacing. This is reflected in the handwriting identification of electronic signature data. Due to the high correlation of electronic signature information, its performance on handwriting edges and overall structural problems is fully improved.

[0073] Normalize the data using min-max normalization, also known as deviation normalization, which is a linear transformation of the original data so that the resulting value is mapped to between [0-1]. The conversion function is as follows: Among them, max is the maximum value of the sample data, and min is the minimum value of the sample data.

[0074] Step S5, using the preprocessed training set to train the VILT-based multimodal model.

[0075] The training uses the Adam training optimizer for gradient optimization. It can be found that the loss value of the function will be lower than 0.01 and remain oscillating after the number of training times is greater than 300. It is believed that the network converges after 300 epochs and the entire network has achieved good training results.

[0076] Step S6 specifically includes:

[0077] The model is optimized and parameters are adjusted by relying on the validation set. The key steps include dividing the validation set, defining the optimization goal, adjusting hyperparameters, cross-validation, and continuously monitoring the validation set performance.

[0078] Validation set division: The validation set is used to evaluate the performance of the model during the training process and helps optimize the model without participating in the training.

[0079] Define the optimization goal: optimize the model by minimizing the loss function or maximizing the performance indicator (such as accuracy, F1 value).

[0080] Hyperparameter adjustment: Adjust hyperparameters such as learning rate, batch size, and number of model layers through grid search, random search, or Bayesian optimization, and select the parameter combination that performs best on the validation set.

[0081] Use K-Fold cross validation to divide the training set and test set in a 5:1 ratio to increase the amount of training data and avoid overfitting problems. The K-Fold cross validation process is as follows: Figure 3 shown.

[0082] Continuously monitor the validation set performance: observe the loss and accuracy on the validation set to ensure that the model is adjusted in the right direction. After the model is trained, it is necessary to evaluate the model based on the accuracy, recall, precision, etc.

[0083] Step S7, taking the electronic signature handwriting of the party to be identified as the sample, obtaining the electronic signature data information and electronic signature image information of the sample and sending them to the trained and optimized VILT multimodal model to obtain the detection result, that is, the electronic signature handwriting similarity value;

[0084] Step S8: Based on the test results of the sample output by the model, the authenticity of the input electronic signature can be judged. According to the national standard for handwriting identification, the test results are divided into three categories: definite opinion, indeterminate opinion, and indeterminate opinion. The specific evaluation criteria are as follows:

[0085] ① When the model's test results for the sample are >90%, it is judged as a definitive opinion and a positive and identical appraisal opinion is made;

[0086] ② When 85% < the test results of the model on the sample ≤ 90%, it is judged as an indeterminate opinion and a highly likely identical appraisal opinion is made;

[0087] ③ When 80% < the model's test results on the sample ≤ 85%, it is judged as an indeterminate opinion and an appraisal opinion is made that is likely to be identical, that is, tends to affirm the identity;

[0088] ④ When 70% < the test result of the model on the sample ≤ 80%, it is judged as an inconclusive opinion and a possible identical appraisal opinion is made;

[0089] ⑤ When 50% < the test result of the model on the sample ≤ 70%, the appraisal opinion is made that it is impossible to judge;

[0090] ⑥ When 40% < the model's test results on the sample ≤ 50%, it is judged as an inconclusive opinion and a possibly inconsistent appraisal opinion is made;

[0091] ⑦ When 35% < the test results of the model on the sample ≤ 40%, it is judged as an indeterminate opinion and an appraisal opinion is made that is likely to be different, that is, tends to deny the same;

[0092] ⑧ When 30% < the test result of the model on the sample ≤ 35%, it is judged as an inconclusive opinion and a highly likely non-identical appraisal opinion is made;

[0093] ⑨ When the model’s test results for the sample are ≤30%, it is judged as a definitive opinion and a negative identification opinion is made.

Claims

1. A method for electronic signature handwriting identification based on VILT multimodal model, characterized in that: The steps are: S1: Collect the electronic signature handwriting information formed by the party using a stylus through an electronic writing device, including electronic signature data information and electronic signature image information; S2: Pre-processing the collected electronic signature data information and electronic signature image information of the parties respectively; S3: Use the pre-processed electronic signature handwriting information of the parties as samples, use K-Fold cross-validation to increase the amount of training data, establish a data set based on the samples, and divide the data set into a training set and a validation set; S4: Construct a multimodal learning model based on the Transformer network, namely the VILT multimodal model, and input the electronic signature data information as the text feature part of the model, and the electronic signature image information as the image feature part of the model; S5: The electronic data information and electronic image information in the sample are respectively sent to the VILT multimodal model to train the model; the electronic signature data information uses word embedding to perform data feature conversion, and adds position encoding to pass it to the transformer encoder for feature extraction; the electronic signature image information uses patch embedding, adds position encoding, and finally passes it to the transformer encoder for attention mechanism calculation; S6: Optimize the model and adjust parameters by relying on the validation set; S7: The electronic signature handwriting of the party to be identified is used as the sample, the electronic signature data information and electronic signature image information of the sample are obtained and sent to the trained and optimized VILT multimodal model to obtain the detection result, that is, the electronic signature handwriting similarity value; S8: Obtain corresponding test results according to the national standard of handwriting identification.

2. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that: In the above-mentioned S1, the electronic signature data information includes the X-axis and Y-axis data of the handwriting position information and the Z-axis data of the handwriting pressure information, and the electronic signature image information refers to the electronic signature image.

3. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that: In the above-mentioned S2, electronic data information is preprocessed by organizing statistics, converting formats, and correcting data, and electronic image information is preprocessed by noise elimination, image binarization, character normalization, signature area extraction, and image size normalization.

4. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 3 is characterized in that: The specific method in S2 is: S21: Based on the data of the X-axis and Y-axis, the trajectory of the writing movement is restored, the distance between the two data points is calculated, and a schematic diagram of the distance changing over time is generated. The speed and its changes can be clearly seen through the diagram; based on the data of the Z-axis, a schematic diagram of the pen force changing over time is generated; finally, the above electronic signature data information is corrected and sorted using the Excel spreadsheet analysis method. S22: Use bilateral filtering to eliminate noise from the electronic signature image; use a variable threshold method to process the electronic signature image to ensure that the signature stroke information is not lost; use a bilinear interpolation algorithm to reduce the image and adjust all signature images to a uniform size.

5. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that: The specific method of S4 is: S41: The VILT multimodal model is a model that combines vision and language. The model structure includes an input representation module, a joint encoder, and an image position encoding module. S42: Input representation module: It includes two parts: image information embedding layer and data information embedding layer. The image information embedding layer processes the electronic signature image into blocks, each block is called a patch, and each patch is linearly embedded into a vector of fixed length. The data information embedding layer is processed by word segmentation, and the text is converted into vector representation through the embedding layer; S43: Joint Encoder: The VILT multimodal model uses a single Transformer, which consists of multiple self-attention layers, including a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections; S44: Position encoding module: Since the Transformer itself is insensitive to the order of input, position encoding is added to distinguish the order of input elements. In VILT, both visual and language inputs need to be added with their respective position information: Image block position encoding: Add the position information of each image patch in the original image; Text word position information: Add position information to the text input to identify the order of words in the sentence; S45: The input electronic signature image will be divided into several patches at the embedding layer and input into the Transformer encoder as several groups of vectors. The electronic signature images are all of uniform pixel size, that is, the image is divided into several grayscale images of blocks 1; the multi-pixel patch is mapped to a one-dimensional vector through linear transformation, and finally a matrix with a compressed specification is obtained to achieve dimensionality reduction processing of the image patch; after obtaining the matrix containing all the image data, it is flattened again, and the obtained vector is convenient for subsequent processing by the subsequent Transformer encoder; S46: For the input electronic signature data information, the sequence length is divided according to the corresponding number in the electronic data embedding layer. Each sequence dimension corresponds to a single type of electronic data information. Since the electronic signature data itself is one-dimensional information, there is no need for vectorization operation. All electronic signature data can be flattened once and sent to the subsequent operation process. After adding position coding and modal coding to the input, the learnable classification coding is added and sent to the Transformer encoder for feature extraction. The multi-layer perceptron part establishes a nonlinear mapping relationship between the multi-modal input information and the generated electronic signature image.

6. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that: In S5 described above, hyperparameters need to be set before training the VILT multimodal model, including the number of training times, batch size, learning rate, and regularization parameters.

7. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that: In the S8, According to the national standard for handwriting identification, the test results are divided into three categories and nine types: definitive opinion, indeterminate opinion and unable to judge. The specific evaluation criteria are as follows. ① When the model's test results for the sample are >90%, it is judged as a definitive opinion and a positive and identical appraisal opinion is made; ② When 85% < the test results of the model on the sample ≤ 90%, it is judged as an indeterminate opinion and a highly likely identical appraisal opinion is made; ③ When 80% < the model's test results on the sample ≤ 85%, it is judged as an indeterminate opinion and an appraisal opinion is made that is likely to be identical, that is, tends to affirm the identity; ④ When 70% < the test result of the model on the sample ≤ 80%, it is judged as an inconclusive opinion and a possible identical appraisal opinion is made; ⑤ When 50% < the test result of the model on the sample ≤ 70%, the appraisal opinion is made that it is impossible to judge; ⑥ When 40% < the model's test results on the sample ≤ 50%, it is judged as an inconclusive opinion and a possibly inconsistent appraisal opinion is made; ⑦ When 35% < the test results of the model on the sample ≤ 40%, it is judged as an indeterminate opinion and an appraisal opinion is made that is likely to be different, that is, tends to deny the same; ⑧ When 30% < the test result of the model on the sample ≤ 35%, it is judged as an inconclusive opinion and a highly likely non-identical appraisal opinion is made; ⑨ When the model’s test results for the sample are ≤30%, it is judged as a definitive opinion and a negative identification opinion is made.

Images