A method for electronic signature handwriting identification based on the VILT multimodal model

Abstract

A method for electronic signature handwriting identification based on a VILT multimodal model is disclosed, comprising the following steps: 1) collecting electronic signature information of the parties involved; 2) preprocessing the collected information; 3) using the preprocessed information as samples; 4) constructing a multimodal learning model based on a Transformer network, namely the VILT multimodal model; 5) feeding the electronic data information and electronic image information from the samples into the VILT multimodal model for model training; 6) optimizing and adjusting the model parameters using a validation set; 7) using the electronic signature handwriting of the parties involved to be identified as evidence, feeding it into the trained and optimized VILT multimodal model to obtain a similarity score; and 9) outputting the detection results. This invention improves the mapping ability of electronic image information to electronic data information, thereby enhancing the accuracy of electronic signature handwriting identification.

Description

Technical Field

[0001] This invention proposes an electronic signature handwriting identification method based on the VILT multimodal model, particularly relating to the field of handwriting identification technology. Background Technology

[0002] An electronic signature is an electronic signature created by writing with a stylus on an electronic writing device. Its handwriting information consists of two parts: electronic signature data and electronic signature image information. Electronic signature data refers to a set of data recorded by the electronic device, related to the entire writing process, and composed of a series of changing quantities. Electronic signature image information refers to the image information that records the outline of the electronic signature handwriting through methods such as saving screenshots and exporting them as PDFs or JPGs, thus reflecting the morphological characteristics of the electronic signature handwriting.

[0003] When a stylus touches the touchscreen of a writing device, the data acquisition device collects the position coordinates of the pen tip on the screen at fixed time intervals, recording them as single numbers or X-axis and Y-axis coordinate pairs. This data allows for the determination of stroke trajectory, writing order, and other details, including the number of data points, position coordinates, the starting position of each stroke, the acquired data, and the writing time. The data acquisition device can also record the pressure applied at each point, reflecting the magnitude of the writing force. Therefore, electronic signature data includes the position coordinates of the writing trajectory and pressure information, which can be used to determine writing speed, acceleration, pen force, and other action characteristics. This plays a crucial role in analyzing the writer's personal attributes and specific state during writing, providing a basis for the verification of electronic signature handwriting.

[0004] Currently, traditional signature handwriting identification methods rely on a single data modality, making it difficult to effectively address the risks of signature forgery and fraud. With the development of artificial intelligence technology, deep learning methods based on multimodal data have made significant progress. Therefore, this invention, a method for electronic signature handwriting identification based on the VILT multimodal model, has important practical significance and application value. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention proposes an electronic signature handwriting identification method based on the VILT multimodal model. By using a multimodal learning model based on the Transformer network, it realizes the multimodal feature learning and fusion of the collected electronic signature data information and electronic signature image information, thereby improving the mapping ability of electronic image information to electronic data information and the accuracy of electronic signature handwriting identification.

[0006] The technical solution adopted in this invention is as follows: A method for electronic signature handwriting identification based on the VILT multimodal model, the steps of which are:

[0007] S1: Collect electronic signature handwriting information formed by the party using a stylus through an electronic writing device, including electronic signature data information and electronic signature image information; electronic signature data information includes X-axis and Y-axis data of handwriting position information and Z-axis data of handwriting pressure information, and electronic signature image information refers to electronic signature image.

[0008] S2: Preprocess the collected electronic signature data and electronic signature image information of the parties respectively; organize and statistically analyze the electronic data information, convert the format, and correct the data preprocessing; perform noise elimination, image binarization, character normalization, signature area extraction, and image size normalization preprocessing on the electronic image information.

[0009] S21: Based on the data of the X-axis and Y-axis, reconstruct the trajectory of the writing motion, calculate the distance between two data points, and generate a diagram showing the change of distance over time. The speed and its changes can be clearly seen through the chart. Based on the data of the Z-axis, generate a diagram showing the change of pen force over time. Finally, use Excel spreadsheet analysis to correct and organize the above electronic signature data.

[0010] S22: Bilateral filtering is used to eliminate noise in the electronic signature image; a variable threshold method is used to process the electronic signature image to ensure that the signature stroke information is not lost; a bilinear interpolation algorithm is used to reduce the image size and adjust all signature images to a uniform size.

[0011] S3: Use the preprocessed electronic signature handwriting information of the parties as samples, use K-Fold cross-validation to increase the amount of training data, build a dataset based on the samples, and divide the dataset into training set and validation set;

[0012] S4: Construct a multimodal learning model based on Transformer network, namely the VILT multimodal model, and input the electronic signature data information as the text feature part of the model and the electronic signature image information as the image feature part of the model.

[0013] S41: The VILT multimodal model is a model that combines vision and language. Its structure includes an input representation module, a joint encoder, and an image location encoding module.

[0014] S42: Input representation module: It consists of two parts: image information embedding layer and data information embedding layer. The image information embedding layer divides the electronic signature image into blocks, each block is called a patch, and each patch is linearly embedded into a fixed-length vector. The data information embedding layer is processed by word segmentation and converts the text into a vector representation through the embedding layer.

[0015] S43: Joint Encoder: The VILT multimodal model uses a single Transformer, which consists of multiple self-attention layers, including a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections.

[0016] S44: Position Encoding Module: Since the Transformer itself is not sensitive to the order of input, position encoding is added to distinguish the order of input elements. In VILT, both visual and language inputs need to have their own position information added: Image Patch Position Encoding: Add position information in the original image to each image patch; Text Word Position Information: Add position information to text input to identify the order of words in a sentence;

[0017] S45: The input electronic signature image will be divided into several patches in the embedding layer, and these patches will be used as several sets of vectors to be input into the Transformer encoder. All electronic signature images have a uniform pixel size, that is, the image is divided into several blocks of grayscale image 1. The pixels of each patch are mapped to a one-dimensional vector through a linear transformation, and finally a compressed matrix is ​​obtained, which realizes the dimensionality reduction of the image patch. After obtaining the matrix containing all image data, it is flattened again, and the resulting vector is convenient for subsequent processing by the Transformer encoder.

[0018] S46: For the input electronic signature data, the sequence length is divided according to the corresponding number in the electronic data embedding layer. Each sequence dimension corresponds to a single type of electronic data information. Since the electronic signature data itself is one-dimensional information, there is no need to perform vectorization operations. All electronic signature data can be flattened once and sent to the subsequent operation process. After adding position encoding and modality encoding to the input, a learnable classification encoding is added and then sent to the Transformer encoder for feature extraction. The multilayer perceptron part establishes a nonlinear mapping relationship between the multimodal input information and the generated electronic signature image.

[0019] S5: Input the electronic data information and electronic image information in the sample into the VILT multimodal model to train the model; the electronic signature data information uses word embedding to perform feature transformation and adds position encoding before passing it to the transformer encoder for feature extraction; the electronic signature image information uses patch embedding and adds position encoding before passing it to the transformer encoder for attention mechanism calculation; before training the VILT multimodal model, hyperparameters need to be set, including training iterations, batch size, learning rate, and regularization parameters.

[0020] S6: Optimize the model and adjust parameters by relying on the validation set;

[0021] S7: Use the electronic signature handwriting of the party to be identified as the evidence, obtain the electronic signature data information and electronic signature image information of the evidence, and send them into the trained and optimized VILT multimodal model to obtain the detection result, namely the electronic signature handwriting similarity value.

[0022] S8: Based on the national standards for handwriting identification, the corresponding test results are obtained.

[0023] According to the national standards for handwriting identification, the test results are classified into three categories and nine types: definitive opinion, indeterminate opinion, and indeterminate opinion. The specific evaluation criteria are as follows.

[0024] ① When the model's detection results for the sample are >90%, it is judged as a definitive opinion, and a positive identification opinion is made;

[0025] ② When 85% < the model's test results for the sample ≤ 90%, it is judged as an indeterminate opinion, and an identification opinion of highly probable identity is made;

[0026] ③ When 80% < the model's test results for the sample ≤ 85%, it is judged as an indeterminate opinion, and an identification opinion of very likely identity is made, that is, a tendency to affirm identity is made;

[0027] ④ When 70% < the model's test results for the sample ≤ 80%, it is judged as an indeterminate opinion, and a possible identical identification opinion is made;

[0028] ⑤ When 50% < the model's test results on the sample ≤ 70%, an expert opinion of indeterminacy shall be issued;

[0029] ⑥ When 40% < the model's test results on the sample ≤ 50%, it is judged as an indeterminate opinion, and a possible different identification opinion is made;

[0030] ⑦ When 35% < the model's test results on the sample ≤ 40%, it is judged as an indeterminate opinion, and an identification opinion that is very likely not identical is made, that is, tends to deny the identity;

[0031] ⑧ When 30% < the model's test results on the sample ≤ 35%, it is judged as an indeterminate opinion, and an identification opinion that is highly likely to be different is made;

[0032] ⑨ When the model's test results on the sample are ≤30%, it is judged as a definitive opinion, and a negative identification opinion is made.

[0033] The beneficial effects of this invention are as follows:

[0034] This invention patent proposes a method for electronic signature handwriting identification based on the VILT multimodal model. It utilizes collected electronic signature handwriting information from the parties involved to construct a sample dataset, which is then used to train the VILT multimodal model. K-Fold cross-validation is employed to enhance the effectiveness of the training data. The electronic signature data and images from the evidence are then input into the optimized VILT multimodal model to generate a detection result—a similarity value for the electronic signature handwriting. Based on national handwriting identification standards, a specific detection conclusion is obtained. This patent provides a scientific reference and evaluation basis for industry research on the examination and identification of such signature handwriting. Attached Figure Description

[0035] Figure 1 Flowchart of electronic signature handwriting identification method;

[0036] Figure 2 VILT model structure diagram;

[0037] Figure 3 K-Fold cross-validation flowchart;

[0038] Figure 4 Simplified diagram of the self-attention mechanism;

[0039] Figure 5 Transformer encoder module structure diagram. Detailed Implementation

[0040] The embodiments of the present invention are described in detail below. The specific examples described herein are merely for illustrative purposes and not for limiting the invention. To simplify the disclosure of the embodiments of the present invention, the components and arrangements of specific examples are described below. Of course, these are merely examples and are not intended to limit the invention. Furthermore, the embodiments of the present invention may repeat reference numerals or reference letters in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments or arrangements discussed.

[0041] The flowchart of the electronic signature handwriting identification method based on the VILT multimodal model is as follows: Figure 1 As shown, it includes the following steps:

[0042] S1: Collect electronic signature handwriting information formed by the party using a stylus through an electronic writing device, including electronic signature data information and electronic signature image information;

[0043] S2: Preprocess the collected electronic signature data and electronic signature image information of the parties involved respectively;

[0044] S3: Use the preprocessed electronic signature handwriting information of the parties as samples, use K-Fold cross-validation to increase the amount of training data, build a dataset based on the samples, and divide the dataset into training set and validation set;

[0045] S4: Construct a multimodal learning model based on Transformer network, namely the VILT multimodal model, and input the electronic signature data information as the text feature part of the model and the electronic signature image information as the image feature part of the model.

[0046] S5: The electronic data information and electronic image information in the sample are fed into the VILT multimodal model to train the model. Among them, the electronic signature data information uses word embedding to perform feature transformation of the data and adds position encoding before passing it to the transformer encoder for feature extraction. The electronic signature image information uses patch embedding and adds position encoding before passing it to the transformer encoder for attention mechanism calculation.

[0047] S6: Optimize the model and adjust parameters by relying on the validation set;

[0048] S7: Use the electronic signature handwriting of the party to be identified as the evidence, obtain the electronic signature data information and electronic signature image information of the evidence, and send them into the trained and optimized VILT multimodal model to obtain the detection result, namely the electronic signature handwriting similarity value.

[0049] S8: Based on the national standards for handwriting identification, the corresponding test results are obtained.

[0050] Step S1 specifically includes:

[0051] The user's electronic signature information is collected through an electronic signature handwriting input device. Specifically, in this embodiment, the electronic signature handwriting tablet includes a smart writing pad, an electronic stylus, and a Bluetooth transmission module. The electronic signature information is transmitted to the data processing platform via the Bluetooth transmission module (HC-05 Bluetooth).

[0052] This invention uses a PC with Ubuntu 20.04 LTS 64-bit operating system as the data processing platform, and uses an NVIDIA GeForce RTX 3090 graphics accelerator card to build a deep learning framework based on PyTorch-GPU, CUDA 11.7 and other commonly used Python libraries for accelerated training.

[0053] Step S2 involves preprocessing the collected electronic signature data and electronic signature image information of the parties involved. The electronic data information undergoes preprocessing such as sorting and statistics, format conversion, and data correction. The electronic image information undergoes preprocessing such as noise reduction, image binarization, character normalization, signature area extraction, and image size normalization.

[0054] Step S3: Use the preprocessed electronic signature handwriting information of the parties as a sample, establish a dataset based on the sample, and divide the dataset into a training set and a validation set.

[0055] The Vision-and-Language Transformer (VILT) neural network is a multimodal learning extension model based on the basic Transformer structure, designed to solve the problem of joint understanding between text and images. The model utilizes the multi-head attention mechanism of the Transformer network to obtain sequential information between different modalities, simplifying image information into a non-convolutional processing method. This allows the model to process image and text information in an approximate manner, balancing the expressive power of both and significantly improving the model's computational speed.

[0056] The VILT network adopts the idea of ​​query sequence local neighborhood features from the Vision Transformer (VIT) network and weights them with multi-head attention, thereby reducing computational costs while ensuring the quality of feature extraction. The main structure of the VILT network can be divided into embedding layer, multi-head attention, Transformer encoder, multilayer perceptron (MLP), and cross-modal attention.

[0057] Similar to the VIT network, the embedding layer is responsible for converting the input text and image information into embedding representations that the model can process. Internally, it can be divided into a language embedding layer for processing text input and a visual embedding layer for processing image input. Throughout the VILT embedding layer, the outputs of the language and visual embedding layers are concatenated into a unified output sequence for subsequent Transformer encoder training. This allows the VILT network to process both text and image information simultaneously, achieving multimodal learning of text and images.

[0058] Self-attention is the core of VILT and other derivative algorithms of Transformer. It enhances the observation of key regions, enabling the model to focus on information related to its current position, quickly extract relevant features, and obtain the correlation between pixels within the input image and the overall image, ultimately improving the speed and accuracy of feature extraction. In the field of vision, self-attention aims to ensure that each feature contains the feature map of the entire image, based on the weights between feature pixels within each input image. Compared to traditional convolutional networks, which have strict temporal characteristics at input, self-attention, through its attention weight mechanism, can use parallel computation and has a better global perspective while maintaining training accuracy.

[0059] The self-attention mechanism can be viewed as similar to the pooling layer in a convolutional network, where its weights are determined by the sequence and its associated positions. A simplified diagram of the self-attention mechanism is shown below. Figure 4 As shown. Self-attention is similar to the regular attention mechanism, also consisting of a query vector, a key vector, and a value vector, all originating from the same image patch X. Assuming the input image patches x1 to x4 are encoded into vectors a1 to a4, after being fed into the Self-attention layer, the input vectors are respectively coupled with three different trainable transformation matrix sets W. q W k W v Multiply to obtain the vector query vector q of its input features. i Key value vector k i AND value vector v i .

[0060] Since the VILT multimodal model integrates information from different modalities, the system needs to make certain selections for different modal information during training, analyze the content under different modalities and identify their correlations, so as to optimize the information processing capabilities of different modalities and optimize the learning speed of the model.

[0061] The VILT network is designed to calculate the similarity between images and text. It focuses on the [class] token part of the vector output to perform a binary classification task, removing unmatched text-image pairs. Simultaneously, it utilizes the Optimal Transport algorithm to calculate the distance between the text and image distributions, and uses this distance as a basis to weight the input text-image information, filtering out information with higher relevance.

[0062] The Transformer encoder is the core component of the network, and its structure consists of multiple stacked Transformer encoder modules. Figure 5As shown, the Transformer encoder module consists of a normalization layer, a multi-head attention mechanism, a Dropout layer, and an MLP Block layer. A normalization layer is deployed at the input of each Transformer encoder to perform layer normalization on the input information. This normalizes the mean and variance of each encoded input, thus standardizing the output of each sub-layer. This effectively alleviates problems such as vanishing and exploding gradients during network training.

[0063] The Multilayer Perceptron (MLP) block is used to learn the nonlinear mapping of input information and the feature representation of the vector sequence. The added learnable classification vector [class] is further processed to generate a global feature representation vector, ultimately completing the image generation task. The structure employs residual connections to deepen the network layers while allowing the model to directly learn the transformations of the input data, avoiding the problems of vanishing or exploding gradients. Finally, a DropOut layer randomly sets a portion of network neurons to zero, preventing the model from over-relying on specific network parts, enhancing the network's generalization ability, and effectively mitigating the risk of overfitting during training.

[0064] Step S4 specifically includes:

[0065] A multimodal learning model based on Transformer network, namely the VILT multimodal model, is constructed. Electronic signature data information is used as the text feature part of the model as input, and electronic signature image information is used as the image feature part of the model as input.

[0066] The input digital signature image is divided into 9 patches in the embedding layer, which are then input as 9 vectors into the Transformer encoder. Each digital signature image is 51*51 pixels in size, meaning the image is divided into 9 grayscale blocks of size 17*17. The data size of each image patch is approximately [17*17*1]. The pixels in each patch are mapped to a one-dimensional vector through a linear transformation, ultimately resulting in a matrix of size [3, 3, 289], achieving dimensionality reduction of the image patch. After obtaining the matrix containing all image data, it is flattened again to obtain a vector of size

[2601] for subsequent processing by the Transformer encoder.

[0067] The original text embedding layer, used for the conversion of electronic data information into embedded representations, has been renamed the electronic data embedding layer. For the input electronic data information, the electronic data embedding layer divides the sequence length according to the corresponding number, with each sequence dimension corresponding to a single type of electronic data information. Since the electronic signature data itself is one-dimensional information, no vectorization is required. All electronic signature data is simply flattened before being fed into the subsequent processing. After adding positional and modal encodings to the input, and then adding learnable classification encodings, it is fed into the Transformer encoder for feature extraction. The multilayer perceptron part then establishes a non-linear mapping relationship between the multimodal input information and the generated electronic signature image.

[0068] During training, the cross-modal attention mechanism is used to determine the correspondence between electronic data information and electronic image information. Information with poor handwriting quality or handwriting deviation in electronic image information is filtered out, which can effectively avoid the model from failing to converge due to errors in the initial identification.

[0069] Since the final training target of the model should be electronic signature data, and different electronic signature data images have a high degree of correlation, in order to make the model network more suitable for the handwriting identification task of electronic signature data, this invention introduces the Arcface loss function into the model training.

[0070] The Arcface loss function is based on the Soft max loss function, which focuses more on the difference in vector correlation. The specific loss function is as follows: Where, x i For y i The feature vector of the i-th sample in the dataset, where N is the batch size (Batchsize), and W... j b is the weight parameter for training the model in column j. j Let b be the bias term of the loss function, where n represents the number of classes. During training, the bias term b... yi Increment from 0, parameter That is, weight W j With eigenvector x i The angle between them, ||x i || is the radius of the hypersphere. L2 is given by the following formula. To highlight the similarity within the same handwriting distribution and the differences between different handwriting distributions during handwriting identification, an angle parameter 'm' is added to reduce the angle between them. This reduces the increase in the loss function caused by increased training depth, optimizes the class weights and feature inputs in the loss function, reduces the distance between similar handwriting distributions, and expands the external distance between dissimilar distributions. The loss function after adding this parameter is shown in the following formula.

[0071]

[0072] The Arcface loss function is used, which improves upon the previous image loss function by not considering the inter-class spacing of the cross-entropy loss function. This is reflected in the handwriting identification of electronic signature data. Due to the high correlation of electronic signature information, its performance in handwriting edge and overall structural issues is significantly improved.

[0073] Normalizing the data using min-max standardization, also known as deviation standardization, is a linear transformation of the original data, mapping the resulting values ​​to the range [0-1]. The transformation function is as follows: Where max is the maximum value of the sample data and min is the minimum value of the sample data.

[0074] Step S5: Train the VILT-based multimodal model using the preprocessed training set.

[0075] The Adam training optimizer was used for gradient optimization during training. It was found that the loss value of the function would be below 0.01 and remain oscillating after more than 300 training iterations. It was assumed that the network converged after 300 epochs, and the entire network had achieved good training results.

[0076] Step S6 specifically includes:

[0077] By optimizing the model and tuning parameters based on the validation set, key steps include partitioning the validation set, defining optimization objectives, tuning hyperparameters, cross-validation, and continuously monitoring the performance of the validation set.

[0078] Validation set partitioning: The validation set is used to evaluate the model's performance during training, helping to optimize the model without participating in the training process.

[0079] Define the optimization objective: optimize the model by minimizing the loss function or maximizing performance metrics (such as accuracy or F1 score).

[0080] Hyperparameter tuning: Adjust hyperparameters such as learning rate, batch size, and number of model layers using methods such as grid search, random search, or Bayesian optimization, and select the parameter combination that performs best on the validation set.

[0081] K-Fold cross-validation is used to split the training and test sets in a 5:1 ratio to increase the amount of training data and avoid overfitting. The K-Fold cross-validation process is as follows: Figure 3 As shown.

[0082] Continuously monitor validation set performance: observe the loss and accuracy on the validation set to ensure the model is adjusting in the right direction. After training, the model needs to be evaluated using metrics such as accuracy, recall, and precision.

[0083] Step S7: The electronic signature handwriting of the party to be identified is used as the evidence. The electronic signature data information and electronic signature image information of the evidence are obtained and sent to the trained and optimized VILT multimodal model to obtain the detection result, namely the electronic signature handwriting similarity value.

[0084] Step S8: Based on the model's output of the test results on the sample, the authenticity of the input electronic signature can be determined. According to the national standard for handwriting identification, the test results are classified into three categories and nine types: definitive opinion, indeterminate opinion, and indeterminate opinion. The specific evaluation criteria are as follows:

[0085] ① When the model's detection results for the sample are >90%, it is judged as a definitive opinion, and a positive identification opinion is made;

[0086] ② When 85% < the model's test results for the sample ≤ 90%, it is judged as an indeterminate opinion, and an identification opinion of highly probable identity is made;

[0087] ③ When 80% < the model's test results for the sample ≤ 85%, it is judged as an indeterminate opinion, and an identification opinion of very likely identity is made, that is, a tendency to affirm identity is made;

[0088] ④ When 70% < the model's test results for the sample ≤ 80%, it is judged as an indeterminate opinion, and a possible identical identification opinion is made;

[0089] ⑤ When 50% < the model's test results on the sample ≤ 70%, an expert opinion of indeterminacy shall be issued;

[0090] ⑥ When 40% < the model's test results on the sample ≤ 50%, it is judged as an indeterminate opinion, and a possible different identification opinion is made;

[0091] ⑦ When 35% < the model's test results on the sample ≤ 40%, it is judged as an indeterminate opinion, and an identification opinion that is very likely not identical is made, that is, tends to deny the identity;

[0092] ⑧ When 30% < the model's test results on the sample ≤ 35%, it is judged as an indeterminate opinion, and an identification opinion that is highly likely to be different is made;

[0093] ⑨ When the model's test results on the sample are ≤30%, it is judged as a definitive opinion, and a negative identification opinion is made.

Claims

1. A method for electronic signature handwriting identification based on the VILT multimodal model, characterized in that, The steps are as follows: S1: Collect electronic signature handwriting information formed by the party using a stylus through an electronic writing device, including electronic signature data information and electronic signature image information; S2: Preprocess the collected electronic signature data and electronic signature image information of the parties involved respectively; S3: Use the preprocessed electronic signature handwriting information of the parties as samples, use K-Fold cross-validation to increase the amount of training data, build a dataset based on the samples, and divide the dataset into training set and validation set; S4: Construct a multimodal learning model based on Transformer network, namely the VILT multimodal model, and input the electronic signature data information as the text feature part of the model and the electronic signature image information as the image feature part of the model. S5: The electronic data information and electronic image information in the sample are fed into the VILT multimodal model to train the model. Among them, the electronic signature data information uses word embedding to perform feature transformation of the data and adds position encoding before passing it to the transformer encoder for feature extraction. The electronic signature image information uses patch embedding and adds position encoding before passing it to the transformer encoder for attention mechanism calculation. S6: Optimize the model and adjust parameters by relying on the validation set; S7: Use the electronic signature handwriting of the party to be identified as the evidence, obtain the electronic signature data information and electronic signature image information of the evidence, and send them into the trained and optimized VILT multimodal model to obtain the detection result, namely the electronic signature handwriting similarity value. S8: Based on the national standards for handwriting identification, the corresponding test results are obtained.

2. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that, In S1, the electronic signature data information includes X-axis and Y-axis data of handwriting position information and Z-axis data of handwriting pressure information, and the electronic signature image information refers to the electronic signature image.

3. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that, In S2, electronic data information is sorted and statistically analyzed, format converted, and data preprocessed by correction. Electronic image information is preprocessed by noise elimination, image binarization, character normalization, signature region extraction, and image size normalization.

4. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 3, characterized in that, The specific method in S2 is as follows: S21: Based on the data from the X and Y axes, reconstruct the trajectory of the writing motion, calculate the distance between two data points, and generate a diagram showing the change of distance over time. The speed and its changes can be clearly seen through the chart. Based on the data from the Z axis, generate a diagram showing the change of pen pressure over time. Finally, use Excel spreadsheet analysis to correct and organize the above electronic signature data. S22: Bilateral filtering is used to eliminate noise in the electronic signature image; a variable threshold method is used to process the electronic signature image to ensure that the signature stroke information is not lost; a bilinear interpolation algorithm is used to reduce the image size and adjust all signature images to a uniform size.

5. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that, The specific method of S4 is as follows: S41: The VILT multimodal model is a model that combines vision and language. Its structure includes an input representation module, a joint encoder, and an image location encoding module. S42: Input representation module: It consists of two parts: image information embedding layer and data information embedding layer. The image information embedding layer divides the electronic signature image into blocks, each block is called a patch, and each patch is linearly embedded into a fixed-length vector. The data information embedding layer is processed by word segmentation and converts the text into a vector representation through the embedding layer. S43: Joint Encoder: The VILT multimodal model uses a single Transformer, which consists of multiple self-attention layers, including a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections. S44: Position Encoding Module: Since the Transformer itself is not sensitive to the order of input, position encoding is added to distinguish the order of input elements. In VILT, both visual and language inputs need to have their own position information added: Image Patch Position Encoding: Add position information in the original image to each image patch; Text Word Position Information: Add position information to text input to identify the order of words in a sentence; S45: The input electronic signature image will be divided into several patches in the embedding layer, and these patches will be used as several sets of vectors to be input into the Transformer encoder. All electronic signature images have a uniform pixel size, that is, the image is divided into several blocks of grayscale image 1. The pixels of each patch are mapped to a one-dimensional vector through a linear transformation, and finally a compressed matrix is ​​obtained, which realizes the dimensionality reduction of the image patch. After obtaining the matrix containing all image data, it is flattened again, and the resulting vector is convenient for subsequent processing by the Transformer encoder. S46: For the input electronic signature data, the sequence length is divided according to the corresponding number in the electronic data embedding layer. Each sequence dimension corresponds to a single type of electronic data information. Since the electronic signature data itself is one-dimensional information, there is no need to perform vectorization operations. All electronic signature data can be flattened once and sent to the subsequent operation process. After adding position encoding and modality encoding to the input, a learnable classification encoding is added and then sent to the Transformer encoder for feature extraction. The multilayer perceptron part establishes a nonlinear mapping relationship between the multimodal input information and the generated electronic signature image.

6. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that, In S5, hyperparameters need to be set before training the VILT multimodal model, including the number of training iterations, batch size, learning rate, and regularization parameters.

7. The electronic signature handwriting identification method based on the VILT multimodal model according to claim 1, characterized in that, In S8, According to the national standard for handwriting identification, the test results are classified into three categories and nine types: definitive opinion, indeterminate opinion, and indeterminate opinion. The specific evaluation criteria are as follows: ① When the model's detection results for the sample are >90%, it is judged as a definitive opinion, and a positive identification opinion is made; ② When 85% < the model's test results for the sample ≤ 90%, it is judged as an indeterminate opinion, and an identification opinion of highly probable identity is made; ③ When 80% < the model's test results for the sample ≤ 85%, it is judged as an indeterminate opinion, and an identification opinion of very likely identity is made, that is, a tendency to affirm identity is made; ④ When 70% < the model's test results for the sample ≤ 80%, it is judged as an indeterminate opinion, and a possible identical identification opinion is made; ⑤ When 50% < the model's test results on the sample ≤ 70%, an expert opinion of indeterminacy shall be issued; ⑥ When 40% < the model's test results for the sample ≤ 50%, it is judged as an indeterminate opinion, and a potentially different identification opinion is made; ⑦ When 35% < the model's test results for the sample ≤ 40%, it is judged as an indeterminate opinion, and an identification opinion that is very likely not identical is made, that is, tends to deny the identity; ⑧ When 30% < the model's test results on the sample ≤ 35%, it is judged as an indeterminate opinion, and an identification opinion that is highly likely to be different is made; ⑨ When the model's test results on the sample are ≤30%, it is judged as a definitive opinion, and a negative identification opinion is issued.

Images