A multi-modal fusion handwritten electronic signature verification method

Abstract

The present invention proposes a multi-modal fusion handwritten electronic signature verification method, which includes the following modality data preprocessing and feature extraction steps: S1 : Collect several modal data of the user's handwritten electronic signature, including signature image pic and signature timing data seq;S2 : Obtain the pressure matrix through the signature time series data pmat ; The signature timing data includes the horizontal coordinates of each moment xt , vertical coordinate yt and pressure pt , first generate the size ymax × xmax Two-dimensional all-zero matrix, then put the matrix in yt OK xt The column is assigned pt ; Interpolate the matrix and unify it to the signature image pic Same size H × W ; S3 :Extract local features through signature time series data, and define time functions at the same time. Extract speed, angle and acceleration features from the time series with coordinate, time and pressure information to obtain the time series with extended dimensions. Through interpolation operations, the writing time differences of each signature are regularized to a uniform length. L .

Description

Technical Field

[0001] The present invention belongs to the field of multimodal learning, and in particular relates to a multimodal fusion handwritten electronic signature verification method. Background Art

[0002] A handwritten electronic signature refers to a signature recorded in a digital way. According to different modal types, it can be divided into two categories: static signature and dynamic signature. Static signature can refer to the signature image generated by the electronic board or the image scanned from the paper signature, which reflects spatial information such as shape, thickness, and texture; dynamic signature refers to the physical data collected by the electronic board during the handwriting process, such as coordinates, pressure, inclination, and pen lifting status at different times, which reflects time information.

[0003] Handwritten electronic signatures are most vulnerable to forgery attacks, where attackers impersonate registered users to sign. These attacks can be divided into: simple forgery, where attackers only know the text content to be written but not the structural shape; and skilled forgery, where attackers can deliberately practice the structure of the written text to better pass signature verification. Since both text content and text structure are easily publicly accessible, the threat posed by skilled forgeries should be focused on. Therefore, handwritten electronic signature verification is required, which is mainly divided into two steps: registration and comparison. In the registration stage, user signature samples are collected and stored in the signature database after feature extraction for subsequent comparison and analysis. In the verification stage, the registered sample template and the sample to be verified are input. After feature comparison, it is determined whether the sample to be verified is the legitimate signature of the declared user. According to the handwritten signature modality used, it can be divided into offline verification, online verification, and verification under multimodal fusion.

[0004] 2004 Year, Vatsa et al. Vatsa M , Singh R , Mitra P , et al . Signature Verification Using Static and Dynamic Features [ C ] / / Neural Information Processing : 11th International Conference ( ICONIP ). Springer , 2004 : 350 - 355.] proposed a multi-modal multi-classifier signature verification algorithm, through a one-dimensional logarithm Gabor Wavelet and Euler number analyze the texture and topological features of the signature image respectively, and combine the real-time pressure value, breakpoint and writing time characteristics of the dynamic signature.

[0005] 2021 Year, Ji et al. Ji N , Liu B , Zhao Z , et al . Content - Independent Online Handwriting Verification Based on Multi - Modal Fusion [ C ] / / 2021 IEEE International Conference on Multimedia and Expo ( ICME ). IEEE , 2021 : 1 - 6 .] proposed a multi-path feature fusion network to fuse static and dynamic signatures obtained by electronic devices to enhance handwriting verification. This method also utilized a content-independent handwriting verification scheme to counter skilled forgery attacks.

[0006] 2022 Year Zhang et al. Zhang P , Jiang J , Liu Y , et al . MSDS : A large - Scale Chinese Signature and Token Digit String Dataset for Handwriting Verification [ J ]. Advances in Neural Information Processing Systems , 2022 , 35 : 36507 - 36519 .] contributed a large-scale multimodal Chinese signature and digit string handwriting dataset MSDS , where two modalities are simply combined for signature verification by splicing, which shows that the verification model has a lower equal error rate than the single modality under multimodal information.

[0007] 2024 Year, Li et al. Li Q , Wang Z , Jin L , et al . MMHSV : A Multimodal Handwritten Signature Verification Fusing Dynamic and Static Feature [ C ] / / 2024 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ). IEEE , 2024 : 4730 - 4734 .] proposed a deep learning-based multimodal handwritten signature verification framework, which fused the complementary signal components of signature image and stroke sound. The framework was based on a signature representation extraction model based on multi-task learning and dual path.

[0008] In addition, Chinese patent applications CN117789309A A method for line recognition of different handwritten signatures is disclosed, which involves the fusion of different modalities, online acquisition of time series data of electronic signatures, obtaining sequence features and image features respectively and fusion, and finally obtaining a representation vector of the data pair, and determining the similarity between the test signature and the template data through the similarity of the representation vector. Chinese patent application CN115482541A A cross-modal handwriting-based quasi-cooperative training method is disclosed, which involves converting paper signatures into binary images, echoing electronic signature sequences to obtain signature images, and then aligning the signature images of the two modal types in the channel dimension to obtain image pairs of electronic and paper signatures, constructing a dual-tower network structure, and performing cross-modal comparison multi-task joint classification training through labeled data and unlabeled data; the identity of the cross-modal handwriting signer is determined by comprehensive classification results. Chinese patent application CN116152815A A cross-modal handwriting comparison method based on an adversarial erasure mechanism is disclosed, which involves constructing positive and negative sample pairs generated by paper signatures and electronic handwritten signature data, removing stroke boundary interference and noise around signatures, and using a dual-branch architecture to construct a deep neural network based on an adversarial erasure mechanism. The self-attention map of the feature map corresponding to the positive and negative sample pairs is obtained through channel pooling operations, and the effective area of ​​the feature map is selectively erased locally. Finally, the probability that the signature pair belongs to the same person is inferred through a binary classifier. Chinese patent application CN115601843A A multimodal signature handwriting identification system based on a dual-stream network is disclosed, which involves the signer's offline paper signature image and online signature image data. A cyclic adversarial generative network is used to migrate the signature image between domains, and the same-modal data are combined and spliced ​​to be converted into images of the same scale. The spliced ​​two types of same-modal signature image data are respectively input into the dual-stream network, and the probability score of the similarity of the signature image pair is output for binary classification judgment.

[0009] In general, the electronic signature verification method under multimodal fusion can have better performance than that under single modality, and the information between static images and dynamic sequences can complement each other, but there are few studies on this kind. Regarding multimodal handwriting data, most of them are to generate dynamic sequences and translated images simultaneously on the electronic signature board, and the situation of placing carbon paper on the handwriting board and scanning it after writing is not suitable for most actual application scenarios. Summary of the invention

[0010] The present invention proposes a multi-modal fusion handwritten electronic signature verification method, which enhances the ability of the signature verification model by fusing the multi-modal information of the handwritten electronic signature and provides users with a verification service for forged handwritten signatures.

[0011] The technical solution of the present invention is implemented as follows: a multi-modal fusion handwritten electronic signature verification method, the method comprising the following modal data preprocessing and feature extraction steps:

[0012] S1: Collect several modal data of the user's handwritten electronic signature, including signature image pic and signature timing data seq;

[0013] S2 : Obtain the pressure matrix through the signature time series data pmat ; The signature timing data includes the horizontal coordinates of each moment xt , vertical coordinate yt and pressure pt , first generate the size ymax × xmax Two-dimensional all-zero matrix, then put the matrix in yt OK xt The column is assigned pt ; Interpolate the matrix and unify it to the signature image pic Same size H × W ;

[0014] S3 :Extract local features through signature time series data, and define time functions at the same time. Extract speed, angle and acceleration features from the time series with coordinate, time and pressure information to obtain the time series with extended dimensions. Through interpolation operations, the writing time differences of each signature are regularized to a uniform length. L ;

[0015] S4 :Extract global features through signature time series data. First, expand the original time series data to obtain the time series data of horizontal / vertical coordinates, speed, horizontal / vertical speed, acceleration, horizontal / vertical acceleration and pressure. Then, calculate the average value, maximum / minimum value difference, standard deviation and maximum value as global features. Based on the global features, get the vector representing the global features of the signature. VG ;

[0016] S5 :For the signature image pic and pressure matrix pmat Concatenate in the channel dimension and then input into a 2D convolutional neural network 2D - CNN , get the deep representation vector of image-level fusion hp ; For the local feature time series, input the one-dimensional convolutional neural network 1D - CNN , and obtain the time-level deep representation vector hs ;

[0017] S6 : Building a deep neural network DNN , mining the input global feature vector VG Information applicable to authenticity classification;

[0018] S7 : Enter a template signature sigtemp And the signature to be tested sigtest Multimodal data of , get the signature feature vector fvtemp and fvtest ,calculate out = FC ( fvtemp || fvtest ),in out It is a two-dimensional vector used for authenticity identification and classification. During the model training process, the binary cross entropy loss function is used. During the test, the user's registration signature is used as a sample signature and is input into the identification network together with the signature to be tested to obtain the verification result.

[0019] As a preferred embodiment, the steps S5 In the process of obtaining the time-level deep representation vector hs After that, use TFN The network performs backend fusion, which is calculated as follows: ,in For the concatenation operation, is the tensor outer product operation, FC is a fully connected layer.

[0020] As a preferred embodiment, the steps S6 Mining the input global feature vector VG When the information in the true and false classification is suitable, the output vector hG , and calculate ,in fv is the signature feature vector that integrates multimodal information.

[0021] As a preferred embodiment, the steps S4 The vector representing the global features of the signature is obtained from VG The following global features are also included: Total signing time T , Number of pens Nup , the location of the maximum pressure point ( xpmax , ypmax ), the ratio of average speed to maximum speed, the ratio of horizontal span to vertical span and the signature path length .

[0022] After adopting the above technical solution, the beneficial effect of the present invention is that by combining the signature image and the signature time series data, the method realizes a comprehensive analysis of the user's signature. The traditional signature verification method usually relies only on static images and is easily affected by forgery and imitation, while the multimodal fusion method introduces dynamic time series data, allowing the system to capture a variety of information including writing pressure, speed, angle, etc., thereby improving the accuracy and reliability of verification.

[0023] The generation and interpolation of the pressure matrix enables the time series data to be compared with the image data at the same scale. This processing method solves the length difference problem caused by different users' writing habits, ensures the consistency of feature extraction, and further enhances the model's ability to recognize signatures.

[0024] The combination of local and global feature extraction enables this method to not only capture microscopic changes in signatures (such as speed and acceleration), but also analyze the overall writing pattern. The construction of the global feature vector effectively summarizes the overall characteristics of the signature by calculating statistics such as the mean and standard deviation, thus improving the accuracy of classification.

[0025] Using a two-dimensional convolutional neural network ( 2D - CNN ) and one-dimensional convolutional neural network ( 1D - CNN ) respectively extracts deep features from images and time series data, achieving effective fusion of multimodal information. This deep learning model can automatically learn and extract the most discriminative features, reducing the subjectivity of manual feature selection and improving the generalization ability of the model.

[0026] The deep neural network constructed ( DNN ) can deeply mine the information in the global feature vector that is suitable for authenticity classification, allowing the system to more accurately determine the authenticity of the signature. Compared with traditional machine learning methods, this deep learning method can handle more complex pattern recognition tasks and further improve the accuracy of verification.

[0027] The binary cross entropy loss function is used in the model training process to ensure the optimization of the model in the training phase. At the same time, the sample signature at the time of registration is used as a benchmark in the test phase to compare with the signature to be tested, which improves the security and accuracy of verification and avoids erroneous verification results.

[0028] Taking into account a variety of feature information, the system's anti-interference ability is enhanced, enabling it to better adapt to the writing styles and habits of different users. This flexibility and adaptability make the method more widely applicable in practical applications and can meet high-security scenarios such as financial transactions and legal document signing. By integrating the information of handwritten signature multimodal data, including image and sequence features, local and global features, signatures can be verified more effectively, providing forgery recognition services for handwritten signatures. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] picture 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Example

[0033] As shown 1 As shown, a multi-modal fusion handwritten electronic signature verification method includes the following steps:

[0034] S1 : Collect multiple modal data of the user's handwritten electronic signature, including signature images pic and signature timing data seq .

[0035] S2 : Obtain the pressure matrix through the signature time series data pmat The signature timing data contains the horizontal coordinates of each moment. xt , vertical coordinate yt and pressure pt , first generate the size ymax × xmax Two-dimensional all-zero matrix, then put the matrix in yt OK xt The column is assigned pt Since signatures of different people or signatures of the same person at different times have variability, the matrix sizes are different, so the matrix is ​​interpolated to unify the signature image. pic Same size H × W .

[0036] S3 :Extract local features from signature time series data. Define the time function, extract features such as speed, angle and acceleration from the time series with coordinate, time and pressure information, and obtain the time series with extended dimensions. Since the writing time of each signature is different, it is regularized to a uniform length through interpolation. L .

[0037] S4 :Extract global features through signature time series data. First, expand the time series data of horizontal / vertical coordinates, speed, horizontal / vertical speed, acceleration, horizontal / vertical acceleration and pressure from the original time series data, and then selectively calculate global features such as average value, maximum / minimum value difference, standard deviation and maximum value. In addition, the following global features are also included: 1 )Total signature time T ;( 2 ) Number of pens Nup ;( 3 ) Location of the maximum pressure point ( xpmax , ypmax );( 4 )The ratio of average speed to maximum speed; 5 )The ratio of the horizontal span to the vertical span; 6 )Signature path length Combining all the above elements, we get a vector representing the global features of the signature VG .

[0038] S5 : Construct a feature fusion network, first of all, the signature image pic and pressure matrix pmat Concatenate in the channel dimension and then input into a 2D convolutional neural network 2D - CNN , get the deep representation vector of image-level fusion hp ; For the local feature time series, input the one-dimensional convolutional neural network 1D - CNN , and obtain the time-level deep representation vector hs , and then use TFN The network performs backend fusion, which is calculated as follows: ,in For the concatenation operation, is the tensor outer product operation, FC is a fully connected layer. 1D and 2D - CNN It is a network with a channel attention mechanism to better capture features related to the classification task. Each layer of the network uses ReLu Activation function.

[0039] S6 : Building a deep neural network DNN , further mining the input global feature vector VG The information in the true and false classification is used to output the vector hG , and calculate ,in fv is the signature feature vector that integrates multimodal information. DNN Each layer of network neurons uses ReLu Activation function.

[0040] S7 : Enter a template signature sigtemp And the signature to be tested sigtest Multimodal data of , get the signature feature vector fvtemp and fvtest ,calculate ,in out is a two-dimensional vector used for authenticity identification and classification. In the model training process, the binary cross entropy loss function is used. During the test, the user's registration signature is used as a sample signature and input into the identification network together with the signature to be tested to obtain the verification result.

[0041] For the interpolation method, linear interpolation can be used. S2 The images in the steps are unified to ( 64 , 192 )of 1 : 3 Height and width values, S3 The long sequences in the step are downsampled and the short sequences are upsampled to the average length.

[0042] The steps S3 In the example, the time function used for feature extraction is [ Zhang P , Jiang J , Liu Y , et al . MSDS : A large - Scale Chinese Signature and Token Digit String Dataset for Handwriting Verification [ J ]. Advances in Neural Information Processing Systems , 2022 , 35 : 36507 - 36519 .], extracting features such as velocity, angle and acceleration from the time series with coordinate, time and pressure information, as shown in Table 1 As shown in the figure, the time series with dimension expansion is obtained. Since the writing time of each signature is different, the long sequence is downsampled and the short sequence is upsampled to a uniform average length through interpolation operations, such as linear interpolation. L .

[0043] Global features are extracted by processing the signature time series data. First, the time series data of horizontal / vertical coordinates, speed, horizontal / vertical speed, acceleration, horizontal / vertical acceleration and pressure are expanded from the original time series data, and then global features such as average value, maximum / minimum value difference, standard deviation and maximum value are selectively calculated from them, as shown in Table 2. 2 In addition, the following global features are included: Total signature time T ; Number of pens Nup ; Location of the maximum pressure point ( xpmax , ypmax ); the ratio of average speed to maximum speed; the ratio of horizontal span to vertical span; ( 6 )Signature path length Combining all the above elements, we get a vector representing the global features of the signature VG ; Steps used S4 In the example above, select the global features to be used. 2 As shown;

[0044] Constructing feature fusion network

[0045] First, the signature image pic and pressure matrix pmat Concatenate in the channel dimension and then input into a 2D convolutional neural network 2D - CNN , get the deep representation vector of image-level fusion hp ; For the local feature time series, input the one-dimensional convolutional neural network 1D - CNN , and obtain the time-level deep representation vector hs , and then use TFN The network performs backend fusion, which is calculated as follows: ,in For the concatenation operation, is the tensor outer product operation, FC is a fully connected layer. 1D and 2D - CNN It is a network with a channel attention mechanism to better capture features related to the classification task. Each layer of the network uses ReLu Activation function.

[0046] Building a Deep Neural Network DNN , further mining the input global feature vector VG The information in the true and false classification is used to output the vector hG , and calculate ,in fv is the signature feature vector that integrates multimodal information. DNN Each layer of network neurons uses ReLu Activation function.

[0047] Input signature sample pairs to train the binary classification model. Input sample signature sigtemp And the signature to be tested sigtest Multimodal data of , get the signature feature vector fvtemp and fvtest ,calculate ,in out is a two-dimensional vector used for authenticity identification and classification. In the model training process, the binary cross entropy loss function is used. During the test, the user's registration signature is used as a sample signature and input into the identification network together with the signature to be tested to obtain the verification result.

[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-modal fusion handwritten electronic signature verification method, characterized in that: The method comprises the following steps of preprocessing and feature extraction of modal data: S1 : Collect several modal data of the user's handwritten electronic signature, including signature image pic and signature timing data seq; S2 : Obtain the pressure matrix through the signature time series data pmat ; The signature timing data includes the horizontal coordinates of each moment xt , vertical coordinate yt and pressure pt , first generate the size ymax × xmax Two-dimensional all-zero matrix, then put the matrix in yt OK xt The column is assigned pt ; Interpolate the matrix and unify it to the signature image pic Same size H × W ; S3 :Extract local features through signature time series data, and define time functions at the same time. Extract speed, angle and acceleration features from the time series with coordinate, time and pressure information to obtain the time series with extended dimensions. Through interpolation operations, the writing time differences of each signature are regularized to a uniform length. L ; S4 :Extract global features through signature time series data. First, expand the original time series data to obtain the time series data of horizontal / vertical coordinates, speed, horizontal / vertical speed, acceleration, horizontal / vertical acceleration and pressure. Then, calculate the average value, maximum / minimum value difference, standard deviation and maximum value as global features. Based on the global features, get the vector representing the global features of the signature. VG ; S5 :For the signature image pic and pressure matrix pmat Concatenate in the channel dimension and then input into a 2D convolutional neural network 2D - CNN , get the deep representation vector of image-level fusion hp ; For the local feature time series, input the one-dimensional convolutional neural network 1D - CNN , and obtain the time-level deep representation vector hs ; S6 : Building a deep neural network DNN , mining the input global feature vector VG Information applicable to authenticity classification; S7 : Enter a template signature sigtemp And the signature to be tested sigtest Multimodal data of , get the signature feature vector fvtemp and fvtest ,calculate out = FC ( fvtemp || fvtest ),in out It is a two-dimensional vector used for authenticity identification and classification. During the model training process, the binary cross entropy loss function is used. During the test, the user's registration signature is used as a sample signature and is input into the identification network together with the signature to be tested to obtain the verification result.

2. As claimed in claim 1 The multi-modal fusion handwritten electronic signature verification method is characterized by: The steps S5 In the process of obtaining the time-level deep representation vector hs After that, use TFN The network performs backend fusion, which is calculated as follows: ,in For the concatenation operation, is the tensor outer product operation, FC is a fully connected layer.

3. As claimed in claim 1 The multi-modal fusion handwritten electronic signature verification method is characterized by: The steps S6 Mining the input global feature vector VG When the information in the true and false classification is suitable, the output vector HkD , and calculate ,in f is the signature feature vector that integrates multimodal information.

4. As claimed in claim 1 The multi-modal fusion handwritten electronic signature verification method is characterized by: The steps S4 The vector representing the global features of the signature is obtained from VG The following global features are also included: Total signing time T , Number of pens Nup , the location of the maximum pressure point ( xpmax , ypmax ), the ratio of average speed to maximum speed, the ratio of horizontal span to vertical span and the signature path length .

Images