Signature handwriting authenticity digital identification method based on multi-modal features

Abstract

The invention discloses a signature handwriting authenticity digital identification method based on multi-modal features, and relates to the technical field of image processing, and the method comprises the following specific steps: 1, obtaining off-line image data and on-line time sequence data of two groups of signatures to be identified, the data preprocessing module carries out standardization processing on the offline image data and the online time sequence data of the signature to be identified to obtain preprocessed offline image data and preprocessed online time sequence data; 2, inputting the preprocessed offline image data into an offline feature extraction network, and extracting and outputting 1024-dimensional offline features; and 3, inputting the preprocessed online time sequence data into an online signature feature extraction network. According to the method, the online features and the offline features of the signature handwriting are fused and combined to form multi-mode signature handwriting digital recognition, the subtle difference of the counterfeited signature handwriting in static structure consistency and dynamic behavior consistency can be effectively captured, and the recognition capability of complex counterfeited means is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method for digitally recognizing the authenticity of signature handwriting based on multimodal features. Background Technology

[0002] With the increasing demands for security and convenience in identity authentication from scenarios such as digital finance (e.g., electronic contract signing, online payment verification) and e-government (e.g., remote government affairs signature confirmation), digital signature data (image or time-series data converted from physical signatures through scanning, electronic pens, touchscreens, etc.) is becoming increasingly widely used in the field of identity authentication as a behavioral characteristic carrier that combines uniqueness and ease of use. Biometric authentication is divided into physiological characteristics (e.g., fingerprints, irises) and behavioral characteristics (e.g., signature handwriting, typing rhythm). Among them, digital signature recognition is a technology that verifies user identity by extracting and analyzing the digital characteristics (static structure, dynamic behavior) of signature handwriting. Because it conforms to traditional signature habits, it is widely used in key areas such as financial contract signing, legal document confirmation, and government service authentication.

[0003] Currently, the field of digital signature recognition is mainly divided into two major technical directions: offline digital signature recognition, which processes offline digital image data of signatures (such as scanned copies of paper signatures and image files generated by taking photos), and focuses on analyzing the static features of the handwriting in the image (such as stroke shape, ink density, and overall structure); and online digital signature recognition, which processes dynamic time-series data collected in real time during the signing process (such as the x / y coordinates of the electronic pen, writing pressure, timestamp, and derived digital parameters such as speed and acceleration), and focuses on mining the dynamic features reflecting writing habits in the time-series data (such as stroke order, speed changes, and pressure curves).

[0004] Traditional signature handwriting recognition solutions based on single-modal digital data (offline image data only or online time-series data only) face severe challenges: Limitations of offline data recognition: Relying solely on static digital image data of the signature handwriting, it loses the dynamic behavioral characteristics of the writing process (such as starting speed, ending pressure, and stroke order logic). When faced with highly realistic digital image data generated from forged signatures, it is difficult to distinguish between "genuine signature handwriting" and "forged signature handwriting." Limitations of online data recognition: Processing only dynamic time-series data, it cannot utilize the static details of handwriting in offline image data (such as ink edge texture, stroke intersection shape, and the combination of paper texture and ink). It is susceptible to replay attacks (stealing and replaying legitimate users' time-series data) and device acquisition noise (pressure sampling deviations of different electronic pens). Poor cross-recording media compatibility: Existing solutions lack adaptability to recording media of digital signature handwriting data (such as PNG format scanned documents, JSON time-series files from electronic pens, and local cached data from tablets). Differences in digital data formats across different media lead to a significant drop in accuracy during cross-media recognition, making it difficult to meet the needs of multi-device collaborative authentication scenarios.

[0005] Therefore, a digital signature handwriting authentication method based on multimodal features is proposed to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a digital identification method for the authenticity of signature handwriting based on multimodal features, so as to solve the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The method for digitally identifying the authenticity of signature handwriting based on multimodal features includes the following steps: Step 1: Obtain two sets of offline image data and online time-series data of the signature to be identified. The data preprocessing module performs standardization processing on the offline image data and online time-series data of the signature to be identified, respectively, to obtain preprocessed offline image data and preprocessed online time-series data. Step 2: Input the preprocessed offline image data into the offline feature extraction network to extract 1024-dimensional offline features. ; Step 3: Input the preprocessed online time series data into the online signature feature extraction network to extract 1024-dimensional online features. ; Step 4: Concatenate a set of signatures to be identified into 1024-dimensional offline features. and 1024-dimensional online features Generate 2048-dimensional fusion features 1; Then, concatenate another set of signatures to be identified using 1024-dimensional offline features. and 1024-dimensional online features Generate 2048-dimensional fusion features 2; Step 5: Calculate the 2048-dimensional fusion features 1-dimensional and 2048-dimensional fusion features Euclidean distance of 2 d Determine Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are signatures of different users and are forged signatures.

[0008] Furthermore, the specific steps for standardizing offline image data to be identified for signature recognition are as follows: Step 111: Standardize the dimensions of offline image data and online time-series data for bilinear interpolation, respectively; Step 112: Perform noise filtering on the two sets of size-standardized data; Step 113: Perform contrast enhancement processing on the two sets of noise-filtered data respectively; Step 114: Normalize the two sets of contrast-enhanced data to obtain preprocessed offline image data.

[0009] Furthermore, the specific steps for standardizing the online time-series data to be identified are as follows: Step 121: Online time series data coordinate calibration; Step 122: Use linear interpolation to fill in the missing data caused by device sampling delay; Step 123: Calculate the derived features of the imputed data; Step 124: Retain the original time series length T for the data from the derived feature calculation to obtain the preprocessed online time series data.

[0010] Furthermore, the offline feature extraction network EConvNeXt includes the backbone network ConvNeXt, an ARM attention refinement module, and a multi-scale bidirectional fusion module.

[0011] Furthermore, the specific steps for step two are as follows: Step 21: Extract feature maps at different levels using the backbone network ConvNeXt; Step 22: The ARM attention refinement module enhances feature maps at different levels; Step 23: The multi-scale bidirectional fusion module fuses the enhanced features at different levels to obtain 1024-dimensional offline features. ; Furthermore, the online signature feature extraction network is SigGRU.

[0012] Furthermore, the specific steps for step three are as follows: Step 31: Receive the preprocessed online time series data; Step 32: The preprocessed online time series data is mapped to a 512-dimensional feature vector through a fully connected Dense layer with 512 neurons, and then the 512-dimensional feature vector is normalized by a Batch Normalization layer. Step 33: Using the tanh activation function and a gating mechanism, capture stroke order dependency, velocity abrupt change, and pressure peak in the time series data, and output T×1024-dimensional time series features; Step 34: A learnable temporal attention pooling mechanism is used to weight and aggregate the T×1024-dimensional temporal features output by the GRU; Step 35: Input the weighted aggregated features into a fully connected Dense layer with 1024 neurons. Through linear transformation and feature mapping, the final output is a 1024-dimensional online signature feature. .

[0013] Furthermore, the specific steps for step four are as follows: Step 41: Extract the 1024-dimensional offline features output by the offline feature extraction network EConvNeXt. The 1024-dimensional online features output by the online signature feature extraction network By splicing, a 2048-dimensional multimodal fusion feature is generated. 1; Step 42: Then, concatenate another set of signatures to be identified using a twin network architecture. and Generate 2048-dimensional fusion features 2.

[0014] Furthermore, step five involves the following specific steps: Step 51: Through 1 and 2. Calculate Euclidean distance d ; Step 52: Determine the Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are signatures of different users and are forged signatures.

[0015] Furthermore, the Euclidean distance d is calculated as follows: ; in, Represented as 2048-dimensional fusion features 1-dimensional and 2048-dimensional fusion features 2-dimensional index number.

[0016] Beneficial effects: This invention combines online and offline features of signature handwriting to form a multimodal digital signature handwriting recognition system. This effectively captures subtle differences in the static structural consistency and dynamic behavioral coherence of forged signature handwriting, significantly improving the ability to identify complex forgery methods (such as tracing combined with temporal imitation). Furthermore, by standardizing the processing of digital data from different recording media, it solves the problem of cross-media recognition compatibility. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 This is a flowchart of the signature handwriting authenticity digital recognition method based on multimodal features of the present invention; Figure 2 This is a schematic diagram of the ARM Attention Refinement Module structure. Figure 3 This is a schematic diagram of the multi-scale bidirectional fusion module structure; Figure 4 1024-dimensional offline features Generate flowchart; Figure 5 The structure diagram of the SigGRU network for online signature feature extraction; Figure 6 This is a schematic diagram of the network structure for digital feature extraction of dual-modal signatures. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The present invention will be further described below with reference to embodiments.

[0021] Example 1, please refer to Figure 1-6 : This embodiment presents a digital signature handwriting authenticity recognition method based on multimodal features; The specific steps are as follows: Step 1: Obtain two sets of offline image data and online time-series data of the signature to be identified. The data preprocessing module performs standardization processing on the offline image data and online time-series data of the signature to be identified, respectively, to obtain preprocessed offline image data and preprocessed online time-series data. Offline image data: such as scanned documents and photographed images, in PNG / JPG format, with a resolution ≥200dpi; Online time-series data: such as JSON format data collected by electronic pen / touch screen, including x / y coordinates, pressure p, timestamp t, sampling rate 50-200Hz; The specific steps for standardizing offline image data to be identified for signature recognition are as follows: Step 111: Standardize the dimensions of offline image data and online time-series data for bilinear interpolation, respectively; Offline image data and online time-series data are scaled to standard sizes using bilinear interpolation. The standard size is 224 pixels × 224 pixels × 3 (width × height × channel); Step 112: Perform noise filtering on the two sets of size-standardized data; Two sets of images of uniform size were subjected to adaptive median filtering to remove scanning noise and paper texture. Step 113: Perform contrast enhancement processing on the two sets of noise-filtered data respectively; Improve the distinction between handwriting and background by limiting contrast adaptive histogram equalization (CLAHE); Step 114: Normalize the two sets of contrast-enhanced data to obtain preprocessed offline image data.

[0022] The normalization process involves normalizing pixel values ​​to the [0,1] range to eliminate the influence of brightness differences.

[0023] The specific steps for standardizing online time-series data to be identified for signature recognition are as follows: Step 121: Online time series data coordinate calibration; Online time-series data coordinate calibration: Maps x / y coordinates to a [0,1] normalized coordinate system to eliminate screen size differences; Step 122: Use linear interpolation to fill in the missing data caused by device sampling delay; Step 123: Calculate the derived features of the imputed data; Based on the original time series data x,y,p,t (where x is the nth...) tThe coordinates of the pen tip in the horizontal direction at each sampling moment are in pixels, with the origin set to the upper left corner of the signature area; y is the coordinate of the pen tip in the horizontal direction at the sampling moment. t The coordinates of the pen tip in the vertical direction at each sampling time are in pixels, with the origin at... x The axes remain consistent; p is the first... t The pen pressure value at each sampling moment reflects the contact force between the pen tip and the writing surface. It is usually normalized and ranges from [0,1]. t is the timestamp of the sampling moment in milliseconds, recording the acquisition time of pen tip position and pressure data. The following 6-dimensional derived features are generated, and the formulas and symbols for each feature are defined as follows: x-direction velocity y-direction velocity (in The difference in x-coordinate between adjacent time steps. This represents the difference in y-coordinates between adjacent time steps. (The time interval between adjacent time steps). X-direction acceleration acceleration in the y direction (in The x-direction velocity of adjacent time steps Difference, The y-direction velocity of adjacent time steps (difference) Pen pressure variation rate (in (The difference in pressure values ​​p between adjacent time steps). Tangent angle (in The difference in x-coordinate between adjacent time steps. (The difference in y-coordinates between adjacent time steps). Step 124: Retain the original time series length T for the data calculated from the derived features to obtain the preprocessed online time series data (without zero padding / truncation to ensure time series integrity). Step 2: Input the preprocessed offline image data into the offline feature extraction network to extract 1024-dimensional offline features. ; The offline feature extraction network EConvNeXt consists of a backbone network ConvNeXt, an ARM attention refinement module, and a multi-scale bidirectional fusion module.

[0024] The specific steps for step two are as follows: Step 21: Extract feature maps at different levels using the backbone network ConvNeXt; Step 22: The ARM attention refinement module enhances feature maps at different levels; Step 23: The multi-scale bidirectional fusion module fuses the enhanced features at different levels to obtain 1024-dimensional offline features. ; The backbone network ConvNeXt contains four feature extraction stages: Stage 1, Stage 2, Stage 3, and Stage 4. Stage 1, Stage 2, Stage 3, and Stage 4 are composed of ConvNeXt Blocks with numbers of 3, 3, 9, and 3 respectively. The feature map resolutions of Stage 1, Stage 2, Stage 3, and Stage 4 are 56 pixels × 56 pixels, 28 pixels × 28 pixels, 14 pixels × 14 pixels, and 7 pixels × 7 pixels respectively. The number of channels of Stage 1, Stage 2, Stage 3, and Stage 4 are 96 pixels, 192 pixels, 384 pixels, and 768 pixels respectively.

[0025] Step 21: Extract feature maps at different levels from the preprocessed offline image data in Stage 1, Stage 2, Stage 3, and Stage 4. The outputs of Stage1, Stage2, Stage3 and Stage4 are all connected in series with a set of ARM attention refinement modules. The ARM attention refinement modules include channel attention sub-modules, spatial attention sub-modules and feature refinement modules. Channel Attention Submodule: Used to perform global max pooling and global average pooling on the feature maps of different levels input from Stage1, Stage2, Stage3 and Stage4 respectively. The results of global max pooling and global average pooling are input into a two-layer MLP with shared weights, and channel weights are generated by Sigmoid activation to highlight the discriminative channel features of "stroke intersection point, start / end point". The number of hidden layer neurons in a two-layer MLP is 1 / 4 of the number of input channels; Spatial Attention Submodule: This module performs max pooling and average pooling on the feature maps of different levels input from Stage1, Stage2, Stage3 and Stage4 along the channel dimension. The results of max pooling and average pooling are concatenated, and the concatenated data is processed by 3×3 convolution and Sigmoid activation to generate spatial weights, focusing on key spatial regions such as "handwriting edges and ink density variation areas". 3×3 convolution with stride 1 and padding 1; Feature Refinement Module: Input feature maps at different levels and multiply them sequentially with channel weights and spatial attention weights, and output an enhanced feature map; Step 22 is performed as follows: Step 221: Perform global max pooling and global average pooling on the feature maps of different input levels respectively, and input the results of global max pooling and global average pooling into a two-layer MLP with shared weights, and generate channel weights by sigmoid activation; Step 222: Perform max pooling and average pooling on the feature maps of different input levels along the channel dimension, concatenate the results of max pooling and average pooling, and generate spatial attention weights by performing 3×3 convolution and sigmoid activation on the concatenated data. Step 223: Multiply the input feature maps of different levels with the channel weights and spatial attention weights in sequence, and output the enhanced feature maps of different levels. The multi-scale bidirectional fusion module includes a unified channel sub-module, a bottom-up fusion module, a top-down fusion module, and a feature aggregation fusion module; Unified Channel Submodule: Unifies the four enhanced feature maps into 256 channels through 1×1 convolution; Bottom-up fusion module: Following the order of Stage4 (7×7) → Stage3 (14×14) → Stage2 (28×28) → Stage1 (56×56), the feature map of the current Stage is upsampled using bilinear interpolation so that the spatial size (width × height) of the upsampled feature map is completely consistent with the spatial size of the feature map of the next Stage (for example, the 7×7 feature map of Stage4 is upsampled to 14×14, which matches the size of the 14×14 feature map of Stage3), and then added element by element with the feature map of the corresponding Stage. Top-down fusion module: Following the order of Stage1 (56×56) → Stage2 (28×28) → Stage3 (14×14) → Stage4 (7×7), the feature map of the current Stage is downsampled using a 3×3 convolution with a stride of 2, so that the spatial size (width × height) of the downsampled feature map is completely consistent with the spatial size of the feature map of the next Stage (for example, the 56×56 feature map of Stage1 is downsampled to 28×28, which matches the size of the 28×28 feature map of Stage2), and then added element by element with the feature map of the corresponding Stage. Feature aggregation and fusion module: The four bidirectionally fused feature maps are concatenated, compressed to 1024 channels by 1×1 convolution, and output as 1024-dimensional offline features. .

[0026] The specific steps for step 23 are as follows: Step 231: Input the enhanced feature maps of different levels into the unified channel submodule, and unify the four enhanced feature maps into 256 channels through 1×1 convolution; Step 232: After the 256 channels are unified, the low-resolution feature maps are upsampled to the same resolution as the high-resolution feature maps in the order of Stage 4 → Stage 3 → Stage 2 → Stage 1 using bilinear interpolation. Figure 1 Then, add each element of the fused feature map to the corresponding Stage feature map to obtain the first fused feature map; Step 233: The first fused feature map is downsampled to the level of the low-resolution feature map by performing a 3×3 convolution with a stride of 2 in the order of Stage1→Stage2→Stage3→Stage4. Figure 1 Then, add the corresponding Stage feature map element by element to obtain the second fused feature map; Step 234: Concatenate the four sets of second fusion feature maps, compress them to 1024 channels through a 1×1 convolution, and output 1024-dimensional offline features. .

[0027] The online signature feature extraction network is SigGRU; SigGRU consists of an input layer, a dimensionality upscaling and normalization layer, a GRU feature extraction layer, a temporal aggregation layer, and an output layer. Input layer: Receives preprocessed online time-series data; (Online time-series data length is T, number of channels is 93, original features: x / y / p; 6 derived features) Dimensionality Upscaling and Normalization Layer: The preprocessed online time series data is upscaled to 512 dimensions through a 512-neuron Dense layer, and the feature distribution is stabilized by a Batch Normalization layer (normalizing along the time dimension) (alleviating cross-device sampling differences). GRU feature extraction layer: It consists of two GRU blocks connected in series. Each block contains 512 hidden units and uses the tanh activation function. It captures stroke order dependence, velocity abrupt change and pressure peak in time series data through gating mechanism and outputs T×1024-dimensional time series features. A dropout layer (dropout rate 0.2) is added after each GRU block to suppress overfitting. Temporal aggregation layer: A learnable temporal attention pooling mechanism is used to perform weighted aggregation of the T×1024-dimensional temporal features output by GRU; Attention weights for each time step are calculated using a single-layer MLP (1024-dimensional input, 1-dimensional output). ,in T This represents the number of raw time steps in the online signature time series data. After Softmax normalization, the time-series features are weighted and summed to obtain... :

[0028] Output the 1024-dimensional time series features at step t for the GRU Output layer: Outputs 1024-dimensional online features through a 1024-neuron Dense layer. ; Step 3: Input the preprocessed online time series data into the online signature feature extraction network to extract 1024-dimensional online features. ; Step three involves the following steps: Step 31: Receive the preprocessed online time series data; Step 32: The preprocessed online time series data is mapped to a 512-dimensional feature vector through a fully connected Dense layer with 512 neurons. Then, the 512-dimensional feature vector is normalized by a Batch Normalization layer to stabilize the feature distribution (alleviating cross-device sampling differences). Step 33: Using the tanh activation function and a gating mechanism, capture stroke order dependency, velocity abrupt change, and pressure peak in the time series data, and output T×1024-dimensional time series features; Step 34: A learnable temporal attention pooling mechanism is used to weight and aggregate the T×1024-dimensional temporal features output by the GRU; Step 35: Input the weighted aggregated features into a fully connected Dense layer with 1024 neurons. Through linear transformation and feature mapping, the final output is a 1024-dimensional online signature feature. .

[0029] Step 4: Concatenate a set of signatures to be identified into 1024-dimensional offline features. and 1024-dimensional online features Generate 2048-dimensional fusion features 1; Then, concatenate another set of signatures to be identified using 1024-dimensional offline features. and 1024-dimensional online features Generate 2048-dimensional fusion features 2; Step four involves the following steps: Step 41: Extract the 1024-dimensional offline features output by the offline feature extraction network EConvNeXt. The 1024-dimensional online features output by the online signature feature extraction network By splicing, a 2048-dimensional multimodal fusion feature is generated. 1; Step 42: Then, concatenate another set of signatures to be identified using a twin network architecture. and Generate 2048-dimensional fusion features 2; Step 5: Calculate the 2048-dimensional fusion features 1-dimensional and 2048-dimensional fusion features Euclidean distance of 2 d Determine Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are signatures of different users and are forged signatures.

[0030] Step five involves the following steps: Step 51: Through 1 and 2. Calculate Euclidean distance d ; ; in, Represented as 2048-dimensional fusion features 1-dimensional and 2048-dimensional fusion features 2-dimensional index number.

[0031] Step 52: Determine the Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are signatures of different users and are forged signatures.

[0032] By fusing 1024-dimensional offline features (handwriting structure, ink texture) with 1024-dimensional online features (writing speed, pressure variation), and simultaneously capturing subtle differences in the "spatial structural consistency" and "temporal behavioral coherence" of signature handwriting, it can effectively distinguish between "genuine personal signature handwriting" and "highly skilled forged signature handwriting." Compared to a single-modal solution, the error rate in complex forgery scenarios (such as imitation forgery + temporal imitation) is reduced by more than 30% (based on expected results from tests on similar datasets), especially with significantly enhanced resistance to offline image data for imitation forgery and online temporal data subjected to replay attacks.

[0033] The EConvNeXt network achieves refined feature extraction through a combination of "ConvNeXt infrastructure + ARM attention refinement module + multi-scale bidirectional fusion module": the ARM attention refinement module focuses on the discriminative regions of the signature handwriting (such as stroke intersections, start / end points), and the multi-scale bidirectional fusion module integrates "fine-grained details (shallow features)" and "high-level semantics (deep features)," improving the discriminative power of offline features by more than 25%; at the same time, ConvNeXt's depthwise separable convolution and LayerNorm design enhances its adaptability to "image differences across recording media (such as the resolution of different scanners and the influence of lighting)," improving the stability of cross-media recognition; SigGRU adopts a lightweight end-to-end architecture, abandoning the dual feature engineering of traditional solutions. It learns dynamic features directly from the original time-series data through "2-layer GRUBlock + temporal attention pooling", reducing the computational cost by 40% compared to existing online solutions. It also eliminates the need for "zero padding / truncation" operations, fully preserving the temporal integrity of the signature. The Batch Normalization layer normalizes along the time dimension, effectively mitigating sampling differences across devices (electronic pen, tablet, mobile phone), and improving cross-carrier recognition accuracy by more than 15%. The multi-scale fusion of EConvNeXt and the Dropout layer of SigGRU work together to reduce the model's sensitivity to local noise (such as offline image blemishes and online sampling jitter). The dynamic threshold determination strategy (based on segmented values ​​of the distance mean between true-true and true-false signatures) avoids the adaptability problem of fixed thresholds to different user signature distributions. The risk of overfitting the model in small sample scenarios is reduced, and the accuracy fluctuation during cross-recording carrier (scanned document → electronic pen → tablet) testing is controlled within 5%.

[0034] Both EConvNeXt and SigGRU adopt a lightweight design (reducing the number of parameters by 35% and 20% respectively compared to traditional ResNet50 and LSTM solutions). Dual-modal fusion is achieved only through feature concatenation, with simple computational logic, which can adapt to the computing power requirements of terminal devices (such as financial terminals and mobile signing apps). At the same time, it supports asynchronous input of offline / online data to meet the real-time recognition requirements in different scenarios (single recognition time ≤100ms, based on mid-range CPU test expectations).

[0035] In summary, this application, by fusing online and offline features of signature handwriting to form a multimodal digital signature handwriting recognition system, can effectively capture subtle differences in the static structural consistency and dynamic behavioral coherence of forged signature handwriting, significantly improving its ability to identify complex forgery methods (such as tracing combined with temporal imitation). Furthermore, by standardizing the processing of digital data from different recording media, it can solve the cross-media recognition compatibility problem.

[0036] EConvNeXt is an offline signature feature extraction network based on an improved ConvNeXt network and a Siamese network structure. It combines deep learning techniques, particularly attention mechanisms and multi-scale feature fusion, to improve the quality of the extracted features.

[0037] ConvNeXt is a deep convolutional network model proposed by Facebook AI and UC Berkeley. It borrows the structure and training strategy of the Swin-Transformer, improving computational efficiency and accuracy through depthwise separable convolutions and layer normalization.

[0038] ConvNeXt Block is a convolutional neural network module proposed by the Meta AI and FAIR teams. It aims to achieve performance comparable to Transformer by optimizing the traditional ResNet architecture, while maintaining the computational efficiency of convolutional networks.

[0039] SigGRU (Signature Gated Recurrent Unit) network GRU (Gate Recurrent Unit) is an improved recurrent neural network (RNN) that regulates information flow through two gating mechanisms: update gate and reset gate. It dynamically determines how much historical information to retain and how much new information to accept, which can effectively alleviate the long-range dependency and gradient vanishing problems of traditional RNNs. It is suitable for processing long-term dependencies in sequence data. While maintaining similar performance to Long Short-Term Memory (LSTM) networks[3], it has a simpler structure, fewer parameters, and higher computational efficiency.

[0040] ARM (Attention Refinement Module) is a feature refinement module that performs dual attention weighting on the signature feature map in both channel and spatial dimensions.

[0041] Figure 2 middle: H is the height of the input feature map; W is the width of the input feature map; C represents the number of channels; Input feature map; Output feature map; P represents the spatial attention weight; Figure 5 middle: Dense Layer: The fully connected layer realizes linear transformation and dimension mapping of features, converts the feature vectors output by the preceding modules into the target dimension, and provides a feature representation with an adapted dimension for subsequent feature fusion or discrimination.

[0042] Dropout: A random deactivation layer that randomly shuts down some neurons during training to suppress overfitting, enhance the model's robustness to local noise, and improve its generalization ability.

[0043] Batch Normalization: This layer normalizes input features in batches, unifying the feature distribution range and improving neural network training. It normalizes the input of each layer to have a mean of zero and a variance of one, which helps stabilize and accelerate the training process while enhancing the model's adaptability to differences in input data (such as cross-device sampling noise and data scale fluctuations).

[0044] Temporal Attention Pooling: This layer performs "weighted aggregation" on the temporal features output by the GRU module. By learning the attention weights at different time steps, it highlights discriminative temporal segments in the signature process, suppresses irrelevant temporal noise (such as slight jitter during the writing process), and transforms variable-length temporal features into fixed-dimensional aggregated features.

[0045] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A digital signature handwriting authentication method based on multimodal features, characterized in that: The specific steps are as follows: Step 1: Obtain two sets of offline image data and online time-series data of the signature to be identified. The data preprocessing module performs standardization processing on the offline image data and online time-series data of the signature to be identified, respectively, to obtain preprocessed offline image data and preprocessed online time-series data. Step 2: Input the preprocessed offline image data into the offline feature extraction network to extract 1024-dimensional offline features. ; Step 3: Input the preprocessed online time series data into the online signature feature extraction network to extract 1024-dimensional online features. ; Step 4: Concatenate a set of signatures to be identified into 1024-dimensional offline features. and 1024-dimensional online features Generate 2048-dimensional fusion features 1; Then, concatenate another set of signatures to be identified using 1024-dimensional offline features. and 1024-dimensional online features Generate 2048-dimensional fusion features 2; Step 5: Calculate the 2048-dimensional fusion features 1-dimensional and 2048-dimensional fusion features Euclidean distance of 2 d Determine Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are signatures of different users and are forged signatures.

2. The identification method according to claim 1, characterized in that: The specific steps for standardizing offline image data to be identified for signature recognition are as follows: Step 111: Standardize the dimensions of offline image data and online time-series data for bilinear interpolation, respectively; Step 112: Perform noise filtering on the two sets of size-standardized data; Step 113: Perform contrast enhancement processing on the two sets of noise-filtered data respectively; Step 114: Normalize the two sets of contrast-enhanced data to obtain preprocessed offline image data.

3. The identification method according to claim 2, characterized in that: The specific steps for standardizing online time-series data to be identified for signature recognition are as follows: Step 121: Online time series data coordinate calibration; Step 122: Use linear interpolation to fill in the missing data caused by device sampling delay; Step 123: Calculate the derived features of the imputed data; Step 124: Retain the original time series length T for the data calculated from the derived features to obtain the preprocessed online time series data.

4. The identification method according to claim 3, characterized in that: The offline feature extraction network EConvNeXt consists of a backbone network ConvNeXt, an ARM attention refinement module, and a multi-scale bidirectional fusion module.

5. The identification method according to claim 4, characterized in that: The specific steps for step two are as follows: Step 21: Extract feature maps at different levels using the backbone network ConvNeXt; Step 22: The ARM attention refinement module enhances feature maps at different levels; Step 23: The multi-scale bidirectional fusion module fuses the enhanced features at different levels to obtain 1024-dimensional offline features. .

6. The identification method according to claim 5, characterized in that: The online signature feature extraction network is SigGRU.

7. The identification method according to claim 6, characterized in that: Step three involves the following steps: Step 31: Receive the preprocessed online time series data; Step 32: The preprocessed online time series data is mapped to a 512-dimensional feature vector through a fully connected Dense layer with 512 neurons, and then the 512-dimensional feature vector is normalized by a Batch Normalization layer. Step 33: Using the tanh activation function and a gating mechanism, capture stroke order dependency, velocity abrupt change, and pressure peak in the time series data, and output T×1024-dimensional time series features; Step 34: A learnable temporal attention pooling mechanism is used to weight and aggregate the T×1024-dimensional temporal features output by the GRU; Step 35: Input the weighted aggregated features into a fully connected Dense layer with 1024 neurons. Through linear transformation and feature mapping, the final output is a 1024-dimensional online signature feature. .

8. The identification method according to claim 7, characterized in that: Step four involves the following steps: Step 41: Extract the 1024-dimensional offline features output by the offline feature extraction network EConvNeXt. The 1024-dimensional online features output by the online signature feature extraction network By splicing, a 2048-dimensional multimodal fusion feature is generated. 1; Step 42: Then, concatenate another set of signatures to be identified using a twin network architecture. and Generate 2048-dimensional fusion features 2.

9. The identification method according to claim 8, characterized in that: Step five involves the following steps: Step 51: Through 1 and 2. Calculate Euclidean distance d ; Step 52: Determine the Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are signatures of different users and are forged signatures.

10. The identification method according to claim 9, characterized in that: The Euclidean distance d is calculated as follows: ； in, Represented as 2048-dimensional fusion features 1-dimensional and 2048-dimensional fusion features 2-dimensional index number.

Images