Signature handwriting authenticity digital recognition method based on multi-modal features

Abstract

The application discloses a signature handwriting authenticity digital recognition method based on multi-modal features, and relates to the technical field of image processing, and the specific steps are as follows: step one: obtaining two groups of offline image data and online time sequence data of signatures to be recognized, and a data preprocessing module respectively performs standardization processing on the offline image data and the online time sequence data of the signatures to be recognized, so as to obtain preprocessed offline image data and preprocessed online time sequence data; step two: inputting the preprocessed offline image data into an offline feature extraction network, and extracting and outputting 1024-dimensional offline features; step three: inputting the preprocessed online time sequence data into an online signature feature extraction network. The application combines the online features and the offline features of signature handwriting to form multi-modal signature handwriting digital recognition, can effectively capture the subtle differences of the counterfeit signature handwriting in static structure consistency and dynamic behavior coherence, and significantly improves the recognition ability for complex counterfeit means.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a signature handwriting authenticity digital recognition method based on multi-modal features. BACKGROUND

[0002] With the increasing requirements of security and convenience of identity authentication in digital finance (such as electronic contract signing, online payment verification), e-government (such as remote government service signing authentication), etc. scenes, signature handwriting digital data (image data or time series data converted from physical signature through scanning, electronic pen, touch screen, etc. Device) as a behavior characteristic carrier with uniqueness and ease of use, its application in the field of identity authentication is becoming more and more widespread. Biological feature authentication is divided into physiological features (such as fingerprints, irises) and behavioral features (such as signature handwriting, typing rhythm), among which signature handwriting digital recognition is a technology that verifies user identity by extracting and analyzing the digital features (static structure, dynamic behavior) of signature handwriting. It is widely used in key fields such as financial contract signing, legal document authentication, government service authentication, etc. because it meets the traditional signing habits.

[0003] The current signature handwriting digital recognition field is mainly divided into two major technical directions: offline signature handwriting digital recognition: the processing object is the offline digital image data of signature handwriting (such as paper signature scans, image files generated by photography), the core is to analyze the static features of handwriting in the image (such as stroke shape, ink density, overall structure); online signature handwriting digital recognition: the processing object is the dynamic time series data collected in the signature process (such as x / y coordinates, writing pressure, time stamp of electronic pen, derived speed, acceleration, etc. Digital parameters), the core is to mine the dynamic features (such as stroke order, speed change, pressure curve) reflecting writing habits in time series data.

[0004] The traditional signature handwriting recognition scheme based on single modal digital data (only offline image data or only online time series data) faces severe challenges: offline data recognition limitations: only relying on the static digital image data of signature handwriting, losing the dynamic behavior characteristics (such as starting speed, pen pressure, and pen order logic) in the writing process, it is difficult to distinguish between "real signature handwriting of the person" and "forged signature handwriting" when facing high simulation of digital image data generated by imitated signature; online data recognition limitations: only processing dynamic time series data, unable to utilize the static details of handwriting in offline image data (such as ink edge texture, stroke intersection shape, and paper texture and ink combination characteristics), vulnerable to replay attacks (stealing time series data of legitimate users and replaying) and device acquisition noise (pressure sampling deviation of different electronic pens) interference; poor cross-record carrier compatibility: existing schemes have insufficient adaptability to the record carrier (such as PNG format of scanned copies, JSON time series file of electronic pens, and local cache data of tablets) of signature handwriting digital data. The format difference of digital data of different carriers leads to a significant decrease in accuracy when recognizing across carriers, making it difficult to meet the needs of multi-device collaborative authentication scenarios.

[0005] Therefore, a signature handwriting authenticity digital recognition method based on multi-modal features is proposed to solve the above problems. SUMMARY

[0006] The purpose of the present application is to provide a signature handwriting authenticity digital recognition method based on multi-modal features to solve the technical problems raised in the background art.

[0007] To achieve the above purpose, the present application provides the following technical solutions:

[0008] The signature handwriting authenticity digital recognition method based on multi-modal features has the following specific steps:

[0009] Step 1: Obtain two sets of offline image data and online time series data of signatures to be recognized, and the data preprocessing module respectively processes the offline image data and online time series data of the signatures to be recognized to obtain preprocessed offline image data and preprocessed online time series data;

[0010] Step 2: input the preprocessed offline image data into the offline feature extraction network to extract and output 1024-dimensional offline features ;

[0011] Step 3: input the preprocessed online time series data into the online signature feature extraction network to extract and output 1024-dimensional online features ;

[0012] Step 4: splice the 1024-dimensional offline features and 1024-dimensional online features Generating 2048-dimensional fusion features 1; and another set of spliced 1024-dimensional offline features of the signature to be identified and 1024-dimensional online features Generating 2048-dimensional fusion features 2;

[0013] Step five: calculating 2048-dimensional fusion features 1 and the Euclidean distance of 2048-dimensional fusion features 2 d , judging whether the Euclidean distance d is ≤ dynamic threshold α , if the judgment is yes, the two sets of signatures to be identified are the same user signature, which is a real signature, if the judgment is no, the two sets of signatures to be identified are different user signatures, which is a fake signature.

[0014] Further, the specific steps of the offline image data standardization processing of the signature to be identified are as follows:

[0015] Step 111: standardizing the size of the offline image data and the online time series data respectively by the bilinear difference value;

[0016] Step 112: noise filtering of the two sets of size-standardized data;

[0017] Step 113: contrast enhancement processing of the two sets of noise-filtered data respectively;

[0018] Step 114: normalization processing of the two sets of contrast-enhanced data to obtain preprocessed offline image data.

[0019] Further, the specific steps of the online time series data standardization processing of the signature to be identified are as follows:

[0020] Step 121: online time series data coordinate calibration;

[0021] Step 122: filling in the missing data caused by device sampling delay by linear interpolation;

[0022] Step 123: calculating the derivative features of the filled-in data;

[0023] Step 124: preserving the original time series length T of the derivative feature calculation data to obtain preprocessed online time series data.

[0024] Further, the offline feature extraction network EConvNeXt includes a backbone network ConvNeXt, an ARM attention refining module, and a multi-scale bidirectional fusion module.

[0025] Further, the specific operation of step two is as follows:

[0026] Step 21: Extract feature maps of different levels through the backbone network ConvNeXt;

[0027] Step 22: The ARM attention refinement module enhances the feature maps of different levels;

[0028] Step 23: The multi-scale bidirectional fusion module fuses the enhanced feature maps of different levels to obtain output 1024-dimensional offline features .

[0029] Further, the online signature feature extraction network is SigGRU.

[0030] Further, step three is specifically operated as follows:

[0031] Step 31: Receive preprocessed online time series data;

[0032] Step 32: Map the preprocessed online time series data to a 512-dimensional feature vector through a fully connected Dense layer containing 512 neurons, and then normalize the 512-dimensional feature vector through a Batch Normalization layer;

[0033] Step 33: Through the tanh activation function, capture the stroke dependence, speed mutation, and pressure peak in the time series data through the gating mechanism, and output T×1024-dimensional time series features;

[0034] Step 34: The learnable temporal attention pooling mechanism weights and aggregates the T×1024-dimensional time series features output by the GRU;

[0035] Step 35: Input the weighted and aggregated features into a fully connected Dense layer containing 1024 neurons, perform linear transformation and feature mapping, and finally output 1024-dimensional online signature features .

[0036] Further, step four is specifically operated as follows:

[0037] Step 41: Concatenate the 1024-dimensional offline features output by the offline feature extraction network EConvNeXt and the 1024-dimensional online features output by the online signature feature extraction network to generate 2048-dimensional multi-modal fusion features 1;

[0038] Step 42: Through another set of twin network architecture, the spliced and of another group of signatures to be recognized generate 2048-dimensional fusion features 2.

[0039] Furthermore, step five involves the following specific steps:

[0040] Step 51: Through 1 and 2. Calculate Euclidean distance d ;

[0041] Step 52: Determine the Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are signatures of different users and are forged signatures.

[0042] Furthermore, the Euclidean distance d is calculated as follows:

[0043] ;

[0044] in, Represented as 2048-dimensional fusion features 1-dimensional and 2048-dimensional fusion features 2-dimensional index number.

[0045] Beneficial effects: This invention combines online and offline features of signature handwriting to form a multimodal digital signature handwriting recognition system. This effectively captures subtle differences in the static structural consistency and dynamic behavioral coherence of forged signature handwriting, significantly improving the ability to identify complex forgery methods (such as tracing combined with temporal imitation). Furthermore, by standardizing the processing of digital data from different recording media, it solves the problem of cross-media recognition compatibility. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0047] Figure 1 This is a flowchart of the signature handwriting authenticity digital recognition method based on multimodal features of the present invention;

[0048] Figure 2 This is a schematic diagram of the ARM Attention Refinement Module structure.

[0049] Figure 3 This is a schematic diagram of the multi-scale bidirectional fusion module structure;

[0050] Figure 4Offline feature for 1024 dimensions Generate flow chart

[0051] Figure 5 Structure diagram of online signature feature extraction network SigGRU

[0052] Figure 6 Structure diagram of bimodal signature digital feature extraction network DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0054] The present application will be further described below in combination with embodiments.

[0055] Embodiment 1, please refer to Figures 1-6 :

[0056] The signature handwriting authenticity digital recognition method based on multi-modal features in the embodiment

[0057] The specific steps are as follows:

[0058] Step one: obtain two groups of offline image data and online timing data of signatures to be recognized, and a data preprocessing module respectively performs standardization processing on the offline image data and online timing data of the signatures to be recognized, to obtain preprocessed offline image data and preprocessed online timing data

[0059] Offline image data: such as scanned copies, photographed images, format PNG / JPG, resolution ≥200dpi

[0060] Online timing data: such as JSON format data collected by electronic pens / touch screens, containing x / y coordinates, pressure p, timestamp t, sampling rate 50-200Hz

[0061] The specific steps of standardization processing of offline image data of signatures to be recognized are as follows:

[0062] Step 111: respectively standardize the size of offline image data and online timing data by bilinear difference value

[0063] Scale offline image data and online timing data to standard size by bilinear difference value

[0064] The standard size is 224 pixels × 224 pixels × 3 (width × height × channel)

[0065] Step 112: Perform noise filtering on the two sets of size-standardized data;

[0066] Two sets of images of uniform size were subjected to adaptive median filtering to remove scanning noise and paper texture.

[0067] Step 113: Perform contrast enhancement processing on the two sets of noise-filtered data respectively;

[0068] Improve the distinction between handwriting and background by limiting contrast adaptive histogram equalization (CLAHE);

[0069] Step 114: Normalize the two sets of contrast-enhanced data to obtain preprocessed offline image data.

[0070] The normalization process involves normalizing pixel values ​​to the [0,1] range to eliminate the influence of brightness differences.

[0071] The specific steps for standardizing online time-series data to be identified for signature recognition are as follows:

[0072] Step 121: Online time series data coordinate calibration;

[0073] Online time-series data coordinate calibration: Maps x / y coordinates to a [0,1] normalized coordinate system to eliminate screen size differences;

[0074] Step 122: Use linear interpolation to fill in the missing data caused by device sampling delay;

[0075] Step 123: Calculate the derived features of the imputed data;

[0076] Based on the original time series data x,y,p,t (where x is the nth...) t The coordinates of the pen tip in the horizontal direction at each sampling moment are in pixels, with the origin set to the upper left corner of the signature area; y is the coordinate of the pen tip in the horizontal direction at the sampling moment. t The coordinates of the pen tip in the vertical direction at each sampling time are in pixels, with the origin at... x The axes remain consistent; p is the first... t The pen pressure value at each sampling moment reflects the contact force between the pen tip and the writing surface. It is usually normalized and ranges from [0,1]. t is the timestamp of the sampling moment in milliseconds, recording the acquisition time of pen tip position and pressure data. The following 6-dimensional derived features are generated, and the formulas and symbols for each feature are defined as follows:

[0077] x-direction velocity y-direction velocity (in The difference in x-coordinate between adjacent time steps. The difference in y-coordinate between adjacent time steps. a time interval for adjacent time steps);

[0078] X-direction acceleration y-direction acceleration wherein x-direction velocity for adjacent time steps difference, y-direction velocity for adjacent time steps difference);

[0079] pen pressure change rate wherein difference of pressure values p for adjacent time steps;

[0080] tangent angle wherein x-coordinate difference for adjacent time steps, y-coordinate difference for adjacent time steps;

[0081] Step 124: The data calculated for the derived features is retained for the original time sequence length T to obtain preprocessed online time sequence data (no zero padding / truncation operation is performed to ensure time sequence integrity);

[0082] Step two: The preprocessed offline image data is input into the offline feature extraction network to extract 1024-dimensional offline features ;

[0083] The offline feature extraction network EConvNeXt includes a backbone network ConvNeXt, an ARM attention refining module, and a multi-scale bidirectional fusion module.

[0084] The specific operation of step two is as follows:

[0085] Step 21: Extract feature maps of different levels through the backbone network ConvNeXt;

[0086] Step 22: The ARM attention refining module enhances the feature maps of different levels;

[0087] Step 23: The multi-scale bidirectional fusion module fuses the enhanced feature maps of different levels to obtain 1024-dimensional offline features ;

[0088] The backbone network ConvNeXt includes four feature extraction stages, Stage1, Stage2, Stage3 and Stage4, which are composed of ConvNeXt Blocks in the number of 3, 3, 9 and 3 respectively, and the feature map resolutions of Stage1, Stage2, Stage3 and Stage4 are 56 pixels x 56 pixels, 28 pixels x 28 pixels, 14 pixels x 14 pixels and 7 pixels x 7 pixels respectively, and the channel numbers of Stage1, Stage2, Stage3 and Stage4 are 96 pixels, 192 pixels, 384 pixels and 768 pixels respectively.

[0089] Step 21: Specifically, different levels of feature maps of the preprocessed offline image data are extracted by Stage1, Stage2, Stage3 and Stage4.

[0090] The output ends of Stage1, Stage2, Stage3 and Stage4 are all connected in series with a group of ARM attention refining modules, which include a channel attention submodule, a spatial attention submodule and a feature refining module.

[0091] Channel attention submodule: used for global max pooling and global average pooling of different levels of feature maps input by Stage1, Stage2, Stage3 and Stage4, respectively, and the results of global max pooling and global average pooling are input into a double-layer MLP with shared weights, and the channel weight is generated by Sigmoid activation, highlighting the "stroke intersection, starting / ending stroke" discriminative channel features.

[0092] The number of hidden layer neurons of the double-layer MLP is 1 / 4 of the number of input channels.

[0093] Spatial attention submodule: used for maximum pooling and average pooling along the channel dimension of different levels of feature maps input by Stage1, Stage2, Stage3 and Stage4, the results of maximum pooling and average pooling are spliced, and the spatial weight is generated by 3x3 convolution and Sigmoid activation, focusing on the "penmanship edge, ink concentration change area" key spatial region.

[0094] The 3x3 convolution has a step of 1 and a padding of 1.

[0095] Feature refining module: input different levels of feature maps and multiply them with channel weight and spatial attention weight in turn, and output enhanced feature maps.

[0096] Step 22: The specific operation is as follows:

[0097] Step 221: Global max pooling and global average pooling are respectively performed on input feature maps of different levels, and the global max pooling and global average pooling results are input into a double-layer MLP with shared weights to generate channel weights through Sigmoid activation;

[0098] Step 222: Max pooling and average pooling are performed on input feature maps of different levels along the channel dimension, the max pooled and average pooled results are spliced, and the spliced data is subjected to 3x3 convolution and Sigmoid activation to generate spatial attention weights;

[0099] Step 223: The input feature maps of different levels are multiplied by the channel weights and the spatial attention weights in sequence, and enhanced feature maps of different levels are output;

[0100] The multi-scale bidirectional fusion module includes a unified channel submodule, a bottom-up fusion module, a top-down fusion module, and a feature aggregation fusion module.

[0101] Unified channel submodule: 4 enhanced feature maps are unified into 256 channels through 1x1 convolution;

[0102] Bottom-up fusion module: the feature maps of the current stage are up-sampled by bilinear interpolation in the order of Stage4 (7x7) -> Stage3 (14x14) -> Stage2 (28x28) -> Stage1 (56x56), so that the spatial size (widthxheight) of the up-sampled feature maps is completely consistent with the spatial size of the feature maps of the next stage (for example, the 7x7 feature maps of Stage4 are up-sampled to 14x14, which matches the size of the 14x14 feature maps of Stage3), and are added element by element with the corresponding Stage feature maps.

[0103] Top-down fusion module: the feature maps of the current stage are down-sampled by 3x3 convolution with a step of 2 in the order of Stage1 (56x56) -> Stage2 (28x28) -> Stage3 (14x14) -> Stage4 (7x7), so that the spatial size (widthxheight) of the down-sampled feature maps is completely consistent with the spatial size of the feature maps of the next stage (for example, the 56x56 feature maps of Stage1 are down-sampled to 28x28, which matches the size of the 28x28 feature maps of Stage2), and are added element by element with the corresponding Stage feature maps.

[0104] Feature aggregation fusion module: the four feature maps after bidirectional fusion are spliced, compressed to 1024 channels by 1x1 convolution, and output 1024-dimensional offline features .

[0105] The specific operation of step 23 is as follows:

[0106] Step 231: input the enhanced feature maps of different levels into the unified channel sub-module, and unify the four enhanced feature maps into 256 channels through 1x1 convolution;

[0107] Step 232: the 256-channel unified feature map is sequentially upsampled to the high-resolution feature map through bilinear interpolation according to Stage4→Stage3→Stage2→Stage1, and then is element-wise added to the corresponding Stage feature map to obtain a first fusion feature map; Figure 1

[0108] Step 233: the first fusion feature map is sequentially downsampled to the low-resolution feature map through a 3x3 convolution with a step of 2 according to Stage1→Stage2→Stage3→Stage4, and then is element-wise added to the corresponding Stage feature map to obtain a second fusion feature map; Figure 1

[0109] Step 234: the four groups of second fusion feature maps are spliced, compressed to 1024 channels through 1x1 convolution, and output 1024-dimensional offline features .

[0110] The online signature feature extraction network is SigGRU;

[0111] SigGRU includes an input layer, a dimension increasing and normalization layer, a GRU feature extraction layer, a time aggregation layer, and an output layer;

[0112] The input layer receives preprocessed online time series data; (online time series data length T, channel number 93, original features: x / y / p; 6 derived features)

[0113] The dimension increasing and normalization layer increases the preprocessed online time series data to 512 dimensions through a 512 neuron Dense layer, and stabilizes the feature distribution through a Batch Normalization layer (normalization along the time dimension) (reduces cross-device sampling differences);

[0114] The GRU feature extraction layer is composed of two GRU blocks in series, each block contains 512 hidden units, uses a tanh activation function, and captures pen order dependence, speed changes, and pressure peaks in time series data through a gating mechanism, outputting Tx1024-dimensional time series features; a Dropout layer (dropout rate 0.2) is added after each GRU block to suppress overfitting;

[0115] The time aggregation layer uses a learnable time attention pooling mechanism to weight and aggregate the Tx1024-dimensional time series features output by the GRU;

[0116] ​​The attention weight of each time step is calculated by a single-layer MLP (input 1024 dimensions, output 1 dimension) , wherein T is the original time step number of the online signature time series data;

[0117] After Softmax normalization, the weighted sum of the time series features is obtained :

[0118]

[0119] is the 1024-dimensional time series feature output by the GRU at the t-th step

[0120] Output layer: output 1024-dimensional online features through a 1024-neuron Dense layer ;

[0121] Step three: input the preprocessed online time series data into the online signature feature extraction network to extract and output 1024-dimensional online features ;

[0122] Step three is specifically implemented as follows:

[0123] Step 31: receive the preprocessed online time series data;

[0124] Step 32: map the preprocessed online time series data to a 512-dimensional feature vector through a fully connected Dense layer with 512 neurons, and then normalize the 512-dimensional feature vector through a Batch Normalization layer to stabilize the feature distribution (to alleviate the sampling difference across devices);

[0125] Step 33: through the tanh activation function, capture the stroke order dependence, speed sudden change, and pressure peak in the time series data through the gating mechanism, and output T x 1024-dimensional time series features;

[0126] Step 34: the learnable temporal attention pooling mechanism aggregates the T x 1024-dimensional time series features output by the GRU by weighting;

[0127] Step 35: input the weighted aggregated features into a fully connected Dense layer with 1024 neurons, perform linear transformation and feature mapping, and finally output 1024-dimensional online signature features .

[0128] Step four: generate 2048-dimensional fusion features 1 from a group of 1024-dimensional offline features and 1024-dimensional online features 1 of a set of signatures to be recognized; and another group of 1024-dimensional offline features and 1024-dimensional online features generate 2048-dimensional fusion features 2;

[0129] Step four is specifically as follows:

[0130] Step 41: concatenate the 1024-dimensional offline features output by the offline feature extraction network EConvNeXt and the 1024-dimensional online features output by the online signature feature extraction network to generate 2048-dimensional multi-modal fusion features 1;

[0131] Step 42: again concatenate the 2048-dimensional fusion features of another group of to-be-identified signatures through a twin network architecture and to generate 2048-dimensional fusion features 2;

[0132] Step five: calculate the Euclidean distance between the 2048-dimensional fusion features 1 and the 2048-dimensional fusion features 2 d , determine whether the Euclidean distance d is ≤ dynamic threshold α , if yes, the two groups of to-be-identified signatures are the same user's signatures, which are real signatures, if no, the two groups of to-be-identified signatures are different user's signatures, which are fake signatures.

[0133] Step five is specifically as follows:

[0134] Step 51: calculate the Euclidean distance ; d

[0135] ;

[0136] wherein, denotes the dimension index number of the 2048-dimensional fusion features 1 and the 2048-dimensional fusion features 2.

[0137] Step 52: determine whether the Euclidean distance d is ≤ dynamic threshold α , if yes, the two groups of to-be-identified signatures are the same user's signatures, which are real signatures, if no, the two groups of to-be-identified signatures are different user's signatures, which are fake signatures.

[0138] ​​By fusing 1024-dimensional offline features (handwriting structure, ink texture) and 1024-dimensional online features (writing speed, pressure change), while capturing the subtle differences in "spatial structure consistency" and "temporal behavior continuity" of signature handwriting, it can effectively distinguish between "real signature handwriting" and "highly skilled forged signature handwriting". Compared with single-mode solutions, the recognition error rate under complex forgery scenarios (such as copy forgery + temporal imitation) is reduced by more than 30% (based on similar data set testing expectations), and the resistance to copy forgery offline image data and replay attack online temporal data is greatly enhanced;

[0139] The EConvNeXt network realizes fine feature extraction through the combination design of "ConvNeXt basic architecture + ARM attention refinement module + multi-scale bidirectional fusion module": the ARM attention refinement module focuses on the discriminative areas of signature handwriting (such as stroke intersection points, starting / ending points), and the multi-scale bidirectional fusion module integrates "fine-grained details (shallow features)" and "high-level semantics (deep features)", which improves the discriminability of offline features by more than 25%; At the same time, the depth separable convolution and LayerNorm design of ConvNeXt enhances the adaptability to "cross-record carrier image differences (such as different scanner resolutions, lighting effects)", and improves the stability of cross-carrier recognition;

[0140] SigGRU adopts a lightweight end-to-end architecture, discards the double feature engineering of traditional solutions, and directly learns dynamic features from raw temporal data through "2-layer GRU Block + temporal attention pooling", reducing the computational complexity by 40% compared with existing online solutions, and without "zero padding / truncation" operation, the signature temporal integrity is fully preserved; Batch Normalization layer normalizes along the time dimension, effectively alleviating the sampling difference across devices (electronic pen, tablet, mobile phone), and improving the cross-carrier recognition accuracy by more than 15%;

[0141] The multi-scale fusion of EConvNeXt and the Dropout layer of SigGRU work together to reduce the sensitivity of the model to local noise (such as offline image stains, online sampling jitter); The dynamic threshold determination strategy (based on true-true / true-false signature pair distance mean segmentation value) avoids the adaptability problem of fixed threshold to different user signature distribution, and reduces the overfitting risk of the model in small sample scenarios, and the accuracy rate fluctuation of the model in cross-record carrier (scanned copy → electronic pen → tablet) test is controlled within 5%.

[0142] EConvNeXt and SigGRU both adopt lightweight design (parameter reduction of 35% and 20% compared to traditional ResNet50 and LSTM schemes, respectively), and the dual-modal fusion is achieved through feature concatenation, with simple calculation logic that can adapt to the computing power requirements of terminal devices (such as financial terminals and mobile signing APPs). It also supports asynchronous input of offline / on-line data, meeting real-time recognition needs in different scenarios (single recognition time ≤100ms, based on mid-end CPU testing expectations).

[0143] In summary, the present application combines online and offline features of signature handwriting to form a multi-modal signature handwriting digital recognition, which can effectively capture the subtle differences in static structural consistency and dynamic behavior coherence of forged signature handwriting, significantly improving the recognition ability of complex forgery methods (such as copying and timing simulation). At the same time, by standardizing the digital data of different recording carriers, the cross-carrier recognition compatibility problem can be solved.

[0144] EConvNeXt: An offline signature feature extraction network based on an improved ConvNeXt network and a twin network structure. It combines deep learning techniques, especially attention mechanisms and multi-scale feature fusion, to improve the quality of extracted features.

[0145] ConvNeXt is a deep convolutional network model proposed by Facebook AI and UC Berkeley. It draws on the structure and training strategies of Swin-Transformer, improving computational efficiency and accuracy through depthwise separable convolution and layer normalization.

[0146] ConvNeXt Block is a convolutional neural network module proposed by Meta AI and FAIR team, aiming to achieve comparable performance to Transformer by optimizing the traditional ResNet architecture, while maintaining the computational efficiency of convolutional networks.

[0147] SigGRU (Signature Gated Recurrent Unit, Signature Gated Recurrent Unit Network)

[0148] GRU (Gate Recurrent Unit) is an improved recurrent neural network (RNN) that uses two gating mechanisms, update gate and reset gate, to control information flow and dynamically determine how much historical information to retain and how much new information to accept. It effectively alleviates the long-range dependence and gradient vanishing problems of traditional RNNs, making it suitable for handling long-term dependencies in sequence data. While maintaining similar performance to Long Short-Term Memory networks (LSTM[3]), it has a simpler structure, fewer parameters, and higher computational efficiency.

[0149] ARM (Attention Refinement Module) is a feature refinement module for channel and spatial dimension dual attention weighting of signature feature maps.

[0150] Figure 2 In:

[0151] H is the height of the input feature map;

[0152] W is the width of the input feature map;

[0153] C is the number of channels;

[0154] is the input feature map;

[0155] is the output feature map;

[0156] P is the spatial attention weight;

[0157] Figure 5 In:

[0158] Dense Layer: Fully connected layer, realizing linear transformation and dimension mapping of features, converting feature vectors output by previous modules into target dimensions, providing adaptive dimensional feature representation for subsequent feature fusion or discrimination.

[0159] Dropout: Random inactivation layer, by randomly turning off part of the neurons in the training process, inhibiting model overfitting, enhancing model robustness to local noise, and improving model generalization ability.

[0160] Batch Normalization: Batch normalization layer, normalizing input features by batch, unifying feature distribution range, used to improve neural network training. It normalizes the input of each layer to have a mean of zero and a variance of one, which helps to stabilize and speed up the training process, and enhances the model's adaptability to input data differences (such as cross-device sampling noise and data scale fluctuations).

[0161] Temporal Attention Pooling: Temporal attention pooling layer, "weighted aggregation" of time sequence features output by GRU module, by learning attention weights of different time steps, highlighting discriminative time sequence fragments in the signature process, suppressing irrelevant time sequence noise (such as slight shaking in the writing process), and converting variable-length time sequence features into fixed-dimensional aggregated features.

[0162] The preferred embodiments of the application disclosed above are only to facilitate the elucidation of the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments. Obviously, many modifications and variations can be made in light of the teachings above. The description is chosen and described in order to provide the best illustration of the application and its practical application to those skilled in the art and to enable those skilled in the art to best utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. A signature handwriting authenticity digital recognition method based on multi-modal features, characterized in that: The specific steps are as follows: Step one: obtain two groups of offline image data and online timing data of the signature to be identified, and the data preprocessing module respectively processes the offline image data and online timing data of the signature to be identified to obtain preprocessed offline image data and preprocessed online timing data; Step two: input the preprocessed offline image data into the offline feature extraction network to extract and output 1024-dimensional offline features ; Step three: input the pre-processed online time series data into the online signature feature extraction network to extract 1024-dimensional online features ; Step four: concatenate 1024 dimensional offline features of a set of signatures to be recognized and 1024 dimensional online features to generate 2048 dimensional fused features ; Another set of to-be-recognized signature spliced 1024-dimensional offline features and 1024-dimensional online features Generate 2048-dimensional fusion features ; Step 5: Calculate the 2048-dimensional fusion features Features fused with 2048 dimensions Euclidean distance d Determine Euclidean distance d Is it ≤ dynamic threshold? α If the determination is yes, the two sets of signatures to be identified are the signatures of the same user and are genuine signatures; if the determination is no, the two sets of signatures to be identified are the signatures of different users and are forged signatures. The offline feature extraction network EConvNeXt includes a backbone network ConvNeXt, an ARM attention refining module, and a multi-scale bidirectional fusion module; The specific operation of step two is as follows: Step 21: extract feature maps of different levels through the backbone network ConvNeXt; Step 22: the ARM attention refining module enhances the feature maps of different levels; Step 23: The multi-scale bidirectional fusion module fuses the enhanced features of different levels to obtain an output of 1024-dimensional offline features ; The online signature feature extraction network is SigGRU; The specific operation of step three is as follows: Step 31: receive the preprocessed online timing data; Step 32: map the preprocessed online timing data to a 512-dimensional feature vector through a fully connected Dense layer containing 512 neurons, and then normalize the 512-dimensional feature vector through a Batch Normalization layer; Step 33: through the tanh activation function, the pen order dependence, speed mutation, and pressure peak in the timing data are captured through the gating mechanism, and a T×1024-dimensional timing feature is output; Step 34: the learnable temporal attention pooling mechanism weights and aggregates the T×1024-dimensional timing feature output by GRU; Step 35: The weighted aggregated features are input into a fully connected Dense layer with 1024 neurons, through linear transformation and feature mapping, finally outputting 1024-dimensional online signature features .

2. The identification method of claim 1, wherein: The specific steps of the offline image data standardization processing of the signature to be identified are as follows: Step 111: respectively standardize the size of the offline image data and online timing data through bilinear difference; Step 112: filter noise from the two groups of size-standardized data; Step 113: respectively perform contrast enhancement processing on the two groups of noise-filtered data; Step 114: normalize the two groups of contrast-enhanced data to obtain preprocessed offline image data.

3. The identification method of claim 2, wherein: The specific steps of the online timing data standardization processing of the signature to be identified are as follows: Step 121: online timing data coordinate calibration; Step 122: use linear interpolation to fill in missing data caused by device sampling delay; Step 123: calculate the derived features of the filled data; Step 124: retain the original timing length T of the derived feature calculation data to obtain preprocessed online timing data.

4. The identification method of claim 3, wherein: The specific steps of step four are as follows: Step 41: 1024-dimensional offline features output by the offline feature extraction network EConvNeXt with 1024-dimensional online features output by the online signature feature extraction network concatenated to generate 2048-dimensional multi-modal fusion features ; Step 42: another set of to-be-recognized signature splicing is generated again through a twin network architecture and 2048-dimensional fusion features are generated .

5. The identification method of claim 4, wherein: The specific steps of step five are as follows: Step 51 : by and calculating the Euclidean distance d ; Step 52: judging the Euclidean distance d whether ≤ dynamic threshold α If the judgment is yes, the two groups of to-be-identified signatures are the same user signature and are real signatures, and if the judgment is no, the two groups of to-be-identified signatures are different user signatures and are counterfeit signatures.

6. The identification method of claim 5, wherein: The Euclidean distance d is calculated as follows: ； wherein, is represented as a 2048-dimensional fusion feature is represented as a 2048-dimensional fusion feature dimension index number.

Images