Offline handwriting reconstruction and signature verification method and system based on transformer architecture

Abstract

The application provides a kind of offline handwriting reconstruction and signature verification method and system based on the architecture of Transformer, comprising: collecting offline signature image and online handwriting data, and preprocessing offline signature image, and converting online handwriting data to obtain handwriting sequence;Construct handwriting reconstruction model, train handwriting reconstruction model according to the offline signature data after preprocessing and handwriting sequence, obtain corresponding handwriting coordinate point sequence;Based on the handwriting coordinate point sequence reconstructed by handwriting reconstruction model and the signature verification data set collected, the input test material and sample pair are processed, and feature extraction, comparison and optimization are carried out to accurately predict the authenticity information of test material.The handwriting reconstruction model recovers handwriting sequence from offline signature image, provides dynamic writing information for signature verification model, significantly improves the recognition ability of high-level fake signature, and achieves the effect of improving signature verification accuracy and reliability.

Description

Technical Field

[0001] This invention relates to the field of handwriting reconstruction technology, specifically to an offline handwriting reconstruction and signature verification method and system based on the Transformer architecture. Background Technology

[0002] Currently, there are many types of signature verification methods. However, for offline signature images, most existing methods treat them as static images and ignore the dynamic information contained in the signature itself. Some methods require the input of handwriting information collected during the image acquisition process when inputting the image, which is very demanding for signature verification tasks and makes it difficult to meet the conditions.

[0003] Patent document CN117315677A discloses a handwriting identification method and system, including: S1: receiving the signature handwriting to be examined and M original sample signature handwritings provided; S2: constructing a handwriting identification model and completing training; the handwriting identification model includes a first handwriting identification model and a second handwriting identification model; the first handwriting identification model is constructed based on the original training samples, and the second handwriting identification model is constructed after enhancing the original training samples through a spatial transformation network; S3: inputting the signature handwriting to be examined and the M original sample signature handwritings into the trained handwriting identification model, and then outputting the identification result.

[0004] However, patent document CN117315677A requires the input of an online handwriting sequence of the signature during the signature verification process. That is, the handwriting sequence of the signature needs to be provided before judging the authenticity of the signature. However, in actual verification, online handwriting sequences are usually difficult to obtain.

[0005] The paper "Research on Offline Chinese Signature Authentication Method Based on Transformer" by Zhang Yan, Hangzhou Dianzi University, May 1, 2023, discloses an offline Chinese signature authentication technique based on an improved Transformer network. This technique divides the signature image into equally sized image blocks, allowing the subsequent network to focus on the stroke details of the signature. Furthermore, by introducing depthwise separable convolutions, the Transformer network can focus not only on global features of the signature image but also on local features, thereby enhancing the network's ability to extract effective information.

[0006] However, this document has the following shortcomings:

[0007] 1. Static image analysis is limited: It relies solely on the static visual features of offline signature images (such as stroke shape and spatial layout), failing to capture dynamic information about the signature generation process (such as stroke order and pen lifting and lowering states). Skilled forged signatures may appear visually similar, but their dynamic features differ significantly.

[0008] 2. Insufficient identification of high-quality forgeries: Purely static analysis methods are unable to distinguish skillfully forged signatures (such as copied signatures) that are "visually similar but have different writing dynamics", leading to misjudgment.

[0009] 3. Lack of multimodal feature fusion: Only the feature extraction of the image itself is optimized, without cross-modal fusion with dynamic stroke sequence information, which limits the model's comprehensive understanding of the uniqueness of the signature. Summary of the Invention

[0010] In view of the shortcomings of the prior art, the purpose of this invention is to provide an offline handwriting reconstruction and signature verification method and system based on the Transformer architecture.

[0011] An offline handwriting reconstruction and signature verification method based on the Transformer architecture provided by the present invention includes:

[0012] Step S1: Collect offline signature images and online handwriting data, preprocess the offline signature images, and convert the online handwriting data to obtain a handwriting sequence;

[0013] Step S2: Construct a handwriting reconstruction model. Train the handwriting reconstruction model based on the preprocessed offline signature data and handwriting sequence to obtain the corresponding handwriting coordinate point sequence.

[0014] Step S3: Based on the collected signature verification dataset and the handwriting coordinate point sequence reconstructed by the handwriting reconstruction model, process the input evidence and sample pairs, and perform feature extraction, comparison and optimization to accurately predict the authenticity of the evidence.

[0015] Preferably, the preprocessing includes grayscale conversion, Gaussian denoising, Otsu's adaptive binarization, and size normalization;

[0016] The conversion process includes converting online handwriting data into handwriting sequences (X). 1-T ,Y 1-T ,S 1-T And reduce the sequence length by resampling.

[0017] Preferably, step S2 includes:

[0018] Step S2.1: Deep learning methods are used to perform multi-layer convolution, pooling, and non-linear activation operations on the preprocessed offline signature image to extract local visual features and texture information in the image; the local visual features include the shape, thickness, edges, and connection relationships between strokes.

[0019] Step S2.2: Based on the local visual features and texture information and the handwriting sequence, perform handwriting point prediction based on the Transformer decoder to obtain the probability distribution of handwriting points at the next time step;

[0020] Step S2.3: The handwriting points at the next time step are made consistent with the actual handwriting points by adjusting the model parameters. The loss function is defined as follows:

[0021] Loss = L x +L y +L s

[0022] L x =CE(P x ,T x )

[0023] L y =CE(P y ,T y )

[0024] L s =CE(P s ,T s )

[0025] Where Lx is the cross-entropy loss of the x-coordinate, Px, Py, and Ps are the predicted probabilities of the coordinate and the state, respectively, Tx, Ty, and Ts are the true values ​​of the coordinate and the state, respectively, and CE is the cross-entropy loss;

[0026] The model is trained using the gradient descent algorithm, and the model parameters are optimized so that the handwriting points predicted by the model are consistent with the real situation, until the loss of the model on the validation set reaches the preset target or tends to converge.

[0027] Preferably, the Transformer-based decoder is composed of multiple identical decoder layers stacked together;

[0028] Each decoder layer includes the following sub-layers: a masked multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward neural network.

[0029] Preferably, the masked multi-head self-attention layer operates on the embedding of the target sequence to capture the internal dependencies between the generated handwriting points;

[0030] The multi-head cross-attention layer uses the output of the self-attention layer as a query, and the image feature representation output in step S2.1 is used as the key and value.

[0031] The feedforward neural network performs a further nonlinear transformation on the output of the cross-attention layer;

[0032] The feature vector obtained after processing by the last decoder layer is used to predict the probability distribution of the x-coordinate, y-coordinate, and stroke state s of the next handwriting point through three independent linear layers.

[0033] Preferably, the embedding operation of the target sequence includes embedding the input handwriting sequence X separately. 1-T Y 1-T S 1-T Each point is mapped to a vector through an embedding layer, and the embedding vectors are concatenated to form a comprehensive embedding representation of the current point. Temporal position encoding is added to the embedding sequence to indicate its order in the sequence.

[0034] Preferably, step S3 includes:

[0035] Step S3.1: From the handwriting coordinate point sequence, extract image blocks sequentially with the sequence points as the center, perform convolution processing on the image block sequence to obtain its vector representation, and add temporal position encoding to indicate the order of the image blocks in the image block sequence, thereby obtaining the comprehensive vector representation of the image block sequence;

[0036] Step S3.2: The model aggregates information from the image patch sequence through the Transformer network, generates a global representation vector for each signature image, and calculates the global distance and local distance;

[0037] Step S3.3: During the training process, a contrastive learning strategy is used for optimization. When the evidence and sample are real signatures, the model parameters are adjusted to simultaneously reduce the global distance and local distance obtained in step S3.2, and vice versa.

[0038] Among them, "evidence" refers to the signature being queried, and "sample" refers to the signature being referenced.

[0039] Preferably, the calculation of the global distance includes calculating the Euclidean distance between the global representation vectors of the evidence image and the sample image. The smaller the distance, the more similar the two signatures are in overall structure, layout and writing style; the larger the distance, the greater the difference.

[0040] The calculation of the local distance includes: using a multi-head cross-attention mechanism to intelligently align the local features of the sample and the specimen image block sequence; for each local feature of the specimen image, calculating the distance between it and all local features of the sample image, and finding the one with the smallest distance; similarly, for each local feature of the sample image, calculating the distance between it and all local features of the specimen image, and finding the one with the smallest distance, thus forming a bidirectional minimum distance set; and taking the largest bidirectional minimum distance in the set as the local distance.

[0041] Preferably, it further includes: using the trained handwriting reconstruction model and signature verification model to verify the authenticity of the reference signature and query signature.

[0042] An offline handwriting reconstruction and signature verification system based on the Transformer architecture provided by the present invention includes:

[0043] Module M1: Collects offline signature images and online handwriting data, preprocesses the offline signature images, and converts the online handwriting data to obtain a handwriting sequence;

[0044] Module M2: Construct a handwriting reconstruction model. Train the handwriting reconstruction model based on the preprocessed offline signature data and handwriting sequence to obtain the corresponding handwriting coordinate point sequence.

[0045] Module M3: Based on the collected signature verification dataset and the handwriting coordinate point sequence reconstructed by the handwriting reconstruction model, it processes the input evidence and sample pairs, and performs feature extraction, comparison and optimization to accurately predict the authenticity of the evidence.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. This invention recovers handwriting sequences from offline signature images through a handwriting reconstruction model, providing dynamic writing information (such as stroke order and pen lifting and lowering states) for the signature verification model, significantly improving the ability to identify high-level forged signatures, and achieving the effect of improving the accuracy and reliability of signature verification.

[0048] 2. This invention combines handwriting sequence and static image information, and enhances the model's ability to capture the individual specificity of signatures through multimodal feature fusion and comparative learning optimization, thereby improving the accuracy and discriminativeness of signature verification.

[0049] 3. This invention adopts a Transformer-based multimodal signature verification model, which dynamically fuses static image features and reconstructed handwriting sequence features through a cross-attention mechanism, giving full play to the complementary advantages of multimodal information and achieving the effect of improving model performance and robustness.

[0050] 4. This invention only requires offline signature images as input and recovers the handwriting sequence through technical means, reducing the dependence on dedicated online acquisition equipment and achieving the effect of expanding the application scenarios and practical value of high-precision signature verification technology. Attached Figure Description

[0051] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0052] Figure 1 This is a schematic diagram of the working method of the present invention. Detailed Implementation

[0053] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0054] This invention proposes to extract the writing order from static offline signature images using a handwriting reconstruction method, and to extract global and local features of the offline signature images based on the writing order, thereby achieving interpretable signature verification.

[0055] Example 1

[0056] According to the present invention, an offline handwriting reconstruction and signature verification method based on the Transformer architecture is provided, such as... Figure 1 As shown, it includes:

[0057] Step S1: Acquire offline signature images and online handwriting data, preprocess the offline signature images, and convert the online handwriting data to obtain a handwriting sequence. The preprocessing includes grayscale conversion, Gaussian denoising, Otsu's adaptive binarization, and size normalization. The online handwriting data is converted into a handwriting sequence (X). 1-T ,Y 1-T ,S 1-T This step aims to construct a training set containing offline signature images and paired sensor handwriting data, and to collect real / forged signature pairs as a signature verification dataset. A standardized data resource library is established through image cleaning, coordinate annotation (x, y), and stroke state marking (lifting / falling strokes).

[0058] Step S2: Construct a handwriting reconstruction model. Train the handwriting reconstruction model based on the preprocessed offline signature data and handwriting sequence to obtain the corresponding handwriting coordinate point sequence. Step S2 includes:

[0059] Step S2.1: Deep learning methods are used to perform multi-layer convolution, pooling, and non-linear activation operations on the preprocessed offline signature image to extract rich, multi-scale local visual features and texture information from the image. The local visual features include key visual cues such as the shape, thickness, edges, and connection relationships between strokes.

[0060] Step S2.2: Based on the local visual features, texture information, and handwriting sequence, predict handwriting points using a Transformer decoder to obtain the probability distribution of handwriting points at the next time step. The Transformer decoder consists of multiple identical decoder layers stacked together. Each decoder layer includes the following sub-layers: a masked multi-head self-attention layer, a multi-head cross-attention layer, and a feed-forward network.

[0061] The masked multi-head self-attention layer operates on the embedding of the target sequence to capture the internal dependencies between generated handwriting points. The causal mask ensures that during training, when predicting the current point, the model only focuses on historically generated points in the sequence, preventing information leakage. The multi-head cross-attention layer uses the output of the self-attention layer as a query, and the image feature representation output in step S2.1 as the key and value. This mechanism allows the decoder to dynamically select and weight the most important visual information from the image features when generating each handwriting point. The feedforward neural network further performs a non-linear transformation on the output of the cross-attention layer. The feature vector obtained after processing by the last decoder layer is then used by three independent linear layers (fully connected layers) to predict the x-coordinate, y-coordinate, and probability distribution (logits) of the stroke state s of the next handwriting point.

[0062] The embedding operation of the target sequence includes embedding the input handwriting sequence separately, X 1-T Y 1-T S 1-T Each point is mapped to a vector through an embedding layer. These embedding vectors are then concatenated to form a comprehensive embedding representation of the current point. Temporal position encoding is added to the embedding sequence to indicate its order within the sequence.

[0063] Step S2.3: The handwriting points at the next time step are aligned with the actual handwriting points by adjusting the model parameters. The loss function is defined as follows:

[0064] Loss = L x +L y +L s

[0065] L x =CE(P x ,T x )

[0066] L y =CE(P y ,T y )

[0067] L s =CE(P s ,T s )

[0068] Where Lx is the cross-entropy loss of the x-coordinate, Px, Py, and Ps are the predicted probabilities of the coordinate and state, respectively, Tx, Ty, and Ts are the true values ​​of the coordinate and state, and CE is the cross-entropy loss. The model is trained using the gradient descent algorithm, and the model parameters are optimized so that the handwriting points predicted by the model are consistent with the real situation, until the loss of the model on the validation set reaches the preset target or tends to converge.

[0069] Furthermore, the model's generation pattern is used to convert new offline signature images into handwriting sequences via an autoregressive generation method. First, the offline image is processed to obtain image feature representations via step S2.1. The decoder uses a special start symbol ( <bos>The initial input is given by the image. Then, at each time step t, the decoder predicts the coordinates (Xt, Yt) and state St of the current t-th handwriting point based on the image features and its own prediction output from time 0 to t-1. The predicted point (Xt, Yt, St) is then used as one of the inputs to the decoder at time t+1. This process iterates until the model predicts a specific end symbol (...). <eos>The sequence length can be determined by either the input image or the input image reaching a preset maximum sequence length. Based on this, for any input offline signature image, a reconstructed handwriting coordinate point sequence (X1-T, Y1-T, S1-T) can be obtained.

[0070] Step S3: Based on the collected signature verification dataset and the handwriting coordinate point sequence reconstructed by the handwriting reconstruction model, process the input evidence (query signature) and sample (reference signature) pairs, and perform feature extraction, comparison, and optimization to accurately predict the authenticity of the evidence. Step S3 includes:

[0071] Step S3.1: From the handwriting coordinate point sequence, extract image patches sequentially centered on the sequence points. Perform convolution processing on the image patch sequence to obtain its vector representation, and add temporal position encoding to indicate the order of the image patches in the image patch sequence. This yields the comprehensive vector representation of the image patch sequence.

[0072] Step S3.2: The model aggregates information from the image patch sequence through the Transformer network, generates a global representation vector for each signed image, and calculates the global distance and local distance.

[0073] Global distance calculation:

[0074] Information Aggregation and Global Representation Generation: For the image patch sequence (step S3.1), information aggregation is performed through a multi-layer Transformer network. This network analyzes the relationships between the various image patches in the sequence and integrates and extracts key information. After processing by the Transformer network, each signature image generates a compact global representation vector. This vector can be understood as a unique numerical description of the entire signature image at a macroscopic level.

[0075] Global distance calculation: After obtaining the global representation vectors of the evidence image and the sample image, the global distance between these two vectors is measured by calculating the Euclidean distance between them. The smaller the distance, the more similar the two signatures are in overall structure, layout, and writing style; the larger the distance, the greater the difference. This distance value is the global distance.

[0076] Local distance calculation:

[0077] Local Feature Extraction and Alignment: For the image patch sequences of the sample and the evidence, a multi-head cross-attention mechanism is used to intelligently align the local features of the evidence and the sample. This means that for each local feature of the evidence image, the model dynamically searches for and focuses on the most relevant region among all local features of the sample image, and vice versa, thus establishing a bidirectional matching relationship.

[0078] Bidirectional minimum distance calculation: For each local feature in the sample image, the model calculates its distance to all local features in the sample image and finds the one with the smallest distance. Similarly, for each local feature in the sample image, the model also calculates its distance to all local features in the sample image and finds the one with the smallest distance. This process forms a bidirectional set of "minimum distances".

[0079] Local distance summary: Based on the above set of minimum distances, the largest bidirectional minimum distance in the set is selected. This distance reflects the differences between the two signatures in terms of detailed strokes and local structure.

[0080] Step S3.3: During the training process, a contrastive learning strategy is used for optimization. When the evidence and sample are real signatures, the model parameters are adjusted to simultaneously decrease the global distance and local distance obtained in step S3.2, and vice versa. Through this method, the model not only learns to judge whether the signatures are similar, but also learns to amplify the similarity between real signatures, while reducing or amplifying the difference between forged signatures and real signatures, thereby improving the accuracy of signature verification.

[0081] Furthermore, this invention utilizes a trained handwriting reconstruction model and signature verification model to verify the authenticity of reference signatures and query signatures. First, the input image is preprocessed and dynamic handwriting information is reconstructed. Then, the signature verification model extracts global and local features, calculates distances, and outputs a authenticity judgment based on a preset threshold. By recording local matching information (step S3.2), the similarity between image patches of the sample and the evidence is visualized, enhancing the interpretability of the results. Finally, the conclusion, similarity score, and visualization results are output.

[0082] Example 2

[0083] The present invention also provides an offline handwriting reconstruction and signature verification system based on the Transformer architecture. The offline handwriting reconstruction and signature verification system based on the Transformer architecture can be implemented by executing the process steps of the offline handwriting reconstruction and signature verification method based on the Transformer architecture. That is, those skilled in the art can understand the offline handwriting reconstruction and signature verification method based on the Transformer architecture as a preferred embodiment of the offline handwriting reconstruction and signature verification system based on the Transformer architecture.

[0084] An offline handwriting reconstruction and signature verification system based on the Transformer architecture provided by the present invention includes:

[0085] Module M1: Acquires offline signature images and online handwriting data, preprocesses the offline signature images, and converts the online handwriting data to obtain a handwriting sequence. The preprocessing includes grayscale conversion, Gaussian denoising, Otsu's adaptive binarization, and size normalization. The conversion process includes converting the online handwriting data into a handwriting sequence (X). 1-T ,Y 1-T ,S 1-T And reduce the sequence length by resampling.

[0086] Module M2: Constructs a handwriting reconstruction model. The model is trained based on preprocessed offline signature data and handwriting sequences to obtain the corresponding handwriting coordinate point sequence. Module M2 includes: Module M2.1: Using deep learning methods, multi-layer convolution, pooling, and non-linear activation operations are performed on the preprocessed offline signature image to extract local visual features and texture information. The local visual features include the shape, thickness, edges, and connection relationships between strokes. Module M2.2: Based on the local visual features, texture information, and handwriting sequence, handwriting point prediction is performed using a Transformer decoder to obtain the probability distribution of handwriting points at the next time step. Module M2.3: The handwriting points at the next time step are aligned with the actual handwriting points by adjusting the model parameters. The loss function is defined as follows:

[0087] Loss = L x +L y +L s

[0088] L x =CE(P x ,T x )

[0089] L y =CE(P y ,T y )

[0090] L s =CE(P s ,T s )

[0091] Where Lx is the cross-entropy loss of the x-coordinate, Px, Py, and Ps are the predicted probabilities of the coordinate and state, respectively, Tx, Ty, and Ts are the true values ​​of the coordinate and state, and CE is the cross-entropy loss. The model is trained using the gradient descent algorithm, and the model parameters are optimized so that the handwriting points predicted by the model are consistent with the real situation, until the loss of the model on the validation set reaches the preset target or tends to converge.

[0092] The Transformer-based decoder consists of multiple identical decoder layers stacked together. Each decoder layer includes the following sub-layers: a masked multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward neural network. The masked multi-head self-attention layer operates on the embedding of the target sequence to capture the internal dependencies between generated handwriting points. The multi-head cross-attention layer uses the output of the self-attention layer as a query, and the image feature representation output by module M2.1 as the key and value. The feedforward neural network performs a further nonlinear transformation on the output of the cross-attention layer. The feature vector obtained after processing by the last decoder layer is used by three independent linear layers to predict the probability distribution of the x-coordinate, y-coordinate, and stroke state s of the next handwriting point. The embedding operation of the target sequence includes embedding the input handwriting sequence X. 1-T Y 1-T S 1-T Each point is mapped to a vector through an embedding layer, and the embedding vectors are concatenated to form a comprehensive embedding representation of the current point. Temporal position encoding is added to the embedding sequence to indicate its order in the sequence.

[0093] Module M3: Based on the collected signature verification dataset and the handwriting coordinate point sequence reconstructed by the handwriting reconstruction model, it processes the input evidence and sample pairs, and performs feature extraction, comparison, and optimization to accurately predict the authenticity of the evidence. Module M3 includes: Module M3.1: From the handwriting coordinate point sequence, image blocks are extracted sequentially with the sequence points as centers. The image block sequence is convolved to obtain its vector representation, and temporal position encoding is added to indicate the order of the image blocks in the image block sequence, thus obtaining a comprehensive vector representation of the image block sequence. Module M3.2: The model aggregates information from the image block sequence through a Transformer network, generates a global representation vector for each signature image, and calculates global and local distances. The calculation of the global distance includes calculating the Euclidean distance between the global representation vectors of the evidence image and the sample image. The smaller the distance, the more similar the two signatures are in overall structure, layout, and writing habits; the larger the distance, the greater the difference. The calculation of the local distance includes: for the image patch sequences of the sample and the evidence, a multi-head cross-attention mechanism is used to intelligently align the local features of the evidence and the sample. For each local feature of the evidence image, the distance between it and all local features of the sample image is calculated, and the smallest distance is found. Similarly, for each local feature of the sample image, the distance between it and all local features of the evidence image is calculated, and the smallest distance is found, thus forming a bidirectional minimum distance set. The largest bidirectional minimum distance in the set is taken as the local distance.

[0094] Module M3.3: During training, a contrastive learning strategy is used for optimization. When both the query signature and the reference signature are genuine, the model parameters are adjusted to simultaneously decrease both the global and local distances obtained through Module M3.2, and vice versa. Here, the query signature refers to the query signature, and the reference signature refers to the reference signature.

[0095] The present invention also includes: using the trained handwriting reconstruction model and signature verification model to verify the authenticity of the reference signature and query signature.

[0096] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units provided by this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0097] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.< / eos> < / bos>

Claims

1. An offline handwriting reconstruction and signature verification method based on the Transformer architecture, characterized in that, include: Step S1: Collect offline signature images and online handwriting data, preprocess the offline signature images, and convert the online handwriting data to obtain a handwriting sequence; Step S2: Construct a handwriting reconstruction model. Train the handwriting reconstruction model based on the preprocessed offline signature data and handwriting sequence to obtain the corresponding handwriting coordinate point sequence. Step S3: Based on the collected signature verification dataset and the handwriting coordinate point sequence reconstructed by the handwriting reconstruction model, process the input evidence and sample pairs, and perform feature extraction, comparison and optimization to accurately predict the authenticity of the evidence. Step S2 includes: Step S2.1: Deep learning methods are used to perform multi-layer convolution, pooling, and non-linear activation operations on the preprocessed offline signature image to extract local visual features and texture information in the image; the local visual features include the shape, thickness, edges, and connection relationships between strokes. Step S2.2: Based on the local visual features and texture information and the handwriting sequence, perform handwriting point prediction based on the Transformer decoder to obtain the probability distribution of handwriting points at the next time step; Step S2.3: The handwriting points at the next time step are made consistent with the actual handwriting points by adjusting the model parameters. The loss function is defined as follows: Among them, Lx, and Let Px, Py, and Ps represent the cross-entropy loss of the x-coordinate, y-coordinate, and state, respectively; let Tx, Ty, and Ts represent the predicted probabilities of the x-coordinate, y-coordinate, and state, respectively; and let CE represent the cross-entropy loss. The model is trained using the gradient descent algorithm, and the model parameters are optimized so that the handwriting points predicted by the model can be consistent with the real situation, until the loss of the model on the validation set reaches the preset target or tends to converge. The Transformer-based decoder consists of multiple identical decoder layers stacked together. Each decoder layer includes the following sub-layers: a masked multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward neural network; The masked multi-head self-attention layer operates on the embedding of the target sequence to capture the internal dependencies between the generated handwriting points; The multi-head cross-attention layer uses the output of the self-attention layer as a query, and the image feature representation output in step S2.1 is used as the key and value. The feedforward neural network performs a further nonlinear transformation on the output of the cross-attention layer; The feature vector obtained after processing by the last decoder layer is used to predict the probability distribution of the x-coordinate, y-coordinate, and stroke state s of the next handwriting point through three independent linear layers.

2. The offline handwriting reconstruction and signature verification method based on the Transformer architecture according to claim 1, characterized in that, The preprocessing includes grayscale conversion, Gaussian denoising, Otsu's adaptive binarization, and size normalization. The conversion process includes converting online handwriting data into handwriting sequences (X). 1-T , Y 1-T , S 1-T And reduce the sequence length by resampling.

3. The offline handwriting reconstruction and signature verification method based on the Transformer architecture according to claim 1, characterized in that, The embedding operation of the target sequence includes embedding the input handwriting sequence X separately. 1-T Y 1-T S 1-T Each point is mapped to a vector through an embedding layer, and the embedding vectors are concatenated to form a comprehensive embedding representation of the current point. Temporal position encoding is added to the embedding sequence to indicate its order in the sequence.

4. The offline handwriting reconstruction and signature verification method based on the Transformer architecture according to claim 1, characterized in that, Step S3 includes: Step S3.1: From the handwriting coordinate point sequence, extract image blocks sequentially with the sequence points as the center, perform convolution processing on the image block sequence to obtain its vector representation, and add temporal position encoding to indicate the order of the image blocks in the image block sequence, thereby obtaining the comprehensive vector representation of the image block sequence; Step S3.2: The model aggregates information from the image patch sequence through the Transformer network, generates a global representation vector for each signature image, and calculates the global distance and local distance; Step S3.3: During the training process, a contrastive learning strategy is used for optimization. When the evidence and sample are real signatures, the model parameters are adjusted to simultaneously reduce the global distance and local distance obtained in step S3.2, and vice versa. Among them, "evidence" refers to the signature being queried, and "sample" refers to the signature being referenced.

5. The offline handwriting reconstruction and signature verification method based on the Transformer architecture according to claim 4, characterized in that, The calculation of the global distance includes calculating the Euclidean distance between the global representation vectors of the evidence image and the sample image. The smaller the distance, the more similar the two signatures are in terms of overall structure, layout and writing style. The greater the distance, the greater the difference. The calculation of the local distance includes: using a multi-head cross-attention mechanism to intelligently align the local features of the sample and the evidence in the image block sequence of the sample and the evidence; For each local feature in the sample image, calculate its distance to all local features in the sample image and find the one with the smallest distance. Similarly, for each local feature in the sample image, calculate its distance to all local features in the sample image and find the one with the smallest distance, thus forming a bidirectional minimum distance set. The largest bidirectional minimum distance in the set is taken as the local distance.

6. The offline handwriting reconstruction and signature verification method based on the Transformer architecture according to claim 1, characterized in that, It also includes: using the trained handwriting reconstruction model and signature verification model to verify the authenticity of reference signatures and query signatures.

7. An offline handwriting reconstruction and signature verification system based on the Transformer architecture, characterized in that, include: Module M1: Collects offline signature images and online handwriting data, preprocesses the offline signature images, and converts the online handwriting data to obtain a handwriting sequence; Module M2: Construct a handwriting reconstruction model. Train the handwriting reconstruction model based on the preprocessed offline signature data and handwriting sequence to obtain the corresponding handwriting coordinate point sequence. Module M3: Based on the collected signature verification dataset and the handwriting coordinate point sequence reconstructed by the handwriting reconstruction model, it processes the input evidence and sample pairs, and performs feature extraction, comparison and optimization to accurately predict the authenticity of the evidence. The module M2 includes: Module M2.1: Employs deep learning methods to perform multi-layer convolution, pooling, and non-linear activation operations on the preprocessed offline signature image to extract local visual features and texture information from the image; the local visual features include the shape, thickness, edges, and connection relationships between strokes. Module M2.2: Based on the local visual features and texture information and the handwriting sequence, perform handwriting point prediction based on the Transformer decoder to obtain the probability distribution of handwriting points at the next time step; Module M2.3: The model parameters are adjusted to match the handwriting points at the next time step with the actual handwriting points. The loss function is defined as follows: Among them, Lx, and Let Px, Py, and Ps represent the cross-entropy loss of the x-coordinate, y-coordinate, and state, respectively; let Tx, Ty, and Ts represent the predicted probabilities of the x-coordinate, y-coordinate, and state, respectively; and let CE represent the cross-entropy loss. The model is trained using the gradient descent algorithm, and the model parameters are optimized so that the handwriting points predicted by the model can be consistent with the real situation, until the loss of the model on the validation set reaches the preset target or tends to converge. The Transformer-based decoder consists of multiple identical decoder layers stacked together. Each decoder layer includes the following sub-layers: a masked multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward neural network; The masked multi-head self-attention layer operates on the embedding of the target sequence to capture the internal dependencies between the generated handwriting points; The multi-head cross-attention layer uses the output of the self-attention layer as a query, and the image feature representation output by module M2.1 as the key and value. The feedforward neural network performs a further nonlinear transformation on the output of the cross-attention layer; The feature vector obtained after processing by the last decoder layer is used to predict the probability distribution of the x-coordinate, y-coordinate, and stroke state s of the next handwriting point through three independent linear layers.

Images