A method for realizing interpretable offline signature authentication based on a multi-modal large model

Abstract

The present application relates to a kind of based on multi-modal big model to realize the method for interpretable offline signature authentication, comprising: constructing data set;Data set is preprocessed;Multi-modal big model is constructed;Wherein, multi-modal big model includes: visual Transformer submodel, big language submodel and local signature visual representation enhancement submodel;Using the data set after pre-processing, multi-modal big model is trained, and authentication model is obtained;Using authentication model, interpretable offline signature authentication is carried out.The present application can complete the authenticity identification of Chinese signature and english signature and give detailed explanation in natural language form.The authenticity identification accuracy of Chinese signature can reach more than 80%, and the authenticity identification accuracy of english signature can reach more than 90%.For different signature input, different test analysis report can be output, and different verification effect and analysis result can be obtained by adjusting reasoning parameter.

Description

Technical Field

[0001] This invention relates to the fields of computer science, deep learning and pattern recognition technology, and in particular to a method for implementing interpretable offline signature authentication based on a multimodal large model. Background Technology

[0002] Handwritten signatures, as an important biometric feature, have long been widely used in key areas such as finance, law, and identity authentication due to their uniqueness and legal validity, to confirm personal identity and intent. Offline signature authentication specifically refers to the process of verifying the authenticity of a signature image already written on paper or other media. Compared to online signature authentication, which requires dedicated equipment to capture the dynamic information of the signature, offline signature authentication uses scanned or photographed signature images for authentication, and its application scenarios are more extensive, such as bank document processing, contract signing verification, and historical document authentication. However, existing technologies still face many challenges in practical applications, making it difficult to meet the requirements of high security, strong generalization, and interpretability.

[0003] In recent years, deep learning methods, especially convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have made some progress in the field of signature authentication. However, existing deep learning methods still face the following problems:

[0004] 1. The "Black Box" Problem and Lack of Explainability: While current deep learning-based signature authentication models offer good performance, their decision-making processes often lack transparency, making it difficult to explain why a signature is judged as genuine or fake. This is unacceptable in high-risk applications such as finance and law. For example, when a bank system rejects a large check, the customer or staff needs to know the specific reason for the rejection (which part of the stroke features is abnormal? Is it due to inadequate structural imitation or signs of hesitation?) in order to conduct a review or appeal. Existing deep learning methods typically only provide a similarity score or a true / false label, failing to offer such fine-grained, human-understandable explanations, thus reducing the system's credibility and practicality.

[0005] 2. Insufficient comprehensive understanding of subtle features and global structure: While traditional CNNs excel at extracting local textures and patterns, there is still room for improvement in integrating global structural information and capturing the subtle dynamic relationships between strokes (such as stroke smoothness and connection methods). This makes them potentially foolable even when faced with high-level forgeries that mimic the overall shape but distort the details. Summary of the Invention

[0006] The purpose of this invention is to provide a method for interpretable offline signature authentication based on a multimodal large model, in order to solve the problems existing in the prior art. By understanding and comparing signature images, the invention ultimately outputs a text-based, human-readable signature authentication result and a signature authentication report. Through the extraction of additional local features and their fusion with global features, the multimodal large model's ability to extract signature features can be enhanced, thereby improving the accuracy of its signature authentication.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A method for implementing interpretable offline signature authentication based on a multimodal large model includes:

[0009] Construct a dataset; wherein the dataset includes: signature images, authentication instructions, and authentication reports;

[0010] The dataset is preprocessed;

[0011] Construct a multimodal large model; wherein the multimodal large model includes: a visual Transformer sub-model, a large language sub-model, and a local signature visual representation enhancement sub-model;

[0012] The multimodal large model is trained using the preprocessed dataset to obtain an authentication model;

[0013] The authentication model described above is used to perform interpretable offline signature authentication.

[0014] Optionally, the signature image includes: a real signature image and a signature image to be compared; wherein the signature data in the images are signature data in different language types;

[0015] The authentication instruction is: an instruction to verify the signature image to be compared based on the real signature image.

[0016] Optionally, preprocessing the dataset includes:

[0017] Each signature image is processed in two ways: the first preprocessing is: no processing, directly read the original image in RGB three-channel format; the second preprocessing is: convert the RGB original image into a grayscale image, remove the white pixels at the edges except for the black strokes, keep only the strokes, and then use Gaussian blur to remove noise.

[0018] For authentication instructions and authentication reports, lexicalization is performed based on a predefined vocabulary, converting them into numerical vectors.

[0019] Optionally, the local signature visual representation enhancement sub-model includes: an input layer, a convolutional layer, a feature pyramid layer, and several average pooling layers.

[0020] Optionally, training the multimodal large model using the preprocessed dataset includes:

[0021] A local signature visual representation enhancement sub-model is pre-trained using grayscale images of signature images. Two signature images are used as input to the network. If both signatures are genuine, the label is 1; if the two signatures are genuine and forged, the label is 0.

[0022] The first preprocessed signature image is input into the visual Transformer sub-model, the second preprocessed signature image is input into the pre-trained local signature visual representation enhancement sub-model, and the authentication command is input into the large language sub-model. The multimodal large model is trained and is required to output the signature authentication result of the text and the explanation report. The manually annotated report is used as a label for supervision, and the training loss function is cross-entropy loss.

[0023] Optionally, using the authentication model to perform interpretable offline signature authentication includes:

[0024] The two signature images are input into the authentication model for inference, and the result of signature authentication and an explanation report are output; wherein, the first signature is a template and the second signature is an unknown query signature.

[0025] Optionally, inputting the two signature images into the authentication model for inference includes:

[0026] Perform the first preprocessing and the second preprocessing on each signature image respectively;

[0027] The two preprocessed signature images are input into the visual Transformer sub-model to extract global visual features.

[0028] The two preprocessed signature images are input into the local signature visual representation enhancement sub-model to extract several local visual feature vectors.

[0029] After performing a unified dimensional mapping on the global visual features and several local visual feature vectors, they are concatenated together;

[0030] The concatenated feature vectors are input into a large language model to obtain the signature authentication results and an explanation report.

[0031] Optionally, inputting the two preprocessed signature images into the local signature visual representation enhancement sub-model includes:

[0032] After inputting the two preprocessed signature images into the input layer, feature maps are extracted through a convolutional layer.

[0033] The extracted feature map is compressed into a feature vector of a preset dimension using a feature pyramid layer and several average pooling layers.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention proposes a method for interpretable offline signature authentication based on a multimodal large model. Leveraging the multimodal large model's ability to accept image input, understand image information, and output natural language text, the method fine-tunes the multimodal large model on an offline signature instruction fine-tuning dataset. This enables it to determine the authenticity of a query signature based on a template signature and output a text verification report detailing the basis for the authenticity judgment in a specific format. Furthermore, to enhance the model's comprehensive understanding of subtle signature features and global structure, this invention proposes a local signature visual representation enhancement model. This model combines the global feature extraction capabilities of the multimodal large model's original visual encoder with enhanced perception of handwritten signature features, thereby improving the accuracy of signature authentication and the quality of the generated report.

[0036] This invention can authenticate both Chinese and English signatures and provide detailed explanations in natural language. The accuracy rate for authenticating Chinese signatures can reach over 80%, and for English signatures, it can reach over 90%. Different verification and analysis reports can be output for different signature inputs, and different verification effects and analysis results can be obtained by adjusting the inference parameters. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of a method for implementing interpretable offline signature authentication based on a multimodal large model, according to an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] This embodiment proposes a method for implementing interpretable offline signature authentication based on a multimodal large model, including:

[0042] Construct a dataset; the dataset includes: signature images, authentication instructions, and authentication reports;

[0043] Preprocess the dataset;

[0044] Construct a multimodal large model; the multimodal large model includes: a visual Transformer sub-model, a large language sub-model, and a local signature visual representation enhancement sub-model;

[0045] The preprocessed dataset is used to train a large multimodal model to obtain an authentication model;

[0046] Utilize an authentication model to perform interpretable offline signature authentication.

[0047] Specifically, in this embodiment, such as Figure 1 As shown, this embodiment mainly includes the following steps:

[0048] (1) Data preparation: Obtain the offline signature instruction fine-tuning dataset, which should include signature images, corresponding authenticity verification conclusions, and detailed analysis reports;

[0049] (2) Data preprocessing: The two signature images are processed by grayscale conversion, normalization and other methods, and then input into the model to analyze their features and make comparisons;

[0050] (3) Model building: Construct an interpretable offline signature authentication multimodal large language model;

[0051] (4) Model training: Input the two signature images into the model, use the corresponding report text as labels, and perform fine-tuning training.

[0052] (5) Model reasoning: Input a real signature as a template and a signature used for testing into the model as a pair of data, and let the model output the verification results and a detailed analysis report.

[0053] This embodiment proposes a multimodal large-scale model for interpretable offline signature authentication. Leveraging the multimodal large-scale model's ability to accept image input, understand image information, and output natural language text, the model is fine-tuned on an offline signature instruction fine-tuning dataset. This allows it to determine the authenticity of a query signature based on a template signature and output a text verification report detailing the basis for the authenticity judgment in a specific format. Furthermore, to enhance the model's comprehensive understanding of subtle signature features and global structure, this embodiment proposes a local signature visual representation enhancement model. This model combines the multimodal large-scale model's original visual encoder (Visual Transformer, ViT) global feature extraction capabilities with enhanced perception of handwritten signature features, thereby improving the accuracy of signature authentication and the quality of the generated report.

[0054] Furthermore, the signature images in the dataset include: real signature images and signature images to be compared; wherein the signature data in the images are signature data of different language types;

[0055] The authentication command is: to verify the signature image to be compared based on the real signature image.

[0056] Specifically, in this embodiment, the model is trained using data consisting of signature images, instructions, and signature authentication reports. Each data sample contains two signature images: the first image is a genuine signature image, used as a template for comparison; the second image is the signature image to be compared, and its authenticity is unknown. Next, there is a signature authentication instruction, which reads either "Please determine whether Signature 2 is genuine or forged by comparing it against Signature 1." (in Chinese) or "Please determine whether Signature 2 is genuine or forged by comparing it against Signature 1." (in English). The authentication report contains the conclusion of the signature authentication, i.e., genuine or forged, followed by a detailed explanation of the criteria for authenticity and forgery. The report format is as follows:

[0057] "Verification result: Signature 2 is genuine / forged compared to signature 1."

[0058] Inspection points: Inspection point 1, Inspection point 2, Inspection point 3,...

[0059] 1. Inspection Point 1: [Detailed Description]

[0060] 2. Inspection Point 2: [Detailed Description]

[0061] 3. Inspection Point 3: [Detailed Description]

[0062] …"(Chinese)

[0063] “Verification Result:Compared to Signature 1,Signature 2 is genuine / forged.

[0064] Verification Aspects:Verification Aspect 1,Verification Aspect 2,Verification Aspect 3,...

[0065] Verification Aspect 1:[Detailed description]

[0066] Verification Aspect 2:[Detailed description]

[0067] Verification Aspect 3:[Detailed description]

[0068] ..."(English)

[0069] This report was used as a label for the training data.

[0070] Further preprocessing of the dataset includes:

[0071] Each signature image is processed in two ways: the first preprocessing is: no processing, directly read the original image in RGB three-channel format; the second preprocessing is: convert the RGB original image into a grayscale image, remove the white pixels at the edges except for the black strokes, keep only the strokes, and then use Gaussian blur to remove noise.

[0072] For authentication instructions and authentication reports, lexicalization is performed based on a predefined vocabulary, converting them into numerical vectors.

[0073] Specifically, in this embodiment, data preprocessing is performed on the signature image, text instructions, and signature authentication report. Each signature image undergoes two processing methods: first, no processing is performed, and the original image is directly read in RGB three-channel format; second, the original RGB image is converted to grayscale, the white pixels at the edges (excluding black strokes) are removed, leaving only the strokes, and then Gaussian blur is applied for noise removal with a Gaussian kernel size of 3×3. Finally, the image is scaled to 96×336 (width×height) while preserving the aspect ratio, and any insufficient areas are padded with white pixels. Both processed images are used as visual input for the model. For the text instructions and signature authentication report, lexicalization is performed according to a predefined vocabulary, converting them into numerical vectors as model input.

[0074] Furthermore, the local signature visual representation enhancement sub-model includes: an input layer, a convolutional layer, a feature pyramid layer, and several average pooling layers.

[0075] Specifically, in this embodiment, a multimodal large model capable of accepting multiple image inputs and outputting text is constructed. This invention references the existing multimodal large model Qwen2-VL, using a VisionTransformer (ViT) model as the global encoder for images and a Large Language Model (LLM) for decoding and text output. Building upon this, this invention proposes a local signature visual representation enhancement network to extract more discriminative signature features. Subsequently, the local visual features are concatenated with the global features extracted by the VisionTransformer, and then input into multiple fully connected layers for feature fusion. Finally, the fused visual features are input into the large language model for decoding.

[0076] For the Visual Transformer Encoder (ViT), this embodiment uses the same settings as the Qwen2-VL-2B model, which includes 32 Transformer modeling layers, 3 input channels, 1280 embedding dimensions, 1536 hidden layers, 14 image blocks, and 675 million parameters. For the Large Language Model Decoder (LLM), this embodiment uses the same settings as the Qwen2-VL-2B model, with a vocabulary size of 151658, 1536 hidden layers, 28 hidden layers, 12 attention heads, and 8960 intermediate layers. The LLM has 1.5 billion parameters.

[0077] This embodiment proposes a local signature visual representation enhancement network, the structure of which is shown in Table 1 below:

[0078] Table 1

[0079]

[0080] The network accepts two signature images as input, each a single-channel grayscale image with a size of 96*336, resulting in an input size of 2*1*96*336. After feature maps are extracted by an intermediate ResNet-18 layer, feature pyramid layers and three average pooling layers are used to compress the feature maps into one-dimensional feature vectors. Each image outputs four corresponding feature vectors, which are combined with the global image features extracted by ViT and input into LLM for further modeling.

[0081] Let I be the two input signature images. r and I q , where I r For the template signature image, I q The authenticity of the signature image is unknown and needs to be compared. Where H r and W r For I r Width and height, H q and W q For I q The width and height are 3, and 3 represents the number of image channels (RGB channels). This embodiment requires the multimodal large model to be compared with template I. r Comparison to determine signature I q The authenticity. According to step (2), the unprocessed I r and I q First, the input is given to ViT to extract global visual features, as shown in formula (1):

[0082]

[0083] Where φ ViT It is a ViT visual encoder. Image I r The global visual representation vectors share N r indivual, Similarly. Then, according to step (2), I... r and I q The second process is used to obtain I r ′ and I q ′ , will I r ′ and I q ′ The input is fed into a local signature visual representation network, which extracts four local visual feature vectors for each image:

[0084]

[0085] φ L This represents the local signature visual representation network proposed in this invention. After obtaining the local feature vectors, multiple multilayer perceptrons (MLPs) are set up to map global and local visual features to a unified dimension. Each MLP consists of two fully connected layers, as shown in the formula:

[0086]

[0087] L i =GELU(L i (4)

[0088]

[0089] In i It is the input feature vector, Out i It is the output feature vector. W i1 and b i1 It consists of the weights and biases of the first fully connected layer, W. i2 and b i2 These are the weights and biases of the second fully connected layer. All global visual feature vectors use the same MLP, while each local feature vector uses a separate perceptron with its own input dimension d. in and output dimension d out As shown in Table 2 below:

[0090] Table 2

[0091] Feature vector <![CDATA[d in ]]> <![CDATA[d out ]]> G 1280 1536 <![CDATA[T1]]> 512 1536 <![CDATA[T2]]> 256 1536 <![CDATA[T3]]> 128 1536 <![CDATA[T4]]> 64 1536

[0092] Therefore, this embodiment sets up a total of 5 MLP modules. Global visual feature vector After mapping by the MLP, the output global feature vector is: Local visual feature vectors and After each MLP, the output local feature vector is: and Next, concatenate the global feature vector and the local feature vector together:

[0093]

[0094] V represents two signed images I r and I q The set of global and local visual features. Next, V is input into the large language model LLM, which is required to understand and compare the similarities and differences between the two signatures, and decode the text content. The format of the text content is as shown in the report above, containing the signature I. qThe authenticity of the information and a detailed analysis report.

[0095] Furthermore, training a large multimodal model using the preprocessed dataset includes:

[0096] A local signature visual representation enhancement sub-model is pre-trained using grayscale images of signature images. Two signature images are used as input to the network. If both signatures are genuine, the label is 1; if the two signatures are genuine and forged, the label is 0.

[0097] The first preprocessed signature image is input into the visual Transformer sub-model, the second preprocessed signature image is input into the pre-trained local signature visual representation enhancement sub-model, and the authentication command is input into the large language sub-model. The multimodal large model is trained and is required to output the signature authentication result of the text and the explanation report. The manually annotated report is used as a label for supervision, and the training loss function is cross-entropy loss.

[0098] Specifically, in this embodiment, during model training, the local signature visual representation enhancement network is first trained separately using signature image data. Two signature images are used as input to the network. If both signatures are genuine, the label is 1; if the two signatures are a genuine signature and a forged signature, respectively, the label is 0. The training settings for this part are as follows:

[0099] Number of iterations: 10000;

[0100] Learning rate: 0.0001, learning rate reduction strategy is cosine annealing, and the learning rate is reduced to 1×10 in the last iteration. -7 ;

[0101] weight decay: 0.01

[0102] The optimizer is AdamW;

[0103] Batch size: 24;

[0104] Secondly, this embodiment combines a separately trained Local Signature Visual Representation Enhancement Network (VIT), a Visual Transformer (ViT), a Large Language Model (LLM), and five MLP modules for mapping to form the interpretable offline signature authentication multimodal large language model proposed in this invention. Using the pre-processed data described above, the input data follows the process outlined in the model construction, i.e., two signature images are input into ViT and the Local Signature Visual Representation Enhancement Network respectively (the image input into ViT requires no processing, while the image input into the Local Signature Visual Representation Enhancement Network undergoes additional processing). The instruction text is input into the LLM, and the model is required to output the signature authentication result of the text and a detailed explanation report. Manually labeled reports from the data are used as labels for supervision, and the training loss function is cross-entropy loss.

[0105] Before training this multimodal large model, we preloaded the pre-trained weights of Qwen2-VL-2B. Then, we froze the parameters of the ViT and Local Signature Visual Representation Augmentation Networks, training only the parameters of the MLP and LLM parts. The training settings were as follows:

[0106] Number of iterations: 18750;

[0107] Optimizer: AdamW;

[0108] Learning rate: 5×10 -6 The learning rate reduction strategy is a cosine annealing strategy, and the learning rate is reduced to 0 in the last iteration.

[0109] weight decay: 0.1;

[0110] Number of graphics cards: 4 NVIDIA A6000;

[0111] Single-card batch size: 2;

[0112] Total batch size: 8;

[0113] Training accuracy: a mix of half-precision and full-precision;

[0114] Based on the training process described above, this embodiment uses Chinese signature data and English signature data to train versions that can be used for Chinese signature authentication and English signature authentication, respectively.

[0115] Furthermore, using the authentication model, interpretable offline signature authentication includes:

[0116] Two signature images are input into the authentication model for inference, and the result of signature authentication and an explanation report are output; the first signature is a template, and the second signature is an unknown query signature.

[0117] Furthermore, inputting the two signature images into the authentication model for inference includes:

[0118] Perform a first preprocessing and a second preprocessing on each signature image separately;

[0119] The two preprocessed signature images are input into the visual Transformer sub-model to extract global visual features.

[0120] The two preprocessed signature images are input into the local signature visual representation enhancement sub-model to extract several local visual feature vectors.

[0121] After mapping the global visual features and several local visual feature vectors to a unified dimension, they are concatenated together.

[0122] The concatenated feature vectors are input into a large language model to obtain the signature authentication results and an explanation report.

[0123] Furthermore, inputting the two preprocessed signature images into the local signature visual representation enhancement sub-model includes:

[0124] After inputting the two preprocessed signature images into the input layer, feature maps are extracted through a convolutional layer.

[0125] The extracted feature map is compressed into a feature vector of a preset dimension using a feature pyramid layer and several average pooling layers.

[0126] Specifically, in this embodiment, two signatures are input into the model for model inference. The present invention inputs two signatures into the model: the first signature is a template, and the second signature is an unknown query signature. The model is required to compare the second signature with the first signature to identify the authenticity of the second signature, and output the signature authentication result and a detailed explanation report in the format of step (2). If the signature authentication is performed on a Chinese signature, the instruction input in this invention is "Please verify whether Signature 2 is genuine or forged by comparing it against Signature 1." If the signature authentication is performed on an English signature, the instruction input in this embodiment is "Please determine whether Signature 2 is genuine or forged by comparing it against Signature 1." In the output report, relevant keywords such as "genuine," "forged," and "authentic" can be extracted and compared with the real labels of the test data to calculate indicators such as accuracy.

[0127] This embodiment is the first to propose a multimodal large model capable of achieving interpretable offline signature authentication. It applies the image understanding and text output capabilities of a multimodal large model to the understanding of offline signatures (i.e., signature images), outputting a human-readable, text-based signature authentication result along with a formatted, detailed explanation. Existing methods cannot output explanatory text. This embodiment demonstrates for the first time that multimodal large models and AI can perform interpretable signature authentication, and has the potential to be applied to professional signature authentication work in actual criminal investigation processes.

[0128] This embodiment innovatively proposes a local signature visual representation network to enhance the signature perception, analysis, and overall signature authentication capabilities of a multimodal large model. By pre-learning signature authentication knowledge and perceiving features such as stroke trends and character structures of different signatures, the network inputs the learned knowledge into the multimodal large model as representation vectors, greatly improving the signature detail understanding and signature authentication capabilities of the multimodal large model. This network is key to enhancing the signature authentication capabilities of multimodal large models.

[0129] This embodiment proposes for the first time a fine-tuning data format specifically for interpretable signature authentication. Each data sample includes two signature images (the first is a template signature, and the second is the signature to be compared), signature authentication instructions, and a standardized, formatted signature authentication report (consisting of the final authentication conclusion and a detailed explanation). The definition of the signature authentication report format is particularly important, as no other invention has previously defined it (nor has any invention performed interpretable signature authentication). This format is specifically adapted for training multimodal large models and can become a unified training and inference data format for interpretable offline signature authentication based on multimodal large models or other AI within the industry.

[0130] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for implementing an interpretable offline signature authentication based on a multi-modal large model, characterized in that, The method comprises the following steps: constructing a data set; wherein the data set comprises: signature pictures, authentication instructions, and authentication reports; the signature pictures comprise: real signature pictures and signature pictures to be compared, and the signature data in the pictures is signature data of different language types; preprocessing the data set; the preprocessing of the data set comprises: processing each signature picture in two ways; wherein the first preprocessing is: reading the original picture directly in the RGB three-channel format without any processing, and the second preprocessing is: converting the original RGB picture into a grayscale picture, removing the edge white pixels except the black handwriting, only keeping the handwriting, and then removing noise by using Gaussian blur; performing word tokenization on the authentication instructions and the authentication reports according to a predefined word list, and converting them into numerical vectors; wherein the authentication instructions are instructions for verifying the signature picture to be compared based on the real signature picture; constructing a multi-modal large model; wherein the multi-modal large model comprises: a visual Transformer sub-model, a large language sub-model, and a local signature visual feature enhancement sub-model; the local signature visual feature enhancement sub-model comprises: an input layer, a convolutional layer, a feature pyramid layer, and a plurality of average pooling layers; training the multi-modal large model by using the preprocessed data set to obtain an authentication model; the training of the multi-modal large model by using the preprocessed data set comprises: pre-training the local signature visual feature enhancement sub-model by using the grayscale picture of the signature picture; wherein two signature pictures are used as the input of the network, if both of the two signature pictures are real signature pictures, the label is 1, and if the two signature pictures are a real signature picture and a fake signature picture respectively, the label is 0; inputting the signature picture preprocessed in the first way into the visual Transformer sub-model to extract global features, inputting the signature picture preprocessed in the second way into the pre-trained local signature visual feature enhancement sub-model to extract local visual features, concatenating the global features and the local visual features, inputting them into a plurality of fully connected layers for feature fusion, inputting the fused visual features into the large language model for decoding; inputting the authentication instructions into the large language sub-model to train the multi-modal large model, and requiring the multi-modal large model to output the signature authentication result of the text and the explanation report; wherein the report manually labeled is used as the label for supervision, and the loss function of the training is cross-entropy loss; using the authentication model to perform interpretable off-line signature authentication.

2. The method for implementing interpretable off-line signature authentication based on a multi-modal large model according to claim 1, characterized in that, using the authentication model to perform interpretable off-line signature authentication comprises: inputting two signature pictures into the authentication model for inference to output the signature authentication result and the explanation report; wherein the first signature is a template, and the second signature is an unknown query signature.

3. The method for achieving interpretable off-line signature authentication based on a multi-modal large model according to claim 2, characterized in that, inputting two signature pictures into the authentication model for inference comprises: respectively performing the first preprocessing and the second preprocessing on each signature picture; inputting the two signature pictures preprocessed in the first way into the visual Transformer sub-model to extract global visual features; inputting the two signature pictures preprocessed in the second way into the local signature visual feature enhancement sub-model to extract a plurality of local visual feature vectors; The global visual feature and the plurality of local visual feature vectors are uniformly dimensionally mapped and spliced together; The spliced feature vector is input into a large language sub-model to obtain a signature authentication result and an interpretation report.

4. The method for implementing interpretable off-line signature authentication based on a multi-modal large model according to claim 3, characterized in that, The second preprocessed two signature pictures are input into a local signature visual representation enhancement sub-model, which includes: After the second preprocessed two signature pictures are input into an input layer, a feature map is extracted through a convolution layer; The extracted feature map is compressed into a feature vector of a preset dimension by using a feature pyramid layer and a plurality of average pooling layers.

Images