Handwritten signature recognition method and system fused with multi-modal technology

Abstract

The invention provides a handwritten signature recognition method and system fused with a multi-modal technology in the technical field of handwritten signature recognition, and the method comprises the steps: S1, obtaining a large number of handwritten signature images, carrying out the preprocessing and marking of each handwritten signature image, constructing a real data set, and carrying out the sample expansion operation of the real data set; s2, a large number of handwritten documents are obtained, and data synthesis operation is executed based on all the handwritten documents to construct a synthesis data set; s3, creating a handwritten signature recognition model for recognizing the handwritten signature based on the ConvNeXt module and the Vi T module; s4, performing first-stage fine tuning on a handwritten signature recognition model through the synthetic data set, and performing second-stage fine tuning on the handwritten signature recognition model through the real data set; and S5, identifying the handwritten signature through the fine-tuned handwritten signature identification model. The method has the advantages that the accuracy and robustness of handwritten signature recognition are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of handwritten signature recognition, and in particular to a handwritten signature recognition method and system integrating multimodal technology. Background Art

[0002] Handwritten signature recognition has always been one of the most attractive and challenging research areas in image processing and natural language processing. It plays an important role in many fields such as finance, law, medical care, e-commerce, and government services. It automatically extracts handwritten signatures and matches them with electronic signatures or identity information in the system, thereby improving the efficiency and security of document processing, reducing human intervention and fraud risks, and playing a key role in ensuring the legal validity of transactions and contracts.

[0003] Although significant research progress has been made in the field of handwritten signature recognition technology, and the introduction of deep learning technology has greatly promoted the development of this technology field, the process of recognizing handwritten signatures is still full of challenges. These challenges mainly stem from the diversity and randomness of the signatures themselves, as well as the huge differences in style between different writers. The irregularity of signatures further increases the difficulty of recognition, because everyone has their own unique habits when writing, and these habits are particularly evident in signatures. In addition, the visual similarity of characters in signatures and the adhesion between characters have put forward higher requirements on the feature extraction capabilities of traditional deep learning models. Furthermore, the background complexity of handwritten signatures is also an issue that cannot be ignored. Noise and interference in the background may interfere with the judgment of the model and affect the accuracy of recognition.

[0004] Therefore, how to provide a handwritten signature recognition method and system that integrates multimodal technology to improve the accuracy and robustness of handwritten signature recognition has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a handwritten signature recognition method and system integrating multimodal technology, so as to improve the accuracy and robustness of handwritten signature recognition.

[0006] In a first aspect, the present invention provides a handwritten signature recognition method integrating multimodal technology, comprising the following steps:

[0007] Step S1, obtaining a large number of handwritten signature images, constructing a real data set after preprocessing and annotating each of the handwritten signature images, and performing a sample expansion operation on the real data set;

[0008] Step S2, obtaining a large number of handwritten documents, and performing a data synthesis operation based on each of the handwritten documents to construct a synthetic data set;

[0009] Step S3, creating a handwritten signature recognition model for recognizing handwritten signatures based on the ConvNeXt module and the ViT module;

[0010] Step S4, performing a first-stage fine-tuning on the handwritten signature recognition model using the synthetic data set, and performing a second-stage fine-tuning on the handwritten signature recognition model using the real data set;

[0011] Step S5: Recognize the handwritten signature using the fine-tuned handwritten signature recognition model.

[0012] Furthermore, the step S1 is specifically as follows:

[0013] Acquire a large number of handwritten signature images, intercept the signature area of ​​each handwritten signature image, remove the defective and unrecognizable handwritten signature images during the interception process to complete the preprocessing of each handwritten signature image, and annotate the signature of each preprocessed handwritten signature image to construct a real data set;

[0014] A sample expansion operation including at least geometric transformation, texture transformation, random cropping and noise injection is performed on each handwritten signature image in the real data set, and the geometric transformation includes at least rotation, scaling, translation and flipping.

[0015] Furthermore, the step S2 is specifically as follows:

[0016] Set a random integer r1 in the range of [2,5] and a random integer r2 in the range of [2,4], wherein the random integer r1 is used to identify the number of samples to be split from each line of text, and the random integer r2 is used to identify the character length of the sample to be split;

[0017] A large number of handwritten documents are obtained, and the text of each handwritten document is split by combining the random integer r1 and the random integer r2 through a horizontal projection method and a vertical projection method to obtain a number of basic samples;

[0018] Image processing including at least dynamic brightness adjustment, random image rotation, random texture superposition and random covering is performed on each of the basic samples to complete the data synthesis operation, thereby constructing a synthetic data set.

[0019] Furthermore, in step S3, the ConvNeXt module is composed of a Stem network and four backbone convolutional networks, and uses a GELU activation function to perform depth-separable convolution and point-by-point convolution on the handwritten signature image to extract local features;

[0020] The Vi T module is used to divide the local features into multiple patches and linearly map them to a high-dimensional space, process the patches in the high-dimensional space through the encoder layer of the Transformer to extract the global features with long-distance dependencies, and output the handwritten signature recognition results based on the local features and the global features through the decoder of the Transformer;

[0021] Each of the encoder layers includes a self-attention mechanism and a multi-layer perceptron.

[0022] Furthermore, the step S4 is specifically as follows:

[0023] The handwritten signature recognition model is fine-tuned in one stage using the synthetic data set to preliminarily fit the diversity, randomness and complexity of handwritten signatures. The handwritten signature recognition model fine-tuned in the first stage is then fine-tuned in a second stage using the real data set to fit the data distribution characteristics of real handwritten signatures.

[0024] In a second aspect, the present invention provides a handwritten signature recognition system integrating multimodal technology, comprising the following modules:

[0025] A real data set construction module is used to obtain a large number of handwritten signature images, pre-process and annotate each of the handwritten signature images to construct a real data set, and perform sample expansion operations on the real data set;

[0026] A synthetic data set construction module, used to obtain a large number of handwritten documents, and perform a data synthesis operation based on each of the handwritten documents to construct a synthetic data set;

[0027] A handwritten signature recognition model creation module, used to create a handwritten signature recognition model for recognizing handwritten signatures based on the ConvNeXt module and the ViT module;

[0028] A handwritten signature recognition model fine-tuning module, used to perform a first-stage fine-tuning of the handwritten signature recognition model using the synthetic data set, and perform a second-stage fine-tuning of the handwritten signature recognition model using the real data set;

[0029] The handwritten signature recognition module is used to recognize the handwritten signature through the fine-tuned handwritten signature recognition model.

[0030] Furthermore, the real data set construction module is specifically used for:

[0031] Acquire a large number of handwritten signature images, intercept the signature area of ​​each handwritten signature image, remove the defective and unrecognizable handwritten signature images during the interception process to complete the preprocessing of each handwritten signature image, and annotate the signature of each preprocessed handwritten signature image to construct a real data set;

[0032] A sample expansion operation including at least geometric transformation, texture transformation, random cropping and noise injection is performed on each handwritten signature image in the real data set, and the geometric transformation includes at least rotation, scaling, translation and flipping.

[0033] Furthermore, the synthetic data set construction module is specifically used for:

[0034] Set a random integer r1 in the range of [2,5] and a random integer r2 in the range of [2,4], wherein the random integer r1 is used to identify the number of samples to be split from each line of text, and the random integer r2 is used to identify the character length of the sample to be split;

[0035] A large number of handwritten documents are obtained, and the text of each handwritten document is split by combining the random integer r1 and the random integer r2 through a horizontal projection method and a vertical projection method to obtain a number of basic samples;

[0036] Image processing including at least dynamic brightness adjustment, random image rotation, random texture superposition and random covering is performed on each of the basic samples to complete the data synthesis operation, thereby constructing a synthetic data set.

[0037] Furthermore, in the handwritten signature recognition model creation module, the ConvNeXt module is composed of a Stem network and four backbone convolution networks, and uses a GELU activation function to perform depth-separable convolution and point-by-point convolution on the handwritten signature image to extract local features;

[0038] The Vi T module is used to divide the local features into multiple patches and linearly map them to a high-dimensional space, process the patches in the high-dimensional space through the encoder layer of the Transformer to extract the global features with long-distance dependencies, and output the handwritten signature recognition results based on the local features and the global features through the decoder of the Transformer;

[0039] Each of the encoder layers includes a self-attention mechanism and a multi-layer perceptron.

[0040] Furthermore, the handwritten signature recognition model fine-tuning module is specifically used for:

[0041] The handwritten signature recognition model is fine-tuned in one stage using the synthetic data set to preliminarily fit the diversity, randomness and complexity of handwritten signatures. The handwritten signature recognition model fine-tuned in the first stage is then fine-tuned in a second stage using the real data set to fit the data distribution characteristics of real handwritten signatures.

[0042] The advantages of the present invention are:

[0043] A real data set is constructed by acquiring a large number of handwritten signature images, preprocessing and annotating them, and then performing sample expansion operations on the real data set; a large number of handwritten documents are acquired, and data synthesis operations are performed based on each handwritten document to construct a synthetic data set; then a handwritten signature recognition model for recognizing handwritten signatures is created based on the ConvNeXt module and the Vi T module, and the handwritten signature recognition model is fine-tuned in one stage through the synthetic data set, and in the second stage through the real data set, and finally the handwritten signature is recognized by the fine-tuned handwritten signature recognition model; that is, in order to address the scarcity problem of handwritten signature images, sample expansion operations and data synthesis operations are performed to expand the sample size and fit the diversity, randomness and complexity of real signature scenes; a handwritten signature recognition model is created based on the ConvNeXt module and the Vi T module, and the ConvNeXt module is used to extract local features of handwritten signatures, and the Vi T module is used to extract local features of handwritten signatures. The T module is used to extract the global features of the handwritten signature. By combining local features and global features, the deep features of the handwritten signature can be obtained, thereby improving the accuracy and robustness of recognition. The handwritten signature recognition model is fine-tuned in two stages using synthetic data sets and real data sets to distribute and optimize the model performance, gradually improving the adaptability and accuracy of the handwritten signature recognition model to the handwritten signature recognition task, ultimately greatly improving the accuracy and robustness of handwritten signature recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0045] Figure 1 It is a flow chart of a handwritten signature recognition method integrating multimodal technology of the present invention.

[0046] Figure 2 It is a structural schematic diagram of a handwritten signature recognition system integrating multimodal technology of the present invention. DETAILED DESCRIPTION

[0047] The technical solution in the embodiments of the present application has the following overall idea: in view of the scarcity problem of handwritten signature images, sample expansion operations and data synthesis operations are performed to expand the sample size and adapt to the diversity, randomness and complexity of real signature scenes; a handwritten signature recognition model is created based on the ConvNeXt module and the Vi T module, the ConvNeXt module is used to extract local features of the handwritten signature, and the Vi T module is used to extract global features of the handwritten signature. By combining local features and global features, the deep features of the handwritten signature can be obtained, thereby improving the accuracy and robustness of recognition; the handwritten signature recognition model is fine-tuned in two stages using synthetic data sets and real data sets, and the model performance is distributed and optimized, gradually improving the adaptability and accuracy of the handwritten signature recognition model to the handwritten signature recognition task, thereby improving the accuracy and robustness of handwritten signature recognition.

[0048] Please refer to Figure 1 to Figure 2 As shown, a preferred embodiment of a handwritten signature recognition method integrating multimodal technology of the present invention comprises the following steps:

[0049] Step S1, obtaining a large number of handwritten signature images, constructing a real data set after preprocessing and annotating each of the handwritten signature images, and performing a sample expansion operation on the real data set;

[0050] Step S2, obtaining a large number of handwritten documents, and performing a data synthesis operation based on each of the handwritten documents to construct a synthetic data set;

[0051] Step S3, creating a handwritten signature recognition model for recognizing handwritten signatures based on the ConvNeXt module and the ViT (Vision Transformer) module;

[0052] Step S4, performing a first-stage fine-tuning on the handwritten signature recognition model using the synthetic data set, and performing a second-stage fine-tuning on the handwritten signature recognition model using the real data set;

[0053] Step S5: Recognize the handwritten signature using the fine-tuned handwritten signature recognition model.

[0054] Aiming at the characteristics of handwritten signatures, the present invention uses the ConvNeXt module to perform preliminary feature extraction to obtain local features, enhances the recognition ability of the handwritten signature recognition model for signature details, and uses the Vi T module to deeply learn multi-dimensional features on synthetic data sets and real data sets to capture the global pattern and structure (global features) of the signature; in view of the possible deficiencies of handwritten signature data, the training set is expanded through sample expansion operations and data synthesis operations to improve the generalization ability of the handwritten signature recognition model, and enhances its robustness in processing real scene data, so that it can maintain stable recognition performance in a variety of application scenarios.

[0055] The step S1 is specifically as follows:

[0056] Acquire a large number of handwritten signature images, intercept the signature area of ​​each handwritten signature image, remove the defective and unrecognizable handwritten signature images during the interception process, and minimize the interception of non-handwritten signature parts to complete the preprocessing of each handwritten signature image, and annotate the signature of each preprocessed handwritten signature image to construct a real data set;

[0057] A sample expansion operation including at least geometric transformation, texture transformation, random cropping and noise injection is performed on each handwritten signature image in the real data set, and the geometric transformation includes at least rotation, scaling, translation and flipping.

[0058] Due to the lack of real data (handwritten signature images), the information contained in the data set itself is limited, so the real data set is expanded through sample expansion operations (data enhancement technology). However, improper sample expansion operations may introduce noise, thereby reducing recognition capabilities, so a combination of geometric transformation, texture transformation, random cropping and noise injection are used to ensure the quality of the real data set and improve recognition performance.

[0059] The step S2 is specifically as follows:

[0060] Set a random integer r1 in the range of [2,5] and a random integer r2 in the range of [2,4], wherein the random integer r1 is used to identify the number of samples to be split from each line of text, and the random integer r2 is used to identify the character length of the sample to be split;

[0061] A large number of handwritten documents are obtained, and the text of each handwritten document is split by combining the random integer r1 and the random integer r2 through a horizontal projection method and a vertical projection method to obtain a number of basic samples;

[0062] The horizontal projection method is used to determine the upper and lower boundaries of each line, while the vertical projection method is used to determine the left and right boundaries of each character, thereby achieving accurate text segmentation;

[0063] Image processing including at least dynamic brightness adjustment, random image rotation, random texture superposition and random covering is performed on each of the basic samples to complete the data synthesis operation, thereby constructing a synthetic data set.

[0064] Since handwritten signature images in real scenes have characteristics such as image degradation and complex background, the present invention aims to simulate the distribution characteristics of handwritten signatures in real environments by applying dynamic brightness adjustment, random image rotation, random texture superposition and random covering to improve the robustness and accuracy of recognition.

[0065] Dynamic brightness adjustment is to set a random decimal between 0.2 and 4.0 as the Gamma value of the image to describe the nonlinear mapping of image brightness. When the Gamma value of the image is greater than 1, the image brightness is increased, and when the Gamma value is less than 1, the image brightness is reduced.

[0066] Random image rotation is to set a random integer ranging from -30 degrees to 30 degrees to randomly rotate the image. Such rotation can increase the diversity of the data set and help improve the generalization ability of the model.

[0067] Random texture overlay introduces random texture effects. By randomly overlaying or mixing different texture patterns, the visual complexity of the image is increased. This processing can simulate different surface features and enhance the model's ability to recognize texture changes.

[0068] Random coverage means covering a certain proportion of the area at random positions on the image to simulate occluded or partially visible scenes. This processing can train the model to make accurate recognition and judgment when faced with incomplete information, thereby enhancing its adaptability in practical applications.

[0069] Since the sample size of handwritten signature images in real scenes is small, it is difficult to meet the requirements of personalization, diversification and complexity of handwritten signatures in real scenes. Therefore, the data set is expanded through data synthesis operations to simulate the distribution characteristics of handwritten signatures in real environments.

[0070] In step S3, the ConvNeXt module is composed of a Stem network and four backbone convolutional networks, and uses a GELU activation function to perform depth-separable convolution and point-by-point convolution on the handwritten signature image to extract local features;

[0071] The Vi T module is used to divide the local features into multiple patches and linearly map them to a high-dimensional space, process the patches in the high-dimensional space through the encoder layer of the Transformer to extract the global features with long-distance dependencies, and output the handwritten signature recognition results (handwritten signature sequence) based on the local features and the global features through the decoder of the Transformer;

[0072] Each of the encoder layers includes a self-attention mechanism and a multi-layer perceptron (MLP).

[0073] The present invention combines the ConvNeXt module and the Vi T module. The ConvNeXt module uses its efficient convolution operation to improve the utilization of local information and capture the subtle features and local patterns in the handwritten signature image. The Vi T module uses its self-attention mechanism to enhance the utilization of global information and identify the long-distance dependency and overall structure in the handwritten signature image. The combination of the two fully extracts the deep features of the handwritten signature, thereby improving the accuracy and robustness of recognition.

[0074] The step S4 is specifically as follows:

[0075] The handwritten signature recognition model is fine-tuned in one stage using the synthetic data set to preliminarily fit the diversity, randomness and complexity of handwritten signatures and improve the model's recognition capability for diversified signatures. The handwritten signature recognition model fine-tuned in the first stage is then fine-tuned in a second stage using the real data set to fit the data distribution characteristics of real handwritten signatures and improve the accuracy and robustness of the model in practical applications.

[0076] A preferred embodiment of a handwritten signature recognition system integrating multimodal technology of the present invention comprises the following modules:

[0077] A real data set construction module is used to obtain a large number of handwritten signature images, pre-process and annotate each of the handwritten signature images to construct a real data set, and perform sample expansion operations on the real data set;

[0078] A synthetic data set construction module, used to obtain a large number of handwritten documents, and perform a data synthesis operation based on each of the handwritten documents to construct a synthetic data set;

[0079] A handwritten signature recognition model creation module, used to create a handwritten signature recognition model for recognizing handwritten signatures based on a ConvNeXt module and a Vi T (Vision Transformer) module;

[0080] A handwritten signature recognition model fine-tuning module, used to perform a first-stage fine-tuning of the handwritten signature recognition model using the synthetic data set, and perform a second-stage fine-tuning of the handwritten signature recognition model using the real data set;

[0081] The handwritten signature recognition module is used to recognize the handwritten signature through the fine-tuned handwritten signature recognition model.

[0082] Aiming at the characteristics of handwritten signatures, the present invention uses the ConvNeXt module to perform preliminary feature extraction to obtain local features, enhances the recognition ability of the handwritten signature recognition model for signature details, and uses the Vi T module to deeply learn multi-dimensional features on synthetic data sets and real data sets to capture the global pattern and structure (global features) of the signature; in view of the possible deficiencies of handwritten signature data, the training set is expanded through sample expansion operations and data synthesis operations to improve the generalization ability of the handwritten signature recognition model, and enhances its robustness in processing real scene data, so that it can maintain stable recognition performance in a variety of application scenarios.

[0083] The real data set construction module is specifically used for:

[0084] Acquire a large number of handwritten signature images, intercept the signature area of ​​each handwritten signature image, remove the defective and unrecognizable handwritten signature images during the interception process, and minimize the interception of non-handwritten signature parts to complete the preprocessing of each handwritten signature image, and annotate the signature of each preprocessed handwritten signature image to construct a real data set;

[0085] A sample expansion operation including at least geometric transformation, texture transformation, random cropping and noise injection is performed on each handwritten signature image in the real data set, and the geometric transformation includes at least rotation, scaling, translation and flipping.

[0086] Due to the lack of real data (handwritten signature images), the information contained in the data set itself is limited, so the real data set is expanded through sample expansion operations (data enhancement technology). However, improper sample expansion operations may introduce noise, thereby reducing recognition capabilities, so a combination of geometric transformation, texture transformation, random cropping and noise injection are used to ensure the quality of the real data set and improve recognition performance.

[0087] The synthetic data set construction module is specifically used for:

[0088] Set a random integer r1 in the range of [2,5] and a random integer r2 in the range of [2,4], wherein the random integer r1 is used to identify the number of samples to be split from each line of text, and the random integer r2 is used to identify the character length of the sample to be split;

[0089] A large number of handwritten documents are obtained, and the text of each handwritten document is split by combining the random integer r1 and the random integer r2 through a horizontal projection method and a vertical projection method to obtain a number of basic samples;

[0090] The horizontal projection method is used to determine the upper and lower boundaries of each line, while the vertical projection method is used to determine the left and right boundaries of each character, thereby achieving accurate text segmentation;

[0091] Image processing including at least dynamic brightness adjustment, random image rotation, random texture superposition and random covering is performed on each of the basic samples to complete the data synthesis operation, thereby constructing a synthetic data set.

[0092] Since handwritten signature images in real scenes have characteristics such as image degradation and complex background, the present invention aims to simulate the distribution characteristics of handwritten signatures in real environments by applying dynamic brightness adjustment, random image rotation, random texture superposition and random covering to improve the robustness and accuracy of recognition.

[0093] Dynamic brightness adjustment is to set a random decimal between 0.2 and 4.0 as the Gamma value of the image to describe the nonlinear mapping of image brightness. When the Gamma value of the image is greater than 1, the image brightness is increased, and when the Gamma value is less than 1, the image brightness is reduced.

[0094] Random image rotation is to set a random integer ranging from -30 degrees to 30 degrees to randomly rotate the image. Such rotation can increase the diversity of the data set and help improve the generalization ability of the model.

[0095] Random texture overlay introduces random texture effects. By randomly overlaying or mixing different texture patterns, the visual complexity of the image is increased. This processing can simulate different surface features and enhance the model's ability to recognize texture changes.

[0096] Random coverage means covering a certain proportion of the area at random positions on the image to simulate occluded or partially visible scenes. This processing can train the model to make accurate recognition and judgment when faced with incomplete information, thereby enhancing its adaptability in practical applications.

[0097] Since the sample size of handwritten signature images in real scenes is small, it is difficult to meet the requirements of personalization, diversification and complexity of handwritten signatures in real scenes. Therefore, the data set is expanded through data synthesis operations to simulate the distribution characteristics of handwritten signatures in real environments.

[0098] In the handwritten signature recognition model creation module, the ConvNeXt module is composed of a Stem network and four backbone convolution networks, and uses a GELU activation function to perform depth-separable convolution and point-by-point convolution on the handwritten signature image to extract local features;

[0099] The Vi T module is used to divide the local features into multiple patches and linearly map them to a high-dimensional space, process the patches in the high-dimensional space through the encoder layer of the Transformer to extract the global features with long-distance dependencies, and output the handwritten signature recognition results (handwritten signature sequence) based on the local features and the global features through the decoder of the Transformer;

[0100] Each of the encoder layers includes a self-attention mechanism and a multi-layer perceptron (MLP).

[0101] The present invention combines the ConvNeXt module and the Vi T module. The ConvNeXt module uses its efficient convolution operation to improve the utilization of local information and capture the subtle features and local patterns in the handwritten signature image. The Vi T module uses its self-attention mechanism to enhance the utilization of global information and identify the long-distance dependency and overall structure in the handwritten signature image. The combination of the two fully extracts the deep features of the handwritten signature, thereby improving the accuracy and robustness of recognition.

[0102] The handwritten signature recognition model fine-tuning module is specifically used for:

[0103] The handwritten signature recognition model is fine-tuned in one stage using the synthetic data set to preliminarily fit the diversity, randomness and complexity of handwritten signatures and improve the model's recognition capability for diversified signatures. The handwritten signature recognition model fine-tuned in the first stage is then fine-tuned in a second stage using the real data set to fit the data distribution characteristics of real handwritten signatures and improve the accuracy and robustness of the model in practical applications.

[0104] In summary, the advantages of the present invention are:

[0105] A real data set is constructed by acquiring a large number of handwritten signature images, preprocessing and annotating them, and then performing sample expansion operations on the real data set; a large number of handwritten documents are acquired, and data synthesis operations are performed based on each handwritten document to construct a synthetic data set; then a handwritten signature recognition model for recognizing handwritten signatures is created based on the ConvNeXt module and the Vi T module, and the handwritten signature recognition model is fine-tuned in one stage through the synthetic data set, and in the second stage through the real data set, and finally the handwritten signature is recognized by the fine-tuned handwritten signature recognition model; that is, in order to address the scarcity problem of handwritten signature images, sample expansion operations and data synthesis operations are performed to expand the sample size and fit the diversity, randomness and complexity of real signature scenes; a handwritten signature recognition model is created based on the ConvNeXt module and the Vi T module, and the ConvNeXt module is used to extract local features of handwritten signatures, and the Vi T module is used to extract local features of handwritten signatures. The T module is used to extract the global features of the handwritten signature. By combining local features and global features, the deep features of the handwritten signature can be obtained, thereby improving the accuracy and robustness of recognition. The handwritten signature recognition model is fine-tuned in two stages using synthetic data sets and real data sets to distribute and optimize the model performance, gradually improving the adaptability and accuracy of the handwritten signature recognition model to the handwritten signature recognition task, ultimately greatly improving the accuracy and robustness of handwritten signature recognition.

[0106] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A handwritten signature recognition method integrating multimodal technology, characterized in that: The steps include: Step S1, obtaining a large number of handwritten signature images, constructing a real data set after preprocessing and annotating each of the handwritten signature images, and performing a sample expansion operation on the real data set; Step S2, obtaining a large number of handwritten documents, and performing a data synthesis operation based on each of the handwritten documents to construct a synthetic data set; Step S3, creating a handwritten signature recognition model for recognizing handwritten signatures based on the ConvNeXt module and the ViT module; Step S4, performing a first-stage fine-tuning on the handwritten signature recognition model using the synthetic data set, and performing a second-stage fine-tuning on the handwritten signature recognition model using the real data set; Step S5: Recognize the handwritten signature using the fine-tuned handwritten signature recognition model.

2. A handwritten signature recognition method integrating multimodal technology as claimed in claim 1, characterized in that: The step S1 is specifically as follows: Acquire a large number of handwritten signature images, intercept the signature area of ​​each handwritten signature image, remove the defective and unrecognizable handwritten signature images during the interception process to complete the preprocessing of each handwritten signature image, and annotate the signature of each preprocessed handwritten signature image to construct a real data set; A sample expansion operation including at least geometric transformation, texture transformation, random cropping and noise injection is performed on each handwritten signature image in the real data set, and the geometric transformation includes at least rotation, scaling, translation and flipping.

3. The handwritten signature recognition method integrating multimodal technology as claimed in claim 1, characterized in that: The step S2 is specifically as follows: Set a random integer r1 in the range of [2,5] and a random integer r2 in the range of [2,4], wherein the random integer r1 is used to identify the number of samples to be split from each line of text, and the random integer r2 is used to identify the character length of the sample to be split; A large number of handwritten documents are obtained, and the text of each handwritten document is split by combining the random integer r1 and the random integer r2 through a horizontal projection method and a vertical projection method to obtain a number of basic samples; Image processing including at least dynamic brightness adjustment, random image rotation, random texture superposition and random covering is performed on each of the basic samples to complete the data synthesis operation, thereby constructing a synthetic data set.

4. A handwritten signature recognition method integrating multimodal technology as claimed in claim 1, characterized in that: In step S3, the ConvNeXt module is composed of a Stem network and four backbone convolutional networks, and uses a GELU activation function to perform depth-separable convolution and point-by-point convolution on the handwritten signature image to extract local features; The ViT module is used to divide the local features into multiple patches and linearly map them to a high-dimensional space, process the patches in the high-dimensional space through the encoder layer of the Transformer to extract the global features with long-distance dependencies, and output the handwritten signature recognition results based on the local features and the global features through the decoder of the Transformer; Each of the encoder layers includes a self-attention mechanism and a multi-layer perceptron.

5. The handwritten signature recognition method integrating multimodal technology as claimed in claim 1, characterized in that: The step S4 is specifically as follows: The handwritten signature recognition model is fine-tuned in one stage using the synthetic data set to preliminarily fit the diversity, randomness and complexity of handwritten signatures. The handwritten signature recognition model fine-tuned in the first stage is then fine-tuned in a second stage using the real data set to fit the data distribution characteristics of real handwritten signatures.

6. A handwritten signature recognition system integrating multimodal technology, characterized in that: Includes the following modules: A real data set construction module is used to obtain a large number of handwritten signature images, pre-process and annotate each of the handwritten signature images to construct a real data set, and perform sample expansion operations on the real data set; A synthetic data set construction module, used to obtain a large number of handwritten documents, and perform a data synthesis operation based on each of the handwritten documents to construct a synthetic data set; A handwritten signature recognition model creation module, used to create a handwritten signature recognition model for recognizing handwritten signatures based on the ConvNeXt module and the ViT module; A handwritten signature recognition model fine-tuning module, used to perform a first-stage fine-tuning of the handwritten signature recognition model using the synthetic data set, and perform a second-stage fine-tuning of the handwritten signature recognition model using the real data set; The handwritten signature recognition module is used to recognize the handwritten signature through the fine-tuned handwritten signature recognition model.

7. A handwritten signature recognition system integrating multimodal technology as claimed in claim 6, characterized in that: The real data set construction module is specifically used for: Acquire a large number of handwritten signature images, intercept the signature area of ​​each handwritten signature image, remove the defective and unrecognizable handwritten signature images during the interception process to complete the preprocessing of each handwritten signature image, and annotate the signature of each preprocessed handwritten signature image to construct a real data set; A sample expansion operation including at least geometric transformation, texture transformation, random cropping and noise injection is performed on each handwritten signature image in the real data set, and the geometric transformation includes at least rotation, scaling, translation and flipping.

8. The handwritten signature recognition system integrating multimodal technology as claimed in claim 6, characterized in that: The synthetic data set construction module is specifically used for: Set a random integer r1 in the range of [2,5] and a random integer r2 in the range of [2,4], wherein the random integer r1 is used to identify the number of samples to be split from each line of text, and the random integer r2 is used to identify the character length of the sample to be split; A large number of handwritten documents are obtained, and the text of each handwritten document is split by combining the random integer r1 and the random integer r2 through a horizontal projection method and a vertical projection method to obtain a number of basic samples; Image processing including at least dynamic brightness adjustment, random image rotation, random texture superposition and random covering is performed on each of the basic samples to complete the data synthesis operation, thereby constructing a synthetic data set.

9. The handwritten signature recognition system integrating multimodal technology as claimed in claim 6, characterized in that: In the handwritten signature recognition model creation module, the ConvNeXt module is composed of a Stem network and four backbone convolution networks, and uses a GELU activation function to perform depth-separable convolution and point-by-point convolution on the handwritten signature image to extract local features; The ViT module is used to divide the local features into multiple patches and linearly map them to a high-dimensional space, process the patches in the high-dimensional space through the encoder layer of the Transformer to extract the global features with long-distance dependencies, and output the handwritten signature recognition results based on the local features and the global features through the decoder of the Transformer; Each of the encoder layers includes a self-attention mechanism and a multi-layer perceptron.

10. The handwritten signature recognition system integrating multimodal technology as claimed in claim 6, characterized in that: The handwritten signature recognition model fine-tuning module is specifically used for: The handwritten signature recognition model is fine-tuned in one stage using the synthetic data set to preliminarily fit the diversity, randomness and complexity of handwritten signatures. The handwritten signature recognition model fine-tuned in the first stage is then fine-tuned in a second stage using the real data set to fit the data distribution characteristics of real handwritten signatures.

Images