Method, system, device, medium and program product for processing handwritten signatures
By using a signature processing model to determine the clarity of handwritten signature images and select post-processing paths, the high complexity and low efficiency of handwritten signature recognition in existing technologies are solved, achieving efficient and accurate signature recognition and improved adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUA DATA TECH (SHANGHAI) CO LTD
- Filing Date
- 2026-04-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing handwritten signature recognition methods are complex, inefficient, have limited applicability to various scenarios, and consume excessive resources, making it difficult to effectively recognize illegible or unclear handwritten signatures.
A signature processing model is used to process handwritten signature images. The post-processing path is selected based on the sharpness discrimination result. If the sharpness is high, decoding is performed, and if the sharpness is low, matching is performed. The model includes image branch, text branch, sharpness classification head and gated neural network. Combined with CLIP model and pre-trained OCR model, efficient recognition of handwritten signatures is achieved.
It improves the efficiency and adaptability of handwritten signature recognition, reduces recognition complexity, reduces resource consumption, adapts to more scenarios, and ensures the accuracy and reliability of recognition results.
Smart Images
Figure CN122116386A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of handwritten text recognition technology, and in particular to a method, system, device, medium and program product for processing handwritten signatures. Background Technology
[0002] With the acceleration of digital transformation and the widespread adoption of AI (Artificial Intelligence) technology, the flow of various business data and documents is gradually becoming fully electronic. However, signatures, as core credentials for identity verification and liability endorsement, still need to be presented in handwritten form even in electronic scenarios. Handwritten signatures contain unique writing rhythm, stroke pressure, and cursive characteristics—personalized biometric information that directly reflects the signatory's subjective will. They clearly demonstrate the signatory's knowledge, acceptance, and confirmation of the document's content. The associated legal responsibility and security endorsement attributes cannot be replaced by standardized printed signatures or automatically generated electronic identifiers. Especially in areas with extremely high security and compliance requirements, such as financial transactions, commercial procurement and consumer transactions, submission of important personal materials, and the determination of liability in high-risk scenarios, handwritten signatures have become a crucial link in defining rights and obligations and mitigating legal risks. Their irreplaceable nature is becoming increasingly prominent in the digital process.
[0003] For handwritten signature recognition, most current methods rely on OCR (Optical Character Recognition) models or multimodal large models for direct recognition. However, handwritten signatures have uncontrollable problems such as illegible handwriting, cursive writing, and unclear characters. Direct recognition is difficult to achieve for this type of signature. Generally, a signature database similar to facial recognition is built for matching and recognition.
[0004] Currently, the two methods are deployed independently, which has problems such as high complexity of recognition methods, low recognition efficiency, limited adaptability to scenarios, and redundant resource consumption. Summary of the Invention
[0005] The technical problem to be solved by this disclosure is to overcome the shortcomings of existing technologies that independently use direct recognition and matching recognition to identify handwritten signatures, such as high recognition complexity, low recognition efficiency, limited adaptability to scenarios, and redundant resource consumption. This disclosure provides a method, system, device, medium, and program product for processing handwritten signatures.
[0006] This disclosure solves the above-mentioned technical problems through the following technical solution:
[0007] This disclosure provides a method for processing handwritten signatures, the method comprising:
[0008] Obtain the handwritten signature image to be processed;
[0009] The handwritten signature image is processed by a trained signature processing model to obtain the sharpness discrimination result and general feature vector of the handwritten signature image. The signature processing model includes an image branch, a text branch, a sharpness classification head and a gated neural network. The general feature vector is obtained by weighted fusion of image features from the image branch and text features from the text branch through the gated neural network.
[0010] Based on the clarity discrimination result, a post-processing path is selected. If the clarity is higher than the clarity threshold, the general feature vector is decoded to obtain the signature text. If the clarity is lower than or equal to the clarity threshold, the general feature vector is matched with the standard features in the signature library to obtain the signature text.
[0011] Optionally, the signature processing model is obtained based on the dual-tower structure of the CLIP (Contrastive Language–Image Pre-training) model;
[0012] And / or,
[0013] The sharpness classification head is a fully connected neural network, the general feature vector has a dimension of 512, and the sharpness discrimination result is a sharpness quantification value;
[0014] And / or,
[0015] The image features are obtained by adjusting the initial image features output by the image feature extraction layer of the signature processing model through residual connections;
[0016] And / or,
[0017] The text features are obtained based on a pre-trained handwritten OCR model.
[0018] Optionally, the step of decoding the general feature vector to obtain the signature text includes:
[0019] The signature text is obtained by using the signature image text recognition and decoding layer of the signature processing model to decode the general feature vector.
[0020] And / or,
[0021] The step of matching the general feature vector with standard features in the signature library to obtain the signature text includes:
[0022] The general feature vector is matched with the features corresponding to the signature graph in the signature library. If the matching degree is higher than the first matching degree threshold, the signature text is obtained. If the matching degree is not higher than the first matching degree threshold, the candidate signature is obtained.
[0023] The candidate signature is matched with the roster. If the matching degree is higher than the second matching degree threshold, the signature text is obtained. If the matching degree is not higher than the second matching degree threshold, the candidate signature is determined by a preset large model to determine whether the handwritten signature image is included.
[0024] If the candidate signature contains the handwritten signature image, the signature text is obtained; if it cannot be determined or the candidate signature does not contain the handwritten signature image, a feedback mechanism is triggered to obtain the signature text.
[0025] Optionally, after obtaining the signature text, the method further includes:
[0026] The signature text is verified using a pre-defined large model to obtain the verification result;
[0027] In response to the verification result indicating that the verification failed, a feedback mechanism is triggered to obtain a new signature text.
[0028] Optionally, the step of obtaining the handwritten signature image to be processed includes:
[0029] The handwritten signature image in the document to be processed is located and extracted using a preset YOLO (an object detection algorithm) object detection model.
[0030] Optionally, the signature processing model is obtained through three-stage training;
[0031] The first stage involves training the image feature extraction layer and text feature extraction layer of the signature processing model;
[0032] The second stage involves training the signature graph text recognition and decoding layer and the text feature extraction layer of the signature processing model.
[0033] The third stage involves enhancing the training of the signature image text recognition and decoding layer.
[0034] This disclosure also provides a handwritten signature processing system, the processing system comprising:
[0035] The signature image acquisition module is used to acquire the handwritten signature image to be processed;
[0036] A signature image processing module is used to process the handwritten signature image through a trained signature processing model to obtain the sharpness discrimination result and a general feature vector of the handwritten signature image. The signature processing model includes an image branch, a text branch, a sharpness classification head, and a gated neural network. The general feature vector is obtained by weighted fusion of image features from the image branch and text features from the text branch through the gated neural network.
[0037] The signature text acquisition module is used to select a post-processing path based on the clarity discrimination result. If the clarity is higher than the clarity threshold, the general feature vector is decoded to obtain the signature text. If the clarity is lower than or equal to the clarity threshold, the general feature vector is matched with the standard features in the signature library to obtain the signature text.
[0038] Optionally, the signature processing model is derived from the dual-tower structure of the CLIP model;
[0039] And / or,
[0040] The sharpness classification head is a fully connected neural network, the general feature vector has a dimension of 512, and the sharpness discrimination result is a sharpness quantification value;
[0041] And / or,
[0042] The image features are obtained by adjusting the initial image features output by the image feature extraction layer of the signature processing model through residual connections;
[0043] And / or,
[0044] The text features are obtained based on a pre-trained handwritten OCR model.
[0045] Optionally, the signature text acquisition module includes:
[0046] The decoding unit is used to perform a decoding operation on the general feature vector using the signature image text recognition decoding layer of the signature processing model to obtain the signature text.
[0047] And / or,
[0048] The signature text acquisition module includes:
[0049] The graph matching unit is used to perform graph matching between the general feature vector and the features corresponding to the signature graph in the signature library. If the matching degree is higher than the first matching degree threshold, the signature text is obtained; if the matching degree is not higher than the first matching degree threshold, the candidate signature is obtained.
[0050] The image-text matching unit is used to perform image-text matching between the candidate signature and the roster. If the matching degree is higher than the second matching degree threshold, the signature text is obtained. If the matching degree is not higher than the second matching degree threshold, the candidate signature is determined by a preset large model to determine whether the handwritten signature image is included.
[0051] The large model judgment unit is used to obtain the signature text when the candidate signature contains the handwritten signature image, and to trigger a feedback mechanism to obtain the signature text when it cannot be judged or the candidate signature does not contain the handwritten signature image.
[0052] Optionally, the processing system further includes:
[0053] The verification processing module is used to verify the signature text using a preset large model to obtain the verification result;
[0054] The feedback mechanism trigger module is used to trigger the feedback mechanism to obtain a new signature text in response to the verification result indicating that the verification failed.
[0055] Optionally, the signature image acquisition module is further configured to locate and extract the handwritten signature image from the document to be processed using a preset YOLO object detection model.
[0056] Optionally, the signature processing model is obtained through three-stage training;
[0057] The first stage involves training the image feature extraction layer and text feature extraction layer of the signature processing model;
[0058] The second stage involves training the signature graph text recognition and decoding layer and the text feature extraction layer of the signature processing model.
[0059] The third stage involves enhancing the training of the signature image text recognition and decoding layer.
[0060] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein the processor executes the computer program to implement the handwritten signature processing method described above.
[0061] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the handwritten signature processing method described above.
[0062] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the handwritten signature processing method as described above.
[0063] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.
[0064] The positive and progressive effects of this disclosure are as follows:
[0065] This disclosure processes handwritten signature images using a signature processing model to obtain the clarity judgment result of the handwritten signature image. Then, based on the clarity judgment result, a post-processing path is selected to obtain the signature text. By setting the clarity judgment of the handwritten signature image in the signature processing model, the detection efficiency is improved, the complexity of the recognition method is reduced, the recognition efficiency is improved, more scenarios can be adapted, and resource consumption is reduced. Attached Figure Description
[0066] Figure 1 This is a first flowchart of the handwritten signature processing method according to Embodiment 1 of this disclosure;
[0067] Figure 2 This is a flowchart of step S103 in the handwritten signature processing method of Embodiment 1 of this disclosure;
[0068] Figure 3 This is a second flowchart of the handwritten signature processing method of Embodiment 1 of this disclosure;
[0069] Figure 4 This is a schematic diagram of the signature processing model in the handwritten signature processing method of Embodiment 1 of this disclosure;
[0070] Figure 5 The diagram illustrates a specific example of the handwritten signature processing method according to Embodiment 1 of this disclosure.
[0071] Figure 6 This is a schematic diagram of the modules of the handwritten signature processing system according to Embodiment 2 of this disclosure;
[0072] Figure 7 This is a schematic diagram of the structure of the electronic device according to Embodiment 3 of this disclosure. Detailed Implementation
[0073] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.
[0074] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.
[0075] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision and disclosure of handwritten signature information comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0076] Example 1
[0077] This embodiment provides a method for processing handwritten signatures, such as... Figure 1 As shown, the processing method includes:
[0078] S101. Obtain the handwritten signature image to be processed;
[0079] S102. The handwritten signature image is processed by the trained signature processing model to obtain the sharpness discrimination result and general feature vector of the handwritten signature image. The signature processing model includes an image branch, a text branch, a sharpness classification head and a gated neural network. The general feature vector is obtained by weighted fusion of image features from the image branch and text features from the text branch through the gated neural network.
[0080] S103. Select the post-processing path based on the sharpness judgment result. If the sharpness is higher than the sharpness threshold, perform a decoding operation on the general feature vector to obtain the signature text. If the sharpness is lower than or equal to the sharpness threshold, match the general feature vector with the standard features in the signature library to obtain the signature text.
[0081] Specifically, the signature processing model includes an image branch and a text branch, which have independent feature extraction capabilities. The image branch is structurally optimized and its functions are enhanced. A fully connected classification head is added to the image branch to achieve adaptive discrimination of signature clarity.
[0082] After the basic features of the handwritten signature image are extracted by the backbone network, a new classification head module, namely the sharpness classification head, is added. The core function of this sharpness classification head is to adaptively determine the sharpness of the handwritten signature image, and output it as the basis vector for subsequent feature fusion. The sharpness determination result is then output through an additional output layer.
[0083] After the sharpness classification head, a gated neural network (GNN) is connected. Its core function is to dynamically adjust the fusion weight ratio between the image feature vector layer and the OCR text feature vector layer. The image feature vector layer mainly represents some morphological features of the handwritten signature, while the OCR text feature vector layer is extracted from the text recognition task and focuses on representing the text features of the signature. The input of the OCR text feature vector layer is the image, and the output is the hidden state feature vector.
[0084] The final feature output layer, based on the dynamic weights output by the gated neural network, performs a weighted vector summation operation on image and text features to generate a unified, universal feature vector. The gated neural network outputs two values, which serve as weights for the two features, and the fused features are obtained by summing them element-wise. Simultaneously, the signature processing model automatically determines subsequent processing logic based on the sharpness classification result output by the sharpness classification head: when the sharpness value is higher than the sharpness threshold, such as 0.7, it is determined to be a high-quality signature, triggering the sequence decoding process to complete direct OCR recognition and obtain the signature text; when the sharpness value is lower than or equal to the sharpness threshold, it is determined to be a low-quality signature, initiating a vector matching process to compare features with a standard signature sample library and obtain the signature text. This achieves seamless integration and unified processing for both clear and unclear scenarios. Handwritten signature images in clear scenarios are high-quality signatures, while handwritten signature images in unclear scenarios are low-quality signatures.
[0085] In this solution, the handwritten signature image is processed by a signature processing model to obtain the clarity judgment result of the handwritten signature image. Then, the post-processing path is automatically switched according to the clarity judgment result to obtain the signature text. This achieves unified and seamless processing of high and low quality signatures. Setting the clarity judgment of the handwritten signature image in the signature processing model improves the detection efficiency, reduces the complexity of the recognition method, improves the recognition efficiency, can be adapted to more scenarios, and reduces resource consumption.
[0086] In one feasible approach, the signature processing model is derived from the dual-tower structure of the CLIP model.
[0087] Specifically, the signature processing model continues the classic multimodal model, the CLIP model's dual-tower structure design, and retains the independent feature extraction capabilities of the image branch and the text branch.
[0088] In this scheme, the signature processing model is based on the dual-tower structure of the CLIP model, which retains the independent feature extraction capabilities of the image branch and the text branch, thus ensuring the accuracy and reliability of the signature processing model in feature extraction.
[0089] In one feasible approach, the sharpness classification head is a fully connected neural network, the general feature vector has a dimension of 512, and the sharpness discrimination result is a sharpness quantification.
[0090] Specifically, the core function of the clarity classification head is to achieve adaptive discrimination of signature clarity. Its output dimension is set to 512 dimensions, which serves as the basis vector for subsequent feature fusion and generates specific clarity quantification values through an additional output layer. The range of the clarity quantification values is mapped to [0,1].
[0091] A fully connected network, for example, can be a three-layer neural network. The first layer receives the basic features of the handwritten signature image and converts them into higher-dimensional features. The second layer further processes the high-dimensional features and learns the feature relationships. The third layer is the output layer, which outputs a 512-dimensional feature. Finally, an additional binary classification layer (512×2) is used to determine whether the handwritten signature image is clear.
[0092] In this scheme, a classification head composed of a fully connected neural network is used to ensure the accuracy and reliability of the output features.
[0093] In one feasible approach, image features are obtained by modulating the initial image features output from the image feature extraction layer of the signature processing model through residual connections.
[0094] Specifically, the image feature vector layer establishes a residual association with the previous basic image feature extraction layer through residual connections (shortcuts), and only fine-tunes and optimizes the basic initial image features. The reason for not directly using basic features in the fusion is to avoid redundant information contained in the original features, such as background noise and irregular stroke perturbations, which would affect the convergence stability of the overall training of the signature processing model. At the same time, the core representational ability of effective features is preserved through residual fine-tuning.
[0095] In this scheme, the initial image features are adjusted through residual connections to obtain the image, which avoids redundant information contained in the original features, retains the core representation ability of effective features, and improves the convergence stability of the overall training of the signature processing model.
[0096] In one feasible approach, text features are obtained based on a pre-trained handwritten OCR model.
[0097] Specifically, the OCR text feature vector layer extracts the last hidden state in the recurrent neural network from the text recognition task as a feature vector, and memorizes all the recognized text features.
[0098] In this scheme, the handwritten OCR model obtains text features, ensuring the accuracy and reliability of the text features.
[0099] In one feasible scheme, the step S103, which involves decoding the general feature vector to obtain the signature text, includes:
[0100] The signature image text recognition decoding layer of the signature processing model performs decoding operations on the general feature vector to obtain the signature text.
[0101] Specifically, the clarity of the obtained handwritten signature image is determined by a clarity classification head, and the generated mixed features are used to enter two different post-processing processes: clear scene or unclear scene. The mixed features are the general feature vector.
[0102] For clear scenarios, the signature text is obtained by directly decoding the general feature vector.
[0103] In this scheme, the signature image text recognition decoding layer of the signature processing model is used to decode the general feature vector to obtain the signature text, thus ensuring the accuracy and reliability of the signature text.
[0104] In a feasible solution, such as Figure 2 As shown, step S103, which involves matching the general feature vector with the standard features in the signature database to obtain the signature text, includes the following steps:
[0105] S1031. Perform graph-to-graph matching between the general feature vector and the features corresponding to the signature graph in the signature library;
[0106] If the matching degree is higher than the first matching degree threshold, then proceed to step S1032 to obtain the signature text;
[0107] If the matching degree is not higher than the first matching degree threshold, then proceed to step S1033 to obtain the candidate signature;
[0108] S1034. Perform image-text matching between candidate signatures and the roster;
[0109] If the matching degree is higher than the second matching degree threshold, then proceed to step S1032 to obtain the signature text;
[0110] If the matching degree is not higher than the second matching degree threshold, then proceed to step S1035: determine whether the candidate signature contains a handwritten signature image by using a preset large model.
[0111] If the candidate signature contains a handwritten signature image, then proceed to step S1032 to obtain the signature text;
[0112] If it cannot be determined or the candidate signature does not contain a handwritten signature image, then step S1036 is executed to trigger the feedback mechanism to obtain the signature text.
[0113] Specifically, the clarity of the obtained handwritten signature image is determined by a clarity classification head, and the generated mixed features are used to enter two different post-processing processes: clear scene or unclear scene. The mixed features are the general feature vector.
[0114] For unclear scenarios, the matching process proceeds to the signature database, which includes several signature images:
[0115] Prioritize image-to-image matching with signature images in the signature library. If the matching degree is higher than the first matching degree threshold, output the result directly, and use the signature image with a matching degree higher than the first matching degree threshold as the signature text; otherwise, generate a batch of candidate signatures. The candidate signatures are derived from the signature images in the image library. The matching degree between the candidate signature and the handwritten signature image is not higher than the first matching degree threshold, but higher than the third matching degree threshold, and the third matching degree threshold is lower than the first matching degree threshold.
[0116] Perform image-text matching between the candidate signatures and the roster. If the match score is higher than the second match score threshold, output the result directly, and use the text in the roster with a match score higher than the second match score threshold as the signature text; otherwise, retain the candidate signatures.
[0117] The system uses a pre-defined large model to determine whether candidate signatures contain a handwritten signature image. If a clear match is found, meaning the candidate signature contains a handwritten signature image, the result is output, yielding the signature text. If it cannot be determined or is clear that the candidate signature does not contain a handwritten signature image, a feedback process is initiated, triggering a feedback mechanism to obtain the signature text. The method for determining whether a candidate signature contains a handwritten signature image is to assess the matching degree between the candidate signature and the handwritten signature image. If the matching degree is higher than a fourth matching degree threshold, the candidate signature contains a handwritten signature image.
[0118] The feedback mechanism realizes a closed loop of abnormal recognition results, manual intervention, and image library iteration. When the preset large model determines that the signature text is invalid, including situations such as unclear signature matching failure or the preset large model being unable to determine, the feedback process is triggered, and the handwritten signature image to be verified is transferred to the manual annotation stage, where humans complete the accurate recognition and information completion of the signature.
[0119] Manually annotated standard signature data, including images and corresponding text, is added to the signature database, simultaneously updating the image-to-image and image-to-text matching sample databases. This mechanism achieves human-machine collaborative optimization. By manually supplementing annotated data and continuously iterating the image database, it addresses the blind spots in the signature processing model's recognition of low-quality signatures and less common handwriting styles. This improves the sample coverage and accuracy of subsequent matching tasks, enhancing the accuracy, adaptability, and robustness of the signature processing model for handwritten signature recognition. It balances efficiency and stability, adapting to complex real-world application scenarios.
[0120] The feedback loop not only corrects single recognition errors, but also gradually strengthens signature matching capabilities through continuous updates to the image library, achieving a dynamic balance between automation efficiency and reliability in complex scenarios.
[0121] In this solution, unclear handwritten signature images are matched multiple times to obtain the signature text, ensuring the accuracy and reliability of the signature text.
[0122] In a feasible solution, such as Figure 3As shown, after obtaining the signature text in step S103, the process further includes:
[0123] S104. The signature text is verified using a pre-set large model to obtain the verification result;
[0124] S105. In response to the verification result indicating that the verification failed, a feedback mechanism is triggered to obtain a new signature text.
[0125] Specifically, the post-processing path is selected based on the clarity judgment result, and either decoding or matching operations are used respectively. For the signature text recognized by these two recognition routes, the accuracy needs to be judged by a preset large model. If the verification is successful, the signature text is directly output; otherwise, the feedback path needs to be followed to obtain a new signature text through the feedback mechanism.
[0126] In addition, when using the signature processing model to process batch handwritten signature images, two post-processing paths are followed simultaneously, and the corresponding signature text is selected according to the clarity. That is, matrix calculation is used to select values in a targeted manner, thereby improving computational efficiency.
[0127] In this solution, the signature texts output by the two post-processing paths are verified by using a pre-set large model, thus ensuring the accuracy of the obtained signature texts.
[0128] In one feasible embodiment, step S101 includes:
[0129] The handwritten signature image in the document to be processed is located and extracted using a preset YOLO object detection model.
[0130] Specifically, the document to be processed is converted into an image format, and then the handwritten signature image is extracted. The YOLO object detection model is used to locate and extract individual handwritten signature images from the document to be processed.
[0131] In this solution, the handwritten signature image in the document to be processed is located and captured by a preset YOLO object detection model, which improves the effectiveness and reliability of the obtained handwritten signature image.
[0132] In one feasible approach, the signature processing model is obtained through three-stage training;
[0133] The first stage involves training the image feature extraction layer and text feature extraction layer of the signature processing model;
[0134] The second stage involves training the signature graph text recognition decoding layer and the text feature extraction layer of the signature processing model.
[0135] The third stage involves enhanced training of the signature image text recognition decoding layer.
[0136] Specifically, the first stage focuses on the basic training of the original image feature extraction layer and text feature extraction layer, with image-text matching loss as the core training objective. In this stage, the image feature extraction layer and text feature extraction layer are initialized using the weight parameters of the CLIP pre-trained model, and a triplet training framework is constructed. For the feature extraction layer of the 12-layer attention structure, only the middle 5th and 6th layers and the last 11th and 12th layers are trained, while the remaining attention layers are frozen. This ensures feature extraction capability while avoiding excessive parameter oscillations in the initial training stage.
[0137] The second stage uses the weight parameters obtained from the first stage training as the initialization basis, fixes the initial image feature extraction layer, and simultaneously promotes the joint optimization of the two training objectives, namely the CTC (Connectionist Temporal Classification) loss function of the signature image text recognition decoding layer, and the contrastive loss function of image-to-image and image-to-text matching of the text feature extraction layer.
[0138] Among them, the weights of the handwritten OCR model are used as the initialization parameters of the OCR text feature vector layer for OCR decoding training. During the training process, CTC loss is calculated only for samples with clarity higher than the set threshold. In the early stage of training, the label value of clarity is directly used to ensure the effectiveness of decoding training.
[0139] Considering that signature images are easily affected by complex environmental factors in real-world scenarios, simply training with image-to-image matching may lead the model to overemphasize irrelevant details such as environmental features and stroke thickness, significantly slowing down convergence. Therefore, an image-to-text matching training objective is introduced to correct the focus of feature extraction. Image-to-text matching needs to mine the similarity features between images and text, naturally tending to focus on core information such as the strokes and handwriting of the signature itself, effectively avoiding environmental interference. However, for some symbol classes and signatures that are difficult for humans to distinguish, image-to-text matching cannot extract corresponding text features, easily causing the loss of core features. Therefore, a joint training approach with two matching objectives is adopted, assigning image-to-text matching a weight of 0.7 and image-to-image matching a weight of 0.3 in the loss function design. It is particularly important to note that there are many training objectives at this stage, and the reasonable initialization of weight parameters is crucial, directly affecting whether the model can converge smoothly. The total loss function at this time is defined as: Total loss = Classification head loss + CTC loss + 0.7 × Image-to-text matching loss + 0.3 × Image-to-image matching loss. Through multi-loss collaborative optimization, the effectiveness and stability of model training are further improved. The classification head loss corresponding to the clarity quantification is incorporated into the total loss function, enabling joint training of feature extraction and clarity discrimination. This ensures that the signature processing model simultaneously optimizes the accuracy of signature quality discrimination and the effectiveness of feature extraction. Through collaborative optimization of multiple loss functions, the goals of image-text matching and image-image matching are balanced, avoiding environmental interference and the loss of core features, thereby improving training effectiveness and stability.
[0140] The classification head loss uses the cross-entropy loss function, and the corresponding formula is as follows:
[0141] ;
[0142] Among them, L bce Represents the classification head loss, y i Let p be the true label of the i-th sample, 0 / 1. i The probability value of a sample being 1 is output by the signature processing model, where N represents the number of samples;
[0143] The loss function formula used in CTC loss is as follows:
[0144] ;
[0145] Among them, L CTC Represents CTC loss, y i Let x be the true label sequence of the i-th sample. i For the i-th input sequence, B -1 This indicates that after deduplication and whitespace removal, the predicted sequence equals the true label sequence, p is the probability of generating this sequence, N represents the number of samples, and π represents a path with a length proportional to x.i The number of time steps is the same, and a symbol is output at each time step;
[0146] The formula for calculating the contrast loss function is as follows:
[0147] ;
[0148] Among them, L contrast The comparison loss function is represented by sim, which indicates the calculation of cosine similarity, and z represents the comparison loss function. i Let z represent the feature vector of the i-th sample. i + Indicates with z i The corresponding feature vector of the positive sample, where N represents the number of samples; τ represents the temperature coefficient, a positive scalar used to adjust the distribution of similarity.
[0149] The third stage focuses on enhancing the training of the signature image text recognition decoding layer, while continuing to freeze the image feature extraction layer and only conducting targeted training on the signature image text recognition decoding layer.
[0150] For model training, a clarity classification dataset and a matching task-specific dataset are constructed, and positive and negative triplet pairs and mixed sampling methods are used to support the model in accurately learning core features.
[0151] The sharpness classification dataset is used to assist in classifying the sharpness of handwritten signature images. Based on the signature processing model trained in the first stage, it performs vectorized matching between handwritten signature images and corresponding text. The matching degree indirectly reflects the sharpness of the handwritten signature image. For example, with 10,000 labeled handwritten signature images, the matching degree is calculated based on the signature processing model. If the matching degree is lower than a threshold, it indicates that the signature is illegible and difficult to recognize, and is classified as unclear; otherwise, it is classified as clear.
[0152] The dedicated dataset for matching tasks, namely the graph-to-graph / graph-to-text matching dataset, is constructed using positive and negative triplet pairs. The format of positive samples is (original signature image, signature in the image, other images with the same signature), and the label is set to (1,1). The format of negative samples is (original signature image, signature not in the image, other signature images), and the label is set to (0,0). It is used to support the training of the two types of matching tasks.
[0153] The decoding layer augmentation training dataset is constructed to supplement the decoding layer training data. It collects an additional batch of single-line text images on top of the existing data to help improve the decoding layer's recognition capabilities.
[0154] The training data is assembled using a mixed sampling method, which includes both clear and unclear signature images in a single batch. When calculating the loss, the matrix calculation is used to select values based on the clarity of the samples in the CTC loss function and the contrastive loss function results, thereby improving computational efficiency.
[0155] The use of dedicated datasets and hybrid sampling methods ensures data diversity and helps the signature processing model accurately learn the core features of the signature.
[0156] In this scheme, a three-stage training approach is adopted, which focuses on basic feature extraction, multi-loss joint optimization, and decoding layer enhancement training in stages. Reasonable initialization of weights and freezing strategy ensure the convergence stability of the signature processing model.
[0157] The working principle of the handwritten signature processing method in this embodiment is explained below with specific examples:
[0158] like Figure 4 As shown, a signature processing model is obtained through three stages of training. The model includes an image branch, a text branch, a feature output layer, and a signature image text recognition decoding layer. The image branch includes an image feature extraction layer, a classification head module, a gated neural network, and an image feature vector layer. The text branch includes a text feature extraction layer and an OCR text feature vector layer. The first stage trains the image feature extraction layer and the text feature extraction layer, with image-text matching loss as the core training objective. The second stage trains the signature image text recognition decoding layer and the text feature extraction layer. Figure 4 The middle decoding layer requires training in both image-text matching and image-image matching for the text feature extraction layer. The third stage involves enhanced training of the signature image-text recognition decoding layer.
[0159] The handwritten signature image is processed by an image feature extraction layer to extract basic image features, and a text feature extraction layer to extract basic text features. These basic image features are then input into a classification head module, which outputs a basic vector and a sharpness determination result. The sharpness determination result indicates whether the handwritten signature image is clear.
[0160] The basic vectors are input into the gated neural network, i.e. Figure 4 In this context, gating is used to dynamically adjust the fusion weight ratio of the image feature vector layer and the OCR text feature vector layer in the gated neural network.
[0161] The feature output layer, based on the dynamic weights of a gated neural network (weight 1 and weight 2), performs a weighted vector summation operation on the image features output from the image feature vector layer and the text features output from the OCR text feature vector layer, outputting a hybrid feature, i.e., a general feature vector. Based on the sharpness judgment result, if the sharpness exceeds a sharpness threshold, the hybrid feature is decoded using a signature image text recognition decoding layer.
[0162] like Figure 5As shown, for the document to be processed, a handwritten signature image is located and captured using a preset YOLO object detection model. Based on the sharpness judgment result output by the classification head module of the signature processing model, if the sharpness is higher than the sharpness threshold (i.e., a sharp scene), the handwritten signature image is decoded to obtain the signature text; if the sharpness is lower than or equal to the sharpness threshold (i.e., an unclear scene), the handwritten signature image is matched to obtain the signature text. The signature text obtained from the decoding or matching operations is verified using a preset large model. If the verification passes, the obtained signature text is output; if the verification fails, a feedback mechanism is triggered.
[0163] The matching process is as follows:
[0164] First, perform graph-to-graph matching with the signature graphs in the signature database. If the matching degree is higher than the first matching degree threshold, output the result directly to obtain the signature text; otherwise, generate a batch of candidate signatures.
[0165] Then, the candidate signatures are matched against the roster. If the match score is higher than the second match score threshold, the result is output directly to obtain the signature text; otherwise, the candidate signature is retained.
[0166] Finally, a pre-defined large model is used to determine whether the candidate signatures contain a handwritten signature image. If a clear match is found, i.e., the candidate signature contains a handwritten signature image, the result is output, yielding the signature text; if it cannot be determined or is clear that the candidate signature does not contain a handwritten signature image, a feedback mechanism is triggered to obtain the signature text.
[0167] In the feedback mechanism, the signature text of the handwritten signature image is manually annotated, and the standard signature data containing the image and corresponding text after manual annotation is added to the signature database.
[0168] Example 2
[0169] Corresponding to the aforementioned embodiments of handwritten signature processing methods, this embodiment provides an embodiment of a handwritten signature processing system, such as... Figure 6 As shown, the processing system includes:
[0170] Signature image acquisition module 1 is used to acquire the handwritten signature image to be processed;
[0171] The signature image processing module 2 is used to process the handwritten signature image through a trained signature processing model to obtain the sharpness discrimination result and general feature vector of the handwritten signature image. The signature processing model includes an image branch, a text branch, a sharpness classification head and a gated neural network. The general feature vector is obtained by weighted fusion of image features from the image branch and text features from the text branch through the gated neural network.
[0172] The signature text acquisition module 3 is used to select the post-processing path based on the clarity judgment result. If the clarity is higher than the clarity threshold, the general feature vector is decoded to obtain the signature text. If the clarity is lower than or equal to the clarity threshold, the general feature vector is matched with the standard features in the signature library to obtain the signature text.
[0173] In this scheme, the signature processing model is derived from the dual-tower structure of the CLIP model;
[0174] And / or,
[0175] The sharpness classification head is a fully connected neural network with a general feature vector dimension of 512, and the sharpness discrimination result is a sharpness quantification value;
[0176] And / or,
[0177] Image features are obtained by adjusting the initial image features output by the sharpness classification head through residual connections;
[0178] And / or,
[0179] The text features are obtained based on a pre-trained handwritten OCR model.
[0180] In this solution, the signature text acquisition module 3 includes:
[0181] Decoding unit 31 is used in the signature image text recognition decoding layer of the signature processing model to perform decoding operations on the general feature vector to obtain the signature text;
[0182] And / or,
[0183] Signature text acquisition module 3 includes:
[0184] The graph matching unit 32 is used to perform graph matching between the general feature vector and the features corresponding to the signature graph in the signature library. If the matching degree is higher than the first matching degree threshold, the signature text is obtained; if the matching degree is not higher than the first matching degree threshold, the candidate signature is obtained.
[0185] Image-text matching unit 33 is used to perform image-text matching between candidate signatures and the roster. If the matching degree is higher than the second matching degree threshold, the signature text is obtained. If the matching degree is not higher than the second matching degree threshold, the candidate signature is judged by a preset large model to determine whether it contains a handwritten signature image.
[0186] The large model judgment unit 34 is used to obtain the signature text when the candidate signature contains a handwritten signature image, and to trigger a feedback mechanism to obtain the signature text when it cannot be judged or the candidate signature does not contain a handwritten signature image.
[0187] In this solution, the processing system also includes:
[0188] Verification processing module 4 is used to verify the signature text using a preset large model and obtain the verification result;
[0189] Feedback mechanism trigger module 5 is used to trigger the feedback mechanism to obtain a new signature text in response to the verification result indicating that the verification failed.
[0190] In this solution, the signature image acquisition module is also used to locate and capture the handwritten signature image in the document to be processed using a preset YOLO object detection model.
[0191] In this scheme, the signature processing model is obtained through three-stage training;
[0192] The first stage involves training the image feature extraction layer and text feature extraction layer of the signature processing model;
[0193] The second stage involves training the signature graph text recognition decoding layer and the text feature extraction layer of the signature processing model.
[0194] The third stage involves enhanced training of the signature image text recognition decoding layer.
[0195] In this embodiment, the handwritten signature image is processed by a signature processing model to obtain the clarity judgment result of the handwritten signature image. Then, a post-processing path is selected based on the clarity judgment result to obtain the signature text. Setting the clarity judgment of the handwritten signature image in the signature processing model improves the detection efficiency, reduces the complexity of the recognition method, improves the recognition efficiency, can adapt to more scenarios, and reduces resource consumption.
[0196] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.
[0197] Example 3
[0198] Figure 7 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the handwritten signature processing method described in any of the above embodiments. Figure 7 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0199] like Figure 7 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).
[0200] Bus 93 includes a data bus, an address bus, and a control bus.
[0201] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.
[0202] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0203] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the handwritten signature processing method provided in any of the above embodiments.
[0204] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.
[0205] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.
[0206] Example 4
[0207] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the handwritten signature processing method provided in any of the above embodiments.
[0208] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.
[0209] Example 5
[0210] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the handwritten signature processing method described in any of the preceding embodiments.
[0211] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.
[0212] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.
Claims
1. A method for processing handwritten signatures, characterized in that, The processing method includes: Obtain the handwritten signature image to be processed; The handwritten signature image is processed by a trained signature processing model to obtain the sharpness discrimination result and general feature vector of the handwritten signature image. The signature processing model includes an image branch, a text branch, a sharpness classification head and a gated neural network. The general feature vector is obtained by weighted fusion of image features from the image branch and text features from the text branch through the gated neural network. Based on the clarity discrimination result, a post-processing path is selected. If the clarity is higher than the clarity threshold, the general feature vector is decoded to obtain the signature text. If the clarity is lower than or equal to the clarity threshold, the general feature vector is matched with the standard features in the signature library to obtain the signature text.
2. The handwritten signature processing method as described in claim 1, characterized in that, The signature processing model is derived from the dual-tower structure of the CLIP model; And / or, The sharpness classification head is a fully connected neural network, the general feature vector has a dimension of 512, and the sharpness discrimination result is a sharpness quantification value; And / or, The image features are obtained by adjusting the initial image features output by the image feature extraction layer of the signature processing model through residual connections; And / or, The text features are obtained based on a pre-trained handwritten OCR model.
3. The handwritten signature processing method as described in claim 1, characterized in that, The step of decoding the general feature vector to obtain the signature text includes: The signature text is obtained by using the signature image text recognition and decoding layer of the signature processing model to decode the general feature vector. And / or, The step of matching the general feature vector with standard features in the signature library to obtain the signature text includes: The general feature vector is matched with the features corresponding to the signature graph in the signature library. If the matching degree is higher than the first matching degree threshold, the signature text is obtained. If the matching degree is not higher than the first matching degree threshold, the candidate signature is obtained. The candidate signature is matched with the roster. If the matching degree is higher than the second matching degree threshold, the signature text is obtained. If the matching degree is not higher than the second matching degree threshold, the candidate signature is determined by a preset large model to determine whether the handwritten signature image is included. If the candidate signature contains the handwritten signature image, the signature text is obtained; if it cannot be determined or the candidate signature does not contain the handwritten signature image, a feedback mechanism is triggered to obtain the signature text.
4. The handwritten signature processing method as described in claim 1, characterized in that, After obtaining the signed text, the method further includes: The signature text is verified using a pre-defined large model to obtain the verification result; In response to the verification result indicating that the verification failed, a feedback mechanism is triggered to obtain a new signature text.
5. The handwritten signature processing method as described in claim 1, characterized in that, The steps for obtaining the handwritten signature image to be processed include: The handwritten signature image is located and extracted from the document to be processed using a preset YOLO object detection model.
6. The method for processing handwritten signatures as described in any one of claims 1-5, characterized in that, The signature processing model is obtained through three-stage training. The first stage involves training the image feature extraction layer and text feature extraction layer of the signature processing model; The second stage involves training the signature graph text recognition and decoding layer and the text feature extraction layer of the signature processing model. The third stage involves enhancing the training of the signature image text recognition and decoding layer.
7. A handwritten signature processing system, characterized in that, The processing system includes: The signature image acquisition module is used to acquire the handwritten signature image to be processed; A signature image processing module is used to process the handwritten signature image through a trained signature processing model to obtain the sharpness discrimination result and a general feature vector of the handwritten signature image. The signature processing model includes an image branch, a text branch, a sharpness classification head, and a gated neural network. The general feature vector is obtained by weighted fusion of image features from the image branch and text features from the text branch through the gated neural network. The signature text acquisition module is used to select a post-processing path based on the clarity discrimination result. If the clarity is higher than the clarity threshold, the general feature vector is decoded to obtain the signature text. If the clarity is lower than or equal to the clarity threshold, the general feature vector is matched with the standard features in the signature library to obtain the signature text.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the handwritten signature processing method according to any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the handwritten signature processing method according to any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the handwritten signature processing method as described in any one of claims 1-6.