Handwriting extraction system, method and equipment based on multiple prior information fusion and medium

Abstract

The invention discloses a handwriting extraction system based on multiple prior information fusion. A prior information processing module encodes signature metadata and content image information into text and image prompt features based on text and image prompt generated by setting segmentation conditions; the multi-modal information fusion module comprises a plurality of encoders and a multi-head cross attention mechanism module, and is used for fusing text and image prompt features with original to-be-segmented signature image features to obtain fusion features; the feature decoder performs feature decoding on the fusion feature to obtain a segmentation mask; and the post-processing optimization module processes and corrects the handwriting corresponding to the segmentation mask. According to the method, multiple prior information is fused, and through joint guidance of text and image prompts, the robust understanding and characterization capability of the model on handwriting attributes is improved, the segmentation performance in a complex practical application scene is effectively enhanced, and controllable, conditional and explainable fine segmentation of handwriting forms is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a handwriting extraction technology based on the fusion of multiple prior information. Background Technology

[0002] Handwriting segmentation is a key task in document image processing, aiming to accurately extract handwriting pixels from the original signature image and separate the handwriting from the background. This technology is widely used in business scenarios such as digital signature verification and authentication; document scanning and OCR processing; signature recognition and identity authentication; and document archiving and digitization. In practical applications, handwriting segmentation faces the following technical problems and challenges:

[0003] Data dependency and overfitting. Existing deep segmentation models (such as U-Net and SegNet) are prone to overfitting under small sample conditions of handwritten signatures, have poor generalization across data sources, and are difficult to stably transfer to different shooting devices, paper and lighting conditions;

[0004] Insufficient controllability and interpretability. End-to-end "black box" learning lacks explicit constraints on attributes such as color, thickness, and cursive, making it difficult to conditionally control them by "segmenting only a person's signature / segmenting only a certain color";

[0005] Insufficient robustness against complex backgrounds. Printed text, underlines, and paper textures are easily missegmented as foreground elements under low contrast or non-uniform lighting, resulting in inadequate restoration of boundary details.

[0006] Semantic-geometric coordination is challenging. It is difficult to balance global semantic understanding with local detail representation; encoder downsampling loses details, and upsampling (transposed convolution) cannot fully restore boundaries.

[0007] It has poor cross-domain adaptability. It lacks compatibility with unfamiliar signers, backgrounds, and devices, resulting in poor stability in actual deployments.

[0008] For example, Chinese invention patent application CN118628746A discloses a text-guided image segmentation method that integrates and interacts with image and text information. With the assistance of asymmetric bilateral cue blocks, text tags are pruned by image guidance to retain optimized information to guide segmentation, while image tags utilize prior text information to enhance feature representation and target prediction. In the decoding stage, an image decoder consisting of N text-guided decoding blocks and a final decoding block is used to upsample the image tags with the help of text information. The output is transmitted through a segmentation head to obtain the image segmentation result of the infected region.

[0009] Chinese invention patent application CN115861618B discloses a single-stage end-to-end panoramic narrative segmentation method. It employs a convolutional neural network to extract and fuse image features at multiple scales; utilizes a transformer-based bidirectional encoder representation technique to extract word vectors from the text, and filters out non-noun word vectors using dataset annotations. For cases where multiple word vectors correspond to the same noun, the average value is used as the feature corresponding to the current noun; embedding local spatial priors into attention modeling enhances visual features at different scales, thereby improving segmentation performance; using the extracted text features as convolution kernels, convolution is performed on the resulting feature map to obtain a mask M; a bidirectional contrastive learning loss is used to regulate semantic consistency between modalities; and other loss functions are added to make the predicted mask's value and distribution approximate the true mask.

[0010] Chinese invention patent application CN202411365663.3 (A Handwriting Sample Extraction System and Method) describes a complete process of "importing target image - preprocessing - processing based on font spacing parameters / font complexity parameters - outputting handwriting samples". It uses indicators such as "font spacing parameters and font complexity parameters" to assist in sample segmentation and extraction, reflecting the parameterized characterization of layout / font complexity, which helps improve the consistency of batch sample generation. However, its sample generalization ability under different devices, paper, and lighting conditions is limited. Although it mentions pixel-level handwriting segmentation and boundary refinement in complex background documents, it does not directly solve the problem of fine-grained foreground masking under the interference of printed text, underlines, paper texture, and shadows.

[0011] The main problems with the existing technologies mentioned above are: 1) The types of prior information are relatively simple, lacking systematic modeling and explicit fusion of multiple priors such as original scene images and acquisition priors; 2) Multimodal interactions are generally designed around single image branches and single text branches, making it difficult to directly accommodate dual image sources and acquisition priors; 3) In scenarios such as handwriting extraction that strongly depend on fine-grained structures and acquisition conditions, existing methods are insufficient in mining structural priors and acquisition priors other than text priors.

[0012] Based on extensive technical research and real-world data analysis, the current technical challenges and bottlenecks regarding handwriting style for paper signatures are as follows:

[0013] (1) The diversity of handwriting attributes significantly increases the difficulty of segmentation.

[0014] 1) Variations in handwriting color and texture: In practical applications, signature handwriting encompasses various colors such as black (most common), blue (common ballpoint pen), and red (annotations / special uses). Different colors exhibit varying contrast and appearance in images. Traditional segmentation models often struggle to accurately identify handwriting of various colors simultaneously, leading to a sharp decline in segmentation performance in multi-color mixed scenarios.

[0015] 2) Handwriting thickness and writing variation: The same person's signature may have significant differences in handwriting thickness and continuity due to the use of different writing tools (such as fountain pen, ballpoint pen, brush) or changes in writing pressure. It is difficult for the model to uniformly model all possible morphological features, and the accurate extraction of details and boundaries is particularly easily affected.

[0016] 3) Diversity of signature content: The handwritten signatures of different users vary greatly, and the stroke direction and continuity of the same user also fluctuate at different times, occasions, and moods. The model not only needs to recognize the handwriting itself, but also needs to distinguish the signature content from the background and interference, so as to deal with complex situations such as multiple signatures, overlaps, and deformations.

[0017] (2) Bottlenecks in segmentation model technology

[0018] 1) Feature level mismatch: Conventional deep segmentation networks lose key geometric details during downsampling feature extraction, resulting in blurred boundaries; while upsampling (such as transposed convolution) restoration processes struggle to fully recover the original handwriting edges, affecting segmentation accuracy and reliability. The detail of handwriting boundaries directly determines the quality of segmentation evaluation metrics, and existing structures struggle to balance macroscopic semantics with microscopic details.

[0019] 2) Balancing global semantics and local details is challenging: The model needs to understand the overall handwriting distribution (macro-level semantic recognition) while also preserving pixel-level boundaries and subtle local divisions. Traditional convolutional neural networks face an irreconcilable contradiction between receptive field expansion and locality preservation, leading to uneven trade-offs between small-region feature representation and overall image semantic control, ultimately affecting segmentation consistency.

[0020] 3) Insufficient cross-domain generalization ability: Existing segmentation models generally rely on a single data source for training and lack effective adaptation strategies for different signers, various backgrounds, and multiple types of acquisition devices. When transferring from one data source to a new scenario, the model performance drops significantly, lacking flexible data augmentation and domain adaptation mechanisms. Summary of the Invention

[0021] To address the aforementioned problems, this invention proposes a handwriting extraction method based on the fusion of multiple prior information. Targeting the challenge of handwriting segmentation in complex backgrounds, it combines deep learning with human prior information to design a handwriting segmentation model that integrates multimodal cues (text and image cues). A structured text and image cue strategy is constructed, encoding signature metadata and content image information into natural language cues. The CLIP pre-trained encoder is fine-tuned to adapt to the specific requirements of handwriting segmentation, and multimodal prior information is fused based on the fine-tuned feature encoder. Semantic understanding is enhanced, and conditional handwriting extraction is performed based on semantics. A post-processing optimization module is added to eliminate interference and enhance the aesthetics of the segmented handwriting. This significantly improves the accuracy and robustness of handwriting segmentation.

[0022] Based on the first aspect of this invention, a handwriting extraction system based on multiple prior information fusion is proposed, comprising: a prior information processing module, a multimodal information fusion module, a feature decoding module, and a post-processing optimization module. The prior information processing module encodes signature metadata and content image information into text and image prompt features based on text and image prompts generated according to set segmentation conditions. The multimodal information fusion module includes multiple encoders and a multi-head cross-attention mechanism module, which fuses the text and image prompt features with the original signature image features to be segmented to obtain fused features. The feature decoder performs feature decoding on the fused features to obtain a segmentation mask. The post-processing optimization module processes and corrects the handwriting corresponding to the segmentation mask.

[0023] Further preferred, the handwriting extraction system is a branched multimodal fusion structure of "dual image encoder + text encoder + multi-head cross attention", in which the content prior image, real scene image and text prior are collaboratively constrained under the same attention framework to achieve the segmentation result.

[0024] Further optimization involves constructing an image encoder and a text encoder based on the CLIP module. The CLIP image encoder 1 extracts image prior prompt features based on printed images of the same content through the content prior unit. The CLIP image encoder 2 obtains the signature image features of the paper signature to be extracted through the original image unit. The CLIP text encoder obtains the text prompt word information and acquires text prior prompt features through the acquisition prior unit.

[0025] Further optimization involves using a Transformer decoder as the core semantic understanding and segmentation engine, receiving high-dimensional features output from the multimodal fusion module. This is transformed into a clear pixel-level segmentation result, resulting in a segmentation mask.

[0026] Further optimization involves employing a multi-level self-attention segmentation modeling feature decoder. The decoder uses a deeply stacked Transformer structure, with each layer containing a self-attention module and a feedforward network. Through multi-path parallel attention heads, implicit associations in different semantic spaces are extracted, and the segmentation target, color, and interference type defined by text / image cues are fused into the global context to achieve dynamic conditional guidance for segmentation behavior. The attention weights of each layer are adjusted according to the fused features, focusing on the condition-specified regions and boundaries.

[0027] Further optimization involves the output of all self-attention modules following a set of deep feedforward fully connected networks to perform fine-grained pixel interpretation, boundary sculpting, and attribute differentiation for each local region; all segmentation feature layers employ residual connections and normalization to overlay deep semantic feature information.

[0028] Further optimization involves the prior information processing module constructing a general template for text prompts and generating text prompts for signature images. The encoder encodes the text and image prompts generated under the above-mentioned segmentation conditions. The prompt dimensions for the text prompts include: acquisition method, handwriting color, signature content, and interference type. Conditional segmentation is implemented for multiple prompt modes, including: segmenting specified signature content, segmenting all signature content, segmenting specified handwriting color, and segmenting the specified method of the general template.

[0029] Further optimization involves fine-tuning the CLIP encoder with domain samples to improve its sensitivity to signature content, font structure, and background interference. The "text and image prompt features obtained from prior encoding" are fused with the "features of the original signature image to be segmented" to achieve controllable conditional segmentation. Feature extraction and information fusion are pre-trained with domain samples to establish a local domain fine-tuning strategy. A local training set of "signature image + text prompt + standard printed prior" is constructed. Gradient descent is used to fine-tune the newly added decoder and some encoder layers, and a phased unfreezing strategy is adopted to gradually update the parameters.

[0030] Based on the second aspect of this application, a handwriting extraction method based on multi-prior information fusion is proposed, comprising: a prior information processing module encoding signature metadata and content image information into text and image prompt features based on text and image prompts generated according to set segmentation conditions; a multi-modal information fusion module fusing the text and image prompt features with the original signature image features to be segmented to obtain fused features; a feature decoder performing feature decoding on the fused features to obtain a segmentation mask; and a post-processing optimization module processing and correcting the handwriting corresponding to the segmentation mask.

[0031] Further optimization involves the following steps in the multimodal information fusion module: the content prior unit extracts image prior prompt features based on printed images of the same content; the original image unit obtains signature image features from the paper signature image to be extracted; and the text prior prompt features are obtained by acquiring text prompt word information through the prior unit.

[0032] Further optimization involves the feature decoder capturing long-distance dependencies between different regions of the image through a multi-layer self-attention mechanism, thus integrating text / image cues. and The defined segmentation target, color, and interference type are integrated into the global context to achieve dynamic conditional guidance for segmentation behavior; the attention weights of each layer are adjusted according to the fusion features to focus on the specified regions and boundaries.

[0033] Further optimization involves the prior information processing module generating text prompts for the signature image based on a general text prompt template. The encoder encodes the text and image prompts generated under the above-mentioned segmentation conditions, setting text prompts based on the acquisition method, handwriting color, signature content, and interference type, and setting conditional segmentation with multiple prompt modes.

[0034] Based on a third aspect of this application, an electronic device is proposed, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method described in any of the preceding claims.

[0035] Based on a fourth aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method described in any of the preceding claims.

[0036] This invention proposes a handwriting extraction method based on multiple prior information fusion. In terms of prior utilization, it upgrades from a two-modal prior of "single image + text description" to a multi-prior joint modeling of "content printed image + original signature image + text acquisition prior," explicitly introducing content form prior and acquisition condition prior for fine-grained handwriting extraction. In terms of model structure, it expands from a two-branch structure of "single image encoder + single text encoder + asymmetric bilateral cue block" to a three-branch multimodal fusion structure of "dual image encoder + text encoder + multi-head cross-attention," enabling the content prior image, real-scene image, and text prior to collaboratively constrain the segmentation results under the same attention framework. In terms of task adaptation, for special targets like handwriting / signatures with structural, fine-grained, and high-noise backgrounds, it introduces a printed content prior and post-processing optimization module to improve the shortcomings of existing text-guided segmentation methods in such scenarios regarding structural continuity, stroke integrity, and noise suppression.

[0037] To address the challenges and problems of existing technologies, this invention aims to improve the model's robust understanding and representation of handwriting attributes through joint guidance of text and image cues. This significantly enhances segmentation performance in complex real-world application scenarios, achieving controllable, conditional, and interpretable fine-grained segmentation of handwriting morphology. By integrating multiple prior information and using joint guidance of text and image cues, the model's robust understanding and representation of handwriting attributes is improved, preserving pixel-level boundaries and subtle local separations. It balances the relationship between small-region feature representation and overall image semantics. The model is trained using real-world application scenarios as priors, effectively enhancing segmentation performance in complex scenarios and achieving controllable, conditional, and interpretable fine-grained segmentation of handwriting morphology. This effectively solves problems such as the significantly increased segmentation difficulty caused by the diversity of handwriting attributes, easy loss of data, blurred boundaries, and impact on segmentation accuracy and reliability. Attached Figure Description

[0038] Figure 1 A schematic diagram of the overall architecture of the handwriting extraction system based on the fusion of multiple prior information in this embodiment of the invention;

[0039] Figure 2 A schematic diagram illustrating the use of the Cross-Attention mechanism to fuse image and text features in this embodiment of the invention;

[0040] Figure 3 This is a structural block diagram of an exemplary electronic device that can be used to implement embodiments of this application. Detailed Implementation

[0041] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0042] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0043] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0044] The terms “a” and “a plurality” used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as “one or more”.

[0045] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0046] This invention proposes a handwriting extraction system based on multi-prior information fusion, supporting flexible segmentation under different conditions. In terms of prior utilization, it upgrades from a two-modal prior of "single image + text description" to a multi-prior joint modeling of "content printed image + original signature image + text acquisition prior," explicitly introducing content form prior and acquisition condition prior for fine-grained handwriting extraction. In terms of model structure, it expands from a two-branch structure of "single image encoder + single text encoder + asymmetric bilateral cue block" to a three-branch multimodal fusion structure of "dual image encoder + text encoder + multi-head cross-attention," enabling the content prior image, real-scene image, and text prior to collaboratively constrain the segmentation results under the same attention framework. In terms of task adaptation, for special targets such as handwriting / signatures with structural, fine-grained, and high-noise backgrounds, a printed content prior and post-processing optimization module is introduced to improve the shortcomings of existing text-guided segmentation methods in such scenarios regarding structural continuity, stroke integrity, and noise suppression.

[0047] Specifically, it includes: a prior information processing module, a multimodal information fusion module, a feature decoding module, and a post-processing optimization module. Prior information processing includes content priors, the original image, and acquisition priors. The multimodal information fusion module includes multiple encoders and a multi-head cross-attention mechanism module. The feature decoding unit uses a Transformer decoder, and the post-processing optimization module processes and corrects the extracted handwriting. In the multimodal information fusion module, CLIP image encoder 1 obtains prior image features from printed images with the same content through content priors. CLIP image encoder 2 obtains signature image features from the original image unit to extract the paper signature image. The CLIP text encoder obtains prior text features from text prompts through the acquisition prior unit. These features are input into the multi-head cross-attention mechanism module for multimodal feature information fusion, and the fused features are decoded using a Transformer decoder.

[0048] The prior information processing module performs conditional segmentation on the signature image based on text prompts.

[0049] 1) Text prompt word design and encoding

[0050] A general template for text prompts is constructed to generate text prompts for signature images. The encoder encodes the text and image prompts generated under the above-defined segmentation conditions.

[0051] This includes: for example, the signature image's "{collection method}, {signature content}, {handwriting color}, and {interference type}". The setting prompt dimensions include: collection method (photograph, document scanner), handwriting color (black, blue, red), signature content (username or category), and interference type (background texture, printed text, underline, etc.).

[0052] Conditional segmentation enables conditional segmentation for various prompt modes, including: segmenting specified signature content, segmenting all signature content, segmenting specified handwriting color, and segmenting a general template by specified method.

[0053] You can set the following splitting conditions to split the data according to the specified criteria.

[0054] a. Mode 1: Segment the signature of a specified user, and extract the specified signature content from the collected image, such as extracting the specified signature content "Zhang San" from the signature image;

[0055] b. Mode 2: Segment all signature content, such as: segment all signature content from the collected image;

[0056] c. Mode 3: Segmenting a specified handwriting color, such as segmenting black handwriting from a captured image;

[0057] d. Mode 4: General mode prompt, "This is a signature image from a document scanner. The signature reads 'Zhang San,' in black ink, with background texture as interference."

[0058] Add additional segmentation information: Further granularity of information can be added to the text dimension. Relevant information can be added according to a general template, such as the following:

[0059] a. Data collection method: brand and model of data collection equipment, data collection time, lighting conditions, etc.;

[0060] b. Writing attributes: thickness of the signature strokes, type of writing tool (ballpoint pen, brush, fountain pen, etc.);

[0061] c. Signature metadata: signer identification code, signature content description, file batch ID, etc.;

[0062] d. Environmental interference: the document material of the signature file, the background mode of the signature file, and specific typographic or underlined marks.

[0063] A text encoder is constructed based on the CLIP model to encode the text information of the text prompts generated under the above-mentioned segmentation conditions.

[0064] a. Input the generated text prompts one by one into the text encoder built on the classic Transformer Text Encoding Network (CLIP) model. Use the Byte Pair Encoding (BPE) method or the text segmentation and tokenization process to segment and embed the text sequence to obtain a fixed-length high-dimensional semantic feature vector.

[0065] b. Select Chinese / English vocabulary lists, and add positional embedding after word segmentation to ensure effective integration of semantics and sequence structure;

[0066] c. For multiple prompt words, one-hot / embedded efficient encoding representations of discrete categories can be concatenated or input in stages, supporting multi-condition joint prompts. Text-level concatenation is prioritized because for multiple prompt words, a complete sentence is synthesized at the text level according to natural language logic before being fed into the CLIP tokenizer and text encoder, rather than arbitrarily adding or concatenating the output vectors of CLIP. This maintains greater consistency between the model and the pre-trained distribution during model use. The staged principle involves inputting multiple conditions sequentially, such as decomposing according to semantic hierarchy: based on splitting from "global to local, from stable to adjustable," for example:

[0067] The first stage involves inputting scenario / category prompts (such as category labels and scenario descriptions) to form basic semantic conditions.

[0068] The second stage input supplements / refines the cues (such as style, color, target attributes), and modulates or reweights the features from the first stage through another network or attention module.

[0069] An image encoder is built based on the CLIP model to realize image prompting and encoding.

[0070] Image cues use "standardized printed images" as priors, especially to guide the model to understand the form of the signature content (such as generating printed samples that are consistent with the current signature content using a standard font library).

[0071] The image is normalized according to the "standardized printed image" standard in terms of resolution and color space to obtain a standardized image, so as to eliminate the influence of redundant variables and improve the stability and transferability of visual features. For example, the normalized image is a grayscale image with a size of 224*224.

[0072] An image encoder encodes standardized image information.

[0073] The image cues are normalized through preprocessing (such as scaling, normalization, brightness / contrast) to obtain a normalized image, which is then input into the CLIP image encoder (Visual Encoder).

[0074] The CLIP image encoder divides the standardized image into several patches (e.g., 16×16), converts them into fixed-length vectors, and extracts global and local visual features through a multi-layer Transformer self-attention module.

[0075] After encoding, high-dimensional semantic features are obtained, which can then be aligned with text prompts in the Cross-Attention and information fusion modules to facilitate semantic-visual conditional guidance.

[0076] Further optimization can be achieved by fine-tuning the CLIP encoder in conjunction with domain samples, enhancing domain adaptability and optimizing sensitivity to signature content, font structure, and background interference.

[0077] The multimodal information fusion module efficiently integrates the "text and image prompt features obtained from prior encoding" with the "original signature image features to be segmented" to achieve high-precision and controllable conditional segmentation.

[0078] Further steps may include two main components: pre-training with domain samples and feature extraction and information fusion.

[0079] The optimal choice is the CLIP pre-trained model. In this exemplary embodiment, the large-scale vision-language contrastive pre-trained model CLIP (Contrastive Language-Image Pre-Training) is pre-selected, which has powerful cross-modal semantic alignment capabilities. The image encoder can adopt Vision Transformer (ViT), and the text encoder is a multi-layer Transformer structure, both of which support high-dimensional feature output and flexible adaptation.

[0080] A local domain fine-tuning strategy is established, using a locally constructed training set of "signature image + text prompt + standard printed prior" to fine-tune the new decoder and some encoder layers using gradient descent (such as AdamW). A phased unfreezing strategy is adopted to gradually update the parameters. This allows the model to better adapt to the subdivision task of "target segmentation of paper signatures in complex backgrounds" while maintaining the original "image-text alignment" capability, and improves the controllability of boundary accuracy, robustness and attribute conditions (such as "signature area", "stamp" and "noisy background").

[0081] a. Data Adaptation: Prior data such as images of paper signatures in real-world scenarios (including multiple devices, lighting, and backgrounds), text prompts, and standard printed images are collected to construct a local segmentation target training set. This training set is used for supervised training and fine-tuning of the segmentation model. Through this training set, the model learns how to robustly separate the signature area from the non-signature area under complex paper backgrounds and different shooting conditions, and how to adjust the segmentation results according to the attributes / conditions specified in the text prompts (such as "segment only handwritten signatures, excluding stamped areas"), thereby achieving controllable conditional segmentation.

[0082] b. Employ a phased fine-tuning approach:

[0083] In the first stage, all parameters of the CLIP image encoder and text encoder are frozen, and only the newly added segmentation decoder and image-text feature fusion module are trained to learn segmentation rules in a fixed cross-modal semantic feature space.

[0084] In the second stage, the high-level Transformer blocks of the image encoder near the output end are gradually unfrozen and jointly trained with the decoder and fusion module. A group learning rate strategy is adopted, using a smaller learning rate on the unfrozen high-level encoder to slowly adjust its feature distribution so that it can adapt to the local feature space of paper signature and document scenarios.

[0085] In the third stage, based on the convergence of the first two stages, the parameters of the entire network are further unfrozen, and fine-tuning is performed in a small number of rounds using a smaller unified or grouped learning rate, thereby achieving efficient fusion of pre-trained knowledge and domain features. Here, "pre-trained knowledge" refers to CLIP's cross-modal semantic alignment capabilities and general visual / linguistic representations obtained on large-scale general image and text data, while "domain features" refers to the statistical characteristics of proprietary patterns such as stroke texture, paper and background noise, seals and patterns in paper signature and related document scenarios.

[0086] c. Setting the learning rate and loss function: A relatively small learning rate on the order of 1e-5 is adopted to gradually update the parameters of the pre-trained encoder with small steps, so as to reduce the risk of destroying the original semantic structure. In terms of the loss function, based on the combination of the standard BCELoss (binary cross-entropy loss function) and Dice Loss (overlap loss function based on Dice similarity coefficient), an attribute-aware weight mechanism is introduced: by assigning higher loss weights to pixels near the segmentation boundary and target regions corresponding to the attributes specified in the text prompt, the model can further strengthen the characterization and constraint of signature contours, key category regions and text conditional attributes while ensuring the overall segmentation accuracy, thereby significantly improving the quality of segmentation boundaries and the controllability of attribute conditions.

[0087] Efficient fusion of multimodal features.

[0088] Feature encoding stage. The original signature image to be segmented, the text prompt, and the standardized image prompt are all encoded separately by the CLIP model branch to obtain three high-dimensional semantic features, specifically:

[0089] a. Signature image →

[0090] b. Text prompt →

[0091] c. Image cues →

[0092] Using a multi-head cross-attention module, text prompt features are used as the Query (Q), original image prompt features as the Key (K), and prior image prompt features as the Value (V). The Query comes from... Key / Value comes from and This approach achieves the guidance and decoupling of semantic priors from visual features. By extracting implicit associations across different semantic spaces using multi-path parallel attention heads, high-dimensional cross-modal fusion is realized, yielding fused feature A.

[0093]

[0094] Implement conditional guidance and prior constraints.

[0095] This exemplary embodiment is as follows Figure 2 The diagram illustrates the use of a Cross-Attention mechanism to fuse image and text features. Specifically, the input text features act as the query in the multi-head attention mechanism, guiding the model to focus on relevant parts of the image, namely the text features obtained from the CLIP text encoder. As a "Query", the signature image and prompt features and As "Key and Value", the image features are used as both the retrieved object (key) and the actual content (value) in attention matching. The high-dimensional feature space "Attention" is obtained by fusing and aligning text and image through a multi-head cross-attention mechanism. After output through the Attention mechanism, the model obtains "fused features" that aggregate information from both modalities, preparing for downstream segmentation or understanding tasks.

[0096] In this exemplary embodiment of “multi-head cross attention”, the attention mechanism is a combination of parallel and multi-path approaches, making the interaction between the two types of features more refined and efficient.

[0097] The fused features are then processed through residual connections and multi-layer semantic transformations to obtain conditional segmentation features that combine global semantics and local details. This provides rich information for downstream decoders.

[0098] The feature decoder performs feature decoding on the "fused features". In this exemplary embodiment, the feature decoding uses a Transformer decoder, but conventional decoders in this field can also be used. This decoder serves as the core semantic understanding and segmentation engine, directly receiving the high-dimensional features output by the multimodal fusion module. This feature has been fused with features including the encoded original signature image. Semantic features of text prompts and standardized image cue features This rich fusion information is accurately transformed into pixel-level segmentation results that meet specific user needs, resulting in segmentation masks with flexible formats, enabling conditional and controlled fine-grained segmentation.

[0099] A multi-level self-attention segmentation modeling decoder is employed. This exemplary embodiment uses a deeply stacked Transformer structure, typically with 8 layers, each containing a self-attention module and a feedforward neural network. This multi-level self-attention mechanism captures long-distance dependencies between image regions, integrating text / image cues... and The defined segmentation target, color, and interference type are integrated into the global context to achieve dynamic conditional guidance for segmentation behavior. Based on the fusion prior, the attention weight of each layer is automatically adjusted, allowing for focused attention on conditionally specified regions and detailed boundaries, and providing strong targeted segmentation capabilities (such as segmenting only Zhang San or only the black region based on prompts).

[0100] Feedforward network and information stabilization mechanism. The output of all self-attention modules follows a set of deep feedforward fully connected networks to perform fine-grained pixel interpretation, boundary sculpting, and attribute differentiation for each local region. All segmentation feature layers employ residual connections and layer normalization to ensure efficient superposition of deep semantic feature information, improve gradient flow performance, effectively prevent gradient vanishing or exploding, and guarantee training stability and generalization effect. For overlapping, blurred, or low-contrast regions, it can automatically focus and correct, making it less likely to miss or misjudge small handwriting.

[0101] Multimodal conditional guidance and segmentation decision-making. Guided segmentation is performed using semantic control. The decoder transmits the segmentation target obtained during the encoding stage, precisely controlling the image to be segmented. A self-attention module aggregates global and local segmentation information to obtain high-precision and sharp boundaries. A segmentation mask with a flexible format is output.

[0102] Guided segmentation mechanism.

[0103] Semantic control: A clear segmentation target (such as specifying the signer, specifying the handwriting color, and specifying the exclusion of interference) is obtained in the fusion encoding stage. This clue is conditionally passed to the segmentation output by the Transformer decoder, so as to achieve fine and controllable task.

[0104] Boundary optimization: By leveraging self-attention to aggregate global and local information, the segmentation accuracy and boundary clarity are significantly higher than traditional methods, and it can handle complex backgrounds and diverse handwriting shapes.

[0105] Flexible output format: The final segmentation mask can be used as a single-class binary image or to generate multi-channel attribute masks (such as name channel and color channel). The results are automatically adjusted according to the input prompts to adapt to multi-task practical scenarios.

[0106] Training and inference are optimized collaboratively. During training, diverse prior data are used as training conditions to improve the cross-domain generalization and anti-interference ability of the multimodal information module; during inference, all input prompts (text, image priors) can affect the segmentation behavior in real time, and the system can further dynamically adjust various details based on the actual input information.

[0107] The soft decision-making process is linked to the segmentation output. The decoder restores the image to its original resolution using an upsampling structure (transposed convolution), and then transforms it into a probability distribution mask using a segmentation head (such as 1×1 convolution + sigmoid activation) to achieve pixel-level decision-making. Finally, it outputs a high-dimensional segmentation feature map.

[0108] Supports parallel segmentation with multiple heads: Employs a multi-head attention mechanism and multiple CLIP encoders to output an independent mask for each condition, addressing multiple business requirements (such as multiple signatures, multiple colors, and multiple backgrounds), thereby enhancing the ability of multi-task parallel inference.

[0109] The inference end can flexibly set the mask probability threshold to ensure the business adaptability and accuracy of the output target. For example, "a mask probability > 0.8 is a valid foreground", which meets the actual document fine extraction standards.

[0110] After obtaining a fine-grained segmentation mask through multimodal coding fusion and Transformer decoder, further "handwriting extraction post-processing" can be performed to enhance the business applicability and visual quality of the segmentation results. This ensures that the final output handwriting region is not only accurate and semantically clear, but also possesses smooth, full boundaries and reliable background noise suppression. Specifically, this includes...

[0111] Segmentation boundary smoothing and subject missing filling can utilize traditional image processing algorithms such as morphological operations to apply dilation, erosion, or opening / closing operations to the segmented handwriting regions, effectively correcting jagged edges and breakpoints caused by model prediction, resulting in natural handwriting edge transitions and full contours; dynamically adjust the size and shape of structural elements (such as circles, ellipses, and line segments) according to business needs to adapt to different handwriting styles, and optimize for thin and thick lines separately to prevent excessive smoothing from causing loss of detail information; for more complex stroke intersection and overlap scenarios, ensure subject continuity and segmentation integrity through region connectivity detection and boundary redrawing.

[0112] Background noise removal and interference suppression. Based on the segmentation results, small irrelevant areas (such as isolated pixels and scattered background fragments) are further detected and removed. Area thresholds are used to filter out parts that do not conform to the statistical characteristics of the signature subject. Adaptive filtering and median filtering are used to reduce background noise that may affect business processing, such as paper texture, blurred shadows, and residual printed text. Low-confidence regions in the deep learning segmentation results are suppressed according to probability masks (such as retaining only pixels with a mask greater than 0.8), further improving the purity and visual expressiveness of the target region.

[0113] Edge enhancement and visual beautification. Edge detection and enhancement algorithms (such as Canny and Sobel) are employed to improve the clarity of key stroke outlines, laying the foundation for subsequent digital storage, signature verification, or document beautification scenarios. Color restoration or filling strategies can be combined to maintain the color authenticity and consistency of each stroke fragment for "attribute segmentation" results (such as segmenting strokes of different colors), avoiding distortion or color block loss during processing. The final output must meet both the precision and aesthetic standards of professional document processing, ensuring smooth integration with various application scenarios (electronic documents, contract signing, forensic identification, etc.).

[0114] This invention designs a structured text and image prompting strategy, encoding signature metadata (acquisition device, handwriting color, signature content, interference type, etc.) and content image information into natural language prompts; it fine-tunes the CLIP pre-trained encoder to adapt it to the specific requirements of handwriting segmentation, and fuses multimodal prior information based on the fine-tuned feature encoder; it employs a Transformer decoder to enhance semantic understanding capabilities and performs conditional handwriting extraction based on semantics; and it further adds a handwriting extraction post-processing optimization module to eliminate interference and enhance the aesthetics of the segmented handwriting.

[0115] like Figure 3 The diagram shows a structural block diagram of an exemplary electronic device that can be used to implement embodiments of this application. The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0116] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, output unit 307, storage unit 308, and communication unit 309. Input unit 306 can be any type of device capable of inputting information to electronic device 300. Input unit 306 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 307 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 308 may include, but is not limited to, disk and optical disk. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0117] The computing unit 301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above. For example, in some embodiments, the reconstruction and decomposition of the muscle movement trajectory based on the original trajectory of the signature stroke, and the decomposition of its logarithmic velocity curve, can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. In some embodiments, the computing unit 301 can be configured in any other suitable manner to perform signature handwriting comparison and verification methods.

[0118] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0122] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0123] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A handwriting extraction system based on the fusion of multiple prior information, characterized in that, include: The system comprises a prior information processing module, a multimodal information fusion module, a feature decoding module, and a post-processing optimization module. The prior information processing module encodes the signature metadata and content image information into text and image prompt features based on the text and image prompts generated by the prior information processing module according to the set segmentation conditions. The multimodal information fusion module includes multiple encoders and a multi-head cross-attention mechanism module, which fuses the text and image prompt features with the original signature image features to be segmented to obtain fused features. The feature decoder performs feature decoding on the fused features to obtain the segmentation mask; the post-processing optimization module processes and corrects the handwriting corresponding to the segmentation mask.

2. The system according to claim 1, characterized in that, The handwriting extraction system is a branched multimodal fusion structure of "dual image encoder + text encoder + multi-head cross attention". The content prior image, real scene image and text prior are collaboratively constrained under the same attention framework to achieve the segmentation result.

3. The system according to claim 2, characterized in that, The multimodal information fusion module constructs two image encoders and one text encoder based on the CLIP module. CLIP image encoder 1 extracts image prior cue features based on printed images of the same content through the content prior unit. CLIP image encoder 2 obtains the signature image to be extracted from the paper signature image through the original image unit and extracts the signature image features. CLIP text encoder obtains text cue word information and obtains text prior cue features through the acquisition prior unit.

4. The system according to claim 1, characterized in that, The feature decoder uses the Transformer decoder as the core semantic understanding and segmentation engine. It receives the high-dimensional features output by the multimodal fusion module, transforms them into explicit pixel-level segmentation results, and obtains the segmentation mask.

5. The system according to claim 4, characterized in that, A multi-level self-attention segmentation modeling feature decoder is adopted. The decoder adopts a deep stacked Transformer structure, with each layer containing a self-attention module and a feedforward network. Through multi-path parallel attention heads, implicit associations in different semantic spaces are extracted. The segmentation target, color, and interference type defined by text / image cues are fused into the global context to achieve dynamic conditional guidance of segmentation behavior. The attention weight of each layer is adjusted according to the fused features, focusing on the condition-specified region and boundary.

6. The system according to claim 5, characterized in that, The output of all self-attention modules is followed by a set of deep feedforward fully connected networks to perform fine-grained pixel interpretation, boundary sculpting, and attribute differentiation for each local region; all segmentation feature layers use residual connections and normalization to overlay deep semantic feature information.

7. The system according to any one of claims 1-6, characterized in that, The prior information processing module constructs a general template for text prompts and generates text prompts for signature images. The encoder encodes the text and image prompts generated under the above-mentioned segmentation conditions. The prompt dimensions for text prompts include: acquisition method, handwriting color, signature content, and interference type. Conditional segmentation is implemented for multiple prompt modes, including: segmenting specified signature content, segmenting all signature content, segmenting specified handwriting color, and segmenting the general template specified method.

8. The system according to any one of claims 1-6, characterized in that, The CLIP encoder is fine-tuned using domain samples to optimize its sensitivity to signature content, font structure, and background interference. The "text and image cue features obtained from prior encoding" are fused with the "features of the original signature image to be segmented" to achieve controllable conditional segmentation. Feature extraction and information fusion are pre-trained using domain samples to establish a local domain fine-tuning strategy. A training set of "signature image + text cue + standard printed prior" is constructed locally. Gradient descent is used to fine-tune the newly added decoder and some encoder layers, and a phased unfreezing strategy is adopted to gradually update the parameters.

9. A handwriting extraction method based on the fusion of multiple prior information, characterized in that, include: The prior information processing module encodes the signature metadata and content image information into text and image prompt features based on the text and image prompts generated by the set segmentation conditions; The multimodal information fusion module fuses text and image prompt features with the original signature image features to be segmented to obtain fused features; The feature decoder performs feature decoding on the fused features to obtain a segmentation mask; the post-processing optimization module processes and corrects the handwriting obtained from the segmentation mask.

10. The method according to claim 9, characterized in that, In the multimodal information fusion module, the content prior unit extracts image prior prompt features based on printed images of the same content, the original image unit obtains the signature image to be extracted from the paper signature image, and the text prior prompt features are obtained by acquiring text prompt word information through the prior unit.

11. The method according to claim 9, characterized in that, The feature decoder captures long-distance dependencies between regions of an image through a multi-layer self-attention mechanism, integrating text / image cues. and The defined segmentation target, color, and interference type are integrated into the global context to achieve dynamic conditional guidance for segmentation behavior; the attention weights of each layer are adjusted according to the fusion features to focus on the specified regions and boundaries.

12. The method according to any one of claims 9-11, characterized in that, The prior information processing module generates text prompts for the signature image based on a general text prompt template. The encoder encodes the text and image prompts generated under the above-mentioned segmentation conditions. It sets text prompts based on acquisition method, handwriting color, signature content, and interference type, and sets conditional segmentation with multiple prompt modes.

13. An electronic device, comprising: processor; And a memory for storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 8-10.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the method according to any one of claims 8-10.

Images