Cross-modal handwriting comparison method, system, device and medium based on adversarial erasing mechanism

Abstract

This application provides a cross-modal handwriting comparison method based on an adversarial erasure mechanism, comprising: acquiring paper signatures and electronic handwritten signatures as training data; preprocessing the training data by removing stroke boundary interference and noise around the signature; constructing positive and negative sample pairs based on the paper and electronic signature data; constructing a deep neural network based on an adversarial erasure mechanism using a dual-branch architecture; obtaining self-attention maps of the feature maps corresponding to the positive and negative sample pairs through channel pooling operations on the deep neural network, and selectively erasing the effective regions of the feature maps locally; and using a binary classifier consisting of two fully connected layers and a nonlinear activation function, through forward propagation, inferring the probability that the handwritten signatures of the paired inputs belong to the same person. This application selectively removes some discriminative information from the fused features, allowing the use of a low-complexity model as the cross-modal handwriting recognition model, resulting in less computation and shorter training time.

Description

Technical Field

[0001] This application relates to electronic signature handwriting recognition technology, and more particularly to a cross-modal handwriting comparison method based on deep feature erasure learning. Background Technology

[0002] Cross-modal handwriting comparison technology refers to the use of multiple forms of data (such as paper handwriting images and handwriting trajectory sequences) for handwriting comparison, and the optimization of system performance by leveraging the correlation between different modalities. Currently, this technology is still in its early research stages, with only a few methods and research approaches used in related work. These methods can be broadly categorized into two types: those based on multimodal joint feature learning and those based on cross-modal relationship modeling. Methods based on multimodal joint feature learning focus on enhancing the effective fusion of features from different modalities. These methods typically require designing complex fusion networks to extract as many rich features as possible from each modality in the multimodal signature data, fuse these features in the optimal way, and finally send the fusion result to the decision layer to obtain the final comparison result. To achieve feature fusion of multiple modalities, these methods mostly employ multi-encoder approaches and improve recognition accuracy by embedding attention modules, resulting in excessive model complexity, computational burden, and long training times. Methods based on cross-modal relationship modeling primarily focus on uncovering the intrinsic correlations between different modalities of data. These methods typically use multiple feature encoders to map handwriting information from various modalities to the same latent common feature space, resulting in common feature vectors that simultaneously contain cross-modal information. Finally, these common feature vectors containing cross-modal information are used for downstream handwriting comparison. To fully capture the intrinsic correlations between different modalities of data, these methods usually rely on massive amounts of multimodal signature data for model pre-training. Furthermore, due to limitations in the degree of correlation between different modalities of data, the model may encounter problems such as difficulty in convergence and optimization during training.

[0003] Compared with other biometric technologies, signature authentication technology has advantages such as sufficient dynamic information, difficulty in imitation, high distinguishability, respect for privacy, and high efficiency in information acquisition. It excels in the collectability of signature features, the acceptable level of harm to the human body, and robustness, demonstrating broad application prospects and value. Electronic signatures are widely used in banking transactions and judicial procedures. However, unlike handwritten Chinese text, Chinese signatures exhibit diverse styles, lack semantic information, have complex font structures, and overlap between characters, leading to poor character segmentation accuracy.

[0004] Existing pre-training methods either focus on unimodal or multimodal tasks, failing to effectively adapt to each other and only utilizing unimodal or limited multimodal data. The Unified Modality Pre-training Architecture (UNIMO) has been proposed, effectively adapting to both unimodal and multimodal understanding and generation tasks. UNIMO leverages a large-scale free text corpus and image sets to improve visual text understanding capabilities, utilizing cross-modal contrastive learning (CMCL) to integrate textual and visual information into a unified semantic space, forming an "image-text pair" corpus. UNIMO utilizes rich unpaired unimodal data, allowing textual and visual knowledge to mutually reinforce each other within the unified semantic space, thereby learning more generalized representations. Experiments show that UNIMO performs best overall on both unimodal and multimodal downstream tasks.

[0005] To facilitate semantic alignment between vision and language at different levels, the original image captions are rewritten at the word, phrase, or sentence level, creating positive and negative image-text pairs. Image-text retrieval techniques are used to extract relevant images and text from unimodal data, which are then treated as unimodal positive samples during cross-modal learning. UNIMO has achieved state-of-the-art results not only on multimodal tasks but also on unimodal tasks. Summary of the Invention

[0006] This invention addresses the problems of excessive model complexity, computational burden, and long training time in multimodal joint feature learning methods for cross-modal handwriting comparison, and provides a simple and effective cross-modal handwriting representation learning scheme.

[0007] According to one aspect of this application, a cross-modal handwriting comparison method based on an adversarial erasure mechanism is provided, comprising: performing paper signature segmentation to obtain a binarized paper handwritten signature image and preprocessing it; inputting the preprocessed binarized paper handwritten signature image and electronic signature image as a pair into a feature encoder; passing the encoded output through a high-level semantic feature map and channel pooling to obtain a self-attention space feature map; obtaining a feature self-attention map through channel average pooling; performing threshold selection based on the self-attention space feature map; obtaining a drop mask; performing function activation based on the self-attention space feature map to obtain a salient feature heatmap; and comparing the salient feature heatmap with the drop mask. The Mask performs a spatial dot product operation to obtain the erasure region of the binary image of the paper handwritten signature. The erasure region is then spatially multiplied with the high-level semantic feature map to complete feature erasure and obtain the binary handwritten signature mask image. The paired binary handwritten signature mask image and electronic signature are used as inputs to the handwriting feature appraiser. The target loss is determined based on the binary cross-entropy loss. When the loss function converges, the model training is completed. The handwriting feature appraiser is forced to infer whether the handwriting features of the paper handwritten signature and the electronic signature belong to the same person based on the feature map after feature erasure.

[0008] Further preferably, the step of obtaining the binarized paper signature image includes: obtaining paper signature data containing the signature and binarized signature mask data, setting the paper signature data as source domain A and the binarized signature mask data as target domain B; a deep convolutional neural network randomly reads unpaired data from source domain A and target domain B as network input; training the deep convolutional neural network until the network converges to obtain a trained source domain A to target domain B conversion model and a target domain B to source domain A conversion model; using the paper signature data as input, and through forward propagation of the source domain A to target domain B conversion model, performing background removal and binarization processing on the paper signature to obtain a binarized paper handwritten signature image.

[0009] Further preferably, the preprocessing includes: removing noise around the signature and noise at the stroke edges; scaling the binarized handwritten signature image to a predetermined resolution; detecting the contours of each connected component; traversing each contour and calculating the area of ​​each contour; filling contours smaller than a set noise area threshold with zero values ​​to complete noise removal; and processing the binarized handwritten signature image I... m Perform dilate and erode operations to obtain the dilated signature mask image (dilate(I)). m ) and eroded signature mask image erode (I m ), call the formula: I d =dilate(I m )-erode(I m Obtain the binarized image of the handwritten signature after removing stroke edge noise. d .

[0010] Further preferably, the feature erasure further includes: using the joint features F∈R output by the encoder. H*W*C As input, a feature self-attention map M is generated through channel pooling. att ∈R H*W Threshold binarization is performed based on the feature self-attention map, setting pixels greater than the threshold to 0 and pixels less than the threshold to 1, resulting in the discard mask M. drop ∈R H*W The sigmoid function is used to activate the feature self-attention map, resulting in a salient feature heatmap M. imp ∈R H*W Perform a spatial dot product operation between the area to be erased and the encoder output to hide the most discriminative part of the encoder output features, and obtain a handwritten signature binarized mask image with completed feature erasure.

[0011] Further preferred, the determination of the target loss based on binary cross-entropy loss includes: setting a binary label y iCalculate the tagged signature features belonging to label y i The probability p(y) i ), and the tagged signature features do not belong to tag y i The probability p(1-y) i According to the formula:

[0012]

[0013] Calculate the loss of the network branches, and thus determine the feature erasure branch loss L. erased and feature unerased branch loss L unerased Call the formula: L Total =L unerased +λ erased L erased Calculate the feature loss of the handwriting identification network, where N represents the total number of paired paper and electronic signatures in the training set, and λ erased L represents the weight coefficient of the feature erasure branch loss. unerased L represents the loss function for the unerased branches of features. erased The loss function for the feature erasure branch is given. Both branch loss functions are calculated using the formula described above, and mathematically, both are binary cross-entropy losses L. BCE .

[0014] Another aspect of this application provides a cross-modal handwriting comparison system based on an adversarial erasure mechanism. The acquisition module acquires paper signatures and electronic signatures. The preprocessing module segments the paper signatures to obtain a handwritten binary signature image, removing noise around the signature and edge noise of the strokes to obtain a handwritten signature binary mask image. The feature encoder encodes the input handwritten signature binary mask image and electronic signature image to output joint features containing a high-level semantic feature map. The feature erasure module hides the most discriminative part of the joint features output by the encoder to obtain the handwritten signature binary mask image. This handwritten signature binary mask image serves as input to a cross-modal handwriting identification module, forcing an inference about whether the handwriting features belong to the same person. The high-level semantic feature map serves as input to a cross-modal handwriting identification network, directly inferring whether the handwriting features belong to the same person.

[0015] Further optimization reveals that the cross-modal handwriting identification network adopts a dual-branch network architecture, including a feature encoder, a feature erasure learning module, and a classifier. The feature encoder uses an 18-layer ResNet-18 residual network as its basic skeleton, removing the last fully connected layer and replacing the convolutions (conv3, conv4, conv5) in convolutional layers 3, 4, and 5 with deformable convolutions with 3x3 kernels. The classifier consists of two fully connected layers and a non-linear activation function. The feature erasure learning module obtains the self-attention map of the corresponding feature map through channel pooling, and then performs local erasure of the effective area of ​​the binary signature feature map of the paper handwriting through threshold selection and activation function (Sigmoid function) normalization.

[0016] Further preferably, the step of determining the cross-modal handwriting identification network loss based on the binary cross-entropy loss function includes: setting a binary label y i Calculate the tagged signature features belonging to label y i The probability p(y) i ), and the tagged signature features do not belong to tag y i The probability p(1-y) i According to the formula:

[0017]

[0018] Calculate the loss of the network branches, and thus determine the feature erasure branch loss L. erased and feature unerased branch loss L unerased Call the formula: L Total =L unerased +λ erased L erased Calculate the feature loss of the handwriting identification network.

[0019] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any of the preceding claims.

[0020] According to another aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in any of the preceding claims.

[0021] This invention provides a simple and effective cross-modal handwriting recognition and comparison technology. It adopts a single encoder to extract the skeleton of features and introduces a feature erasure module to selectively remove some useful and discriminative information from the fused features. This forces the model to make accurate predictions based on the remaining information, thereby enabling the encoder to extract more comprehensive and effective representations. It can use a low-complexity model as the cross-modal handwriting recognition model, with less computation and shorter training time. Attached Figure Description

[0022] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 This is a flowchart of a cross-modal handwriting comparison method based on adversarial erasure learning according to an exemplary embodiment of this application;

[0024] Figure 2 A schematic diagram of a deep neural network model with a dual-branch architecture according to an exemplary embodiment of this application is shown;

[0025] Figure 3 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of this application is shown. Detailed Implementation

[0026] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0027] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0032] Cross-modal contrastive generative adversarial networks (GANs) are used to train GAN models for text-to-image synthesis. By maximizing the mutual information between images and text through inter-modal and intra-modal contrastive learning, they address the cross-modal contrastive loss problem in text-to-image generation. The term "erasure" refers to removing some useful, discriminative information, hoping the network can still make accurate predictions using the remaining information. This naturally improves the network's ability to extract more comprehensive features, resulting in more accurate contrastive associations (CAMs). This enhances the network's feature extraction capabilities.

[0033] Erasing methods generally include: direct erasure, erasure using predictive information (CAM), and Dropout. Direct erasure is performed on the original image, usually randomly according to certain rules, and is often used for data augmentation. Erasure utilizes information predicted by the network. For example, the network input is the original image, and real-time CAM erasure is used during training before loss calculation. This training-erasure-retraining-erasure approach is often called adversarial erasure. Another example is directly using CAM as a mask applied to the feature map. Adversarial complementary erasure involves first training class A, then finding the corresponding feature map for that class. During training class B, this feature map is erased. Because B is trained under supervision, it can learn other regions of that class. The two regions are complementary, and finally, the two learned regions from A and B are combined to form the complete target. However, this method presents risks and difficulties for detecting small targets, as the entire target cannot be erased.

[0034] The scheme of this application is described below with reference to the accompanying drawings.

[0035] Figure 1 The diagram illustrates a cross-modal handwriting comparison process based on a feature erasure learning mechanism, as exemplified in this application. The signature (Liu Ziwen) is extracted from a paper document, and paper signature segmentation is performed. Surrounding noise and edge noise are removed, and edge noise is eliminated to obtain a segmented image of the paper handwritten signature. The electronic signature (Liu Ziwen) and the segmented image of the paper handwritten signature are then input into a handwriting identification network for signature handwriting identification.

[0036] The handwriting identification network is constructed using a two-branch handwriting identification network and trained using adversarial training.

[0037] Paper signatures are acquired using scanners and document scanners. However, these signatures may be surrounded by interference from seals, fingerprints, and other text, necessitating paper signature segmentation. To achieve more robust signature segmentation, a pre-trained unsupervised image transformation model, CycleGAN, is used to indirectly acquire the paper signature image. Specifically, this embodiment employs the following method:

[0038] Obtain paper data containing the signature and binarized signature mask data, setting the paper data as source domain A and the binarized signature mask data as target domain B; use a Cyclegan deep convolutional neural network to randomly read unpaired data from source domain A and target domain B as network input; train the Cyclegan deep convolutional neural network until the network converges, obtaining a trained source domain A to target domain B transformation model and a target domain B to source domain A transformation model; using the paper signature as input, perform background removal on the paper signature through the trained source domain A to target domain B transformation model to obtain a preliminary segmentation result; binarize the preliminary segmentation result to obtain the segmented result binarized paper signature.

[0039] The main purpose of preprocessing the binarized paper signature after post-processing segmentation is to remove noise, including removing noise from stroke edges and noise around the signature. The removal process begins with removing noise around the signature, followed by removing noise from the stroke edges. Noise removal around the signature: In addition to blurred or noisy stroke edges, handwritten paper signatures also contain additional noise. This noise mainly comes from interference in the paper background that resembles the texture of the strokes, such as printed text. To remove this noise interference, this embodiment can use the following steps:

[0040] The paper handwritten signature segmentation result is scaled to a predetermined resolution, such as 128x256. The contours of each connected component in the signature mask data are detected; the contour is the sum of the point coordinates. The contours of each connected component in the paper handwritten signature segmentation result are detected using OpenCV's built-in contour detection algorithm. A noise area threshold is set based on empirical knowledge. Each contour is traversed one by one, the area of ​​each contour is calculated, and contours with areas smaller than the set noise area threshold are filled with zero values ​​to complete noise removal.

[0041] Noise Removal from Stroke Boundaries: Due to the arbitrariness of the writing background and image degradation, the edges of segmented signature strokes often exhibit varying degrees of blurring and noise interference. To mitigate the impact of this noise or edge blurring, the following preprocessing method is used for the acquired binarized handwritten signature mask image: For the binarized handwritten signature mask image I... m Perform dilate and erode operations to obtain the dilated signature mask image (dilate(I)). m ) and eroded signature mask image erode (I m ), call the formula:

[0042] I d =dilate(I m )-erode(I m )

[0043] Obtain the handwritten signature mask image I after removing boundary interference. d This embodiment is mainly based on the built-in cv.dilate and cv.erode methods of OpenCV, where the kernel size used for the dilation operation is 5x5 and the kernel size used for the erosion operation is 3x3.

[0044] A dual-branch handwriting identification network is constructed using a dual-branch network architecture. The handwriting identification network consists of a feature encoder, a feature erasure module, and a classifier.

[0045] like Figure 2 The diagram shows a dual-branch architecture deep neural network model according to an exemplary embodiment of this application. The feature erasure module erases some handwriting features. In the deep neural network model, the feature erasure fusion branch forces the model to make accurate predictions based on the remaining information after some useful information has been erased. The feature encoder and classifier constitute the handwriting identification branch, which is used to identify the identity of the input signature pair.

[0046] In this embodiment, the feature encoder uses an 18-layer ResNet-18 residual network as its basic skeleton. After removing the fully connected layer at the end, it encodes features from the input data. Furthermore, considering the inherent characteristics of character strokes, such as severe deformation and complex spatial structure, the convolutions in convolutional layers 3, 4, and 5 of the ResNet-18 are replaced with deformable convolutional networks with 3x3 kernels. This provides the model with a more flexible receptive field to cope with the deformation variations of the stroke features themselves. In other words, the feature encoder uses deformable convolutions with three 3x3 kernels to encode features from the input data and output joint features.

[0047] The classifier consists of two fully connected layers and a non-linear activation function, forming a binary classifier. The classifier determines whether the input signature pairs were written by the same person.

[0048] The feature erasure module structure can be designed similarly to the erasure module in ADL (Attention-based Dropout Layer for Weakly Supervised Object Localization), primarily used to selectively erase features from specific regions of the feature map. Its forward propagation process is as follows:

[0049] The feature erasure module uses the joint features F∈R output by the encoder. H*W*C As input, a feature self-attention map M is generated through channel pooling. att ∈R H*W Based on M att ∈R H*WThreshold binarization is performed, setting pixels greater than the threshold to 0 and pixels less than the threshold to 1, resulting in a discard mask, M. drop ∈R H*W The threshold can be set to 0.7; the Sigmoid function is used to activate the feature self-attention map to obtain the salient feature heatmap M. imp ∈R H*W Perform a spatial dot product operation between the salient feature heatmap and the discard mask to obtain the region to be erased; perform a spatial dot product operation between the region to be erased and the encoder output to hide the most discriminative part of the encoder output features, thus completing feature erasure. Where: R represents the feature vector space, H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

[0050] The handwriting recognition network training adopts an adversarial erasure learning mechanism, which is mainly reflected in the learning mode of two branches that are mutually adversarial. That is, the handwriting recognition branch performs handwriting recognition normally based on the complete joint vector encoded by the encoder, while the feature erasure fusion branch is forced to use the remaining information to perform accurate handwriting recognition based on the joint vector after erasing useful information. Through this contradictory mechanism, it is beneficial to improve the model's ability to learn more comprehensive representations.

[0051] During training, the handwriting feature identification network takes paired binary signature mask data and electronic signatures as input. First, it performs joint feature learning through the feature encoder. Then, the joint features are processed in parallel by the feature erasure module to erase useful information. Finally, it performs handwriting identification through the classifier and directly through the classifier.

[0052] The target loss can be calculated by using the difference between the identification result and the true label based on the binary cross-entropy loss.

[0053] Training the handwriting identification network. The handwriting identification network takes paired binary signature mask data and electronic signatures as input. It first performs joint feature learning through a feature encoder, and then the joint features are processed in parallel by a feature erasure module to erase useful information.

[0054] The target loss is determined primarily by calculating the difference between the identification result and the true label based on the binary cross-entropy loss. Specifically, this can be achieved by setting a binary label y. i The binary label can be set to 0 or 1, where 0 is a positive sample label and 1 is a negative sample label. The signature feature with the label is calculated to belong to label y. i The probability p(y) i ), and the tagged signature features do not belong to tag y i The probability p(1-y) i According to the formula:

[0055]

[0056] Calculate the loss of the feature erasure branch, and thus determine the feature erasure loss L. erased and feature not erased loss L unerased Call the formula:

[0057] L Total =L unerased +λ erased L erased

[0058] Calculate the feature loss of the handwriting recognition network. Where λ erased The weighting coefficients representing the feature erasure branch loss can be empirically set to 0.5, y i It is a binary label 0 or 1, p(y) i ) belongs to the tag y i The probability is calculated by taking the logarithm, where log is the logarithm, N represents the total number of paired paper and electronic signatures in the training set, and λ is the probability. erased L represents the weight coefficient of the feature erasure branch loss. unerased L represents the loss function for the unerased branches of features. erased The loss function for the feature erasure branch is given. Both branch loss functions are calculated using the formula described above, and mathematically, both are binary cross-entropy losses L. BCE .

[0059] When the loss function converges, the model training is complete, and the handwriting identification model is obtained. Multimodal signature data, such as binarized images of handwritten paper signatures and stroke sequences of electronic signatures, are input into the handwriting identification model. A classifier performs handwriting identification and determines whether the signature belongs to the same signatory based on the handwriting style.

[0060] The handwriting identification network comprises a feature encoder, a feature erasure learning module, and a classifier. It employs a dual-branch network architecture, including an erasure learning branch and a handwriting classification branch. The erasure learning branch forces the model to make accurate predictions even using residual information, while the handwriting classification branch directly identifies the handwriting. The feature encoder uses ResNet-18 as its basic framework, with the final fully connected layer removed, to encode signature features. The feature erasure learning module obtains the self-attention map of the corresponding feature map through channel pooling, and then selectively erases effective regions of the feature map through threshold selection and Sigmoid activation function normalization. A binary classifier, consisting of two fully connected layers and a non-linear activation function, is responsible for calculating the probability that the handwritten signature input pairs belong to the same person.

[0061] The following specific example further illustrates the implementation of the present invention.

[0062] Binarized images of a paper handwritten signature and an electronic signature image of "Du Shasha" with identical content are input into a feature encoder. The encoded output is processed through a high-level semantic feature map and then through channel pooling (Fi) to obtain a self-attention spatial feature map (Fu). This self-attention spatial feature map is then obtained through channel average pooling. Threshold selection is performed based on the self-attention spatial feature map, and the mask is discarded to obtain the DropMask. The self-attention spatial feature map is then activated using the Sigmoid function to obtain a feature saliency heatmap. The feature saliency heatmap is spatially multiplied with the DropMask to obtain the region to be erased. This region is then spatially multiplied with the high-level semantic feature map to perform feature erasure. The feature map Fe after feature erasure is used as input to a feature detector, which forces an inference about whether the handwriting features belong to the same person. The feature detector is a feature network trained with a loss function.

[0063] Obtain paper-based handwritten signatures and offline electronic handwritten signatures. Preprocess the paper-based handwritten signature data as training data. Preprocessing includes removing stroke boundary interference and noise around the signature. Scaling the paper-based handwritten signature images to fix their size yields 128x256 images. Binarizing the scaled images yields corresponding handwritten signature mask images. Dilation is applied to the mask images to obtain coarsened mask images, and erosion is applied to obtain thinner mask images. The kernel size for dilation and erosion operations is related to the resolution; for example, based on image size, the kernel size for dilation can be set to 5, and the kernel size for erosion can be set to 3. The convolutional kernels used perform dilation and erosion operations on the strokes; dilation thickens the strokes, and erosion thins them.

[0064] By performing a spatial subtraction operation between the coarsened and thinned handwritten mask images, the backbone of the handwritten signature is obtained, which is the handwritten signature mask image after removing boundary noise.

[0065] Besides blurred stroke edges and noise, handwritten signatures on paper also contain additional noise. This noise mainly comes from interference in the paper background that resembles the texture of the strokes, such as printed text. To remove this noise interference, the following method is used: Contour detection is performed on the scaled-down handwritten signature to obtain several contour sets.

[0066] Traverse the contours one by one and calculate the area of ​​each contour. Based on experience, set a noise area threshold. Combine the contour and corresponding area information to fill the contours and their contained areas with zero values ​​to remove noise.

[0067] In a deep neural network based on an adversarial erasure learning mechanism, the feature erasure learning module obtains the self-attention map of the corresponding feature map through channel pooling operation. Then, it selectively erases the effective region of the feature map through operations such as threshold selection and Sigmoid function normalization. A binary classifier consisting of two fully connected layers and a nonlinear activation function is responsible for calculating the probability that the handwritten signature input pairs belong to the same person.

[0068] The feature encoder uses ResNet-18 as its basic skeleton, removes the final fully connected layer, and replaces the convolutions in convolutional layers 3, 4, and 5 with deformable convolutions with 3x3 kernels. This provides the model with a more flexible receptive field to cope with the deformation changes of the stroke features themselves. It is used to encode signature features.

[0069] Based on the processed paper and offline electronic handwritten signature data, positive and negative sample pairs are constructed to form a training set. Feature training is performed on the deep neural network based on the adversarial erasure learning mechanism to obtain the trained deep learning model.

[0070] The signature feature encoder encodes the handwritten signature input from the user to obtain a high-level semantic feature map containing high-level semantic information. The feature erasure module performs local feature erasure on the high-level semantic feature map and forces handwriting identity inference. The high-level semantic feature map is processed by channel pooling to obtain a self-attention spatial feature map. The self-attention spatial feature map is selected based on a threshold to obtain a drop mask. The self-attention spatial feature map is activated by the sigmoid function to obtain a feature saliency map. The feature saliency map and the drop mask are spatially multiplied to obtain the region to be erased. The region to be erased is spatially multiplied with the high-level semantic feature map to perform feature erasure.

[0071] The feature map after feature erasure is used as input to the handwriting identification network, which is forced to infer whether the handwriting features belong to the same person. The high-level semantic feature map is used as input to the handwriting identification network, which directly infers whether the handwriting features belong to the same person. The forced identification involves making an identity determination after some useful information has been erased. This forces the model to mine other features that can be used to determine identity from the remaining unerased information. This is constrained by the loss function. The model directly uses the complete feature information for identity determination, which is relatively mild, but the useful information mined is limited.

[0072] The loss functions for feature erasure and non-feature erasure are determined based on binary cross-entropy, and the model weight parameters are updated through backpropagation.

[0073] Based on handwritten signature input from multiple individuals, a pre-trained deep neural network is used to directly infer the probability that the signature belongs to the same person through forward propagation. During the inference phase, the feature erasure learning branch does not participate in the calculation; after the handwritten signature is encoded by the feature encoder, the final result is directly output through the classifier's inference.

[0074] An exemplary embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this application.

[0075] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0076] An exemplary embodiment of this application also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of this application.

[0077] like Figure 3 The diagram illustrates a structural block diagram of an electronic device 300 that can serve as a server or client in this application, and is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0078] Electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 302 or a computer program loaded from storage unit 308 into random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304. Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, output unit 307, storage unit 308, and communication unit 309. Input unit 306 can be any type of device capable of inputting information to electronic device 300. Input unit 306 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 307 can be any type of device capable of displaying information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 308 may include, but is not limited to, disks and optical discs. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0079] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above.

[0080] The program code implementing the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0081] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0082] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0083] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0084] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0085] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A cross-modal handwriting comparison method based on an adversarial erasure mechanism, characterized in that, Paper signature segmentation is performed to obtain binarized paper handwritten signature images, which are then preprocessed. The preprocessed binarized paper handwritten signature images and electronic signature images are then input into a feature encoder for feature encoding. The output high-level semantic feature map is processed by channel pooling to obtain a self-attention spatial feature map. Channel average pooling is then used to obtain a feature self-attention map. Threshold selection is performed based on the self-attention spatial feature map to determine and obtain the drop mask. Function activation is performed based on the self-attention spatial feature map to obtain a salient feature heatmap. The salient feature heatmap and the drop mask are multiplied by a spatial dot to obtain the erasure region of the paper handwritten signature binarized image. The erasure region is multiplied by a spatial dot to complete the feature erasure and obtain the binarized handwritten signature mask image. Using paired binary handwritten signature mask images and electronic signatures as inputs to the handwriting feature detector, the target loss is determined based on binary cross-entropy loss. When the loss function converges, the model training is complete. The handwriting feature identification device uses the feature map after feature erasure to force an inference as to whether the handwriting features of the paper handwritten signature and the electronic signature belong to the same person.

2. The cross-modal handwriting comparison method as described in claim 1, characterized in that, The process of obtaining a binarized paper signature image includes: acquiring paper signature data containing the signature and binarized signature mask data, setting the paper signature data as source domain A and the binarized signature mask data as target domain B; a deep convolutional neural network randomly reads unpaired data from source domain A and target domain B as network input; training the deep convolutional neural network until the network converges to obtain a trained source domain A to target domain B conversion model and a target domain B to source domain A conversion model; using the paper signature data as input, and through forward propagation of the source domain A to target domain B conversion model, performing background removal and binarization processing on the paper signature to obtain a binarized paper handwritten signature image.

3. The cross-modal handwriting comparison method as described in claim 1, characterized in that, The preprocessing includes: removing noise around the signature and noise at the stroke edges; scaling the binarized handwritten signature image to a predetermined resolution; detecting the contours of each connected component; traversing each contour and calculating the area of ​​each contour; filling contours smaller than a set noise area threshold with zero values ​​to complete noise removal; and preprocessing the binarized handwritten signature image I... m Perform dilate and erode operations to obtain the dilated signature mask image (dilate(I)). m ) and eroded signature mask image erode (I m ), call the formula: I d =dilate(I m )-erode(I m The handwritten signature binarized image Id after removing stroke edge noise is obtained.

4. The cross-modal handwriting comparison method as described in any one of claims 1-3, characterized in that, The feature erasure further includes: using the joint features F∈R output by the encoder. H*W*C As input, a feature self-attention map M is generated through channel pooling. att ∈R H*W Threshold binarization is performed based on the feature self-attention map, setting pixels greater than the threshold to 0 and pixels less than the threshold to 1, resulting in the discard mask M. drop ∈R H*W The sigmoid activation function is used to activate the feature self-attention map, resulting in a salient feature heatmap M. imp ∈R H*W Perform a spatial dot product operation between the area to be erased and the encoder output to hide the most discriminative part of the encoder output features, and obtain a binary mask image of the handwritten signature with erased features. Here, R represents the feature vector space, H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

5. The cross-modal handwriting comparison method as described in any one of claims 1-3, characterized in that, The method for determining the target loss based on binary cross-entropy loss includes: setting a binary label y. i Calculate the tagged signature features belonging to label y i The probability p(y) i ), and the tagged signature features do not belong to tag y i The probability p(1-y) i According to the formula: Calculate the loss of each network branch to determine the loss for the feature-erased branch and the loss for the non-erased branch, using the formula: L Total =L unerased +λ erased L erased Calculate the feature loss of the handwriting identification network, where N represents the total number of paper and electronic signatures trained on the ensemble, and λ erased L represents the weight coefficient of the feature erasure branch loss. unerased L represents the loss function for the unerased branches of features. erased The loss function is the feature erasure branch.

6. A cross-modal handwriting comparison system based on an adversarial erasure mechanism, characterized in that, include: The system comprises an acquisition module, a preprocessing module, and a cross-modal handwriting identification network. The cross-modal handwriting identification network includes a feature encoder, a feature erasure module, and a classifier. The acquisition module acquires paper signatures and electronic signatures. The preprocessing module segments the paper signatures to obtain a binary image of the handwritten signature, and removes noise around the signature and edge noise to obtain a binary mask image of the handwritten signature. The feature encoder encodes the input binary mask image of the handwritten signature and the electronic signature image, outputting a joint feature containing a high-level semantic feature map. The feature erasure module performs channel pooling on the high-level semantic feature map to obtain a self-attention space feature map, and then performs channel average pooling to obtain a feature self-attention map. Based on the self-attention space feature map... Threshold selection is performed on the feature map to determine the discard mask. Function activation is performed based on the self-attention spatial feature map to obtain a salient feature heatmap. The salient feature heatmap and the discard mask are spatially multiplied to obtain the erasure region of the handwritten signature binarized image. The erasure region is spatially multiplied with the high-level semantic feature map to complete feature erasure and obtain the binarized handwritten signature mask image. The most discriminative part of the joint features output by the hidden encoder is obtained to obtain the handwritten signature binarized mask image. The classifier is forced to infer whether the handwriting features belong to the same person based on the input handwritten signature binarized mask image, and directly infers whether the handwriting features belong to the same person based on the high-level semantic feature map.

7. The cross-modal handwriting comparison system as described in claim 6, characterized in that, The cross-modal handwriting identification network adopts a dual-branch network architecture. The feature encoder uses an 18-layer residual network ResNet-18 as its basic skeleton, removes the last fully connected layer, and replaces the convolutions in convolutional layers 3, 4, and 5 with deformable convolutions with 3x3 kernels. The classifier consists of two fully connected layers and a non-linear activation function. The feature erasure learning module obtains the self-attention map of the corresponding feature map through channel pooling. Then, it performs local erasure of the effective area of ​​the binary signature feature map of paper handwriting through threshold selection and Sigmoid activation function normalization. The loss of the cross-modal handwriting identification network is determined based on the binary cross-entropy loss function.

8. The cross-modal handwriting comparison system as described in claim 7, characterized in that, The method for determining the cross-modal handwriting identification network loss based on the binary cross-entropy loss function includes: setting a binary label y. i Calculate the tagged signature features belonging to label y i The probability p(y) i ), and the tagged signature features do not belong to tag y i The probability p(1-y) i According to the formula: Calculate the loss of the network branches, and thus determine the feature erasure branch loss L. erased and feature unerased branch loss L unerased Call the formula: L Total =L unerased +λ erased L erased Calculate the feature loss of the handwriting identification network, where N represents the total number of paper and electronic signatures trained on the ensemble, and λ erased This represents the weighting coefficients of the feature erasure branch loss.

9. An electronic device, comprising: processor; And a memory for storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the cross-modal handwriting comparison method according to any one of claims 1-5.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the cross-modal handwriting comparison method according to any one of claims 1-5.

Images