Offline handwritten signature segmentation system and method based on a two-branch neural network

Abstract

The application discloses an offline handwritten signature segmentation method based on a double-branch neural network, relates to electronic signature recognition technology, learns rough semantic information of a document image, divides regions of the document image, segments a signature image based on a coding-decoding branch network, and obtains a rough segmented signature; details of a blurred region of a signature edge are inferred, feature coding is continuously performed, boundary details around the signature are learned, and a more fine segmented signature is obtained; feature categories are divided according to regions, feature centers are adaptively distributed, so that image features belonging to the same category are close to the feature centers; signature space structure capture is performed, a generative adversarial loss function is generated, true and false evaluation is performed on the fine segmented signature, the double-branch network is fooled, and a higher-quality signature output is obtained. Small-size electronic signatures can be accurately segmented and recognized on any position and complex background. The application can be applied to electronic signature handwriting recognition in complex scenes and blurred backgrounds.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information processing, and particularly relates to a handwritten electronic signature segmentation and recognition method. BACKGROUND

[0002] As a unique active behavior biometric feature of the signatory, the handwritten signature is difficult to be imitated by others, is effectively associated with the personal identity, and is a composite information carrier of the signatory's conscious and subconscious. Therefore, it constitutes an alternative identity authentication used in various applications, such as bank checks, contracts, certificates and forensic medicine. In the past decade, the application development of handwritten signature verification technology has been rapid, and currently, the original handwriting signature products formed based on handwriting biometric feature recognition technology have been widely applied in many fields such as finance, government affairs, justice, public security and people's livelihood.

[0003] Publication No. CN112070779B, entitled "A Remote Sensing Image Road Segmentation Method Based on Convolutional Neural Network Weak Supervision Learning", discloses a remote sensing image road segmentation method based on convolutional neural network weak supervision learning. The sparse supervision information provided by the road center line data is used to propagate semantic features from the road center line to the unlabeled pixels through a context-aware label propagation algorithm, and a deep learning framework is used to train a convolutional neural network with a double-branch encoding-decoding structure to predict road surface data from remote sensing images. The method can adapt to different scale remote sensing image road surface segmentation, and the label propagation algorithm is used to generate weak labels by propagating semantic information from the road center line to the unlabeled pixels; a convolutional neural network model with a double-branch encoding-decoding structure is constructed, including an encoding part, a decoding part and a dilated convolution spatial pyramid pooling part between them, wherein the decoding part includes two sub-branches of a semantic segmentation branch and an edge detection branch, which are used to obtain a semantic segmentation map and an edge detection map, and an edge detection operator is used to generate a rough edge for the training set image to supervise the edge detection branch to assist the semantic segmentation branch; the training set image is input into the convolutional neural network model with a double-branch encoding-decoding structure, the loss function is calculated according to the weak label, the rough edge and the output semantic segmentation map and edge detection map, the model is trained on the GPU through back propagation, and after the training is completed, the test set image is input into the trained model to predict the road surface, and the final road segmentation result is obtained.

[0004] A kind of off-line handwritten signature identification method disclosed in Chinese invention patent with publication number CN106778586B and name "off-line handwritten signature identification method and system" is disclosed, which pre-processes the off-line signature sample in off-line signature sample library into the width and height of statistical signature, and cuts according to proportion;Multiple features are extracted, shape, texture and pseudo-dynamic features of signature are extracted respectively;Shape features include moment features, which are used to describe overall structural features including signature shape, character position inclination and barycenter offset;Texture features include local binary pattern features and gray level co-occurrence matrix features, which are used to intuitively reflect visual features of signature image, and describe signature image through gray level distribution of pixel and its surrounding space neighborhood;Pseudo-dynamic features include impulse-coupled neural network features, which are used to indirectly represent the change of dynamic information including pressure when signer writes signature through gray level transformation;Specifically, impulse-coupled neural network features are extracted from gray level image of cut boundary, local binary pattern features and gray level co-occurrence matrix features are extracted from normalized gray level image, and low-order moment features are extracted from normalized binary image;Multiple features of extracted off-line signature sample are trained to obtain standard sample library after training;Get the signature to be tested, and pre-process the signature to be tested to obtain multiple features of the signature to be tested;Match the multiple features of the signature to be tested with the corresponding features of the off-line signature sample in the standard sample library, and identify whether the signature to be tested is a real signature or a fake signature.

[0005] The method of the prior art usually adopts the technical route of semantic segmentation, regards the signature region as two categories of background and signature foreground, and classifies it pixel by pixel;Due to the small scale and complex stroke spatial structure of handwritten signature, the segmentation result obtained by the conventional semantic segmentation method is relatively rough, the stroke edge is prone to excessive segmentation and has sawtooth phenomenon, in addition, the segmentation model does not consider the correlation between pixels in the process of classifying the signature region pixel by pixel, which is prone to cause the problem of missing part of the stroke in the segmentation result.

[0006] Due to the diversity of signing environment, signature can be signed on any position and complex background, and factors such as complexity of background and small size of region of interest make the automatic electronic signature segmentation become a difficult task. So far, only a few methods are used for signature segmentation or text segmentation. These methods usually adopt the idea of semantic segmentation, give the original document image, and directly classify it at pixel level. Due to the fact that neural network is easy to remember or learn simple visual features, the segmentation-based method tends to learn rough foreground region, and for small-scale signature, the result obtained is usually rough. Secondly, the segmentation network usually only classifies a single pixel, without considering the potential relationship between pixels, which leads to the problem of missing pixels in the segmentation result such as partial strokes. In addition, due to the complexity of the scene, some complex background textures such as (seal, fingerprint, printed matter, etc.) may damage the local features of the signature, thereby causing segmentation inference failure. SUMMARY

[0007] The present application aims at the prior art, the stroke segmentation result is relatively rough, the stroke edge is easy to over-segment and has the phenomenon of sawtooth, and the association between pixels is not considered in the process of classifying the signature region pixel by pixel, which is easy to cause the segmentation result to have the problem of missing part of the stroke. The present application mainly takes the document image data as the research background, and the goal is to find the handwritten signature information in the document image and realize the fine segmentation of the region. An additional image generation task is introduced to optimize the segmentation result to realize the fine-grained segmentation of the handwritten signature. Secondly, the center loss (A Discriminative Feature Learning Approach for Deep Face Recognition, CenterLoss) is used during training to continue to optimize the segmentation result, and the "intra-class consistency" between different class features is improved to further enhance the robustness of the model in complex scenes.

[0008] The technical scheme for solving the above technical problems is that, on the one hand, an offline handwritten signature segmentation system based on a double-branch neural network is proposed, which includes a signature detection module, an illegal signature verification module, a signature segmentation module, and a signature post-processing module. The signature detection module is used to locate the position of the handwritten signature in the image and determine the candidate region. The illegal signature verification module filters out the obviously illegal candidate region based on the confidence of the detection frame and the candidate frame to obtain the region of interest image. The signature segmentation module further includes a low-level semantic learning branch module, a high-level boundary learning branch module, a spatial attention fusion module, and a spatial structure discrimination subnetwork. The low-level semantic learning branch module uses an encoding-decoding network skeleton to preliminarily segment the signature body and the signature boundary to obtain a semantic feature map. The high-level boundary learning branch module learns the signature boundary details through a dilated spatial gold tower pooling upsampling operation to obtain a convolution feature map. The spatial attention fusion module generates a signature region prediction based on the attention mechanism fusion of the semantic feature map and the convolution feature map. The spatial structure discrimination subnetwork uses a binary classification neural network to evaluate whether the signature region prediction is sufficient true by taking it as input, so as to deceive the signature segmentation network to generate a higher-quality intermediate segmented signature. The signature post-processing module removes the noise and interference in the intermediate segmented signature through image processing operations, filters and optimizes the stroke structure to obtain the signature output.

[0009] Further optimization involves using multiple convolutional layers in the low-level semantic learning branch module to encode features of the region of interest image in the encoding part. The decoding part uses the same number of deconvolutional layers as the encoding part to restore the spatial dimension of the signature image and distinguish the signature subject and the boundary region. The encoding and decoding parts are connected by a compression excitation network (SE) module to focus on the image feature channels. The encoding part performs feature encoding to obtain feature maps at multiple stages, which are denoted as E1 to E4 features in order. Feature E4 is weighted by channel importance through the SE module. Then, the features E1 to E3 of the first three stages are fused with different stages of the decoding network to output a feature map with three channels, where each feature map represents a category.

[0010] Further optimization involves using the dilated spatial pyramid pooling (ASPP) module in the high-level boundary learning branch module. By expanding the receptive field of the convolutional kernel, the ASPP module aggregates features from different receptive fields, enabling the boundary features around the signature to effectively connect the signature foreground and document background information, thus performing high-level detail prediction and achieving more accurate signature segmentation. In the ASPP module, the first five convolutional layers perform convolution operations on the feature maps output by the encoding network of the low-level semantic learning branch module. The last convolutional layer uses a 1x1 convolutional kernel to concatenate the channels of the output feature maps of the first five convolutional layers and then performs channel compression to output the convolutional feature map.

[0011] Further optimization involves a low-level semantic learning branch module that includes a feature clustering part. This part categorizes features by region and assigns corresponding feature center vectors based on the spatial locations of the signature subject, signature boundary, and background. This causes image features belonging to the same category to converge towards the assigned feature centers, prompting the encoding network to learn discriminative encodings of the signature subject, signature boundary, and background. The feature clustering is primarily based on the enhanced "intra-class feature consistency" strategy in Center Loss (A Discriminative Feature Learning Approach for Deep Face Recognition). During training, the following formula is used: Calculate the feature center vector, where I E This represents the three-channel output of the last convolutional layer in the low-level semantic learning branch. Indicates three-channel output I E The i-th eigenvector in the middle, Indicates pre-allocation Category y i The feature center is N, which represents the total number of pixels in the feature map.

[0012] Further preferably, the spatial structure discriminator sub-network realizes signature spatial structure capture, the spatial discriminator sub-network and the double-branch signature segmentation network generate an adversarial loss function, on one hand, the spatial discriminator sub-network maximizes the distinction between the real signature mask and the predicted signature mask, and simultaneously forces the double-branch signature segmentation network to generate a higher-quality predicted signature mask, according to the signature mask prediction result Real signature mask label I alpha , the double-branch signature segmentation network G, the spatial structure discriminator D, and the calling formula: Calculate the adversarial loss function L adv (G, D), according to the to-be-segmented document image I sig , the semantic label I of the signature body and the signature boundary region label , the signature boundary region label I border , the real signature mask label I alpha , the cross-entropy loss function L seg , the mean square error loss function L detail , the calling formula: L total =λ seg L seg (I sig , I label )+λ detail L detail (I sig , I border )+λ adv (I sig , I alpha )Calculate the loss function of the double-branch signature segmentation network, wherein λ seg , λ detail , λ adv respectively represent the cross-entropy loss function, the mean square error loss function, and the adversarial loss function weight coefficient.

[0013] The application further provides an offline handwritten signature segmentation method based on a double-branch neural network, which locates the position of a handwritten signature in an image to determine a candidate region, obtains an image of a region of interest based on the confidence of a detection frame and the candidate frame, preliminarily segments the signature body and the boundary around the signature in the image of the region of interest based on an encoding-decoding branch network to obtain a semantic feature map, and divides the feature categories according to regions; the semantic feature map is subjected to a dilated spatial gold tower pooling upsampling operation to learn the signature boundary details and obtain a convolution feature map; the signature region prediction is generated based on the attention mechanism fusion of the semantic feature map and the convolution feature map; a binary classification neural network is used to take the signature region prediction as input to evaluate whether it is sufficient, so as to fool the double-branch signature segmentation network to generate a higher-quality intermediate segmented signature; the noise and interference in the intermediate segmented signature are removed through an image processing operation, filtering and stroke structure optimization are performed, and the signature output is obtained.

[0014] Further preferably, the region of interest image is encoded by using multiple convolution layers, the signature image space dimension is restored by using deconvolution, and the region where the signature subject and the boundary are located is distinguished, the image feature channel is focused by using a squeeze and excitation network SE module, feature encoding is performed to obtain multiple stage feature maps, and the features are sequentially recorded as E1-E4 features. The feature E4 is weighted by the SE module according to the importance of the channel, and then the features E1-E3 of the previous three stages are respectively fused with different stages of the decoding network to output a feature map with three channels, and each feature map represents a category.

[0015] Further preferably, the ASPP module is used to aggregate features of different receptive fields by expanding the receptive field of the convolution kernel, so that the boundary features around the signature effectively connect the signature foreground and the document background information, and high-level detail prediction is performed to realize more accurate signature segmentation; six convolution layers are set to generate a convolution feature map, wherein the first five convolution layers are used to perform convolution operation on the feature map output by the low-level semantic learning branch module coding network, and the last convolution layer is used to perform channel compression on the output feature map channel connected by the first five convolution layers to output a convolution feature map.

[0016] Further preferably, the feature category is further divided according to the spatial position of the signature subject, the signature boundary and the background, and the corresponding feature center vector is assigned, so that the image features belonging to the same category are close to the assigned feature center, the center loss is used to strengthen the “intra-class consistency” between different class features in the output feature map, and a three-channel region prediction map is output after back propagation, the features output by the last convolution layer of the encoder are compressed to change each feature channel into a real number, the weight of the compressed features is generated by excitation operation, and the weight is weighted on the features channel by channel to complete the channel attention calibration of the features.

[0017] Further preferably, the spatial structure discrimination subnetwork captures the spatial structure of the signature, the spatial discrimination subnetwork and the double-branch signature segmentation network generate an adversarial loss function in mutual opposition, on the one hand, the spatial discrimination subnetwork is trained to maximize the distinction between the real signature mask and the predicted signature mask, and at the same time, the double-branch signature segmentation network is forced to generate a higher quality predicted signature mask, and the signature mask prediction result is used as the final signature segmentation result. Real signature mask label I alpha , double-branch signature segmentation network G, spatial structure discriminator D, and the formula is called. Calculate the adversarial loss function L a L adv (G, D), according to the to-be-segmented document image I sig , the semantic label I of the signature subject and the signature boundary region label , the signature boundary region label I border, real signature mask label I alpha , cross-entropy loss function L seg , mean square error loss function L detail , call formula: L total = lambda seg L seg (I sig , I label )+ lambda detail L detail (I sig , I border )+ lambda adv (I sig , I alpha )Calculate the loss function of the double-branch signature segmentation network, wherein, lambda seg , lambda detail , lambda adv Cross-entropy loss function, mean square error loss function, and adversarial loss function weight coefficient.

[0018] The present application fuses the advantages of image segmentation method and image generation method respectively, realizes fine-grained segmentation of signature pixels, alleviates the interference problem caused by complex background, improves the "intra-class and inter-class consistency" between different categories of features in the segmentation network, learns discriminative feature encoding, solves the complexity of background and the small size of signature area, enhances the robustness of the model in complex scenes, in addition, to overcome the problem of partial stroke defects, an additional discriminant network is introduced to capture the structural prior of stroke details. Considering the correlation between pixels, the segmentation result is complete, and small size electronic signature can be accurately segmented and recognized on any position and complex background. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 Schematic diagram of an end-to-end offline signature automatic segmentation system

[0020] Figure 2 Double-branch offline signature segmentation network DETAILED DESCRIPTION

[0021] For the convenience of understanding the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0022] As Figure 1The end-to-end offline signature segmentation system of the application is shown in the schematic diagram, which comprises a low-level semantic learning branch module, a high-level boundary learning branch module, a spatial attention fusion module, and a spatial structure discrimination subnetwork. The low-level semantic learning branch module is based on an encoding-decoding network skeleton and performs preliminary segmentation on the signature body and the boundary around the signature. The high-level boundary learning branch module shares the high-level semantic feature map at the end of the low-level semantic branch encoder, performs up-sampling operation through Atrous Spatial Pyramid Pooling (ASPP), and realizes specialized learning of the signature boundary details. The spatial attention fusion module uses the low-level semantic learning branch module to predict the effective fusion of the signature area of the convolution feature map at the end of the two branches, and generates a signature area prediction. The spatial structure discrimination subnetwork uses a binary neural network to input the generated signature area prediction and evaluate whether it is sufficient to be true, so as to fool the signature segmentation network to generate a signature area prediction with higher quality.

[0023] The low-level semantic learning part is used to learn the rough semantic information of the document image, and the image can be divided into three parts: signature foreground, image background, and fuzzy transition area between foreground and background. The signature image can be segmented using an encoding-decoding branch network to obtain a rough signature segmentation result. The high-level boundary detail learning part mainly focuses on inferring the details of the fuzzy area of the signature edge. Based on the signature segmentation result feature map obtained by the last layer of the low-level semantic learning branch part encoder, the ASPP module is used to continue high-level feature encoding to learn the boundary details around the signature, and a more detailed signature segmentation is obtained.

[0024] The encoding part of the low-level semantic learning branch module uses multiple (such as 4) convolution layers to encode the features of the input signature image. The decoding part uses the same number of deconvolution layers as the encoding part to restore the spatial dimension of the signature image and distinguish the areas of the signature body and the boundary. The SE (Squeeze-and-excitation networks) module of the attention mechanism is used between the encoding part and the decoding part to focus on the image feature channels and automatically learn the importance of different channels, which is beneficial to the model processing classification problems. The encoding network in the low-level semantic learning branch part encodes the features of the input document image data to obtain multiple stage feature maps, which are denoted as E1-E4 features in sequence. The feature E4 is weighted by the SE module according to the importance of the channels, and then the features E1-E3 of the previous 3 stages are fused with different stages of the decoding network to output a feature map with three channels. Each feature map represents a class, which is the signature foreground, the document image background, and the fuzzy unknown area between the signature foreground and the background, respectively.

[0025] Further, to enhance the ability of the low-level semantic learning branch to deal with complex background interference (such as seals, fingerprints, printed matter, etc.), a feature clustering part can be added to perform clustering processing on the output feature map of the last convolution layer of the low-level semantic learning branch part, so that the features belonging to the same category are attracted to the assigned feature center, and the "intra-class consistency" between different categories of features is enhanced to improve the distinction between different categories of features. The feature clustering part is used to improve the "intra-class consistency" of the feature map, processes the output of the last layer of convolution at the end of the low-level semantic learning branch, classifies the categories according to the regions, adaptively allocates the corresponding feature center, and in the learning process, the features belonging to the same category are attracted to the assigned feature center, so as to enhance the "intra-class and inter-class consistency" between different categories of features.

[0026] During the training process, the center loss (A Discriminative Feature Learning Approach for Deep Face Recognition, CenterLoss) is used to further enhance the "intra-class consistency" between different categories of features in the output feature map of the last convolution layer. After the document image is input into the low-level semantic learning branch and forward propagated, a three-channel region prediction map is output (channel 0 represents the region prediction of the document background, channel 1 represents the region prediction of the signature body, and channel 2 represents the region prediction of the signature boundary). During the training process, the CenterLoss allocates the corresponding feature center vector according to the spatial position of the signature body, the signature boundary and the background, and encourages the features belonging to the same region to be attracted to the same feature center (the specific center vector is initially a randomly initialized 3x3 tensor, and the tensor value can be updated through back propagation during the training process). Specifically, it encourages the encoding network to learn the discriminative encoding of the signature body, the signature boundary and the background. The implementation of feature clustering is mainly based on the "intra-class feature consistency" strategy in the CenterLoss (A Discriminative Feature Learning Approach for DeepFace Recognition, CenterLoss). During the training process, the formula is called: determine the feature center vector, where I E represents the three-channel output of the last convolution layer of the low-level semantic learning branch, represents the third channel output I E the i-th feature vector in the three-channel output, represents the feature center of the category y i , and N represents the total number of pixels in the feature map.

[0027] The feature X output by the convolution layer at the end of the encoder is taken as the input of the SE module, and the number of feature channels of the SE module is C. First, the Squeeze operation is performed on X to change each two-dimensional feature channel into a real number. Then, the Excitation operation is performed on the compressed feature (1x1xC) to generate a weight w. The weight w is weighted to the feature X channel by channel to complete the channel attention calibration of the feature X. The encoding part uses the pooling layer operation to reduce the spatial dimension, which easily causes the loss of some stroke details, resulting in a certain roughness of the result obtained by upsampling in the decoding part. Therefore, the jump structure is used to optimize the output after upsampling the results of different convolution layers in the encoding part.

[0028] The high-level boundary learning branch module uses the Arous Pyramid Pooling (ASPP) module to expand the receptive field of the convolution kernel, guide the model to aggregate features of different receptive fields, effectively connect the signature foreground and document background information around the signature, and perform high-level detail prediction to achieve more accurate signature segmentation. In an optimal manner, the ASPP module sets 6 convolution layers, wherein the convolution kernel size is set to 1x1 and 3x3, and the 3x3 size convolution kernel has a hollow rate of 1, 2, 4, and 8.

[0029] The first five convolution layers in the ASPP module perform convolution operations on the feature maps output by the encoding network in the low-level semantic learning branch module, and the last convolution layer uses a 1x1 convolution kernel to perform channel compression on the output feature maps of the first five convolution layers after channel connection to obtain the output of the high-level boundary learning branch module.

[0030] The outputs of the last convolution layers of the two branches are merged to generate a fused signature output.

[0031] To constrain the segmented signature result to maintain the integrity of the spatial structure, a spatial structure discriminator subnetwork is used as a structure model to capture the spatial structure prior of the character strokes. The spatial structure discriminator captures the spatial structure based on the spatial structure discriminator network, and based on the game idea of generative adversarial, evaluates the true and false of the more refined signature segmentation output obtained by the double-branch network to fool the double-branch network to obtain a higher quality signature output. The discriminator network takes the signature region mask generated by the fusion signature output by the double-branch network and the real signature mask label as input, discriminates whether the fusion result is sufficient, and prompts the double-branch neural network to decouple and segment a higher quality signature.

[0032] The spatial structure discrimination sub-network is composed of three convolutional layers and a fully connected layer, and is a binary classification network for evaluating the integrity of the predicted signature region. The spatial structure discrimination sub-network determines whether the input sample is real signature mask data or predicted signature mask data, and outputs a probability value belonging to the real signature mask. In particular, in order to make the model more stable during training and faster convergence, the Least Squares Generative Adversarial Networks (LSGAN) is used for adversarial training. During training, the spatial discrimination sub-network and the signature segmentation network are mutually antagonistic. On the one hand, the spatial discrimination sub-network maximizes the accuracy of identifying real signature masks and predicted signature masks, while forcing the signature segmentation network to generate higher quality predicted signature masks. According to the signature mask prediction result Real signature mask label I alpha , a double-branch neural network G, a spatial structure discriminator D, according to the formula: Calculate the adversarial loss function L adv (G, D), where E represents the mathematical expectation, P data represents the distribution of signature region mask data in the training set, represents the signature region mask predicted by the double-branch signature segmentation network, I alpha represents the real signature mask, L adv represents the generative adversarial loss function, represents the input I alpha obeys the distribution P data ; represents the input label obeys the distribution P data .

[0033] During the network training process of the offline signature module, multiple loss functions are used to calculate the difference between the outputs of the low-level semantic learning branch, the high-level detail learning branch, the spatial attention module, and the spatial structure discrimination sub-network and the corresponding true values. Through back propagation, the weight parameters of the network are updated. Joint optimization of multiple loss functions obtains the final double-branch neural network loss function. Specifically, according to the document image I sig to be segmented, the semantic labels of the signature subject and the signature boundary region I label , the signature boundary region label I boder , the real signature mask label I alpha , the cross-entropy loss function L seg , the mean square error loss function L detail , the generative adversarial loss function L adv , the formula is called:

[0034] L total = λ seg Lseg (I sig ,I label )+λ detail L detail (I sig ,I border )+λ adv (I sig ,I alpha )computing the double-branch neural network loss function, wherein λ seg , λ detail , λ adv respectively represent the cross-entropy loss function, the mean square error loss function, and the weight coefficient of the adversarial loss function.

[0035] The spatial attention fusion module adopts an attention mechanism to guide the selective fusion of the convolution feature outputs at the end of the two branches, uses the low-level semantic learning branch end related character area prediction (signature subject + signature boundary) as the region attention information, and performs a point multiplication operation with the feature output at the end of the high-level boundary learning branch to select the signature-related features in the branch, and then performs a channel connection operation on the selected features and the low-level learning branch end convolution layer output for the final signature region mask prediction.

[0036] As shown in Figure 2 , the end-to-end offline signature automatic segmentation system structure schematic diagram of the application includes a signature detection module, an illegal signature verification module, a signature segmentation module (double-branch signature segmentation network), and a signature post-processing module. The signature detection module uses a lightweight text detector (Real-time Scene Text Detection with Differentiable Binarization, DBnet) (attention signature area detector) to quickly locate the specific position of the handwritten signature in the image; the illegal signature verification module filters out some obviously illegal candidate regions based on the provided detection box confidence and the pre-defined candidate box width-height ratio, and performs handwriting area candidate verification; the signature segmentation module uses the double-branch signature segmentation network described above, directly performs signature segmentation after obtaining the region of interest through the candidate box, and obtains the intermediate signature segmentation result of the rough handwriting output; the signature post-processing module further refines the obtained intermediate signature segmentation result, removes some obvious noise or other interference in the signature segmentation result through image processing operations such as erosion, dilation, and connected component calculation, and performs signature filtering and stroke structure optimization to obtain the signature output.

[0037] The following further describes the implementation of the offline handwritten signature segmentation of the application with a specific example.

[0038] S1. Handwritten signature material collection

[0039] 1) Collect various styles of handwritten signatures in large-scale population as the collection template with A4 paper with 10x5 table; 2) Use high-speed scanner to scan the A4 paper with handwritten traces to obtain corresponding image data; 3) Use Hough line detection algorithm to detect horizontal lines and vertical lines in the high-speed scanner image respectively, and calculate the intersection points for positioning a single signature, and then obtain a plurality of single handwritten signatures; 4) regarding the obtained single handwritten signature as two categories of foreground and background, using KMeans clustering algorithm to cluster it, and obtaining fine signature region mask marking;

[0040] S2. Document image background material collection

[0041] A crawler program can be constructed to crawl a large amount of document image data from the Internet, and then based on the Labelme (general image detection toolkit) tool, the representative styles such as fingerprints, seals, underlines, grids, printed bodies, etc. in the crawled data are locally marked to obtain representative background style material data;

[0042] S3. Training and test data construction

[0043] 1) Synthesize the signature data to be segmented: based on the single handwritten signature and the corresponding signature mask marking obtained in S1 and the document image background material obtained in S2, the signature foreground and the document background are fused to obtain the signature data to be segmented;

[0044] 2) Build signature edge transition area: dilate and erode the signature mask marking obtained in S1, and then subtract the results to obtain the signature edge transition area;

[0045] 3) Build semantic segmentation label: regarding the eroded signature mask in 2) and the signature edge transition area as two categories, and assigning a category label, wherein the eroded signature mask (as the main part of the signature) is assigned a label value of 1. The signature edge transition area (as the boundary of the signature) is assigned a label value of 2;

[0046] 4) Triplet input pair construction: organize the synthesized signature data to be segmented, semantic segmentation label, and signature region mask label into the format of triplets according to the corresponding relationship, and input the model training;

[0047] 5) Divide all the constructed triplets into training set and test set according to the proportion;

[0048] S4. Build a double-branch signature segmentation network model.

[0049] The double-branch signature segmentation network model comprises a high-level detail prediction branch, a low-level semantic learning branch, a spatial attention fusion module, and a spatial structure sub-discriminative network. The low-level semantic learning branch is based on an encoding-decoding network skeleton and is used for preliminary segmentation of a signature body and a signature surrounding boundary. The high-level boundary detail branch shares a high-level semantic feature map at the end of the low-level semantic branch encoder, and is subjected to an Atrous Spatial Pyramid Pooling (ASPP) operation and an up-sampling operation to realize specialized learning of signature boundary details. The spatial attention fusion module uses the low-level semantic learning branch to effectively fuse the convolution feature maps at the end of the two branches in the signature region prediction. The spatial structure discriminative sub-network uses a binary classification neural network, takes the generated signature region prediction as input, and evaluates whether it is sufficient to be true, so as to fool the signature segmentation network to generate a signature region prediction of higher quality.

[0050] The low-level semantic learning branch: the encoding part uses four convolutional layers to encode the features of the input signature image, and the decoding part uses four deconvolutions to restore the spatial dimensions of the signature image and predict the region where the signature body and boundary are located. The SE (Squeeze-and-excitation networks) module between the encoding part and the decoding part can automatically learn the importance of different channels while the model focuses on the image feature channels. For example, the SE module takes the output feature X of the last convolutional layer of the encoder as input, and the number of feature channels is C. First, the output feature X is subjected to a Squeeze operation, which changes each two-dimensional feature channel to a real number. Second, the 1x1xC compressed feature generates a weight w through an Excitation operation. Finally, the channel of the feature is weighted, for example, the weight generated by the Excitation operation can be weighted to the feature X through a multiplication operation, to complete the channel attention calibration of the feature X. The encoding part uses a pooling layer operation in the encoding process, which reduces the spatial dimension and easily causes the loss of some stroke details, resulting in a certain roughness in the result obtained by the up-sampling of the decoding part. Therefore, a jump structure is used to up-sample and optimize the output of the results of different convolutional layers in the encoding part.

[0051] High-level detail prediction branch: this example uses an empty spatial pyramid pooling (ASPP) to expand the receptive field of the convolution kernel, guiding the model to aggregate features in different receptive fields, so that the boundary features around the signature effectively link the signature foreground and document background information. In the ASPP module, 6 convolution layers are set, among which the convolution kernel size is set to 1x1 and 3x3, and the 3x3 size convolution kernel is set to 1, 2, 4, and 8. The first five convolution layers in the ASPP module are convolution operations on the feature maps output by the encoding network in the low-level semantic learning branch, which can use a 3x3 convolution kernel, and the last convolution layer can use a 1x1 convolution kernel to connect the output feature maps of the first five convolution layers and then perform channel compression to obtain the final output.

[0052] The signature segmentation network introduces a spatial structure discriminator subnetwork to evaluate the integrity of the predicted signature region. The spatial discriminator network in this embodiment can be selected to consist of three convolution layers and one fully connected layer. The spatial discriminator judges whether the input sample is a real signature mask or a predicted signature mask data, and outputs a probability value belonging to a real signature mask. During the training process, the spatial discriminator and the signature segmentation network are mutually antagonistic. On the one hand, the spatial discriminator network is trained to maximize the correct rate of distinguishing the real signature mask from the predicted signature mask, while forcing the signature segmentation network to generate higher quality signature mask prediction. The adversarial loss function is calculated according to the formula: where E represents the mathematical expectation, P data represents the data distribution of the signature region mask data, represents the signature region mask predicted by the double-branch signature segmentation network, I alpha represents the real signature mask label, L adv represents the generative adversarial loss function.

[0053] Spatial attention fusion module: an attention mechanism is used to guide the selective fusion of the convolution feature outputs at the end of the two branches. This module uses the character region prediction (signature body + signature boundary) at the end of the low-level semantic learning branch as regional attention information, and performs point multiplication operation with the feature output at the end of the high-level boundary learning branch to select the signature-related features in this branch. Then, the selected features are connected with the output of the low-level learning branch end convolution layer through channel connection operation for the final signature region mask prediction.

[0054] S5. Train the double-branch signature segmentation model using the signature segmentation training set to obtain a trained deep learning model.

[0055] The following illustrates the training process of the double-branch signature segmentation network.

[0056] The preprocessed document image to be segmented is input into the low-level semantic learning branch, and passes through the encoder, SE module and decoder in sequence to output a three-channel region prediction map (e.g., channel 0 outputs the region prediction of the document background, channel 1 outputs the region prediction of the signature body, and channel 2 outputs the region prediction of the signature boundary).

[0057] The high-level boundary detail branch shares the convolutional output at the end of the low-level semantic learning branch encoder. After passing through the ASPP upsampling module, it outputs a single-channel signature boundary prediction.

[0058] The spatial attention fusion module uses the character region prediction (signature body + signature boundary) at the end of the low-level semantic learning branch as region attention information. It performs a dot product operation with the feature output at the end of the high-level boundary learning branch to select the signature-related features in that branch. Then, it performs a channel connection operation with the convolutional layer output at the end of the low-level learning branch. After a 3x3 convolution operation, the final signature region mask prediction is obtained.

[0059] The spatial structure discrimination subnetwork takes the signature region mask prediction as input, and after forward propagation, outputs a probability value belonging to the real signature mask. The differences between the outputs of the low-level semantic learning branch, the high-level detail learning branch, the spatial attention module, and the spatial structure discrimination subnetwork and their corresponding real values ​​are calculated using multiple loss functions. These differences are then used to update the network's weight parameters through backpropagation. Specifically, this can be achieved using the formula:

[0060] L total =λ seg L seg (I sig I label )+λ detail L detail (I sig I boder )+λ adv (I sig I alpha Calculate the weight parameter L total Among them, I sig Represents the document image to be segmented, I label I represents the semantic markers indicating the signature body and the signature boundary region. boder Indicates the signature boundary region marker, I alpha Indicates the real signature mask mark, I seg L represents the cross-entropy loss function. detail Let λ represent the mean squared variance loss function. seg , λ detail , λ adv These represent the weight parameters of the cross-entropy loss function, the mean squared variance loss function, and the generative adversarial loss function, respectively.

[0061] The above steps are repeated until the model converges, obtaining a trained double-branch signature segmentation network. For an image to be segmented, it is directly input into the trained signature segmentation network to obtain the corresponding signature segmentation result.

[0062] In the actual inference process of the embodiment, since the network includes two branches, the high-level boundary learning branch may have inaccurate output results or even fail in the inference stage, affecting the quality of the final signature region mask and the segmented signature. To alleviate the above problems, the signature can be further optimized to obtain a more accurate signature. Specifically, according to the final output of the double-branch signature segmentation network Further refining the rough signature foreground output in the low-level semantic learning branch, specifically, according to the signature region mask prediction output by the last convolution layer of the double-branch network Signature region main body and signature boundary region prediction in the low-level semantic learning branch And Call formula: Calculate the final signature segmentation result I sig_mask .

[0063] Although the present application is explained in detail in connection with a limited number of embodiments, the present application is not limited to the use listed in the specification and embodiments, and can be applied to various fields suitable for the present application. For those skilled in the art, modifications, supplements and substitutions can be easily realized, and therefore the present application should not be considered as limited by the foregoing description, unless the general concept defined by the claims and their equivalent scope is deviated.

Claims

1. A method for offline handwritten signature segmentation based on a two-branch neural network, characterized in that, The position of the handwritten signature in the image is determined, a candidate region is determined, a region of interest image is obtained based on confidence of a detection frame and the candidate frame; A semantic feature map is obtained by performing preliminary segmentation on the region of interest image based on an encoding-decoding branch network, and feature categories are divided according to regions; Convolution feature maps are obtained by performing a dilated spatial gold tower pooling upsampling operation on the semantic feature map, and signature boundary details are learned; A predicted signature region mask is generated based on attention mechanism fusion of the semantic feature map and the convolution feature map; a binary classification neural network is used to evaluate whether the signature region prediction is sufficient as input, so as to fool the double-branch signature segmentation network to generate a higher-quality segmented signature; noise and interference in the segmented signature are removed through image processing operations, filtering and stroke structure optimization are performed, and finally the signature output is obtained.

2. The method of claim 1, wherein, The preliminary segmentation to obtain the semantic feature map further includes: feature encoding of the region of interest image is performed by using multiple convolution layers, signature image spatial dimensions are restored by using deconvolution, and regions where the signature subject and the boundary are located are distinguished, a SE module is used to compress and excite the network to focus on image feature channels, feature encoding is performed to obtain feature maps at multiple stages, and the features are sequentially recorded as E1-E4, wherein the feature E4 is weighted by the SE module according to the importance of the channels, and then the features E1-E3 of the previous three stages are respectively fused with different stages of the decoding network, and a feature map with three channels is output, each of which represents a category.

3. The method of claim 1, wherein, The dilated spatial gold tower pooling upsampling operation on the semantic feature map further includes: an ASPP module is used to aggregate features of different receptive fields by expanding the receptive field of the convolution kernel, so that the signature boundary features effectively link the signature foreground and the document background information, high-level detail prediction is performed, and more accurate signature segmentation is realized; six convolution layers are set to generate convolution feature maps, wherein the first five convolution layers perform convolution operations on the feature maps output by the low-level semantic learning branch module encoding network, and the last convolution layer uses a 1x1 convolution kernel to connect the output feature maps of the first five convolution layers and then performs channel compression to output the convolution feature map.

4. The method according to claim 1 or 2, characterized in that, The feature category division according to regions further includes: corresponding feature center vectors are assigned according to the spatial positions of the signature subject, the signature boundary and the background, so that image features belonging to the same category are attracted to the assigned feature center, and the "intra-class consistency" between different categories of features in the output feature map is strengthened by using a center loss; after forward propagation, a three-channel region prediction map is output, the features output by the last convolution layer of the encoder are compressed, each feature channel is changed into a real number, the compressed features are generated by excitation operation, the weights are weighted to the features channel by channel, and the channel attention of the features is completed.

5. The method according to any one of claims 1 to 3, characterized in that The generating predicted signature region mask further comprises capturing signature space structure, and the spatial discrimination sub-network is mutually opposed with the double-branch signature segmentation network. On one hand, the spatial discrimination sub-network maximizes the distinction between the real signature mask and the predicted signature mask, and simultaneously forces the double-branch signature segmentation network to generate a higher quality predicted signature mask. According to the predicted signature mask, the real signature mask is marked Real signature mask mark I alpha , the double-branch signature segmentation network G, the spatial structure discriminator D, and the calling formula is: Calculate the adversarial loss function L adv (G, D), according to the to-be-segmented document image I sig , the semantic mark I of the signature body and the signature boundary region label , the signature boundary region mark I border , the real signature mask mark I alpha , the cross-entropy loss function L seg , the mean square error loss function L detail , and the calling formula is: L total = λ seg L seg (I sig , I label ) + λ detail L detail (I sig , I border ) + λ adv (I sig , I alpha ) is a loss function of the double-branch signature segmentation network, wherein λ seg , λ detail , λ adv respectively represent a cross-entropy loss function weight coefficient, a mean square error loss function weight coefficient, and an adversarial loss function weight coefficient.

6. An offline handwritten signature segmentation system based on a two-branch neural network, characterized in that, The method comprises the following steps: The signature detection module determines the position of the handwritten signature in the image and determines a candidate region; The illegal signature verification module filters out legal candidate regions to obtain a region of interest image based on confidence of a detection frame and the candidate frame; The signature segmentation module further comprises a low-level semantic learning branch module, a high-level boundary learning branch module, a spatial attention fusion module, and a spatial structure discrimination subnetwork. The low-level semantic learning branch module performs preliminary segmentation on the signature body and the signature surrounding boundary based on an encoding-decoding network skeleton to obtain a semantic feature map. The high-level boundary learning branch module learns signature boundary details through an empty spatial gold tower pooling upsampling operation on the semantic feature map to obtain a convolution feature map. The spatial attention fusion module fuses the semantic feature map and the convolution feature map based on an attention mechanism to predict a signature region mask. The spatial structure discrimination subnetwork uses a binary classification neural network to evaluate whether the signature region mask is sufficient by taking the signature region mask as input, so as to fool the signature segmentation module to generate a higher-quality segmented signature. The signature post-processing module performs filtering and stroke structure optimization to obtain a final signature output.

7. The system of claim 6, wherein, In the low-level semantic learning branch module, the encoding part uses multiple convolution layers to encode features of the region of interest image, and the decoding part uses an inverse convolution corresponding to the number of convolution layers in the encoding part to restore the spatial dimension of the signature image and distinguish the regions where the signature body and the boundary are located. The SE module is used to focus on the image feature channels between the encoding part and the decoding part. The encoding part encodes features to obtain multiple stage feature maps, which are denoted as E1 to E4 in sequence, wherein the feature E4 is weighted by the SE module according to the importance of the channels, and then the features E1 to E3 of the previous three stages are fused with different stages of the decoding network to output a feature map with three channels, each of which represents a category.

8. The system of claim 6, wherein, The high-level boundary learning branch module uses the ASPP module to aggregate features of different receptive fields by expanding the receptive field of the convolution kernel, so that the signature surrounding boundary features effectively link the signature foreground and the document background information, perform high-level detail prediction, and realize more accurate signature segmentation. In the ASPP module, the first five convolution layers respectively perform convolution operations on the feature maps output by the encoding network of the low-level semantic learning branch module, and the last convolution layer uses a 1x1 convolution kernel to connect the output feature maps of the first five convolution layers and then performs channel compression to output a convolution feature map.

9. The system according to any of claims 6-8, characterized in that The spatial attention fusion module uses the attention mechanism, uses the low-level semantic learning branch end character region prediction containing the signature body and the signature boundary as regional attention information, and performs a point multiplication operation with the feature output at the end of the high-level boundary learning branch. The signature features in the high-level boundary learning branch are selected, and the selected signature features are connected with the output of the convolution layer at the end of the low-level semantic learning branch through a channel connection operation to predict the final signature region mask.

10. The system according to any of claims 6-8, characterized in that The low-level semantic learning branch module further comprises a feature clustering part which divides feature categories according to regions, and assigns corresponding feature center vectors according to the spatial positions of the signature body, the signature boundary and the background, so that the image features belonging to the same category converge to the assigned feature center. The formula is called: The feature center vector loss is determined, wherein, represents the feature vector of the i th pixel in the three-channel output, represents the feature center of the category, and N represents the total number of pixels in the feature map.

11. The system of claim 10, wherein, The center loss is used to strengthen the intra-class consistency between different class features in the output feature map of the last convolution layer in the low-level semantic learning branch module, and after back propagation, a three-channel region prediction map is output. The feature X output by the last convolution layer of the encoder is compressed, each two-dimensional feature channel is changed into a real number, and the compressed feature generates a weight through the excitation operation. The weight is weighted to the feature X channel by channel, and the channel attention calibration of the feature X is completed.

12. The system according to any of claims 6-8, characterized in that The spatial structure discriminator subnetwork and the double-branch signature segmentation network are mutually antagonistic. On one hand, the spatial structure discriminator subnetwork maximizes the distinction between a real signature mask and a predicted signature mask, and at the same time forces the double-branch signature segmentation network to generate a predicted signature mask of higher quality. According to the signature mask prediction result Real signature mask label I alpha , the double-branch signature segmentation network G, the spatial structure discriminator D, and the calling formula: Computing the adversarial loss function L adv (G, D), according to the document image I sig , semantic labeling of the signature body and the signature border region I label , signature border region labeling I border , real signature mask labeling I alpha , cross-entropy loss function L seg , mean square error loss function L detail , call formula: L total = λ seg L seg (I sig , I label ) + λ detail L detail (I sig , I border ) + λ adv (I sig , I alpha ) calculate the loss function of the double-branch signature segmentation network, wherein λ seg , λ detail , λ adv respectively represent the cross-entropy loss function weight coefficient, the mean square error loss function weight coefficient, and the adversarial loss function weight coefficient.

Images