Cross-modal handwriting-based co-training method, handwriting comparison system, device and medium

Abstract

The application discloses a cross-modal handwriting-based class collaborative training method and system, a handwriting comparison method and a medium. Paper handwritten signature image is collected and converted into a binary signature image, electronic signature sequence normal writing data and imitation writing data are echoed as signature image data; the signature images of two modal types are aligned, spliced in the channel dimension to construct corresponding electronic-paper signature image pairs; the image pair data set is divided into a labeled data set and an unlabeled data set; a double-tower network structure is constructed, the labeled data set is randomly sampled twice and sequentially input into different networks, the unlabeled data set is randomly sampled once and sequentially input into different networks, the soft labels output by the cross network are combined to generate adversarial sample data by pixel disturbance on the unlabeled data, the corresponding non-cross network is input, cross-modal comparison multi-task joint classification training is performed, and cross-modal handwriting signer identity is determined according to the comprehensive classification result. The application is widely used in places where cross-modal signature recognition is required.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information processing, and particularly relates to a cross-modal handwriting comparison method based on a class collaborative training mode. BACKGROUND

[0002] With the wide application of original handwriting signature products based on handwriting biometric recognition technology in many fields such as finance, government affairs, justice, public security and people's livelihood, the application demand of the handwriting signature scene is further expanded, and gradually develops from single-modal paper handwriting signature handwriting comparison to single-modal electronic handwriting signature handwriting comparison, and then to cross-modal electronic-paper handwriting signature handwriting comparison. In this transition process, the signing mode, device, scene and data form of the signer have changed a lot, which also brings great challenges to the existing signature handwriting comparison method. The present application is directed to the cross-modal handwriting comparison which is more difficult.

[0003] In the actual application under the current era background, the application scene of taking offline paper handwriting signature image as the identification sample and taking electronic handwriting signature data as the evidence is more than that of taking online electronic handwriting signature sequence sample as the evidence, so the comparison of cross-modal signature data is one of the future trends, but the comparison of cross-modal signature data is a very challenging technical problem. As known, due to the wide distribution range of the difference in form, style and habit of the signature handwriting signed by different signers, even the single-modal paper or electronic handwriting signature signed by the same signer under normal state may have large visual difference, and the cross-modal handwriting signature increases more uncertainty on this basis, that is, due to the change of signing medium and writing method, the form difference is relatively more obvious, and especially the cross-modal writing difference of different signers is difficult to be measured by a unified standard to a certain extent.

[0004] Publication number: CN111680490A, name: Chinese invention patent application for a cross-modal document processing method, device and electronic device. The text modality data and image modality data of a first document are obtained; based on a natural language processing model, the text modality data is converted into a word feature embedding vector, and the first text element feature is extracted according to the text modality data; based on a computer vision algorithm, the target position is located and the text content of the first document is identified through a target detection algorithm and an optical character recognition algorithm, and the second text element feature is extracted according to the image modality data, and the elements are aligned to obtain the structural features of the first document; then, the embedded representation of the structural features and the multi-dimensional features of the document is combined to obtain a meta-knowledge graph model that includes a representation of the first document. The text content of the document is converted into a meta-knowledge graph model, and the embedded representation is learned using a graph representation learning model to obtain a meta-knowledge graph model that includes a representation of the first document. The cross-modalities involved are language text modality and image modality, and do not involve signature images and electronic signatures converted from paper materials. Summary of the Invention

[0005] In view of the above-mentioned defects in the prior art cross-modal signature handwriting recognition and comparison and in combination with actual application scenarios, the present invention proposes a cross-modal handwriting comparison method and comparison system based on a collaborative training method.

[0006] According to one aspect of the present application, a cross-modal handwriting-based collaborative training method includes: collecting paper handwritten signature images and converting them into binary signature images, obtaining electronic signature sequences to generate orthogonal electronic signature images, generating electronic signature imitation data through electronic signature sequence data, and generating imitation electronic signature images; aligning signature images of two modal types, splicing them in the channel dimension to construct corresponding electronic-paper signature image pairs; performing data set segmentation on the electronic signature image pairs; constructing an image pair data set including a labeled data set and an unlabeled data set; constructing a classification backbone network including a cross network and a non-cross network dual-branch network structure, randomly sampling the labeled data set twice and inputting them into two branch networks in sequence, randomly sampling the unlabeled data set once and inputting them into two branch networks, performing pixel perturbation on the unlabeled data in combination with the soft labels output by the cross network to generate adversarial sample data, which are respectively input into the non-cross network, and performing cross-modal comparison multi-task joint classification training until the training is completed when the classification loss value is minimized.

[0007] Further optimization, the generation of electronic signature imitation data includes: dividing the electronic signature sequence into single words to produce a single word sequence font library, searching for single word sequences with consistent content from the font library according to the corresponding electronic signature image content, performing signature splicing and then signature echoing to generate electronic signature imitation image data.

[0008] Further optimization, the electronic signature copy data generation includes: according to the corresponding electronic signature normal writing data sequence, the part of strokes are disturbed to generate the electronic signature copy image data.

[0009] Further optimization, the paper handwritten binary signature image and the electronic signature image data alignment includes alignment in thickness, scale and spatial position, the paper handwritten binary signature image is obtained by inflation, corrosion and wild point removal to obtain a binary signature image with the same scale as the electronic signature; the paper handwritten binary signature angle is improved by affine change operation, the maximum and minimum word spacing in the paper binary signature is adjusted according to the corresponding electronic signature, the center operation is performed according to the pixel boundary box to make the paper binary signature in the image center position, and the scaling operation is performed to a fixed size; the skeleton thinning processing is adopted to make the strokes of the two modal signature images have the same thickness.

[0010] Further optimization, the electronic signature image pair is divided into a labeled data set D and an unlabeled data sample set U, the labeled data sample set D is divided into a first labeled data subset d1 and a second labeled data subset d2, which are respectively input into a first branch network n1 and a second branch network n2 in the classification backbone network, the unlabeled data set U is input into the first branch network and the second branch network by generating multiple unlabeled subsets through pixel disturbance adversarial sample generation, the classification backbone network can classify the labeled data sample set and the unlabeled data sample set, and also assign labels to the unlabeled adversarial samples, and thus determine the joint classification training loss of the cross-modal data.

[0011] Further optimization, the classification of the labeled sample set and the unlabeled sample set includes: batch sampling the labeled data set D to obtain data subsets d1 and d2, inputting the data subsets d1 and d2 into networks n1 and n2 respectively, and outputting the classification results corresponding to the data subsets d1 and d2 as logit_n1_d1, logit_n1_d2, logit_n2_d1 and logit_n2_d2 respectively, calculating the cross-entropy classification loss according to the corresponding labels of d1 and d2 to obtain the corresponding loss, which is marked as loss_d, batch sampling the unlabeled data set U to obtain a data subset u1, inputting the data subset u1 into networks n1 and n2 simultaneously, and outputting the classification results corresponding to the data subset u1 as logit_n1_u1 and logit_n2_u1 respectively; generating first and second unlabeled adversarial samples G_n1_u1 and G_n2_u1 by twice disturbance of the data subset u1, inputting the samples generated by the above disturbance into sub-networks n1 and n2 respectively, and taking the classification results output by the networks as their respective labels, that is, assigning labels to the above two unlabeled adversarial samples.

[0012] Further optimization, determine the joint classification training loss of cross-modal data as: using the labeled data set binary classification cross entropy loss loss_d, the integrated binary classification cross entropy loss of the unlabeled data set and the unlabeled adversarial generated data set loss_u, the prediction distribution consistency loss of the unlabeled adversarial generated data set loss_g_u, according to the formula:

[0013] Total_loss = α*loss_d + β*loss_u + γ*loss_g_u, calculate the joint classification training loss Total_loss of cross-modal data, wherein α, β, γ represent the weights of the above three losses respectively.

[0014] According to another aspect of the application, a handwriting comparison system based on cross-modal handwriting class collaborative training includes a data acquisition module, a data preprocessing module, a classification backbone network, and a class collaborative training module. The data acquisition module acquires paper handwritten signature materials signed by different signers, corresponding signer electronic signature sequence writing data and other signers' electronic signature sequence imitation data, and labels the corresponding identity authenticity label information. The data preprocessing module converts the paper handwritten signature materials into corresponding binary paper handwritten signature images through detection and matting, echoes the electronic signature sequence imitation data as electronic handwritten signature images, aligns the signature images of the two modal types, concatenates and constructs corresponding electronic-paper signature image pairs in the channel dimension, and divides the electronic signature image pairs in the data set into labeled data sets and unlabeled data sets according to ID. The classification backbone network adopts two double-branch networks with the same structure, the class collaborative training module inputs the labeled data set into two different branch networks of the classification backbone network twice in turn, inputs the unlabeled data set into two different branch networks at the same time once, generates corresponding adversarial sample data by pixel disturbance on the unlabeled data combined with the soft label output by the cross network, and inputs the data into the non-cross network respectively, performs cross-modal comparison multi-task joint classification training, and completes the training when the classification loss value Total_loss is the minimum. The cross-modal reserved sample signature test comparison module is obtained, which compares and recognizes the cross-modal signature handwriting.

[0015] Further optimization, including, performing dataset splitting on the electronic signature image pair to obtain a labeled data sample set D and an unlabeled data sample set U, dividing the labeled data sample set D into a first labeled data subset d1 and a second labeled data subset d2, and respectively sending them into a first branch network n1 and a second branch network n2 in the classification backbone network, and generating a plurality of unlabeled subsets by generating adversarial samples from the unlabeled data set U, and simultaneously inputting the first branch network and the second branch network, and the classification backbone network classifies the labeled data sample set and the unlabeled data sample set, generates adversarial samples according to the disturbance of the unlabeled data sample set, assigns labels to the unlabeled adversarial samples, and determines the joint classification training loss of the cross-modal data.

[0016] Further optimization, the classification backbone network performs batch sampling on the labeled data set D to obtain data subsets d1 and d2, respectively inputs the data subsets d1 and d2 into the networks n1 and n2, respectively outputs the classification results corresponding to the data subsets d1 and d2, and marks them as logit_n1_d1, logit_n1_d2, logit_n2_d1, and logit_n2_d2, respectively calculates the sum of the cross-entropy classification losses according to the corresponding labels of d1 and d2 to obtain the corresponding loss, and marks it as loss_d, performs batch sampling on the unlabeled data set U to obtain a sub-data set u1, simultaneously inputs the sub-data set u1 into the networks n1 and n2, respectively outputs the classification results corresponding to the sub-data set u1, and marks them as logit_n1_u1 and logit_n2_u1, generates first and second unlabeled adversarial samples G_n1_u1 and G_n2_u1 by twice disturbing the sub-data set u1, respectively inputs the samples generated by the above disturbance into the sub-networks n1 and n2, and takes the classification results output by the networks as their respective labels, that is, assigns labels to the above two unlabeled adversarial samples.

[0017] Further optimization, the joint classification training loss of the cross-modal data is determined as follows: the binary classification cross-entropy loss loss_d of the labeled data set, the binary classification cross-entropy loss loss_u of the unlabeled data set and the unlabeled adversarial generated data set, and the prediction distribution consistency loss loss_g_u of the unlabeled adversarial generated data set are combined according to the formula:

[0018] Total_loss = α*loss_d + β*loss_u + γ*loss_g_u, wherein α, β, and γ represent the weights of the above three losses.

[0019] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method of any one of the above.

[0020] According to another aspect of the present application, the computer instructions are for causing the computer to perform the method described above.

[0021] The present application solves the difficult problem of electronic-paper handwritten signature handwriting comparison in the field of electronic signature handwriting authentication and the field of comparison and recognition. First, due to the large difference in modalities, the cross-device, cross-writing method and cross-scene conditions of the signor during signing are complex, and the task is difficult. Second, due to the high time and money cost of simply collecting data of different signers in different scenes and different states, especially in order to ensure the richness of the imitation data, the imitation data corresponding to the paper handwritten signature is generated and supplemented by data generation. Third, due to the label uncertainty problem of the normal writing data and the imitation data that is prone to occur in the data collection or collection under the cross-modality data difference, a double-tower model is constructed to learn the model of different data from two perspectives based on the class collaborative training method, which can improve the label uncertainty problem to a certain extent. Through semi-supervised learning, the predicted value is combined with the pixel disturbance generated by the adversarial training to narrow the distribution of the feature space under the double perspective, thereby improving the robustness of the model. Finally, based on the integrated learning method, the prediction results under the double perspective are fused to comprehensively judge the true identity of the signor. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Schematic diagram of the cross-modality handwriting comparison training process;

[0023] Figure 2 Schematic diagram of the cross-modality handwriting comparison test process;

[0024] Figure 3 The electronic device structure schematic diagram according to the exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Embodiments of the present application will be described in more detail by referring to the attached drawings. Although certain embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are for exemplary purposes only and are not intended to limit the scope of protection of the present application.

[0026] It should be understood that each step described in the method embodiments of the present application can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0027] The term "include," and derivations thereof, is an open term that means "including, but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related terms shall be construed accordingly. It should be noted that "a" or "an" entity as used herein indicates "one or more" of that entity. The terms "first," "second," and the like as used herein do not imply these components must be in a certain order, but these terms are used to distinguish one component from another. The terms "coupled" and "connected" as used herein mean the joining of cause and effect, i.e., having a causal relationship. Both "coupled" and "connected" shall not be construed as possessing solely mechanical characteristics.

[0028] It should be noted that the use of "a" or "an" herein does not denote a limitation of quantity, but rather a limitation of at least one. Further, the use of "multiple" herein does not denote a limitation of quantity, but rather a limitation of more than one.

[0029] The names of the messages or information exchanged between the devices in the embodiments of the present application are used only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0030] The present application will be further described in greater detail by way of specific embodiments, with reference to the attached drawings in which:

[0031] The embodiment collects paper handwritten signature materials signed by different signers, collects or generates electronic signature sequence normal writing data and other signers' electronic signature sequence imitation data; converts the paper handwritten signature materials into corresponding binary signature image representations after signature detection and matting operations, and echoes the electronic signature sequence normal imitation data as signature image data representations; aligns the signature images of the two modal types as much as possible in vision through image denoising, refinement and scale change preprocessing methods, and constructs corresponding electronic-paper signature image pairs in the channel dimension; divides the constructed image pair dataset into labeled dataset and unlabeled dataset according to a fixed ratio; constructs a double-tower network structure, the two branch networks have the same structure, randomly samples the labeled dataset twice during training and inputs them into different networks in turn, randomly samples the unlabeled dataset once and inputs it into different networks in turn, simultaneously generates corresponding adversarial sample data by pixel disturbance of the unlabeled data combined with the soft labels output by the cross network, and inputs them into the corresponding non-cross network for cross-modal comparison and multi-task joint classification training; during testing, the original handwritten signature materials and electronic sequences are preprocessed to form electronic-paper signature image pairs, and the double-tower model classification results are combined to determine the authenticity of the cross-modal handwriting signer identity.

[0032] The scheme of the present application is described below with reference to the accompanying drawings.

[0033] First, paper handwritten signature materials and electronic signature sequence data are obtained, and the obtained data are preprocessed.

[0034] The preprocessing of the paper handwritten signature material part mainly includes: paper handwritten signature material to image, detection and extraction of handwritten signature image. The paper handwritten signature material is converted into corresponding electronic signature material image by a high-speed scanner. Since the converted signature material image contains not only a single signature image but also a certain degree of background, a target detection algorithm based on yolo_v3 series is used to first detect the handwritten signature position area of the paper scan image. Based on the fact that the handwritten signature position area may contain noise such as printed matter, ink pad, seal, etc., and we hope that the handwritten signature image is a binary representation type, further signature matting is performed based on the existing signature area, i.e. excluding other noise interference, extracting the complete handwritten signature image from the original detection area for binary conversion and storage, and finally obtaining the preprocessed binary signature image. The signature extraction method can use an image semantic segmentation algorithm based on UNet architecture, or other methods known to those skilled in the art.

[0035] The preprocessing of the electronic handwritten signature sequence data part includes: data verification and repair of electronic signature sequence, conversion of electronic signature sequence data to electronic signature image data, and generation of electronic signature imitation data.

[0036] Data verification is performed on the electronic signature sequence data, including quality screening, sequence repair, and resampling. The quality screening mainly judges the distribution of the values of the signature track coordinates (x, y), pressure p, pen-up and pen-down state s, and time t in the current electronic signature sequence data, and removes sequence data that does not conform to the regular distribution. The sequence repair mainly supplements and resets missing values and abnormal values in the electronic signature sequence data. The resampling is a downsampling process for part of the long sequence in the electronic signature, aiming to remove some redundant points and shorten the sequence length.

[0037] The electronic signature sequence data is converted into electronic signature image data, and the coordinates (x, y) and the pen-up and pen-down state s in the electronic signature sequence data are used for binary image echo. The image width*height*channel can be 224*112*1.

[0038] The electronic signature copy image data is generated. Since there is a large proportion of unevenness in the actual collection and collection of positive copy data, data enhancement and expansion are performed on the copy data. There are two main ways: one is to divide all electronic signature sequences into single-character sequences to make a single-character sequence library. According to the content of the corresponding electronic signature writing image, the single-character sequence with the same content is searched from the library to generate a new electronic signature sequence by signature splicing. The second is to generate a new electronic signature sequence by disturbing part of the strokes according to the corresponding electronic signature sequence. The generated electronic signature sequence is converted into electronic signature copy image data by binary echo.

[0039] Through the above preprocessing, the paper-based handwritten signature material data collected can be converted into corresponding paper-based handwritten binary signature image data through detection and matting. The electronic signature sequence data collected can be echoed as corresponding electronic signature image data. The current electronic signature image data (regarded as electronic signature writing image data) can also generate corresponding electronic signature copy image data.

[0040] Then, the paper-based handwritten binary signature image data and the electronic signature image data obtained through the above preprocessing are aligned and image pairs are constructed.

[0041] The alignment and preprocessing of the paper-based handwritten binary signature image and the electronic signature image data mainly include: aligning the paper-based handwritten binary signature image and the electronic signature image in thickness, scale, and spatial position.

[0042] 1) Scale alignment, the size of the paper handwritten binary signature image extracted is also dynamically changing, and the signature size may be small due to noise points. Therefore, dilation and erosion and noise point removal operations are used to remove background noise and background edge of the paper handwritten binary signature image; 2) Spatial position alignment, due to the problems of signature angle, word spacing and non-centralization of signature in the paper handwritten binary signature image, the signature angle is improved through affine transformation operation, the maximum and minimum word spacing in the paper signature is dynamically adjusted by referring to the word spacing of the corresponding (same ID mark) electronic signature, the signature is centralized according to the pixel boundary box, so that the signature is in the center position of the image, and the whole image is scaled to a fixed size (such as 224*112*1); 3) Thickness alignment, the paper handwritten binary signature image is relatively thick in visual performance, and the thickness range of different signatures jumps relatively large, the electronic signature image is binary echoed, and the stroke thickness is consistent, and the skeleton thinning algorithm is used for processing, so that the two reach the same thickness level. The purpose of the multi-aspect alignment processing of the paper handwritten binary signature image and the electronic signature image data is to further improve the difference between the cross-modal data in the image visual level. After this processing, the corresponding paper handwritten binary signature image and electronic signature image data with small visual performance difference can be obtained. Since the electronic signature image data is mainly used as a corresponding reference object, it will not change basically, and the paper handwritten binary signature image as an adjustment object will be greatly changed in scale, spatial position and thickness granularity.

[0043] Constructing image pairs, all electronic signature images and their corresponding ID paper handwritten binary signature images are matched in 1V1 mode and spliced on the channel, so as to facilitate subsequent multi-task classification training of cross-modal signature data.

[0044] Splicing the corresponding electronic-paper signature image pairs, the signature image pairs are divided into cross-modal training data and test data according to the ID of the signatory. For example, a 9:1 allocation ratio can be used, in which 9 / 10 is cross-modal training data and 1 / 10 is cross-modal test data.

[0045] Constructing a cross-modal handwriting comparison training module based on a class collaborative training method.

[0046] As shown in Figure 1 The cross-modal handwriting comparison training flowchart of one of the embodiments of the application.

[0047] The electronic signature sequence is echoed as an electronic signature image, a paper signature image is detected and cut out to obtain a paper handwritten binary signature image; the electronic signature image and the paper handwritten binary signature image are spliced through a channel to obtain an electronic-paper signature image pair; the electronic signature image pair is cut into a data set to obtain a labeled data set D and an unlabeled data set U, the labeled data set D is divided into a labeled subset d1 and a labeled data set d2, which are input into network structure 1 and network structure 2 (branch 1 and branch 2) in the classification backbone network, the unlabeled data set U is output as an unlabeled data set u1 at a fixed viewing angle, and multiple unlabeled subsets are generated through the generation of adversarial samples, such as: unlabeled subset (G-n1-u1) and unlabeled subset (G-n2-u1), and all the unlabeled data subsets are input into network structure 1 and network structure 2 in the classification backbone network; network structure 1 and 2 are classified and output.

[0048] Specifically: online dynamic data preprocessing, double-tower network structure and multi-task joint training.

[0049] (1) Online dynamic data preprocessing. Divide the labeled data set and the unlabeled data set, sample the double-view labeled data set, and generate the unlabeled adversarial sample. Use the signer ID as the signature label, and divide the training data into labeled data set and unlabeled data set according to the signer ID (according to the proportion of 9:1, such as 9 / 10 as labeled data set, and 1 / 10 as unlabeled data, which is considered as unlabeled data). In the training process, the labeled data U is randomly divided into two data subsets d1 and d2 with the same data amount, d1 and d2 have the same data amount, but due to the inconsistent arrangement of actual data, the training iteration order is inconsistent, which can be considered as two types of training viewing angles. The labeled data subsets d1 and d2 are respectively randomly introduced into network structure 1 and network structure 2 in the classification network twice. The unlabeled data is only randomly introduced into network structure 1 or network structure 2 once, the viewing angle is fixed, and the corresponding unlabeled data subset u1 is output; in order to further constrain the difference and complementarity of model training under the condition of double viewing angles, the first unlabeled sample G_n1_u1 and the second unlabeled sample G_n2_u1 are generated by using the twice disturbance adversarial sample generation method (FGSM) on the unlabeled data set u1, and the labels of the above two unlabeled samples are assigned, the classification result logit_G_n1_u1 output by network structure 1 after inputting G_n1_u1 is taken as the label of G_n1_u1, and the classification result logit_G_n2_u1 output by network structure 2 after inputting G_n2_u1 is taken as the label of G_n2_u1.

[0050] (2) The classification backbone network adopts a double-tower network structure, and adopts a collaborative training method to construct different training perspectives for the same training set. The double-tower network includes network structure 1 and network structure 2, and network structure 1 and network structure 2 adopt the same structure (for example, a commonly used feature extraction backbone network resnet50 in deep learning can also be used, or other network structures can be used). The precondition of this training method is that it is believed that there are actually two views of the current training set data, and training two models can achieve complementary or different results. However, in the actual process, collaborative training will make the results of the two models more consistent, so generated adversarial sample data is needed.

[0051] (3) Multi-task joint training. The multi-task joint training mainly includes the following tasks: a main task (labeled data binary classification), which is mainly to determine the signature identity of the labeled electronic-paper signature image pair data, that is, to determine whether the image pair is signed by one person and consistent with the true label; a secondary main task 1 (unlabeled data binary classification + unlabeled adversarial data classification), which is mainly to determine the signature identity of the original and generated unlabeled electronic-paper signature image pair data, and whether it is consistent with the label output by the classification result of the other branch network; a secondary main task 2 (unlabeled adversarial data prediction distribution consistency discrimination), which is mainly to respectively determine the distribution consistency of the feature vectors of the generated unlabeled data and the original unlabeled data. The classification backbone network classifies the labeled sample set and the unlabeled sample set, and generates adversarial samples. The loss function of the classification backbone network is determined according to the labeled data set, the unlabeled data set, and the adversarial samples generated by disturbing the unlabeled data set. Through training, the labeled data is classified into two categories, the unlabeled data is classified into two categories, the unlabeled adversarial data is classified, and the distribution consistency of the unlabeled adversarial data is predicted and discriminated.

[0052] (4) The following will further illustrate the implementation of multi-task joint training by taking a specific example.

[0053] After batch sampling the labeled datasets D1 and D2, sub-datasets d1 and d2 are obtained. The sub-datasets d1 and d2 are input into the networks n1 and n2 respectively to obtain the corresponding classification results logit_n1_d1, logit_n1_d2, logit_n2_d1, logit_n2_d2. According to the classification results of d1 and d2, the cross entropy classification loss is used to calculate and sum the corresponding loss loss_d. Batch sampling is performed on the unlabeled dataset U to obtain an unlabeled subset u1. The perturbed adversarial sample generation method (FGSM) is used on this subset to generate the corresponding first unlabeled sample G_n1_u1 and second unlabeled sample G_n2_u1. The unlabeled data subsets are fed into networks n1 and n2, respectively, to obtain the classification outputs logit_n1_u1 and logit_n2_u1. Labels are assigned to these two unlabeled samples. The classification output of G_n1_u1 after feeding it into network n1 is used as the label for G_n1_u1, and the classification output of G_n2_u1 after feeding it into network n2 is used as the label for G_n2_u1. Based on these classification results, cross-entropy classification is used to calculate and sum the corresponding losses loss_u and loss_g_d. The loss function of the classification backbone network is determined based on the labeled dataset samples, unlabeled samples, and unlabeled adversarial samples. Specifically, it can be as follows: calculate the binary cross entropy for the unlabeled samples and the unlabeled adversarial generated samples and sum them up to obtain the corresponding binary cross entropy loss loss_u. At the same time, in order to further supervise the consistent expression of training under dual perspectives, add view differentiation constraints, and use JS divergence to measure the similarity of the feature expression output by the unlabeled adversarial generated data network. The corresponding loss function value is JS divergence, which is regarded as loss_g_u. Therefore, the joint classification training loss of the dual-tower model constructed based on the collaborative training method using cross-modal data is constructed by using the binary cross entropy loss loss_d of the labeled dataset, the binary cross entropy loss loss_u of the unlabeled dataset and the unlabeled adversarial generated dataset, and the predicted distribution consistency loss loss_g_u of the unlabeled adversarial generated dataset. Specifically, according to the formula:

[0054] Total_loss = α*loss_d + β*loss_u + γ*loss_g_u, which calculates the joint classification training loss Total_loss. α, β, and γ represent the classification loss weights for the labeled dataset, the classification loss weights for the unlabeled dataset, and the distribution consistency loss weights for the unlabeled adversarial generation dataset under two views, respectively.

[0055] The network is trained by using the training set samples, and a cross-modal sample-retained test material comparison module is obtained after the training is completed.

[0056] The training process can be adjusted in time according to the loss convergence. The final goal of the cross-modal handwriting comparison data joint classification training based on the collaborative training mode is to minimize the joint training classification loss value Total_loss, that is, the training is completed when the classification loss value Total_loss is minimized, and a cross-modal double-tower classification model is obtained. When Total_loss is trained to be relatively optimal, it can be considered that the double-tower model training is good, the cross-modal classification effect under different angles is optimal, and the consistency under the two angles is relatively high.

[0057] As shown in Figure 2 The present application is a cross-modal signature handwriting comparison test flow diagram. The electronic signature sequence echo is an electronic signature image, the paper signature image is detected and cut out to obtain a paper handwritten binary signature image; the electronic signature image and the paper handwritten binary signature image are spliced through the channel to obtain an electronic-paper signature image pair; the first network and the second network in the cross-modal sample-retained signature test comparison module classify the signature image pair according to the input electronic-paper signature image pair, and output two classification results respectively, and the two classification results are combined to obtain the prediction result of whether the paper signature and the electronic signature are signed by the same person.

[0058] The above cross-modal test data after data preprocessing are used for comparison test, and finally a good classification accuracy can be obtained. At the same time, after the online end-to-end test is carried out in the actual deployment of the server test environment, the user experience effect is better.

[0059] According to the data preprocessing operation in the data training process in the present application, the actual handwritten signature handwriting comparison or identification scene is generally as follows. The paper handwritten signature material can be regarded as sample-retained data, the electronic signature sequence data collected by a touch screen or the like can be regarded as test material data. In the test, the paper handwritten signature material sample-retained is converted into a paper handwritten binary signature image through signature detection and cutout processing, the electronic handwritten signature sequence test material data is binary echoed into a standard size electronic signature image, and the two are spliced through the channel after spatial position, scale and thickness alignment and other preprocessing operations. The trained double-tower classification model (cross-modal sample-retained signature test comparison module) is inputted to perform the same identity discrimination, that is, the double-branch network model is used to discriminate whether the image pair is signed by one person, two classification prediction results can be obtained, and finally the final discrimination result is obtained by comprehensively judging the two classification prediction values. The comprehensive judgment basis is that if both are true, the signatory is the same person, and if one is false, the signatory identity does not correspond, which is a copy.

[0060] The embodiment of the application provides a cross-modal handwriting identification system based on a class co-training mode, which comprises a data acquisition module, a data preprocessing module, a classification backbone network based on a double-tower network, and a cross-modal sample retention test comparison module.

[0061] The data acquisition module acquires offline paper handwritten signature materials signed by different signers, corresponding electronic signature sequence writing data of the signers and electronic signature sequence imitation data imitated by other signers, and labels corresponding identity authenticity label information.

[0062] The data preprocessing module pre-processes the paper handwritten signature materials and the electronic handwritten signature sequence data, aligns the paper and electronic handwritten signature images, and constructs the image pairs.

[0063] The obtained paper handwritten signature materials are converted into corresponding paper handwritten binary signature image representations after signature detection and matting operations, and the electronic signature sequence imitation data is converted into electronic handwritten signature image data representations through binaryzation. Since the proportion of actual imitation data is not balanced, the imitation electronic handwritten signature image data can be expanded through word splicing and pixel adversarial generation. The signature images of the two modal types are aligned through image denoising, thinning and scale change preprocessing methods, and all electronic signature images and their corresponding ID paper handwritten binary signature images are spliced in the channel dimension to construct corresponding electronic-paper signature image pairs. The image pair dataset is constructed, the dataset is ID divided, and the dataset is divided into a labeled dataset and an unlabeled dataset.

[0064] The classification backbone network adopts two branch structures of the same double-tower network, performs twice random sampling on the labeled dataset and inputs the two different branch networks for classification, performs single random sampling on the unlabeled subset in the unlabeled dataset to obtain a single unlabeled image pair, and inputs the single unlabeled image pair into two different branch networks for classification. It should be noted that one of the two branch networks is used for classification, and the other is used for pseudo-label generation. We can define the former as a non-cross network and the latter as a cross network. The soft label output by the cross network is used to generate corresponding adversarial sample data through pixel disturbance of the unlabeled data, and the adversarial sample data is input into the corresponding non-cross network for cross-modal comparison and multi-task joint classification training. The backbone network used in the double-tower network is a conventional convolutional neural network Resnet50 for image pair feature extraction, which adopts a three-layer residual module for implementation. Compared with the previous conventional convolutional neural network AlexNet and VGG network, the network has more layers, and the specially designed residual structure can greatly improve the gradient vanishing problem caused by too deep network layers.

[0065] The multi-task joint classification training is completed to obtain a cross-modal sample test comparison module, original handwritten signature materials and electronic sequences are preprocessed to form electronic-paper signature image pairs, and the classification results of the double-tower model are combined to determine the authenticity of the cross-modal handwriting signer identity.

[0066] The present application is aimed at cross-modal comparison between paper material modal and electronic image modal, and more intuitive multi-task cross-modal handwriting comparison is performed based on a class collaborative manner; further, the FGSM pixel disturbance generation manner is used to generate an adversarial sample, which is based on metadata pixel disturbance to generate an adversarial sample similar to the metadata, and the generation manner is more stable; in order to ensure the richness of the imitation data, the paper handwritten signature corresponding imitation data is generated and supplemented through the data generation manner. In addition, due to the label uncertainty problem of the normal writing data and the imitation data which is prone to occur in the data collection or acquisition under the cross-modal data difference, the class collaborative training manner is used to construct a double-tower model to learn the model from different data in a double perspective, which can improve the label uncertainty problem to a certain extent, and a small amount of training data is divided into unlabeled data, and through the semi-supervised learning manner, the pixel disturbance generation manner is used for adversarial training combined with the prediction value, so as to narrow the distribution of the feature space under the double perspective, thereby improving the robustness of the model; finally, based on the ensemble learning manner, the prediction results under the double perspective are fused to comprehensively determine the real identity of the signer.

[0067] The exemplary embodiments of the present application also provide an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication. The memory stores a computer program capable of being executed by the at least one processor, and the computer program is used to make the electronic device execute the method according to the embodiments of the present application when executed by the at least one processor.

[0068] The exemplary embodiments of the present application also provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program is used to make the computer execute the method according to the embodiments of the present application when executed by the processor of the computer.

[0069] The exemplary embodiments of the present application also provide a computer program product, comprising a computer program, wherein the computer program is used to make the computer execute the method according to the embodiments of the present application when executed by the processor of the computer.

[0070] Figure 3Shown is a block diagram of an electronic device 300 that can be used as a server or client of the present application, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0071] Electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. RAM 303 may also store various programs and data required for the operation of device 300. Computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0072] Multiple components within electronic device 300 are connected to I / O interface 305, including an input unit 306, an output unit 307, a storage unit 308, and a communication unit 309. Input unit 306 can be any type of device capable of inputting information into electronic device 300. Input unit 306 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 307 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 308 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 309 allows electronic device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0073] The computing unit 301 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs various methods and processes described above.

[0074] Program code implementing methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0075] In the context of the present application, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0076] As used in the present application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or data to a programmable processor.

[0077] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0078] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0079] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A cross-modal handwriting-based collaborative training method, characterized in that: include: The paper handwritten signature materials signed by different signatories are collected, and the corresponding electronic signature sequence data of the signer and the imitation electronic signature sequence data of other signatories are annotated with the corresponding identity authenticity label information; the paper handwritten signature material detection and clipping are converted into the corresponding binary signature image, the positive and imitation electronic signature sequence data are echoed as electronic signature images, the imitation electronic signature image is expanded through spelling generation and pixel adversarial generation, the two modal types of signature images are aligned through image refinement and scale change, and the corresponding electronic-paper signature image pairs are constructed by splicing in the channel dimension; the electronic-paper signature image pairs are divided into datasets; an image pair dataset including labeled datasets and unlabeled datasets is constructed; a classification backbone network with a cross network and a non-cross network dual-branch network structure is constructed, the labeled dataset is randomly sampled twice and input into the two branch networks in sequence, the unlabeled dataset is randomly sampled once and input into the two branch networks, the unlabeled data is pixel-perturbed with the soft labels output by the cross network to generate adversarial sample data, and the data are input into the non-cross network respectively, and cross-modal comparison multi-task joint classification training is performed until the classification loss value is minimized.

2. The class collaborative training method according to claim 1, characterized in that Generating the imitated electronic signature image data includes: dividing the electronic signature sequence into single words to produce a single word sequence font library, searching for single word sequences with the same content from the font library according to the corresponding written electronic signature image content, performing signature splicing, and then signature echoing to generate the corresponding imitated electronic signature image.

3. The class collaborative training method according to claim 1, wherein: Generating the simulated electronic signature image data includes: according to the corresponding orthographed electronic signature sequence, perturbing a part of the stroke data sequence therein and then performing signature echo to generate the simulated electronic signature image.

4. The class collaborative training method according to claim 1, wherein: Aligning signature images of two modal types includes: aligning paper handwritten binary signature images and electronic signature images in terms of thickness, scale and spatial position, applying dilation, erosion and wild point removal to paper handwritten binary signature images to obtain binary signature images with the same scale as the electronic signature; improving the angle of paper handwritten binary signatures through affine transformation operations, adjusting the maximum and minimum character spacing in paper binary signatures with reference to the character spacing of corresponding electronic signatures, performing centering operations based on pixel bounding boxes so that paper handwritten binary signatures are located in the center of the image, and performing scaling operations to a fixed size at the same time; and using skeleton refinement processing to make the stroke thickness of the two modal signature images the same.

5. The class collaborative training method according to any one of claims 1 to 4, characterized in that: The dataset of the signature image pairs is split to obtain a labeled dataset D and an unlabeled data sample set U. The labeled data sample set D is divided into a first labeled data subset d1 and a second labeled data subset d2, which are respectively sent to the first branch network n1 and the second branch network n2 in the classification backbone network. The unlabeled dataset U generates multiple unlabeled subsets through pixel perturbation adversarial samples and inputs them into the first branch network and the second branch network at the same time. The classification backbone network classifies the labeled data sample set and the unlabeled data sample set, also assigns labels to the unlabeled adversarial samples, and determines the joint classification training loss of cross-modal data.

6. The class collaborative training method according to claim 5, characterized in that: Classifying the labeled data sample set D and the unlabeled data sample set U includes: batch sampling D to obtain data subsets d1 and d2, inputting the data subsets d1 and d2 into sub-networks n1 and n2 respectively, and sub-networks n1 and n2 outputting the classification results corresponding to the data subsets d1 and d2 respectively as logit_n1_d1 and logit_n1_d2, and using the cross entropy classification loss to calculate and sum the corresponding labeled data loss loss_d according to logit_n2_d1, logit_n2_d2 and the corresponding labels of d1 and d2. Batch sampling is performed on U to obtain the data subset u1, and the data subset u1 is input into the networks n1 and n2 at the same time. The classification results corresponding to the data subset u1 are output and marked as logit_n1_u1 and logit_n2_u1 respectively; two perturbation adversarial samples are generated for the data subset u1 to generate the corresponding first unlabeled adversarial sample G_n1_u1 and second unlabeled adversarial sample G_n2_u1, and the first and second unlabeled adversarial samples are input into the sub-networks n1 and n2 respectively. The classification results output by n1 and n2 are used as labels to be assigned to the above two unlabeled adversarial samples respectively.

7. The class collaborative training method according to any one of claims 1 to 4 and 6, characterized in that: The joint classification training loss of cross-modal data is determined as follows: using the labeled data loss loss_d, the binary cross entropy loss loss_u of the unlabeled dataset and the unlabeled adversarial generated dataset, and the unlabeled adversarial generated dataset prediction distribution consistency loss loss_g_u, according to the formula: Total_loss = α*loss_d + β*loss_u + γ*loss_g_u, calculates the joint classification training loss Total_loss for cross-modal data, where α, β, and γ represent the weights of the three losses mentioned above, respectively. Loss_u is the binary cross entropy loss obtained by calculating and summing the binary cross entropy based on the unlabeled samples and the unlabeled adversarially generated samples, and loss_g_u is the loss corresponding to the similarity measurement of the feature expression of the unlabeled adversarially generated data using JS divergence.

8. A handwriting comparison system based on cross-modal handwriting collaborative training, characterized by: It includes a data acquisition module, a data preprocessing module, a classification backbone network, and a class collaborative training module. The data acquisition module collects paper handwritten signature materials signed by different signatories, the corresponding signatory's electronic signature sequence positive data and the other signatory's imitation electronic signature sequence imitation data, and marks the corresponding identity authenticity label information; the data preprocessing module detects and converts the paper handwritten signature material into a corresponding binary signature image, and echoes the positive and imitation electronic signature sequence data as an electronic signature image, expands the imitation electronic signature image through spelling generation and pixel adversarial generation, aligns the two modal types of signature images through image refinement and scale change, and compares all electronic signature images with the paper binary signature images of their corresponding IDs in the channel dimension. The corresponding electronic-paper signature image pair dataset is constructed by row splicing; the signature image pairs are divided into labeled datasets and unlabeled datasets according to ID; the classification backbone network adopts two dual-branch networks with the same structure, and the collaborative training module randomly samples the labeled dataset twice and inputs them into the two branch networks of the classification backbone network in sequence. The unlabeled dataset is randomly sampled once and input into the two branch networks at the same time. The unlabeled data is pixel-perturbed with the soft label output by the cross network to generate the corresponding adversarial sample data, and then input it into the non-cross network for cross-modal comparison and multi-task joint classification training. The training is completed when the classification loss value Total_loss is minimized, and the cross-modal retained signature test comparison module is obtained to compare and recognize the cross-modal signature handwriting.

9. The system according to claim 8, wherein The method further includes dividing the data set of the electronic-paper signature image pair to obtain a labeled data sample set D and an unlabeled data sample set U, dividing the sample set D into a first labeled data subset d1 and a second labeled data subset d2, which are respectively sent to the first branch network n1 and the second branch network n2 in the classification backbone network, the data set U generates multiple labeled subsets through adversarial samples and inputs them into the first branch network and the second branch network at the same time, the classification backbone network classifies the labeled data sample set and the unlabeled data sample set, generates adversarial samples based on the perturbation of the unlabeled data sample set, assigns labels to the unlabeled adversarial samples, and determines the joint classification training loss of cross-modal data.

10. The system according to claim 9, wherein: The classification backbone network performs batch sampling on the labeled dataset D to obtain data subsets d1 and d2, and inputs the data subsets d1 and d2 into the networks n1 and n2 respectively, and outputs the classification results logit_n1_d1, logit_n1_d2, logit_n2_d1, and logit_n2_d2 corresponding to the data subsets d1 and d2 respectively. The cross-entropy classification loss is calculated and summed according to the corresponding labels of d1 and d2 to obtain the label data loss loss_d. The unlabeled dataset U is batch sampled to obtain the data subset u1, and u1 is input into the networks n1 and n2 at the same time, and the classification result labels logit_n1_u1 and logit_n2_u1 corresponding to u1 are output respectively; the data subset u1 is perturbed twice to generate adversarial samples, generating the corresponding first unlabeled adversarial sample G_n1_u1 and second unlabeled adversarial sample G_n2_u1, which are input into the sub-networks n1 and n2 respectively. The output classification results are used as their respective labels to assign labels to the two unlabeled adversarial samples.

11. The system according to any one of claims 8 to 10, characterized in that: The joint classification training loss of cross-modal data is determined as follows: using the binary cross entropy loss loss_d of the labeled dataset, the binary cross entropy loss loss_u of the unlabeled dataset and the unlabeled adversarial generated dataset, and the prediction distribution consistency loss loss_g_u of the unlabeled adversarial generated dataset, according to the formula: Total_loss = α*loss_d + β*loss_u + γ*loss_g_u, calculates the joint classification training loss Total_loss of cross-modal data, where α, β, and γ represent the above three loss weights respectively.

12. An electronic device comprising: processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images