Electronic signature verification method, system, device and medium based on graph convolutional neural network

Abstract

This application discloses an electronic signature verification method based on a graph convolutional neural network. The method collects handwritten electronic signature handwriting data, including both authentic and imitation data. The method classifies the handwriting data based on different attribution dimensions to obtain multiple sample attribute labels containing high-level semantics, and constructs a multi-label classification sample database for handwritten electronic signatures. The method then trains the backbone network constructed using a convolutional neural network to obtain a feature extractor, which extracts feature vectors for all samples in the sample database. The method then calculates sample similarity based on the feature vectors. A graph structure of similar samples is constructed, and a graph convolutional neural network model is trained using the graph structure to obtain a handwritten signature handwriting graph convolution classifier. Based on the multi-dimensional attribute information of the handwritten electronic signature to be verified, a test graph structure is constructed using the feature vectors extracted by the feature extractor. The authenticity of the electronic signature is then verified using the graph convolution classifier. This method can be widely used in online signature verification scenarios such as e-commerce and e-government.

Description

Technical Field

[0001] The present invention relates to the field of computer information processing technology, and in particular to a handwritten electronic signature handwriting comparison method based on a graph convolutional neural network. Background Art

[0002] With the further development of Internet technology and the widespread promotion of original handwriting signature products based on biometric recognition technology in many fields such as finance, government affairs, justice, public security, and people's livelihood, the application scenarios of handwritten electronic signatures are gradually expanding. However, with the application of handwritten electronic signatures in different important fields, various forms of attacks and challenges are also emerging. This has also brought a greater threat to the accuracy and anti-attack ability of handwritten electronic signature handwriting comparison. Therefore, the need to improve and optimize the electronic handwritten signature handwriting comparison method is relatively urgent.

[0003] Existing handwritten electronic signature handwriting comparison methods are generally categorized into two types: one involves manually designing and extracting handwriting features based on prior knowledge, and then applying these features to traditional machine learning methods such as random forests for classification and discrimination. The other directly utilizes deep learning methods for feature extraction, eliminating the need for manual feature design, and then performing classification and discrimination at the fully connected layer. While these methods can solve most problems to a certain extent, they are mostly based on extracting features from a single handwritten electronic signature sample and then performing classification and comparison. This can lead to the possibility of signature handwriting authentication failure if the quality of a single signature is poor or the style varies significantly. This results in relatively limited security and user experience, making it difficult to achieve high recognition and comparison accuracy, and failing to meet user needs.

[0004] Publication number: CN113313092A, titled "Handwritten Signature Recognition Method, Automated Claims Processing Method, Apparatus, and Device," discloses a handwritten signature recognition method that obtains business document data of a target object; parses the business document data to obtain image data corresponding to the signature area of ​​the business document; recognizes the handwritten signature in the image data to obtain feature data of the handwritten signature to be recognized in the image data; then recognizes the handwritten signature of the target object to be recognized and obtains the corresponding recognition result. This method extracts signature features based on manual design and deep learning methods. The feature vector can indeed represent the signature data to a certain extent, but it focuses more on the content of the signature and does not further process or determine the authenticity of the signature style.

[0005] Publication number: CN114220178A, name: Signature identification system and method based on channel attention mechanism, including: converting the collected electronic signature handwriting data into a signature image in combination with coordinates, pressure and time information, dynamically sampling and pairing the original and imitated signature data in the data set into image pairs, further enhancing the data, and then splicing and combining the image pairs as the final input image, and distinguishing the authenticity of the handwritten handwriting signature data based on the convolutional neural network of the attention mechanism, which can capture the fine-grained features between the original and imitated signatures, so as to better distinguish the difference in handwriting signature styles. However, when it is echoed, the pressure is integrated into the image, and the situation that the pressure is difficult to obtain on devices such as mobile phones and tablets is not fully considered. The applicability may be limited. In addition, this method does not further consider the adjacency relationship between signature images, but only performs preliminary classification and judgment based on feature vectors. Summary of the Invention

[0006] In light of this, this application proposes a handwritten electronic signature verification method based on a graph convolutional neural network. This method combines the relationships between handwritten electronic signature data and further optimizes the model's classification and discrimination capabilities by strengthening the learning of similar and confusing sample relationships. This method extends the one-to-one comparison test to one-to-many, integrating the multi-dimensional attributes and rich stylistic characteristics of electronic signatures to further improve the efficiency and security of the entire signature handwriting comparison system.

[0007] The present invention takes into account the existence of two attributes of signature handwriting, namely content and style. First, the feature representation of the handwriting signature is fully extracted based on the multi-task joint training method. Secondly, in the data preprocessing stage, the differences between devices are taken into account, and no restrictive information is added when the online sequence data is echoed. At the same time, in order to further distinguish the authenticity of the signature, based on the correlation between the signature images and the signature feature vector extracted by the aforementioned feature extractor, a graph convolutional neural network is constructed to perform authenticity binary classification. The purpose is to combine the actual application scenarios. Signature data is likely to have quality problems, cross-device problems, cross-time problems, multiple signature styles problems, inconsistent collection scenarios, etc. Assuming that only the attributes of a single data are considered during training or verification, the result may be biased and noisy. However, if it can be further combined with the information of its neighboring nodes (similar samples or homologous samples) and the attributes of samples that are similar in time and space, more complete feature information can be obtained to a certain extent. In this way, the relationship between handwritten electronic signature handwriting data can be effectively combined. It is no longer just for single data sample to make judgments or constrain usage scenarios and user behaviors. Instead, the neighboring data with close distance and similarity in the feature space of the signature data to be identified is selected to build a graph relationship. By strengthening the learning of similar and confused sample relationships, the model classification and discrimination capabilities can be further optimized, and finally a better handwriting comparison effect can be obtained. According to one aspect of the present application, a method for electronic signature verification based on a graph convolutional neural network is proposed, which collects handwritten electronic signature handwriting data and imitation data; classifies the electronic signature handwriting data from different attribution dimensions to obtain multiple sample attribute labels with high-level semantics, and constructs a handwritten electronic signature multi-label classification sample database; trains the backbone network model constructed by the convolutional neural network to obtain a feature extractor, and extracts the feature vectors of all samples in the sample database; calculates the sample similarity based on the feature vector; constructs a graph structure of similar samples, and uses the graph structure to train the graph convolutional neural network model to obtain a handwritten signature handwriting graph convolution classifier; based on the graph convolution classifier, the authenticity of the electronic signature is verified according to the multi-dimensional attribute information of the handwritten electronic signature to be verified.

[0008] Further preferred, based on the handwritten electronic signature handwriting sequence data with category labels in the constructed electronic signature multi-label classification sample database, a convolutional neural network is used to perform multi-task joint classification training to obtain a corresponding signature feature extractor; the signature feature extractor is used to extract the positive and negative sample feature vectors in the sample library, and the sample similarity is calculated based on the feature vector, and a signature graph is constructed according to similar samples and a graph convolutional neural network model is trained to obtain a handwritten signature handwriting comparison classifier; the test sample after data preprocessing is subjected to sample deep features extracted by the feature extractor, and then the signature classifier is used to authenticate the identity of the signature handwriting.

[0009] Further preferably, the construction of a multi-label classification sample database for handwritten electronic signatures includes: classifying the pre-processed positive and negative sample paired images from different attribution dimensions and assigning corresponding attribute labels; the label of the signature authenticity part is a binary classification label, and the label of the signature content part is a multi-classification label; for negative samples paired with random content for which a unique content label cannot be determined, their content labels are marked as special numerical representations; the sample size is ensured to be relatively balanced in each dimension.

[0010] Further preferably, the feature extractor training includes: inputting the multi-label sample images of the electronic signature into the convolutional neural network for joint training of authenticity binary classification and content multi-classification, and samples with content labels as special characters only participate in the authenticity binary classification training, and training is completed until the loss value reaches a convergence state, completing the training of the convolutional neural network, and using the entire trained backbone network as a feature extractor, wherein the binary classification and multi-classification loss functions both adopt the cross entropy loss function, and the training loss weights are set to 0.8 and 0.2, respectively.

[0011] Further preferably, the feature vector extracted by the signature feature extractor for each sample image pair is taken as a graph vertex, the cosine distance between any two graph vertices is calculated, and the nearest neighbor algorithm is used to find the first K nearest neighbor nodes of each vertex according to the cosine distance, and the corresponding adjacency matrix A is constructed according to the neighbor relationship of each node. ij , for the adjacency matrix A ij Perform Laplace transform to get matrix A ij The Laplace transform matrix According to the formula: Constructing the Matrix The degree matrix Call the formula based on the degree matrix: Calculate the feature vector H output by the graph convolution layer L+1 l+1 ,in, for The degree matrix, δ is the nonlinear activation function, W l is the parameter matrix corresponding to the L+1 layer, H l is the feature vector of layer L.

[0012] According to another aspect of the present application, in a handwritten electronic signature verification system based on a graph convolutional neural network, a signature collection module collects the original and imitation data of handwritten electronic signature handwriting to construct a multi-label classification sample database of handwritten electronic signatures; a backbone network constructed by training the convolutional neural network is obtained to obtain a feature extractor, and the feature extractor extracts signature handwriting data features based on the electronic signature multi-label classification samples and calculates feature vectors; similar samples are obtained by calculating sample similarity based on the feature vectors; a graph structure of similar samples is constructed, and a training module uses the graph structure to train a graph convolutional neural network model to obtain a handwritten signature handwriting graph convolution classifier; based on the graph convolution classifier, the authenticity of the electronic signature is verified according to the multi-dimensional attribute information of the handwritten electronic signature to be verified.

[0013] It is further preferred to use a convolutional neural network as the backbone network, and the multi-label sample images of electronic signatures are input into the convolutional neural network for joint training of authenticity binary classification and content multi-classification. Samples with content labels as special characters only participate in the authenticity binary classification training. The training is completed until the loss function reaches the preset loss weight, and the training of the convolutional neural network is completed. The entire backbone network that has completed the training is used as a feature extractor, wherein the binary classification and multi-classification loss functions both use the cross entropy loss function, and the training loss weights are set to 0.8 and 0.2, respectively.

[0014] Further preferably, the graph convolutional neural network includes: a graph convolution layer, a hidden layer, and a classification layer, and the hidden layer and the classification layer adopt fully connected layers.

[0015] Further preferably, the feature vector extracted by the signature feature extractor for each sample image pair is taken as a graph vertex, the cosine distance between any two graph vertices is calculated, and the nearest neighbor algorithm is used to find the first K nearest neighbor nodes of each vertex according to the cosine distance, and the corresponding adjacency matrix A is constructed according to the neighbor relationship of each node. ij , for the adjacency matrix A ij Perform Laplace transform to get matrix A ij The Laplace transform matrix According to the formula: Constructing the Matrix The degree matrix Call the formula based on the degree matrix: Calculate the feature vector H output by the graph convolution layer L+1 l+1 ,in, for The degree matrix, δ is the nonlinear activation function, W l is the parameter matrix corresponding to the L+1 layer, H l is the feature vector of layer L.

[0016] According to another aspect of the present application, an electronic device is proposed, comprising: a processor; and a memory for storing a program, wherein the program comprises instructions that, when executed by the processor, cause the processor to execute the above-described handwritten electronic signature verification method based on graph convolutional neural network.

[0017] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute the above-mentioned handwritten electronic signature verification method based on graph convolutional neural network.

[0018] The use of graph convolutional neural networks effectively integrates the relationships between handwritten electronic signature data. Instead of relying solely on single data samples for identification or constraining usage scenarios and user behavior, the system selects nearby and similar neighboring data within the signature feature space to construct a graph relationship. By strengthening the learning of relationships between similar and confusing samples, the model's classification and discrimination capabilities are further optimized. The authenticity of the signature style is not further processed or determined, and the test fully considers multiple signature samples from the signer, extending the one-to-one comparison test to one-to-many. The system integrates the multi-dimensional attributes and rich style and habit characteristics of multiple samples, further improving the efficiency and security of the entire signature handwriting comparison system. This system is widely applicable in online signature verification scenarios such as e-commerce and e-government. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the handwritten electronic signature handwriting comparison training and testing process according to an embodiment of the present application;

[0020] Figure 2 Schematic diagram of a graph convolutional network structure according to an exemplary embodiment of the present application;

[0021] Figure 3 This is a graph convolution classifier training process according to an exemplary embodiment of the present application;

[0022] Figure 4 Shown is a structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present application. DETAILED DESCRIPTION

[0023] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0024] It should be understood that the various steps described in the method embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0025] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] In order to further elaborate the technical solution of the present method, the present invention will be further described in conjunction with specific embodiments and drawings.

[0029] Figure 1 The figure shows a schematic diagram of the handwritten electronic signature handwriting comparison training and testing process in an embodiment of the present application, including: training data collection, data preprocessing, sample library construction, feature extractor training, graph convolution classifier training; test samples, data preprocessing, feature extractor, graph convolution classifier, and output classification comparison results.

[0030] Data collection: Collect rich data on handwritten electronic signatures, including their authentic and imitation handwriting, through open source or online methods.

[0031] Data preprocessing: Preprocess the handwritten electronic signature data to obtain signature handwriting image data of relatively unified standards;

[0032] Signature sample library construction. Classify handwritten electronic signature data based on different attribution dimensions, obtain multiple positive and negative sample category labels with high-level semantics, and build a multi-label classification sample database for handwritten electronic signatures;

[0033] Feature extractor training. Based on the handwritten electronic signature image data with category labels in the constructed electronic signature multi-label classification sample database, a convolutional neural network is used to perform multi-task joint classification training to obtain the corresponding signature feature extractor;

[0034] Signature classifier training. Use the signature feature extractor to extract all positive and negative sample feature vectors from the sample library. Calculate sample similarity based on the extracted feature vectors. Use the nearest neighbor algorithm to obtain the adjacency relationship between similar signature handwritings. Build a graph structure, and further train the graph convolutional neural network model to obtain a handwritten signature comparison classifier.

[0035] Sample testing: After data preprocessing, the test sample can be used to extract the sample feature vector through the feature extractor, and then pass it through the signature classifier to determine the authenticity of the signature handwriting.

[0036] Data collection: This embodiment collects data mainly through two channels, including:

[0037] 1) Collect data from existing open source handwritten electronic signature databases;

[0038] 2) Combined with the actual application scenario, organize the collection of offline handwritten electronic signature data, and several signatories sign their personal names. Then each signatory randomly draws the names of other signatories to imitate their signatures. There are three forms of imitation:

[0039] After observing the signature for a short period of time, imitate it;

[0040] After several attempts at imitating the signature, write again;

[0041] High-fidelity signature data is collected through mask rubbing.

[0042] This can obtain relatively rich identity ID authentic imitation electronic signature handwriting data;

[0043] Data preprocessing. The preprocessing part mainly includes data verification, signature echo, and image pairing. Data verification involves quality screening, sequence repair, and resampling of handwritten electronic signature sequence data. Quality screening mainly detects anomalies in sequence feature coordinates, pressure, pen start and pen lift status, and other information. Sequence repair mainly supplements and repairs missing values ​​and outliers. Resampling aims to further unify the length magnitude of different signatures. Signature echo mainly echoes the signature image based on the coordinates and pen start and pen lift status in the electronic signature sequence data. The image drawing width*height*channel can be 224*224*1. Image pairing includes the construction of positive and negative sample image pairs. The construction of positive sample image pairs involves first selecting an image of the signer's signature as a reference image, and then randomly selecting other positive images of their signature as pairings. The construction of negative sample image pairs includes: using the signature imitation data of other signatories as a pair, or randomly selecting samples of other different signature contents as negative sample pairs. Both positive and negative sample paired images are spliced ​​on the channel. The combined image pair width*height*channels can be 224*224*2.

[0044] Construction of a signature sample library. Preprocessed positive and negative sample paired images are classified along different attribution dimensions and assigned corresponding attribute labels. The attribution dimensions primarily include the signer's identity (i.e., signature authenticity) and signature content. The authenticity label is a binary label, represented by 0 or 1 (e.g., a signature labeled 0 indicates the signer's identity is genuine, while a signature labeled 1 indicates a signature impersonation, indicating a forged signer's identity). The signature content label is a multi-class label. This label can be set by counting the categories of different signature contents in the training sample, primarily in the form of discrete labels. Assuming there are 10,000 different signature contents in the training sample, the label will range from 0 to 9999.

[0045] Therefore, a signature sample library can be constructed based on the rich multi-label signature data of signatories. The sample library is mainly composed of positive sample pairs and negative sample pairs output after the preprocessing stage. All paired images of each signatory contain true and false binary classification labels. However, due to the presence of negative samples with random content pairing (that is, the positive sample signature has fixed content, but the negative sample content is inconsistent with the positive sample content), it is impossible to determine a unique content label for them. Therefore, their content labels are marked with special values ​​(for training convenience, they can be determined as -1) to distinguish them. In addition, it is important to ensure that the sample size of each signatory is relatively balanced in all dimensions.

[0046] Feature extractor training: Based on the aforementioned multi-label signature sample image library, a convolutional neural network (for example, the relatively common ResNet34) is used as the backbone network for joint training of authenticity binary classification and content multi-classification. Samples with content labels containing special characters participate only in the authenticity binary classification training and are not included in the calculation of the content multi-classification loss function.

[0047] In this embodiment, both the binary and multi-classification loss functions use the conventional cross-entropy loss function, but other loss functions can also be used. The multi-task joint training method mainly uses the same training data to train two tasks simultaneously. The current primary task is the true and false binary classification, and the secondary task is the content multi-classification. The latter mainly promotes better feature extraction in the former. During training, the training loss weights of the two are set according to the magnitude of the loss value during the iteration process. In this embodiment, the optimal settings can be 0.8 and 0.2 respectively.

[0048] The convolutional neural network is trained jointly on multiple tasks using positive and negative samples from a library of multi-label sample images. When the loss converges, the convolutional neural network training is complete, and the trained backbone network serves as the final feature extractor. This feature extractor is primarily used to extract feature representations that capture signature authenticity and style. These feature representations serve as input to the downstream graph convolution classification network, and the quality of this feature extractor directly determines the results of the downstream signature authenticity classification task.

[0049] Signature classifier training. Although the features extracted by the trained feature extractor have good authenticity discrimination capabilities after classification training, in practice, since users generally leave signature samples multiple times, and the styles of signature samples left on different devices and at different times are variable, the classification method based on a single sample has significant limitations. It cannot integrate the signature features of users with multiple styles, nor can it obtain more abundant potentially difficult-to-forge sample data. Therefore, we construct a graph structure based on the feature vectors extracted by the above feature extractor to further train the signature classifier.

[0050] like Figure 3 The following is a graph convolution classifier training process in an exemplary embodiment of this application. It includes: calculating sample similarity, selecting neighbor nodes, building a graph structure, and training the classifier. The specific implementation steps include:

[0051] 1) Calculate sample similarity. The signature feature extractor trained above extracts features from each sample image pair in the signature sample library. Each extracted feature vector is considered a graph node, and the cosine distance between any two graph nodes is calculated.

[0052] 2) Neighbor node selection. This embodiment uses the nearest neighbor algorithm (KNN) to find the first K nearest neighbor nodes of each vertex based on the above cosine distance. The size of K can be determined according to the actual data situation. For a feature vector x that needs to be predicted, it is only necessary to find a set of K vectors closest to vector x in the training data set, and then predict the category of x as the category with the largest number of categories among these K samples. The specific steps are: 1. According to the above cosine distance, find the K nodes closest to the node among all nodes, and record the set represented by these K nodes as N_k(x); 2. Based on the labels of the nodes in the set, determine the category to which the node belongs according to the principle of majority voting.

[0053] 3) Construct the graph structure. Assuming that there are N sample image pairs in the sample library, an N*N adjacency matrix A can be constructed based on the neighbor relationship of each node. ij , determine the adjacency matrix A according to the following formula ij Elements in, from which the graph structure relationship between nodes can be constructed:

[0054]

[0055] 4) Graph convolution classification training. Figure 2 The diagram shows a schematic diagram of the graph convolutional network structure in an exemplary embodiment of the present application, including: a graph convolution layer, a hidden layer (fully connected layer), and a classification layer (fully connected layer). The graph convolution layer is mainly used to extract graph features from the input graph data. The input is the feature vector of each node and the relationship between its corresponding adjacent nodes (adjacency matrix), and the output is the fused graph feature matrix. Its essence is to fuse the relationship between neighboring nodes so that the features between nodes can learn from each other. The hidden layer is mainly used for feature mapping, which is to increase the feature dimension and enrich the feature information. The classification layer is used to perform the final true and false binary classification. The network design is relatively shallow, containing only one layer of graph convolution structure, a fully connected hidden layer, and a fully connected binary classification layer, so the convergence speed is relatively fast. The graph convolutional neural network considers the attributes of the node itself and the attributes of the node's adjacent nodes to obtain the comprehensive feature vector of the node, and finally realizes the graph node classification task. Like the convolutional neural network, it performs a weighted summation of the adjacent nodes (pixels) of the node (pixel), and then uses the activation function to map and extract features. After multiple rounds of feature extraction, the activation mapping function such as the softmax function is used for final classification.

[0056] The specific description is as follows:

[0057] For the input adjacency matrix A ij Laplace transform Get matrix A ij The Laplace transform matrix Among them I nis the identity matrix, according to the formula: Constructing the Matrix The degree matrix According to the formula: Calculate the feature vector H output by the graph convolution layer l+1 l+1 .

[0058] in, is the Laplace matrix, A ij is the adjacency matrix, for The degree matrix, δ is the nonlinear activation function, W l H is the weight parameter matrix output by the l+1 layer, i.e., the graph convolution layer. This parameter is the learnable weight parameter of the graph convolution layer. l is the input data of the l-th layer, i.e., the graph convolution layer, which is the sample feature vector extracted by the above feature extractor, H l+1 It is the fused feature matrix output by the l+1 layer, which is the graph convolution layer. The dimension of the feature matrix is ​​N*256.

[0059] The N 1*256 feature vectors in the fused N*256-dimensional feature matrix are assigned to the corresponding N nodes respectively as the feature vectors of each node. The feature vectors of each node are input into the subsequent fully connected hidden layer to perform feature dimension mapping and upgrade to 1*512 dimensions. The upgraded feature vectors and their corresponding true and false binary classification labels (the positive and negative sample labels corresponding to the signature sample library mentioned above) are then input into the fully connected classification layer together for true and false signature binary classification training. The classification loss function is the cross entropy loss. Finally, the trained graph convolution classifier can be obtained, which is the corresponding signature classifier.

[0060] Graph convolution-based classification aims to combine with actual application scenarios. Signature data is likely to have quality problems, cross-device problems, cross-time problems, multiple signature styles problems, inconsistent collection scenarios, etc. Assuming that we only consider the attributes of a single data during training or verification, the result may be biased and noisy. However, if we can further combine the information of its neighboring nodes (similar samples or homologous samples) and the attributes of samples with similar time and space, we can obtain more complete feature information to a certain extent.

[0061] Sample test. The sample test process of this embodiment can be referred to Figure 1After data preprocessing of the test samples, multiple image sample pairs consisting of a single test sample and multiple signature samples are obtained. Each image sample pair is input into the trained signature feature extractor to obtain the corresponding signature feature vector. The corresponding adjacency matrix between the test samples is calculated based on the cosine similarity of the feature vectors. The graph structure of the corresponding samples is constructed using the nearest neighbor algorithm KNN. The feature vector and its corresponding adjacency matrix are input into the trained graph convolution classifier to distinguish between true and false signatures, and finally perform identity authentication.

[0062] The method based on graph convolution classification takes into account the relationship between samples to a certain extent. It is no longer limited to the information of a single sample. The testing method is expanded from one-to-one to one-to-many, further weighing the style richness, time and space span and other information of the signature samples, and further improving the accuracy of signature verification and recognition, which is a significant improvement for user experience and product security.

[0063] The exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of the present application.

[0064] An exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0065] An exemplary embodiment of the present application further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to cause the computer to perform the method according to the embodiment of the present application.

[0066] refer to Figure 4 , a block diagram of an electronic device 300 that can serve as a server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0067] like Figure 4As shown, electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of device 300 can also be stored in RAM 303. Computing unit 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0068] Multiple components within electronic device 300 are connected to I / O interface 305, including an input unit 306, an output unit 307, a storage unit 308, and a communication unit 309. Input unit 306 can be any type of device capable of inputting information into electronic device 300. Input unit 306 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 307 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 308 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 309 allows electronic device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0069] The computing unit 301 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above. For example, in some embodiments, the reconstruction and decomposition of the muscle movement trajectory based on the original trajectory of the signature stroke, as well as the decomposition of its logarithmic velocity curve, can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. In some embodiments, the computing unit 301 can be configured to execute the signature handwriting dynamic acquisition implementation method by any other appropriate means (e.g., by means of firmware).

[0070] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0071] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0072] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0073] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0074] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0075] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. An electronic signature verification method based on graph convolutional neural network, characterized in that: Collect handwritten electronic signature handwriting data, including both original and imitation data; classify the electronic signature handwriting data based on different attribution dimensions, obtain multiple sample attribute labels containing high-level semantics, and construct a multi-label classification sample database for handwritten electronic signatures; The backbone network model constructed by training the convolutional neural network is obtained to obtain the signature image feature extractor, which extracts the signature handwriting data features based on the multi-label classification samples of the electronic signature and calculates the feature vector; The sample similarity is calculated based on the feature vector; a graph structure of similar samples is constructed, and the graph convolutional neural network model is trained using the graph structure to obtain a handwritten signature graph convolution classifier; based on the multi-dimensional attribute information of the handwritten electronic signature to be verified, a test graph structure is constructed using the feature vector extracted by the feature extractor, and the authenticity of the electronic signature is verified based on the graph convolution classifier.

2. The method according to claim 1, characterized in that The construction of a handwritten electronic signature multi-label classification sample database includes: classifying the pre-processed positive and negative sample paired images from different attribution dimensions and assigning corresponding attribute labels; the label of the signature authenticity part is a binary classification label, the label of the signature content part is a multi-classification label, and for negative samples paired with random content for which a unique content label cannot be determined, the content label is marked as a special numerical representation, and the sample size is ensured to be relatively balanced in each dimension; based on the handwritten electronic signature handwriting sequence data with category labels in the constructed electronic signature multi-label classification sample database, a convolutional neural network is used to perform multi-task joint classification training to obtain a corresponding signature feature extractor; the signature feature extractor is used to extract the positive and negative sample feature vectors in the sample library, and the sample similarity is calculated based on the feature vector, and the signature graph structure is constructed according to the sample similarity and the graph convolutional neural network model is trained to further obtain a handwritten signature handwriting comparison classifier.

3. The method according to claim 1, characterized in that The feature extractor training includes: inputting multi-label sample images of electronic signatures into the convolutional neural network for joint training of authenticity binary classification and content multi-classification. Samples with content labels of special characters only participate in the authenticity binary classification training. After training to the loss convergence state, the training of the convolutional neural network is completed, and the trained backbone network is used as the signature feature extractor. During the training process, the binary classification and multi-classification loss functions both adopt the cross entropy loss function, and the training loss weights are set to 0.8 and 0.2, respectively.

4. The method according to any one of claims 1 to 3, characterized in that: Take each sample image pair in the signature sample database and extract the feature vector obtained by the signature feature extractor as a graph node, calculate the cosine distance between any two graph nodes, find the first K nearest neighbor nodes of each node based on the cosine distance, and construct the corresponding adjacency matrix A based on the neighbor relationship of each node. ij , for the adjacency matrix A ij Perform Laplace transform to get matrix A ij The Laplace transform matrix According to the formula: Constructing the Matrix The degree matrix Call the formula based on the degree matrix: Calculate the feature matrix H after the fusion of the graph convolution layer L+1 layer l+1 ,in, for The degree matrix, δ is the nonlinear activation function, W l is the parameter matrix corresponding to the L+1 layer, H l is the feature matrix of the L layer.

5. The method according to claim 4, characterized in that The N 1*256 feature vectors in the fused N*256-dimensional feature matrix are assigned to the corresponding N nodes respectively as the feature vector of each node. The feature vector of each node is input into the subsequent hidden layer to perform feature dimension mapping and upgrade to 1*512 dimensions. The upgraded feature vector and its corresponding true and false binary classification label are then input into the fully connected classification layer for true and false signature binary classification training. The cross entropy loss is used as the classification loss function to obtain the trained graph convolution classifier as the signature classifier.

6. A handwritten electronic signature verification system based on graph convolutional neural network, characterized by: The signature collection module collects handwritten electronic signature handwriting data, including both authentic and imitation data, classifies the electronic signature handwriting data based on different attribution dimensions, obtains multiple sample attribute labels containing high-level semantics, and constructs a multi-label classification sample database for handwritten electronic signatures. The backbone network model constructed by training the convolutional neural network is obtained as a feature extractor, which extracts feature vectors for all samples in the sample database. Similar samples are obtained by calculating sample similarity based on feature vectors; a graph structure of similar samples is constructed, and the training module uses the graph structure to train a graph convolutional neural network model to obtain a handwritten signature handwriting graph convolution classifier; based on the graph convolution classifier, the authenticity of the electronic signature is verified according to the multi-dimensional attribute information of the handwritten electronic signature to be verified.

7. The system according to claim 6, characterized in that The graph convolutional neural network includes: graph convolution layer, hidden layer, and classification layer. The hidden layer and the classification layer use fully connected layers. The graph convolution layer extracts graph features from the input graph data. The input is the feature vector and its adjacency matrix of each node, and the output is the fused graph feature matrix. The hidden layer performs feature mapping, and the classification layer performs true and false binary classification. Using the convolutional neural network as the backbone network, the multi-label sample images of electronic signatures are input into the convolutional neural network for joint training of true and false binary classification and content multi-classification. Samples with content labels as special characters only participate in true and false binary classification training. The training is completed until the loss value reaches a convergence state, and the training of the convolutional neural network is completed. The trained backbone network is used as a feature extractor. The cross entropy loss function is used for both binary classification and multi-classification loss functions.

8. The system according to claim 6, wherein: The construction of a handwritten electronic signature multi-label classification sample database includes: classifying the pre-processed positive and negative sample paired images from different attribution dimensions and assigning corresponding attribute labels; the label of the signature authenticity part is a binary classification label, the label of the signature content part is a multi-classification label, and for the negative samples of random content pairing for which a unique content label cannot be determined, the content label is marked as a special numerical representation, and the sample size is ensured to be relatively balanced in each dimension; based on the handwritten electronic signature handwriting sequence data with category labels in the constructed electronic signature multi-label classification sample database, a convolutional neural network is used to perform multi-task joint classification training to obtain a corresponding signature feature extractor; the signature feature extractor is used to extract the positive and negative sample feature vectors in the sample library, and the sample similarity is calculated based on the feature vector, a signature graph is constructed based on the similar samples and a graph convolutional neural network model is trained to obtain a handwritten signature handwriting comparison classifier; for the test samples that have undergone data preprocessing, the feature extractor extracts the sample deep features, and then the signature classifier is used to judge the authenticity of the signature handwriting identity.

9. The system according to any one of claims 6 to 8, characterized in that: The feature vector extracted by the signature feature extractor for each sample image pair is used as a graph node. The cosine distance between any two graph nodes is calculated. The first K nearest neighbor nodes of each node are found based on the cosine distance. The corresponding adjacency matrix A is constructed based on the neighbor relationship of each node. ij , for the adjacency matrix A ij Perform Laplace transform to get matrix A ij The Laplace transform matrix According to the formula: Constructing the Matrix The degree matrix Call the formula based on the degree matrix: Calculate the feature vector H output by the graph convolution layer L+1 l+1 ,in, for The degree matrix, δ is the nonlinear activation function, W l is the parameter matrix corresponding to the L+1 layer, H l is the feature vector of layer L.

10. An electronic device comprising: processor; And a memory for storing a program, characterized in that, wherein the program includes instructions, which, when executed by the processor, enable the processor to execute the electronic signature verification method based on a graph convolutional neural network according to any one of claims 1-4.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to enable the computer to execute the electronic signature verification method based on a graph convolutional neural network according to any one of claims 1 to 4.

Images