Signature verification system and method

Abstract

To perform learning quickly with a small amount of training data and to verify signatures with high accuracy.SOLUTION: A signature verification system is as described in the following. An image containing one or more handwritten characters is divided into a plurality of patches of a predetermined size for each character. For each of the divided patches, an embedding feature, which is data representing pixel and positional features of the patch, is calculated. Based on the calculated embedding features of each patch, the type of each patch is identified, and based on the identified patch types and the positions of the patches, a context of each patch in the image is calculated. A discrete representation model that applies an attention mechanism, MSA, to the overall embedding features is stored. A signature verification model is provided, which receives, as input, two images each containing one or more handwritten characters and outputs a probability that the two input images are images of characters written by the same writer. A new signature verification model is generated by replacing a part of the signature verification model with the discrete representation model.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a signature verification system and method. [Background technology]

[0002] A handwritten signature, like other biometrics such as face, fingerprint, iris, vein, and DNA, is one piece of information that can be used to verify a person's identity. Signature matching technology, which determines whether two signatures were written by the same person, plays an important role in many fields, including banking, insurance, healthcare, and securities. A technical challenge in signature matching is that the shape of a signature varies from one handwriting to the next, reducing matching accuracy. Furthermore, signatures forged by skilled craftsmen can be very similar to genuine signatures, so an effective method is needed to accurately distinguish the subtle features of forged and genuine signatures.

[0003] Here, we can imagine two cases where signature verification is required. The first is a case where a user registers their own signature in the system in advance, and the system then determines whether the entered signature is that of the registered user or a forgery. The second is a case where it determines whether two different signatures entered into the system were written by the same person. In particular, the second case involves detecting fraudulent contracts by employees in the insurance business, etc. Insurance companies do not usually store their customers' signatures, but if it is discovered that the same person, such as a subcontractor or representative, has written different signatures, such signatures are considered fraudulent. Therefore, insurance companies strive to detect such cases as much as possible. However, it is clear that the latter case is much more difficult to verify than the former.

[0004] There are two basic approaches to signature verification: writer-independent and writer-dependent. In the writer-independent approach, a single model is trained for all writers, either in a similarity space (e.g., authentic-authentic pairs) or a dissimilarity space (e.g., forged-authentic pairs) to distinguish between enrollment signatures (reference signatures) and input signatures (query signatures). In contrast, in the writer-dependent approach, a single model is trained per writer and used to authenticate signatures. As a result, when a new writer is added, a new model must be trained for that writer. In contrast, because writer-independent approaches do not require retraining models when a new writer is added to the system, most research has focused on writer-independent approaches. This paper focuses on this approach.

[0005] The basic processing flow of handwriting verification involves three main steps: image preprocessing, image feature extraction, and signature classification as to whether the signature is genuine or forged. Preprocessing includes various tasks such as noise reduction, image normalization, binarization, skew correction, and skeletonization. Next, distinguishing features for the character string are extracted from the preprocessed signature image, and various signature features are obtained. Finally, a classifier is used to classify the signature as genuine or forged. Recently, with the spread of deep learning, machine learning-based models are often used for signature verification. The Transformer is one well-known such model (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. (2017). "Attention Is All You Need", arXiv:1706.03762. Summary of the Invention [Problem to be solved by the invention]

[0007] However, generating a signature verification model generally requires a large amount of data for training. Handwritten signatures (signatures) have a wide range of handwriting variations (individual differences), and even signatures written by the same handwriter vary from time to time. Furthermore, the character types used in Japanese signatures are diverse, making comprehensive training difficult. Considering specific examples of Japanese signatures, there are an enormous number of characters: 2,965 types in Shift_JIS Standard 1, 3,390 types in Shift_JIS Standard 2, 1,259 types in Shift_JIS Standard 3, and 2,436 types in Shift_JIS Standard 4. As a result, to achieve an acceptable level of accuracy (typically 90% or higher) that can be considered sufficiently reliable for handwriting verification, it is necessary to train a signature verification model using a large number of signature images. However, collecting such a large number of signature images requires significant human and time resources, making it a challenge to achieve high-precision handwriting verification technology with a small amount of training data.

[0008] The present invention has been made in view of the above circumstances, and its object is to provide a signature verification system and method that can quickly learn using a small amount of training data and verify signatures with high accuracy. [Means for solving the problem]

[0009] One of the present inventions for solving the above problem is a signature verification system that includes: a storage device that stores a discrete representation model that divides an image containing one or more handwritten characters into a plurality of patches of a predetermined size for each character; calculates embedded features for each patch, which are data on the pixels and position of each divided patch; identifies the type of each patch based on the calculated embedded features for each patch; calculates the context of each patch in the image based on the identified type of each patch and each patch position; and applies MSA (Multihead Self-Attention), which is an attention mechanism, to the entire embedded features; and a computing device that receives two images containing one or more handwritten characters, and replaces a part of a signature verification model that outputs the probability that the two input images are images of characters written by the same writer, with the stored discrete representation model, to generate a new signature verification model. [Effects of the Invention]

[0010] According to the present invention, it is possible to quickly learn using a small amount of learning data and to verify signatures with high accuracy. [Brief explanation of the drawings]

[0011] [Figure 1] 10 is a flowchart illustrating an example of processing performed by the signature verification system according to the present embodiment. [Figure 2] FIG. 10 is a diagram showing an example of a data collection form displayed during data collection processing. [Figure 3] FIG. 10 is a diagram illustrating an example of a process for generating a discrete expression model by a signature verification system. [Figure 4] 10A and 10B are diagrams illustrating an example of a process for generating a signature verification model according to the present embodiment. [Figure 5] 10A and 10B are diagrams illustrating another example of the generation process of the signature verification model according to the present embodiment. [Figure 6] FIG. 1 illustrates a general architecture of a signature verification system consisting of a front end and a back end. [Figure 7]FIG. 10 is a diagram showing an example of a GUI (signature verification GUI) displayed in the verification process. [Figure 8] FIG. 10 is a diagram showing an example of a GUI (fraud detection GUI) displayed in the fraud detection process. [Figure 9] 10 is a flowchart illustrating an example of a matching process. [Figure 10] 10 is a flowchart illustrating an example of fraud detection processing. DETAILED DESCRIPTION OF THE INVENTION

[0012] First, an overview of the signature verification system of this embodiment will be described. Typical handwritten characters, including signatures, are written in different handwriting styles and forms. Images of such characters (handwritten character images) can be discretized into predetermined area units (discretization units), such as words, characters, and radicals. In this signature verification system, discretization units within the same text image are assumed to have a "positive" relationship, i.e., characteristics of handwriting written by the same hand. In contrast, in this signature verification system, discretization units within different handwritten character images are assumed to have a "negative" relationship, i.e., characteristics of handwriting written by different hand. The signature verification system then trains a discrete representation model of handwritten text using a large number of available (unlabeled) handwritten character images. In this embodiment, a signature verification system using a Vision Transformer as such a model will be described. The signature verification system is created by replacing a portion of the discrete representation model with a separately generated discrete representation model. The discrete representation model includes a quantization representation layer that discretizes each piece of data (divided patches) obtained by dividing a handwritten character image into predetermined discretization units (e.g., radical units). The signature verification system inputs each discrete unit into a transformer layer and outputs the relationship (context) between the discrete units. The system then uses InfoNCE loss to weight the relationships between the discrete units and train a discrete representation model to relax the relationships between the discrete units in different handwritten character images. The signature verification system then replaces the feature extractor of the signature verification model with the discrete representation model. This allows the signature verification model to be generated with less collected data. The signature verification system can then automatically determine whether two different writers have written the same signature or whether a single writer has written two different signatures. This allows it to detect cases such as fraudulent cancellation of insurance contracts or fraudulent contracts using forged signatures, thereby uncovering fraudulent attempts to receive insurance benefits by forging the policyholder's signature. Furthermore, cloud computing allows the signature verification system to be used conveniently and simultaneously by many users remotely, without the need for special devices.Furthermore, by introducing such a signature verification system, the human effort required to check signatures can be reduced.

[0013] The signature verification system is implemented on a cloud computing architecture and has the ability to interact with users through HTTP requests and responses via a GUI. The GUI has the ability to adjust parameters related to the division into discretization units and to visualize the division of the discretization units. Furthermore, the signature verification system has a verification function that determines whether two identical signatures are genuine or forged (whether they were written by the same hand), and a fraud detection function that checks whether a folder containing handwritten images of a specific hand is also containing handwritten images of signatures written by other handwriters.

[0014] In the following description, deep learning techniques such as convolutional neural networks (CNNs), transformers, linear layers, and multi-layer perceptrons are understood by those skilled in the art, and specific details and knowledge are not provided in detail. The source code for the method proposed in this invention can be written in any programming language. There are no particular restrictions on the experimental environment, including hardware, software, and computer networks.

[0015] Hereinafter, embodiments of the present invention will be described with reference to Figures 1 to 10. Note that the gray rectangles in the figures represent layer outputs, and the solid rectangles represent neural network layer blocks, flowchart processes, system components, etc.

[0016] FIG. 1 is a flowchart showing an example of processing performed by the signature verification system according to this embodiment. First, the signature verification system performs data collection processing to collect a small number of handwritten character images (001). Similarly, the signature verification system also collects a large number of unlabeled handwritten character images available on the Internet, etc. (002). Next, the signature verification system generates a discrete representation model using the collected unlabeled handwritten character images (003). Next, the signature verification system further trains the discrete representation model and replaces the feature extractor in the signature verification model with the trained discrete representation model (004). Thereafter, the signature verification system fine-tunes the two signature verification models using the collected handwritten character images (005). Finally, the signature verification system selects multiple best signature verification models and creates a signature verification application programming interface (API) (006). This completes the processing.

[0017] FIG. 2 shows an example of a data collection form displayed during data collection processing. The data collection form is displayed, for example, on the user's (collaborator's) computer. The data collection form consists of three sections: a text box 100 for entering consent, instructions, and author information, a signature entry area 101, and a registration button 104. First, the collaborator carefully reads the consent and instructions in the text box 100 before signing. If the collaborator agrees to the consent, they enter basic information such as a unique ID previously assigned by the administrator, their name, email address, phone number, and department. This information may be used in the future if changes are made to the consent or if additional signatures are collected. The signature verification system randomly selects several names (e.g., 10 names) from a list of, for example, 50,000 commonly known Japanese names and displays each name in the name display field 102 in the signature entry area 101. The scribe (collaborator) uses a digital pen on a tablet to handwrite the name displayed in the name display field 102 into the writable ink box 103. Once the writing is complete, the scribe clicks the registration button 104. The user's computer then sends various information about the collaborator and image data (handwritten character image) of the written name (signature) to the signature verification system. This allows the signature verification system to collect signature images from each collaborator and use each image for training the discrete representation model.

[0018] The data collection form shown in Figure 2 may also be used as the signature registration function of a signature verification system. If the signature verification system is designed so that users can basically use it by entering and registering the minimum amount of information required, for example, their own signature only once, registration can be completed in a short amount of time, improving convenience, but if another person's signature is entered and registered as shown in Figure 2, more of the user's handwriting characteristics can be used, which has the advantage of improving verification accuracy.

[0019] FIG. 3 is a diagram illustrating an example of the process (003) for generating a discrete representation model by a signature verification system. The signature verification system separates a batch B of (unlabeled) handwritten character images 200, 201 generated in the data collection process into multiple connected elements. These elements are, for example, radicals, parts of radicals, or groups of radicals if they are adjacent to each other. The signature verification system further divides these separated elements into fixed-size patches 202, 203. The signature verification system pads small connected elements or non-conforming connections to fit the fixed size. The signature verification system then inputs these patches 202, 203 into a linear projection layer 204, and generates embedded features 205Z, which are feature vectors of the patches (images) and their positions. B ={Z B1 , Z Bm , ..., Z BM}(Embedded features). M is the dimension of the feature, Z B is the continuous feature map of image B.

[0020] The signature verification system then inputs the embedded features 205 into a quantized representation layer 206, which receives a codebook of trainable radicals e1, e2, ..., e k Here, K is the number of types or radicals in the given codebook multiplied by c (c is a constant). B ) is discretized by mapping in the quantization representation layer 206 and replaced by the nearest code in the codebook. B ={Z^ B1 , Z^ B2 , ..., Z^ BM}) is obtained (Z^ represents the circumflux for Z, and so on). Then, the signature verification system obtains the quantized sequence 207(Z^ B ) and the patch position into the Transformer 208, B ={C B1 , C B2, ..., C BM} to obtain a context vector 211.

[0021] The signature verification system trains the discrete representation model using a loss function representing the InfoNCE loss212 (e.g., van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.). This allows the discrete representation model to recognize a “positive” relationship209 between discretized units within the same handwritten character image, and a “negative” relationship210 between discretized units in different handwritten character images. This InfoNCE loss is defined as follows:

number

[0022] where Q is the discretization unit (negative sample) of other text images in the batch, T is the transpose vector, and W m are the trainable weights. In this way, the discrete representation model is trained using unlabeled handwritten character images collected in the data collection process (001).

[0023] Using such a loss function produces the following effects. Since characters in the same handwritten character image are generally written by the same person, such characters (discretization units) are classified into the same class and are learned as having a "positive" relationship with each other. Similarly, since characters in two handwritten character images randomly selected from a large number of handwritten character images are written by almost different writers, the discretization units in the different handwritten character images can be learned as being classified into different classes and having a "negative" relationship. By performing such processing, a discrete representation model can be trained to distinguish between handwriting features of the same writer and handwriting features of different writers from handwritten character images that do not have information about the writer attached to them, i.e., are not labeled. This makes it possible to train a highly accurate discrete representation model without collecting a large number of signature images from a large number of subjects.

[0024] The discretization unit is, for example, the radical of a kanji character. Although there are thousands of types of kanji characters, there are only 214 types of radicals. Therefore, the number of types K of the codebook mentioned above can be set to 214 so that the characteristics of the radicals can be learned. If the features of radical units can be learned, it is basically possible to cover the handwriting characteristics of all types of kanji characters, and this is expected to improve learning efficiency.

[0025] Furthermore, a frequently occurring part other than the radical can also be used as the discretization unit. In this case, this can be achieved by setting K to a value other than 214. In this case, the discretization unit can be, for example, a part of the radical, or a part that spans multiple radicals. By using a frequently occurring part as the discretization unit, it becomes possible to support any language, including characters other than Chinese characters, such as Hangul and Hindi.

[0026] The discretization units can be displayed using a GUI (graphic user interface). When the radicals of kanji are used as the discretization units, the character can be divided into radical units. By displaying a bounding box for each unit, it is possible to visualize whether the division into radical units has been performed correctly. However, this GUI is not intended for end users to check, but rather for the operator who trains the discrete representation model to check the discretization units. This can be used to optimize parameter settings, for example, by adjusting the hyperparameters of the discrete representation model if the division into radical units is not successful. Alternatively, character templates for each radical can be prepared in advance, and the radical positions can be accurately calculated by template matching. The results can then be input into the discrete representation model. This allows the discrete representation model to more reliably train on a radical basis, which is expected to improve the accuracy of handwriting matching.

[0027] 4 is a diagram illustrating an example of the generation process (004) of a signature verification model according to this embodiment. This generation method is based on a Siamese neural network, which uses a set of identical neural networks for training (e.g., F. Schro, D. Kalenichenko, J. Philbin, "Facenet: A unified embedding for face recognition and clustering," in: CVPR, 2015, pp. 815-823).

[0028] This Siamese neural network is composed of two identical Vision Transformers 304, 305 (Vision Transformers) with the same architecture and shared weights. The construction method of such a Transformer is disclosed, for example, in "Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020)." First, in each training process, the signature verification system trains a pair of different handwritten character images 300, 301 (I 1n , I 2n ) are divided into patches 302, 303 of size k × l in the same way as the discrete representation model described above. Specifically, for example, handwritten character images 300, 301 are flattened from left to right and top to bottom of the image. The signature verification system then inputs each patch 302, 303 into a trainable linear projection layer 306 and maps it to obtain embedding features of the image (patch) and its position. The patches 302, 303 are similar to tokens in traditional Transformer-based language processing models (e.g., Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. (2017) “Attention is all you need”, arXiv preprint arXiv:1706.03762.).

[0029] The signature verification system then inputs the embedded features into a Transformer 307 and applies an attention mechanism called Multihead Self-Attention (MSA) to the entire model (see Vaswani et al., supra). Applying MSA to the embedded features allows the model to capture a larger spatial context between patches. While a typical CNN feature extractor consisting of convolution and max-pooling has the drawback of halving the spatial information in the feature map at each step, the signature verification model of this embodiment does not have this drawback.

[0030] The signature verification system then reduces the number of embedded features by inputting the embedded features encoded by the MSA into a multilayer perceptron network 308 (MLP) containing two linear layers, resulting in a feature map F 1n , F 2n The signature verification system then trains a model using a predetermined contrastive loss 309. The contrastive loss 309 is calculated by dividing the two feature maps F 1n , F 2n If the two signatures are of the same class (class 1, a pair of authentic signatures or a pair of signatures written by the same person), the distance between them is maximized. 1n , F 2n The contrast loss 309 can be expressed as, for example, the following equation (2):

number

[0031] where α and β are constants, D n Here are two images I 1n ,I 2n The Euclidean distance y between the feature maps is 0 for different pairs (a pair of a forged signature and an authentic signature) and 1 for identical pairs (a pair of an authentic signature and an authentic signature).

[0032] 5 is a diagram illustrating another example of the signature verification model generation process (004) according to this embodiment. This learning method is based on cross-modal attention (e.g., Yu, J., Zhang, W., Lu, Y., Qin, Z., Hu, Y., Tan, J., & Wu, Q. (2020). Reasoning on the relation: Enhancing visual representation for visual question answering and cross-modal retrieval. IEEE Transactions on Multimedia, 22(12), 3196-3209.).

[0033] As in the case of the Siamese neural network model, the signature verification system obtains embedded features for each of the handwritten signature images 400, 401 by inputting the two signature images 400, 401 into two vision transformers 404, 405. Here, the embedded features of the two handwritten character images of writer n encoded by the vision transformers 404, 405 are denoted as f1 and f2.

[0034] The signature verification system inputs the embedding feature f1 into one linear layer 408 to generate a d-dimensional 1n (Query 410), K 1n (Key 411), and V 1n Similarly, the signature verification system inputs the embedding feature f2 into the other linear layer 409 to obtain a d-dimensional 2n (Query 413), K 2n (Key 414), and V 2n Then, the signature verification system obtains the correlation between the embedding features of the two handwritten character images by inputting Q, K, and V into cross-modal attention 418. For example, the signature verification system calculates each dot product 416, 417 (attention) using the following equations (3) and (4).

number

number

[0035] Then, after calculating the dot products 416 and 417, the signature matching system concatenates the attentioned embedding features and inputs the concatenated features into a multi-layer perceptron (MLP) 419 to obtain the probability that the pair of two input handwritten signature images 400 and 401 is dissimilar (class 0) or similar (class 1). The model is then trained by a focused loss 420 that focuses on learning samples that are easily misclassified. The focused loss 420 is expressed, for example, by the following equations (5)-(7).

number

number

number

[0036] γ and α are constants. The larger the value of γ, the lower the loss for correctly classified pairs, so the model will focus on learning pairs that fail to be classified. α is a parameter that gives different weights to each class, allowing you to balance the importance of similar or dissimilar pairs.

[0037] The signature verification system then replaces the feature extractor (e.g., including a linear projection layer and a transformer encoder) in the existing signature verification model with the discrete representation model transformer (the vision transformers 304, 305 for Siamese neural networks or the vision transformers 404, 405 for cross-modal attention) trained as described above. The signature verification model is then fine-tuned using several samples of handwritten text image data collected by the data collection process (data collection form). This enables the signature verification system to use the optimally tuned signature verification model.

[0038] FIG. 6 is a diagram showing the general architecture of a signature verification system consisting of a front end 500 and a back end 504. The front end 500 is an information processing device used by, for example, a transcriber who is an end user. The front end 500 has the functions of a signature registration GUI 501, a signature verification GUI 502, and a fraud detection GUI 503 in a folder, which are the data collection forms shown in FIG. 2. On the other hand, the back end 504 is an information processing system that performs signature verification. The back end 504 has a registration controller 505, a database 506, a signature verification API 507, and a signature verification model 508.

[0039] As shown in FIG. 2, the signature registration GUI 501 of the front end 500 may accept the end user's input of a dummy signature different from their own name, or may accept only the input of their own signature. This can be freely determined depending on the system design. After finishing writing the signature, when the signature registration GUI 501 accepts the writer's click on the registration button, it sends the data to the registration controller 505 of the back end 504 via an HTTP request. The registration controller 505 analyzes the data, constructs a record, and registers the constructed record (signature data of each writer) in the database 506. The registration controller 505 returns a message indicating whether the data registration was successful or failed to the signature registration GUI 501 via an HTTP response.

[0040] The signature verification GUI 502 is the interface used to verify whether a user's signature is authentic or forged (whether it was written by the same scribe or a different scribe), i.e., to compare two signature images. The signature verification GUI 502 sends a request to the backend 504 to compare the signature images.

[0041] The fraud detection GUI 503 is an interface used to check whether a forged signature or a signature written by another creator exists in a specified folder that collectively manages signature images of the same scribe. The fraud detection GUI 503 sends a check request to the backend 504.

[0042] The signature verification API 507 receives a request from the signature verification GUI 502 or the fraud detection GUI 503 via the HTTP protocol. Upon receiving the request, the signature verification API 507 accesses the database 506 to acquire information about the writer and the signature, and inputs the acquired signature information into a signature verification model 508 to verify the signature or detect fraud. The signature verification API 507 creates a message of the verification result or detection result and returns the created message to the signature verification GUI 502 or the fraud detection GUI 503 via an HTTP response. The signature verification API 507 can be developed as a Lambda RESTful API of Amazon Web Service (AWS) (registered trademark) using, for example, the Python Flask library.

[0043] Next, we will explain, with reference to the drawings, the process of verifying whether signatures were written by the same writer (verification process) and the process of checking whether there are signatures written by other writers in the folder (fraud detection process).

[0044] FIG. 7 is a diagram showing an example of a GUI (signature verification GUI 502) displayed during the verification process. FIG. 8 is a diagram showing an example of a GUI (fraud detection GUI 503) displayed during the fraud detection process. FIG. 9 is a flowchart showing an example of the matching process. FIG. 10 is a flowchart illustrating an example of fraud detection processing.

[0045] As shown in FIG. 9 , in the verification process, first, the front end 500 displays the signature verification GUI 502 (described later) and accepts input of the user's own ID by a predetermined method (e.g., jQuery) (process 800). Then, the signature verification API 507 of the back end 504 obtains user information from the database 506, and the front end 500 displays the user's name in the signature verification GUI 502. The front end 500 also displays each registered signature in the signature verification GUI 502 (process 801). Next, the front end 500 accepts the selection of a signature image to be verified by the user clicking button 605 through the signature verification GUI 502 (process 802). Then, the front end 500 sends a request to the back end 504 to inquire whether the selected signature images were written by the same hand.

[0046] The signature verification API 507 creates all combinations (pairs) of the received signature image with each registered signature image. Here, it is assumed that there are p registered images, i.e., p pairs. The signature verification API 507 calls the signature verification model 508 to perform proof verification processing for each pair (process 803) and calculates a similarity rate to determine whether the two signatures were written by the same hand (process 804). For example, if the signature verification model 508 performs verification processing for v pairs written by the same hand, the calculated similarity rate is v / p × 100%. For example, if this rate exceeds a threshold of 50%, it means that the verified signatures were written by the same hand.

[0047] If the similarity rate exceeds a predetermined threshold (e.g., 50%), the signature verification API 507 creates a response message indicating that the signatures were written by the same hand and sends the created response message to the front end 500. The front end 500 displays the message in the signature verification GUI 502 (process 805).

[0048] If the similarity rate does not exceed a predetermined threshold (e.g., 50%), the signature verification API 507 creates a response message indicating that the signatures were written by different writers and sends the created response message to the front end 500. The front end 500 displays the message in the signature verification GUI 502 (process 806).

[0049] The signature verification API 507 may calculate the distance between the two images instead of the similarity ratio. In particular, the contrast loss shown in FIG. 3 calculates the distance between the two images, and this distance value may be displayed as a dissimilarity. However, it is difficult for a user to interpret the meaning of the distance value even if it is displayed directly. Therefore, the signature verification API 507 may convert the distance value into the "probability of the same handwriting or authenticity" using two distributions: the frequency distribution of distance values ​​between the same handwriting and the frequency distribution of distance values ​​between different handwritings. For example, this probability can be calculated based on the likelihood ratio of the two distributions. The advantage of calculating authenticity based on the distance value is that it is possible to calculate the authenticity in detail even if only one signature image is registered, and that the strictness of the judgment can be easily adjusted by setting a threshold for distinguishing between the authentic and a different person.

[0050] As shown in FIG. 7, the signature verification GUI 502 includes a text box 600 for accepting the user's ID, a text box 601 for accepting input of the user's name, an area 602 for displaying the signature image to be verified, a scrollable grid box 603 for displaying registered signatures, a text box 604 for displaying the path of the signature image to be verified, a button 605 for selecting the signature image to be verified, a verification button 606 for making a verification inquiry, and a text box 607 for displaying a response message.

[0051] 10, the front end 500 displays a fraud detection GUI 503 (described later) and accepts the user's selection of a folder to be checked (process 900). The front end 500 displays a list of signature images in subfolders of the selected folder in the fraud detection GUI 503 (process 901).

[0052] The front end 500 then sends a request to the back end 504 to inquire whether there are any fraudulent signatures written by other writers in the subfolders (process 902). The signature matching API 507 then creates pairs by combining each image in each subfolder with all remaining images in that subfolder. For example, if there are p remaining images, p pairs are generated. The signature matching API 507 then invokes the signature matching model 508 to perform a determination process for each pair to determine whether the two signatures were written by the same writer (process 903), and calculates the similarity rate of the signatures (process 904). For example, if the signature matching model 508 has processed v pairs written by the same writer, the similarity rate calculated in process 904 is v / p × 100%.

[0053] If the similarity rate exceeds a predetermined threshold (e.g., 50%), the signature verification API 507 creates a response message indicating that each signature was written by the same hand (authentic signature) and sends the created response message to the front end 500. The front end 500 displays the message on the fraud detection GUI 503 (process 905).

[0054] If the similarity rate does not exceed a predetermined threshold (e.g., 50%), the signature verification API 507 creates a response message indicating that the signatures were written by different writers (fraudulent signatures) and sends the created response message to the front end 500. The front end 500 displays the message on the fraud detection GUI 503 (process 906).

[0055] As in the matching process, a method of calculating the identity based on the likelihood ratio of the frequency distribution may be applied instead of the similarity rate.

[0056] The signature verification API 507 repeats these processes for all other images in the subfolder.Then, for each subfolder (writer), the front end 500 displays the total number of images, the total number of signatures written by other writers, the percentage of tampering, and the forged images in the fraud detection GUI 503 (process 907).

[0057] 8, the fraud detection GUI 503 includes a button 706 for selecting a folder, a text box 705 displaying the name of the selected folder, a scrollable grid view box 700 displaying a list of signature images in the selected folder (subfolders), a fraud detection button 707 for inquiring whether there are any fraudulent signatures written by other writers in the subfolder, and a text box 708 displaying the results of the fraud detection process (such as a response message). Note that it is assumed here that each subfolder stores signature images written by one writer, and that each folder is assigned the name of the writer.

[0058] Each grid 701 in the grid view box 700 displays information about each writer. Specifically, each grid 701 displays the corresponding writer's ID 702, name 703, and signature image 704.

[0059] The present invention is not limited to the above-described embodiments, and can be implemented using processes and components other than those specifically described in this specification without departing from the scope of the present invention. Therefore, the above-described embodiments and modifications are merely examples, and the present invention is not limited to these contents as long as the characteristics of the present invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these, and other aspects conceivable within the scope of the technical idea of ​​the present invention are also included in the scope of the present invention.

[0060] As described above, the signature verification system of this embodiment divides a handwritten character image into multiple patches 202, 203 of a predetermined size for each character, calculates embedded features for each patch, which are data on the pixel and position characteristics of each divided patch (linear projection layer 306), identifies the type of each patch (e.g., radical) based on the calculated embedded feature 205 for each patch (quantized representation layer 206), calculates the context of the handwritten character image for each patch based on the identified patch type and patch position, and applies MSA (Multihead Self-Attention), an attention mechanism, to the entire embedded feature (Transformer 307), and stores a discrete representation model.The signature verification system then generates a new signature verification model by replacing a portion of the signature verification model, which receives two handwritten character images and outputs the probability that the two input images are images of characters written by the same hand, with the discrete representation model.

[0061] In this way, the signature verification system of this embodiment divides a handwritten signature image into patches based on the basic components of the character, discretizes the patch features into individual codes using vector quantization, generates a signature verification model using a discrete representation model that learns the relationships between them, and performs signature verification using this signature verification model. Such a signature verification model can quickly learn signatures with a small amount of training data and achieve high accuracy in signature recognition. For example, it can compare a user's own signature with other signatures, or check whether a specified folder contains signatures written by other writers. As described above, the signature verification system of this embodiment can quickly learn with a small amount of training data and verify signatures with high accuracy.

[0062] Furthermore, applying MSA to the embedding features in the discrete representation model allows the model to capture a larger spatial context between patches.Usually, CNN feature extractors consisting of convolution and max-pooling have the drawback that the spatial information of the feature map is halved at each step, but this drawback does not exist in the signature verification model of this embodiment.

[0063] Furthermore, the signature verification system of this embodiment inputs the output values ​​of the discrete representation model into the InfoNCE loss evaluation function, which evaluates that patches in the same handwritten image are positively correlated and that patches in different handwritten images are negatively correlated, thereby learning weight values ​​in the discrete representation model so that patches in the same handwritten image are positively correlated and patches in different handwritten images are negatively correlated.

[0064] In this way, inputting the output values ​​of the discrete representation model into the InfoNCE loss function allows us to capture a larger spatial context between patches.

[0065] In addition, in the signature verification system of this embodiment, Q is the discretization unit of other images in the batch, T is the transposed vector, and W m is the learnable weight, the InfoNCE loss evaluation function is Defined as TIFF2025164277000009.tif13170.

[0066] By defining the loss evaluation function (focus loss) in this way, the larger the value of γ, the lower the loss for correctly classified pairs, so the model will focus on learning pairs that fail to be classified.

[0067] Furthermore, in the signature verification system of this embodiment, the discretization unit is the radical of a Chinese character, so that the handwriting characteristics of basically all types of Chinese characters can be covered, and therefore improvement in learning efficiency can be expected.

[0068] Furthermore, the signature verification system of this embodiment calculates the dot product of the embedded features of the two input images, and outputs the probability of similarity between the two input images.

[0069] The signature verification system of this embodiment also includes a GUI that allows the writer to register his or her own information and signature in the system; a signature verification GUI that displays a response message indicating the verification result, as determined by the signature verification model, whether the signatures were written by the same writer or whether the signatures are genuine or forged; a folder check GUI that displays a response message indicating the verification result, as determined by the signature verification model, whether each signature stored in a specified folder is written by the same writer or whether there is a forged signature in the specified folder; a registration controller that acquires the writer's information and signature and stores them in a database; and a signature verification unit that receives requests from the signature verification GUI and the folder check GUI, accesses the database to acquire the writer's information and signature, calls the signature verification model to create a response message indicating the verification result or the determination result, and returns the created response message to the signature verification GUI or the folder check GUI.

[0070] A signature verification system with a GUI built on cloud computing like this allows many users to use it remotely and simultaneously without the need for special devices. The introduction of such a signature verification system also reduces the human labor required to check signatures.

[0071] Furthermore, the signature verification system of this embodiment acquires and encodes the embedded features of the two signature images 400 and 401 by inputting the two signature images 400 and 401 into two vision transformers 404 and 405, respectively, and replaces the feature extractor in the existing signature verification model with the discrete representation model transformer trained as described above (the vision transformers 304 and 305 related to Siamese neural networks or the vision transformers 404 and 405 related to cross-modal attention). The signature verification system is then fine-tuned using several samples of handwritten text image data collected by the data collection process (data collection form).

[0072] In this way, the signature verification system uses several samples of handwritten text image data collected by the data collection process (data collection form) to fine-tune the signature verification model, allowing the signature verification system to use an optimally tuned signature verification model. [Explanation of symbols]

[0073] 001 A process for collecting a small amount of handwritten signature data, 002 A process for collecting unlabeled handwritten character images, 003 A process for training a discrete representation model, 004 A process for replacing the feature extractor of a signature matching model with a discrete representation model, 005 A process for fine-tuning a signature matching model, 006 A process for selecting an optimal signature matching model to create a signature verification API.

Claims

1. a storage device that stores a discrete representation model that divides an image containing one or more handwritten characters into a plurality of patches of a predetermined size for each character, calculates an embedded feature for each patch, which is data on the pixel and position characteristics of each divided patch, identifies the type of each patch based on the calculated embedded feature for each patch, calculates the context of each patch in the image based on the identified type of each patch and each patch position, and applies MSA (Multihead Self-Attention), which is an attention mechanism, to the entire embedded feature; and a computing device that receives two images containing one or more handwritten characters, outputs a probability that the two input images are images of characters written by the same hand, and generates a new signature verification model by replacing a part of the signature verification model with the stored discrete representation model; A signature verification system comprising:

2. 2. The signature verification system according to claim 1, The storage device storing an InfoNCE loss evaluation function that evaluates patches in the same handwritten image as being positively correlated and patches in different handwritten images as being negatively correlated; The computing device inputting two images each containing one or more characters handwritten by the same handwriter into the discrete representation model, and outputting, for each image, a context for each patch in the image; By inputting each output context into the InfoNCE loss evaluation function and obtaining their values, weight values ​​in the discrete representation model are learned so that patches in the same handwritten image are positively correlated and patches in different handwritten images are negatively correlated. Signature verification system.

3. Q is the discretization unit of other images in the batch, T is the transpose vector, W m is a learnable weight, The InfoNCE loss evaluation function is is defined as The signature verification system according to claim 2 .

4. The discretization unit is the radical of a Chinese character. The signature verification system according to claim 3 .

5. The signature verification model is outputting a probability of similarity between the two input images by calculating a dot product of the embedding features of the two input images; The signature verification system according to claim 1 .

6. 2. The signature verification system according to claim 1, A GUI for the scribe to register his / her information and signature in the system; a signature verification GUI for displaying a response message indicating a verification result as to whether the signatures were written by the same hand, or whether the signature is genuine or a forgery, as determined by the signature verification model; a folder check GUI that displays a response message indicating the determination result of whether each signature stored in a predetermined folder is a signature of the same handwriting or whether there is a forged signature in the predetermined folder, as determined by the signature verification model; a registration controller that captures the scribe's information and signature and stores them in a database; a signature verification unit that receives requests from the signature verification GUI and the folder check GUI, accesses the database to obtain information about the writer and the signature, calls the signature verification model to create a response message indicating the verification result or the determination result, and returns the created response message to the signature verification GUI or the folder check GUI; A signature verification system further comprising:

7. The signature verification model described above is training the discrete representation model by inputting two signature images into two identical vision transformers, respectively, to obtain and encode the embedded features of the two signature images; replacing the feature extractor in an existing signature verification model with the trained transformer of the discrete representation model; and then fine-tuning the signature verification system using the collected one or more samples of signature images. The signature verification system according to claim 1 .

8. A signature verification method implemented by a signature verification system including a storage device that stores a discrete representation model and a computing device that generates a signature verification model, the method comprising: storing an InfoNCE loss evaluation function in the storage device that evaluates patches in the same handwritten image as being positively correlated and patches in different handwritten images as being negatively correlated; The computing device inputs two images, each containing one or more characters handwritten by the same writer, into the discrete representation model, and outputs the context of each patch in the image for each image. The computing device inputs each of the output contexts into the InfoNCE loss evaluation function to obtain their values, thereby learning weight values ​​in the discrete representation model so that patches in the same handwritten image have a positive correlation and patches in different handwritten images have a negative correlation; A signature verification method comprising:

Images