A method and system for fine-grained classification of different handwriting of the same electronic signature

Abstract

The present application belongs to the technical field of computer vision, and discloses a fine-grained classification method and system for different handwriting of the same electronic signature, which is mainly used for classification of the same name and different signature modes, such as distinguishing natural handwriting signature, template signature and mechanical arm imitated signature, etc. The handwriting feature map of the input electronic signature image is extracted by a ResNet101 feature extractor and input into a module of extensible attention electronic signature fine-grained features. The module can assist the model in distinguishing the relationship between the current handwriting fine-grained features and the adjacent handwriting features in a feature map, and is helpful for the model to distinguish the sub-class types of different handwriting of the same signature. The fine-grained classification method of the electronic signature handwriting proposed by the present application is superior to other latest methods, so that the classification model can effectively cluster different discriminative features, thereby realizing high-precision classification of different handwriting of the electronic signature.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a fine-grained classification method for different handwriting of the same electronic signature. BACKGROUND

[0002] Electronic signature sets a model, a mechanical arm imitates and copies natural handwriting to write similar handwriting. Therefore, copying and mechanical arm imitation are two common signature attack methods. Copying signature is completed in a non-natural state, and has "awkward" characteristics in terms of writing direction, pressure and speed; mechanical arm signature is an action under computer control, and has inherent regularity in writing speed, acceleration, pressure and writing method. The inherent characteristics formed by the above two signature attacks have many fine-grained differences with the personalized characteristics of natural human signatures. The traditional classification method is difficult to distinguish the fine-grained differences of the above handwriting. Therefore, the electronic signature classification technology based on fine-grained feature classification needs to be developed.

[0003] The fine-grained classification model at the present stage mainly includes strong supervision learning strategy with fine-grained feature labeling and weak supervision learning strategy without corresponding labeling. Since it is not practical to label all fine differences, and manual labeling may not be the best fine-grained discriminative region. Therefore, the fine-grained classification method at the present stage mostly adopts the weak supervision learning strategy. The fine-grained classification model of electronic signature also tends to the weak supervision learning strategy, which can more accurately classify the fine differences of electronic signature under the condition of only providing handwriting category label. The model can effectively distinguish different types of electronic signature by adding fine-grained discriminative module in the existing classification network. The added discriminative module can effectively aggregate unannotated data under the weak supervision condition, thereby improving the classification performance of the fine-grained classification model.

[0004] Most of the research on fine-grained images includes the color of the category, the behavior action between subcategories, etc., for example, the local area (head, foot, wing, skin color) of the foreground category (bird) is regarded as a subcategory with a discriminative area. However, unlike general fine-grained classification data, electronic signatures only include single-color lines. Therefore, the discriminative line subcategory of the handwriting category only has fine-grained features of different handwriting lines. General fine-grained classification models do not focus on the division of single features, so it is difficult to distinguish the fine-grained differences of electronic signature sub-handwriting. For example, natural handwriting and robot imitated signatures have little difference in color and overall shape. Robot imitated handwriting only has unnatural handwriting line differences when carefully imitating the original handwriting. In addition, the difference between the mold handwriting and the real handwriting is very small, and there may be only a slight texture difference, and the shape features are almost the same. The current fine-grained model lacks the scalability from the local texture difference of handwriting to the overall shape feature, and may not effectively capture these subtle but important differences, resulting in the loss of information between categories and affecting the accuracy of handwriting classification.

[0005] Through the above analysis, the problems and defects of the prior art are that the traditional classification method is difficult to distinguish the fine-grained differences of handwriting. SUMMARY

[0006] In view of the problems existing in the prior art, the present application provides a fine-grained classification method for different handwriting of the same electronic signature. Specifically, the model determines the correlation of the local features and the adjacent features, the adjacent features continue to expand outward, and the correlation of the adjacent features is determined until the entire image is traversed.

[0007] The present application is realized in a fine-grained classification method for different handwriting of the same electronic signature, comprising the following steps:

[0008] S1: extracting the coarse-grained features of the input electronic signature image through the ResNet101 feature extractor; wherein the ResNet101 contains five stages of electronic signature feature outputs in total;

[0009] S2: dividing the electronic signature feature maps of the last three stages of S1 into image blocks of equal size, and assigning a position number to each image block to determine the position of each image block corresponding to the original feature map;

[0010] S3: inputting the three-stage feature maps with position numbers in S2 into the scalable attention mechanism; wherein the scalable attention mechanism expands to the surrounding according to the small block feature map, and determines the correlation of the image blocks expanded to the current block according to the similarity calculation;

[0011] S4: The high-order output of the scalable attention in S3 is upsampled and fused with the adjacent previous scalable attention layer to serve as the final output of the scalable attention in each stage. Then, the outputs of the three stages are concatenated together and input into the general classification network model.

[0012] S5: A discriminative fine-grained feature of the electronic signature is obtained through a scalable attention mechanism. The general classifier distinguishes electronic signatures with different handwriting based on the above fine-grained features.

[0013] Furthermore, step S1 specifically includes:

[0014] The original electronic signature image is input into the feature extractor ResNet101, which consists of 5 different residual layers. The output of each subsequent layer is closer to the semantic information and the size of the output feature map gradually decreases.

[0015] Furthermore, step S2 specifically includes:

[0016] The outputs of the last three layers are chosen as the subsequent inputs because these three layers are closer to the semantic features of electronic signatures, and the feature map size is smaller, allowing for the acquisition of more valuable electronic signature features with lower computational cost. Given an input feature map x∈R... C×H×W H and W represent the height and width of the electronic signature feature image, respectively; x is divided into H×W blocks, each block representing a pixel in the image, which is represented as a value in the image matrix in the calculation of the convolutional neural network. Then, a position code is added to each block to record the position of the original feature map corresponding to each block.

[0017] Furthermore, step S3 specifically includes:

[0018] In the electronic signature feature extraction stage, an expandable attention learning module is added. Based on the current handwriting features, the expandable attention module continuously expands outward to learn features with coexisting relationships, effectively locating the discriminative differences between template-based handwriting, robotic arm-imitation handwriting, and natural handwriting.

[0019] The scalable attention mechanism draws inspiration from the attention mechanism. In the scalable attention mechanism, the current small block is continuously extended outward to find the most relevant small blocks. These small blocks continue to expand outward until the entire feature map is traversed.

[0020] Let x be a small block (i.e., a pixel) in the electronic signature feature map x. q The offset weights in the scalable layer are used to calculate the value related to x. q The N adjacent expandable elements, each belonging to another pixel in x, are then used to calculate the similarity of the matrices to obtain x. qCorrelation with N elements. Specifically, let W q denote the linear layer for calculating the weight of x q . Let and denote the scalable linear layer, respectively, and x is input into the above two linear layers to obtain the key scalable weight and value scalable weight of x q . The weight of x q is learned through dot product operation to obtain the correlation with the key scalable weight. Finally, the similarity score is normalized using softmax to aggregate the value scalable point. Therefore, the scalable attention calculation of x q can be expressed as:

[0021]

[0022] In scalable attention, N is usually less than 10. Therefore, each x q only needs to focus on N elements.

[0023] Since the entire scalable attention mechanism calculation belongs to parallel operation, after expanding the first sampling position, it does not continue to expand outward directly from the obtained N elements, but calculates the attention of the points adjacent to the first position until the entire electronic signature feature map is traversed. This not only allows the model to expand the features of a certain x q , but also gradually associates all relevant points of the entire signature feature map to x q . The purpose is to make each feature point not only determine which feature point coexists in the same class of electronic signature itself, but also which feature point is the background or other handwriting. In summary, the computational complexity of ExAttention(x) can be given by O(N q C 2 +NC 2 +NN q C), where N q represents the number of pixel points of x, N q >>N>>C. Therefore, the complexity can be simplified to O(NN q C), which is equivalent to O(NHWC) in pixel-level feature mapping; considering the scalable attention, the key elements of different sampling positions can transmit information between the corresponding layers; specifically, although the attention of a layer only utilizes N key elements, these elements have already obtained the information of other attention values in the previous corresponding layer; after multiple scalable attentions, the output can be aggregated with all key elements; by modeling the relationship between x q in the feature map through the above scalable attention at the pixel level, the coexisting partial region can be extracted;

[0024] In order to ensure the generalization ability of the model, each pixel point is not only expanded once, that is, each pixel point is calculated for a plurality of times of offset and correlation, and a plurality of attention results obtained finally are aggregated together to form a new fine-grained electronic signature feature image matrix; assuming that each pixel point is expanded for M times of coexistence features, the multiple expandable attention of the input feature map x can be expressed as:

[0025]

[0026] wherein, W m represents a weight vector;

[0027] The increase of the expandable attention layer can enhance the discriminant analysis of the handwriting features and improve the robustness of target classification in the data scene; the output of the previous layer is taken as the input of the next layer, and with the deepening of the layer, the expandable ability of the model is gradually excited, and the electronic signature handwriting with the coexistence relationship is completely covered, and L is the number of layers, and the multi-head multi-layer expandable attention Φ(·) can be expressed as:

[0028] Φ(1) = MHExAttention(x);

[0029] Φ(2) = MHExAttention(Φ(1));

[0030] ...

[0031] Φ(L) = MHExAttention(Φ(L-1));

[0032] When L is equal to 7, the classification accuracy of the handwriting is the highest;

[0033] In addition, in the feature map of the electronic signature in a batch during training, the expandable attention can indirectly learn the expandable elements in other feature maps from a sample position in the map, and the obtained elements can associate the handwriting features from different images and having the coexistence relationship, and also help the model to distinguish other handwriting features which do not have the coexistence relationship.

[0034] Further, the step S4 specifically comprises:

[0035] The high-order output of the scalable attention is fused with the adjacent previous scalable attention layer through upsampling, as the final output of the scalable attention of each stage, specifically, the fifth layer is obtained through upsampling and the feature map of the fourth layer is the same size, the feature map after upsampling is fused with the fourth layer feature map in the form of matrix addition, the fusion feature map is taken as the output of the fourth layer, the fusion image obtained in the fourth layer is fused with the feature map of the third layer through upsampling, and the fusion result is taken as the output of the third layer, the fifth layer is not fused, and the output is directly output, and then the outputs of the three stages are spliced together and input into a general classification network model;

[0036] The outputs of the attention layers at different scales are fused with each other through upsampling, so as to help the high-order stage to correct the previous stage and help the output of the model to restore the full spatial resolution, finally, the fusion stage is aggregated, the scalable feature information is enriched, and the performance of the electronic signature handwriting fine-grained feature classification is improved, on this basis, the model after correcting the scalable parameters can infer the handwriting coexistence feature under the full spatial resolution.

[0037] Further, the step S5 specifically comprises:

[0038] The scalable attention mechanism for fine-grained feature extraction is combined with the general classifier, so as to become a fine-grained classification model for electronic signature handwriting recognition, that is, the fine-grained feature vector with the handwriting coexistence relationship is input into the subsequent general classifier, and the classification effect of the electronic signature handwriting is adjusted through the cross-entropy loss function.

[0039] Another object of the present application is to provide a fine-grained classification system for different handwriting of the same electronic signature, which applies the fine-grained classification method for different handwriting of the same electronic signature, and the fine-grained classification system for different handwriting of the same electronic signature comprises:

[0040] The feature extraction module is used for extracting the coarse-grained features of the input electronic signature image through the ResNet101 feature extractor, wherein the ResNet101 contains five stages of electronic signature feature outputs in total;

[0041] The image segmentation module is connected with the feature extraction module and is used for segmenting the electronic signature feature maps of the last three stages into image blocks with equal size, and giving each image block a position number, so as to determine the position of each image block corresponding to the original feature map;

[0042] The attention module is connected with the image segmentation module, and is used for inputting three-stage feature maps with position numbers into scalable attention mechanisms respectively.

[0043] The feature fusion module is connected with the attention module, and is used for fusing high-order outputs of S3 scalable attention through upsampling and adjacent previous scalable attention layers, taking the fused outputs as final outputs of scalable attention of each stage, and then splicing the outputs of the three stages to input into the classification module.

[0044] The classification module is connected with the feature fusion module, and a classifier distinguishes different electronic signatures according to the fused fine-grained features.

[0045] Another object of the present application is to provide a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the fine-grained classification method for different handwriting of the same electronic signature.

[0046] Another object of the present application is to provide a computer readable storage medium storing a computer program, and the computer program is executed by the processor to make the processor execute the steps of the fine-grained classification method for different handwriting of the same electronic signature.

[0047] Another object of the present application is to provide an information data processing terminal for realizing the fine-grained classification system for different handwriting of the same electronic signature.

[0048] In combination with the above technical solutions and solved technical problems, the technical solution to be protected by the present application has the following advantages and positive effects:

[0049] Firstly, the fine-grained classification method for different handwriting of the same electronic signature can be used for model recognition of fine-grained features of electronic signature handwriting, so that the model can learn to determine the correlation between local features and global features of handwriting, and can effectively distinguish single discriminative fine-grained features in electronic signature, thereby assisting in improving the classification of general fine-grained classification models for sub-class handwriting of electronic signature.

[0050] For each step, the following is the significant technical progress achieved thereby:

[0051] Step 1: A high-performance computer and a large number of electronic signature sample data are selected. This step is to obtain suitable hardware and data resources, which is the basis for achieving high-precision electronic signature recognition. A high-performance computer can process a large amount of data and speed up the training; and a large amount of diverse electronic signature sample data can ensure the generalization ability of the model, so that the model can achieve good results on unseen data.

[0052] Step 2: Feature extraction using ResNet101. ResNet101 is a deep residual network that can effectively solve the problem of gradient disappearance and gradient explosion in deep neural networks, allowing the network to learn features in the picture deeply. In addition, dividing the feature map and assigning a location number can increase the model's understanding of feature location information, which helps to improve the performance of the model.

[0053] Step 3: Use the scalable attention mechanism to process the feature map. This step is to extract higher-order features, so that the model can pay attention to more important information and improve the model's understanding of electronic signature fine-grained features. In addition, by upsampling and fusing with the previous scalable attention layer, the spatial information of the features is further extracted and retained, enhancing the model's discrimination ability.

[0054] Step 4: Concatenate the output and input it into a general classification network model. This step is to integrate all the features before, and use a general classification network model for classification. This way, different classification network models can be selected according to different application requirements, so that the model can achieve optimal results in specific tasks.

[0055] Step 5: Obtain discriminative electronic signature fine-grained features and use a general classifier to classify electronic signatures. This step realizes high-precision recognition of electronic signatures, making electronic signatures more secure and unique, which is of great significance to improving the recognition accuracy and reliability of electronic signatures.

[0056] In summary, this technical solution combines deep learning and attention mechanisms to achieve high-precision recognition of electronic signatures, improving the security and reliability of electronic signatures, and has high practical value and application prospects.

[0057] Second, the fine-grained classification method of electronic signature handwriting proposed by the present application is superior to other latest electronic signature handwriting classification methods and can be applied to the classification of electronic signature handwriting such as template, mechanical arm imitation, and natural handwriting.

[0058] Thirdly, the application provides an auxiliary electronic signature attack identification algorithm model design, which can distinguish between copy signature attacks and mechanical arm signature attacks in the signature process of the prosecution. The application has technical advantages and innovation, realizes a reliable inspection process, and meets social needs. Commercial value: a reliable inspection process can improve employee motivation and satisfaction, enhance employee sense of belonging and identity, improve employee work efficiency and performance, and provide impetus for the development of inspection business.

[0059] Fourthly, at present, there are few electronic signature attack identification technologies based on inspection. The application provides a fine-grained classification technology as support, and researches a fine-grained classification method for different handwriting of the same electronic signature, which fills the technical gap in the industry at home and abroad. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiments of the application will be briefly introduced as follows. Obviously, the drawings described below are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0061] Figure 1 is a flowchart of the fine-grained classification method for different handwriting of the same electronic signature provided by the embodiments of the application;

[0062] Figure 2 is an electronic signature expandable attention mechanism diagram provided by the embodiments of the application;

[0063] Figure 3 is an overall process diagram provided by the embodiments of the application;

[0064] Figure 4 is a copy signature classification visualization effect provided by the embodiments of the application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the application more clear, the application will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0066] In view of the problems in the prior art, the application provides a fine-grained classification method for different handwriting of the same electronic signature, which will be described in detail below in combination with the drawings.

[0067] As shown in Figure 1 , the fine-grained classification method for different handwriting of the same electronic signature provided by the embodiments of the application includes the following steps:

[0068] S1, extracting coarse-grained features of an input electronic signature image through a ResNet101 feature extractor. The ResNet101 includes five stages of electronic signature feature outputs in total;

[0069] Step S1 specifically involves inputting an original electronic signature image into the feature extractor ResNet101. The ResNet101 is divided into five different residual layers in total, and the output of each subsequent layer is closer to semantic information, and the size of the output feature map gradually decreases.

[0070] S2, dividing the electronic signature feature maps of the last three stages in S1 into image blocks of equal size, and assigning a position number to each image block to determine the position of each image block corresponding to the original feature map;

[0071] Step S2 specifically involves selecting the output of the last three layers as the subsequent input. Since the last three layers are closer to the semantic features of the electronic signature, and the size of the feature map is low, more valuable electronic signature features can be obtained with lower computational complexity. Given an input feature map x∈R C×H×W , H and W represent the height and width of the electronic signature feature image respectively. x is divided into HxW small blocks, each small block represents a pixel point of the image, which represents a value in the image matrix in the calculation of the convolutional neural network, and then each small block is added with a position code to record the position of each small block corresponding to the original feature map.

[0072] S3, inputting the three-stage feature maps with position numbers in S2 into an extendable attention mechanism. The extendable attention mechanism extends to the surrounding according to the current small block feature, and determines the relevance of the extended image block to the current block according to the similarity calculation;

[0073] S3 specifically involves adding an extendable attention learning module in the electronic signature feature extraction stage. The extendable attention module continuously extends outward to learn features with coexistence relationship according to the current handwriting features. The discriminative differences between the natural handwriting and the mechanical arm imitated handwriting and the template are effectively located.

[0074] The extendable attention mechanism of the embodiment of the present application is derived from the attention mechanism. In the extendable attention mechanism, the current small block is continuously extended outward, and a number of most relevant small blocks are found, which continue to extend outward until the entire feature map is traversed (as shown in Figure 2 ).

[0075] A small block (i.e. a pixel point) in the electronic signature feature map x is denoted as x q . Through the offset weight calculation in the extendable layer, the most relevant small blocks to xq N adjacent extendable elements, N elements belong to other pixel points in x, and the similarity of the matrix is calculated respectively to obtain x q and the correlation of N elements. Specifically, let W q represent the linear layer, which is used to calculate the weight of x q . Let and represent the extendable linear layer, respectively, and x is input into the above two linear layers to obtain the key extendable weight and value extendable weight of x q . The weight of x q is learned through dot product operation to obtain the correlation with the key extendable weight. Finally, softmax is used to normalize the similarity score to aggregate the value extendable point. Therefore, the extendable attention calculation of x q can be represented as:

[0076]

[0077] In the extendable attention, N is usually less than 10. Therefore, each x q only needs to focus on N elements.

[0078] Since the entire extendable attention mechanism calculation belongs to parallel operation, after expanding the first sampling position, it does not directly continue to expand from the obtained N elements, but calculates the attention of the points adjacent to the first position until the entire electronic signature feature map is traversed. This not only allows the model to expand the features of a certain x q , but also gradually associates all related points of the entire signature feature map to x q . The purpose is to make each feature point not only determine which feature point coexists in the same class of electronic signature itself, but also which feature point is the background or other handwriting. In summary, the computational complexity of ExAttention(x) can be given by O(N q C 2 +NC 2 +NN q C), where N q represents the number of pixel points of x, N q >>N>>C. Therefore, the complexity can be simplified as O(NN q C), which is equivalent to O(NHWC) in pixel-level feature mapping. Considering the extendable attention, the key elements of different sampling positions can transmit information between the corresponding layers. Specifically, although the attention of a layer only utilizes N key elements, these elements have already obtained the information of other attention values in the corresponding layers of the previous layers. After multiple extendable attentions, the output can be aggregated with all key elements. Through the above extendable attention, the feature map xq The coexistence part region can be extracted by modeling the relationship between the two.

[0079] To ensure the generalization ability of the model, each pixel point is not only expanded once, that is, each pixel point is calculated for a plurality of times of offset and correlation, and a plurality of attention results obtained finally are aggregated together to form a new fine-grained electronic signature feature image matrix. Assuming that each pixel point is expanded for M times, the multiple expandable attention of the input feature map x can be expressed as:

[0080]

[0081] wherein, W m represents a weight vector.

[0082] The increase of the expandable attention layer can enhance the discriminant analysis of the handwriting features and improve the robustness of the target classification in the data scene. The output of the previous layer is taken as the input of the next layer, and with the deepening of the layer, the expandable ability of the model is gradually excited, and the electronic signature handwriting with the coexistence relationship is completely covered. Let L be the number of layers, and the multi-head multi-layer expandable attention Φ(·) can be expressed as:

[0083] Φ(1)=MHExAttention(x)

[0084] Φ(2)=MHExAttention(Φ(1))

[0085] ...

[0086] Φ(L)=MHExAttention(Φ(L-1)),

[0087] Experiments show that when L is equal to 7, the classification accuracy of the handwriting is the highest.

[0088] In addition, in the feature map of the electronic signature in a batch during training, the expandable attention can indirectly learn the expandable elements in other feature maps from a sample position in the map, and the obtained elements can associate the handwriting features from different images and having the coexistence relationship, and also help the model to distinguish other handwritings that do not have the coexistence relationship.

[0089] S4, the high-order output of the expandable attention of S3 is fused with the adjacent previous expandable attention layer through up-sampling, as the final output of the expandable attention of each stage, and then the outputs of the three stages are spliced together and input into a general classification network model; step S4 is specifically,

[0090] The high-order output of the scalable attention is fused with the adjacent previous scalable attention layer through upsampling as the final output of the scalable attention of each stage. Specifically, the fifth layer is obtained through upsampling to have the same size of the feature map of the fourth layer, the feature map after upsampling is fused with the fourth layer feature map in the form of matrix addition, and the fusion feature map is taken as the output of the fourth layer. The fusion image obtained in the fourth layer is upsampled and fused with the feature map of the third layer in the same way, and the fusion result is taken as the output of the third layer. The fifth layer is directly output without any fusion operation. Then, the outputs of the above three stages are spliced together and input into a general classification network model.

[0091] The outputs of the attention layers at different scales are fused with each other through upsampling, thereby helping the high-order stage to correct the previous stage and helping the output of the model to restore the full spatial resolution. Finally, the fusion stage is aggregated, the scalable feature information is enriched, and the performance of classifying the fine-grained features of the electronic signature is improved. On this basis, the model after correcting the scalable parameters can infer the coexistence features of the handwriting under the full spatial resolution.

[0092] S5, obtaining discriminative fine-grained features of the electronic signature by the scalable attention mechanism, and distinguishing different electronic signatures of the handwriting according to the fine-grained features by the general classifier.

[0093] Step S5 is specifically that the general classifier and the scalable attention mechanism for fine-grained feature extraction are combined to form a fine-grained classification model for the electronic signature handwriting recognition, that is, the fine-grained feature vector with the discriminative coexistence relationship of the handwriting is input into the subsequent general classifier, and the classification effect of the electronic signature handwriting is adjusted through the cross-entropy loss function.

[0094] Fusing the scalable attention mechanisms of the three stages as the fine-grained discriminative module can amplify the linear difference between the subclasses, thereby helping the subsequent supervised classifier to more accurately classify the electronic signatures of different handwriting. Because the model can learn the handwriting features and rules with the coexistence relationship from the unannotated data through the scalable attention, thereby helping the general classifier to better distinguish different types of electronic signatures.

[0095] In summary, the weakly supervised fine-grained classification model of the electronic signature is a model that can effectively distinguish different types of electronic signature handwriting. It amplifies the linear difference between the subclasses by increasing the fine-grained discriminative module, and improves the classification performance by using the unannotated data under the weak supervision condition. The model has important application value for the classification of the electronic signature handwriting.

[0096] Figure 3 The whole process from cutting to fusion in the embodiment of the application is described.

[0097] The application embodiment of the present application provides a computer device, the computer device comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the fine-grained classification method for different handwriting of the same electronic signature.

[0098] The application embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the fine-grained classification method for different handwriting of the same electronic signature.

[0099] The application embodiment of the present application provides an information data processing terminal, which is used to realize the fine-grained classification system for different handwriting of the same electronic signature.

[0100] It should be noted that the embodiments of the present application can be realized by hardware, software or a combination of software and hardware. The hardware part can be realized by special logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above-mentioned devices and methods can be realized by computer executable instructions and / or included in processor control code, such as provided on a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The devices of the present application and their modules can be realized by hardware circuits, such as very large scale integrated circuits or gate arrays, semiconductors, such as logic chips, transistors, etc., or programmable hardware devices, such as field programmable gate arrays, programmable logic devices, etc., can also be realized by software executed by various types of processors, and can also be realized by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0101] In order to further verify the effectiveness of the model of the present application, that is, the expansion ability from local features to overall features. Four natural handwriting and traced handwriting of signatures are randomly selected from the existing data set, and the heat map of the difference between the traced signature and the natural handwriting signature is drawn using the category activation mapping technology. The deeper the color of the region in the heat map, the more attention the model pays to the region when classifying. As shown in Figure 4 The model without the expandable mechanism can only focus on one or two regions when identifying different handwriting signatures, while the model with the expandable mechanism can locate multiple positions with discriminative differences, which helps the model to make the final classification.

[0102] The following are two specific embodiments of the present application:

[0103] Embodiment 1:

[0104] 1. Use a high-performance computer equipped with an Nvidia GeForce RTX 3090 graphics card to conduct experiments. First, download a public dataset containing a large number of electronic signature samples from the Internet.

[0105] 2. Use ResNet101 to extract features from the dataset. Select the output of the last three stages as the input of the fine-grained classification model. Divide the electronic signature feature maps of the last three stages into equal-sized image blocks and assign a position number to each image block.

[0106] 3. Input the three-stage feature maps with position numbers into the scalable attention mechanism. Use the scalable attention mechanism to calculate the feature maps and fuse the resulting high-order output with the previous scalable attention layer through upsampling.

[0107] 4. Concatenate the outputs of the three stages and input them into a general classification network model. This general classifier can be ResNet, VGG, Inception, etc.

[0108] 5. Obtain discriminative fine-grained features of electronic signatures through the scalable attention mechanism. Finally, use the general classifier to classify electronic signatures and obtain the results.

[0109] Example 2:

[0110] 1. Use a high-performance computer equipped with an Nvidia Titan V graphics card to conduct experiments. First, obtain a batch of electronic signature sample data from the enterprise.

[0111] 2. Use the ResNet101 model to extract features from the electronic signature data. Select the output of the last three stages as the input of the fine-grained classification model, and divide these feature maps into equal-sized image blocks and assign a position number to each image block.

[0112] 3. Use the scalable attention mechanism to process the three-stage feature maps with position numbers. And fuse the high-order output of the scalable attention through upsampling with the previous scalable attention layer.

[0113] 4. Concatenate the outputs of the three stages and input them into a general classification network model. This classifier can be AlexNet, DenseNet, EfficientNet, etc.

[0114] 5. Obtain discriminative fine-grained features of electronic signatures through the scalable attention mechanism. Finally, use the general classifier to classify electronic signatures and obtain the results.

[0115] The two embodiments are only for reference and are not limited to the use of the specified hardware or specific classification network model. Suitable hardware and model can be adjusted and selected according to specific needs and scenes.

[0116] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any modification, equivalent replacement, and improvement within the technical range disclosed by the present application and within the spirit and principle of the present application should be covered within the protection scope of the present application.

Claims

1. A method for fine-grained classification of different handwriting of the same electronically signed signature, characterized by, The method comprises the following steps: S1: extracting coarse-grained features of an input electronic signature image through a ResNet101 feature extractor; wherein the ResNet101 contains five stages of feature outputs of the electronic signature; S2: dividing the electronic signature feature maps of the last three stages after S1 into image blocks of equal size, and assigning a position number to each image block to determine the position of each image block corresponding to the original feature map; S3: inputting the three-stage feature maps with position numbers in S2 into an extendable attention mechanism; wherein the extendable attention mechanism extends to the surrounding according to the current small block feature map, and determines the relevance of the extended image block and the current block according to the similarity calculation; S4: fusing the high-order output of the extendable attention in S3 with the adjacent previous extendable attention layer through upsampling to obtain the final output of the extendable attention of each stage, and then splicing the outputs of the three stages to input into a general classification network model; S5: obtaining discriminative electronic signature fine-grained features through the extendable attention mechanism, and distinguishing different electronic signature handwriting according to the above fine-grained features by a general classifier; The extendable attention module continuously extends outward to learn features with coexistence relationship according to the current handwriting features, effectively locating the discriminative differences between the natural handwriting and the mechanically copied handwriting; The extendable attention mechanism is inspired by the self-attention mechanism, which continuously extends outward according to the current small block and finds the most relevant small blocks, which continue to expand outward until the entire feature map is traversed. Let a small block in the electronic signature feature map x be denoted as ; by calculating the offset weights in the scalable layer, the result is obtained with The N adjacent expandable elements, each belonging to another pixel in x, are then used to calculate the similarity of the matrices, resulting in the following values: Correlation with N elements; specifically, let A linear layer used for computation The weights; let and Let x represent the two scalable linear layers respectively. Then, x is input into each of these two scalable linear layers, resulting in... The key can be expanded with weights and the value can be expanded with weights; The weights are learned through dot product operations to obtain the relevance to the key's expandable weights; finally, softmax is used to normalize the similarity scores to aggregate the value's expandable points; therefore, Scalable attention computation can be represented as: ， In order to ensure the generalization ability of the model, each pixel point is expanded more than once, that is, each pixel point is calculated for a plurality of times of offset and correlation, and the finally obtained plurality of attention results are aggregated to form a new fine-grained electronic signature feature image matrix; assuming that each pixel point is expanded for M times of coexistence features, the multiple extendable attention of the input feature map x can be represented as: ； wherein denotes a weight vector.

2. The method for fine-grained classification of different handwriting of the same electronically signed signature as claimed in claim 1, wherein, Step S1 specifically comprises: The original electronic signature image is input into the feature extractor ResNet101, wherein the ResNet101 is divided into 5 different residual layers, and the output of each subsequent layer is closer to the semantic information and the size of the output feature map is gradually reduced.

3. The method for fine-grained classification of different handwriting of the same electronically signed document as claimed in claim 1, wherein, Step S2 specifically comprises: The output of the last three layers is selected as the subsequent input, since the last three layers are closer to the semantic features of the electronic signature, and the size of the feature map is smaller, more valuable electronic signature features can be obtained with lower computational cost; given an input feature map x ∈ R C×H×W , H and W represent the height and width of the electronic signature feature image respectively; x is divided into HxW small blocks, each small block represents a pixel point of the image, which represents a value in the image matrix in the calculation of the convolutional neural network, and then position encoding is added to each small block to record the position of the original feature map corresponding to each small block.

4. The method for fine-grained classification of different handwriting of the same electronically signed document as recited in claim 1, wherein, Step S3 further comprises: The increase of the scalable attention layer can enhance the discriminant analysis of the handwriting features, and improve the robustness of target classification in the data scene; the output of the previous layer is taken as the input of the next layer, and with the increase of the number of layers, the scalable ability of the model is gradually stimulated, and the electronic signature handwriting with coexistence relationship is completely covered. L is the number of layers, and the multi-head multi-layer scalable attention may be represented as: ； ； ； When L is equal to 7, the classification accuracy of the handwriting is the highest; In addition, in the feature map of the electronic signature in a batch during training, the extendable attention can indirectly learn the extendable elements in other feature maps from a certain sample position in the map, and the obtained elements can associate the handwriting features from different images and with coexistence relationship, and also help the model to distinguish other handwriting that does not have coexistence relationship.

5. The method for fine-grained classification of different handwriting of the same electronically signed document as recited in claim 1, wherein, Step S4 specifically comprises: The high-order output of the scalable attention is fused with the adjacent previous scalable attention layer through upsampling as the final output of the scalable attention of each stage; specifically, the fifth layer obtains a feature map of the same size as the fourth layer through upsampling, and the feature map after upsampling is fused with the fourth layer feature map in the form of matrix addition, and the fused feature map is taken as the output of the fourth layer, and the fused image obtained in the fourth layer is upsampled and fused with the feature map of the third layer in the same way, and the fusion result is taken as the output of the third layer, and the fifth layer is directly output without any fusion operation, and then the outputs of the three stages are spliced together and input into a general classification network model; The outputs of the attention layers at different scales are fused with each other through upsampling, thereby helping to correct the previous stage in the high-order stage and helping the model to restore the full spatial resolution of the output; finally, the fusion stage is aggregated, the scalable feature information is enriched, and the performance of classifying the fine-grained features of the electronic signature handwriting is improved; on this basis, the model with the corrected scalable parameters can infer the handwriting coexistence features at the full spatial resolution.

6. The method for fine-grained classification of different handwriting of the same electronically signed document as recited in claim 1, wherein, The step S5 specifically includes: In combination with the general classifier and the scalable attention mechanism for fine-grained feature extraction, the fine-grained feature vector with the handwriting coexistence relationship is input into the subsequent general classifier, and the classification effect of the electronic signature handwriting is adjusted through the cross-entropy loss function.

7. A system for fine-grained classification of different handwriting of the same electronically signed document, applying the method for fine-grained classification of different handwriting of the same electronically signed document according to any one of claims 1 to 6, characterized in that, The fine-grained classification system for different handwriting of the same electronic signature includes: A feature extraction module is configured to extract coarse-grained features of an input electronic signature image through a ResNet101 feature extractor; wherein the ResNet101 includes five stages of electronic signature feature outputs; An image segmentation module is connected with the feature extraction module and is configured to segment the electronic signature feature maps of the last three stages into image blocks of equal size, and assign a position number to each image block to determine the position of the original feature map corresponding to each image block; An attention module is connected with the image segmentation module and is configured to input the three-stage feature maps with position numbers into a scalable attention mechanism respectively; wherein the scalable attention mechanism expands the current small block feature map to the four directions, and determines the relevance of the expanded image block and the current block according to the similarity calculation; A feature fusion module is connected with the attention module and is configured to fuse the high-order output of the scalable attention of S3 with the adjacent previous scalable attention layer through upsampling as the final output of the scalable attention of each stage, and then splice the outputs of the three stages together and input into a classification module; A classification module is connected with the feature fusion module, and a classifier distinguishes different handwriting of the electronic signature according to the fused fine-grained features.

8. A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to make the processor execute the steps of the fine-grained classification method for different handwriting of the same electronic signature according to any one of claims 1-6.

9. An information data processing terminal for implementing the fine-grained classification system for different handwriting of the same electronic signature as claimed in claim 7.

Images