A handwritten signature authentication method and device based on cross-modal feature fusion

Abstract

The application relates to the technical field of deep learning, and provides a handwriting signature authentication method and device based on cross-modal feature fusion. The method comprises the following steps: jointly performing signature authentication by using a signature image and a corresponding signature sequence; the authentication method comprises the following steps: extracting image features from the signature image by using a pre-trained ViT model, and extracting time sequence features from the signature sequence by using a pre-trained Unet model; performing feature fusion on the image features and the time sequence features to obtain fused features; and mapping the fused features into a plurality of low-dimensional feature vectors, and performing signature authentication by using the low-dimensional feature vectors. According to the application, the signature sequence is introduced into the authentication, the image features are extracted from the signature image, the time sequence features are provided from the signature sequence, and the feature fusion is performed, so that the accuracy of the signature authentication can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a handwritten signature authentication method and device based on cross-modal feature fusion. BACKGROUND

[0002] In the field of artificial intelligence and deep learning, handwritten signature authentication technology is an important research topic, which involves extracting features from handwritten signatures and verifying the security of personal identity. Handwritten signatures are considered as a biometric method due to their unique personal characteristics. Each person's signature has its unique stroke, pressure and rhythm pattern, which is difficult to copy. Therefore, handwritten signature authentication has a wide application prospect in the fields of finance, law and security.

[0003] With the development of deep learning technology, handwritten signature authentication methods based on deep neural networks have made significant progress. These methods usually rely on large data sets and complex network structures to extract the spatiotemporal features and dynamic features of signatures. However, traditional deep learning methods often focus on the extraction of single modal features, ignoring the importance of cross-modal feature fusion, resulting in low accuracy of handwritten signature authentication.

[0004] Therefore, it is urgent to overcome the defects of the prior art in the technical field. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a handwritten signature authentication method and device based on cross-modal feature fusion to improve the accuracy of handwritten signature authentication.

[0006] The present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a handwritten signature authentication method based on cross-modal feature fusion, which uses signature images and corresponding signature sequences together for signature authentication. The authentication method includes:

[0008] extracting image features from the signature images using a pre-trained ViT model and extracting time sequence features from the signature sequences using a pre-trained Unet model;

[0009] performing feature fusion on the image features and the time sequence features to obtain fusion features;

[0010] mapping the fusion features into a plurality of low-dimensional feature vectors and using the low-dimensional feature vectors for signature authentication.

[0011] Preferably, the extraction of image features from the signature images using the pre-trained ViT model specifically includes:

[0012] Preprocessing the signature image to obtain a standard image;

[0013] The standard image is divided into a plurality of patches by a convolution layer, and the feature of each patch is unfolded to obtain a patch feature. Then, a position code is added to each patch feature to obtain an input embedding vector. The position code is obtained according to the spatial position information of the patch feature.

[0014] The input embedding vector is input into an N-layer transformer encoder layer to obtain the image feature.

[0015]

[0016] Preferably, the preprocessing of the signature image to obtain a standard image specifically includes:

[0017] The handwritten signature image is converted into a grayscale image, and then the grayscale image is converted into a black-and-white image to remove grayscale noise.

[0018] The black-and-white image is converted into an RGB image, the RGB image is cropped, the cropped sub-image is scaled into a square picture of a preset size, and the square picture is normalized to obtain the standard image, that is, the i-th pixel in the standard image. wherein, I μ and I σ are a preset mean and a preset standard deviation, I μ =(0.5, 0.5, 0.5), I σ =(0.5, 0.5, 0.5), I represents a pixel value in the RGB image, I min represents the minimum value of the pixel in the RGB image, and I max represents the maximum value of the pixel in the RGB image.

[0019] Preferably, the feature fusion of the image feature and the time sequence feature to obtain a fusion feature specifically includes:

[0020] The image feature and the time sequence feature are projected into the same dimension feature space through a projection layer to obtain a same dimension image feature and a same dimension time sequence feature.

[0021] The same dimension image feature and the same dimension time sequence feature are fused to obtain a fusion feature.

[0022] Preferably, the projection of the image feature and the time sequence feature into the same dimension feature space through the projection layer to obtain the same dimension image feature and the same dimension time sequence feature specifically includes:

[0023] The image feature Fimg and the timing feature F temp Projecting to the same dimensional feature space through the projection layer to obtain the same dimensional image feature F proj_img img (F img ) and the same dimensional timing feature F proj_temp temp (F temp ).

[0024] Preferably, the same dimensional image feature and the same dimensional timing feature are fused to obtain a fusion feature, specifically including:

[0025] The same dimensional image feature and the same dimensional timing feature are cross-attention fused with each other to obtain a fused image feature F att_img proj_img proj_temp proj_temp and a fused timing feature F att_temp proj_temp proj_img proj_img ; wherein CrossAtt() represents a cross-attention fusion operation;

[0026] The fused image feature and the fused timing feature are input into an MLP network to predict the weight of the fused image feature and the weight of the fused timing feature using a Sigmoid function, specifically: the weight of the fused image feature is α = Sigmoid(MLP(F proj_img )), and the weight of the fused timing feature is β = Sigmoid(MLP(F proj_temp ));

[0027] The same dimensional image feature and the fused image feature are weighted and added using a residual connection to obtain a reference image feature F fused_img proj_img att_img ; the same dimensional timing feature and the fused timing feature are weighted and added using a residual connection to obtain a reference timing feature F fused_temp proj_temp att_temp ;

[0028] The reference image feature and the reference timing feature are spliced to obtain the fusion feature F final fused_img fused_temp .

[0029] Preferably, the low-dimensional feature vector is used for signature authentication, specifically including: ​​​​​​​​​​​​​​

[0030] calculating a similarity between the low-dimensional feature vector corresponding to the original signature and a low-dimensional feature vector corresponding to the signature to be verified;

[0031] determining whether the signature to be verified and the original signature are written by the same signer according to the similarity.

[0032] Preferably, the pre-trained Unet model is obtained by pre-training, and specifically includes:

[0033] adding random noise and random black windows to the electronic signature sequence to obtain a signature sequence sample for training, wherein the electronic signature sequence includes one or more of horizontal coordinates, vertical coordinates, pressure, time stamps, and press-up identifiers;

[0034] extracting a time function from the signature sequence sample, wherein the time function includes one or more of a horizontal coordinate difference function with respect to time, a vertical coordinate difference function with respect to time, a speed function, a cosine function of a tangent angle, a sine function of a tangent angle, a speed change difference function, a tangent angle difference function, a logarithmic curvature radius function, a speed change amplitude function, an acceleration function, and a pressure function;

[0035] generating a training set using the signature sequence sample and the time function, training a Unet model using the training set, and obtaining the pre-trained Unet model.

[0036] In a second aspect, the present application further provides a handwritten signature authentication device based on cross-modal feature fusion, which is used to implement the handwritten signature authentication method based on cross-modal feature fusion in the first aspect, and the device comprises:

[0037] at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the handwritten signature authentication method based on cross-modal feature fusion in the first aspect.

[0038] In a third aspect, the present application further provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method in the first aspect.

[0039] In a fourth aspect, a chip is provided, which comprises a processor and an interface, and is used to call and run a computer program stored in a memory to execute the method in the first aspect.

[0040] In a fifth aspect, a computer program product comprising instructions which, when executed on a computer or processor, cause the computer or processor to carry out the method of the first aspect is provided.

[0041] The present application can effectively improve the accuracy of signature authentication by incorporating signature sequences into authentication, extracting image features from signature images, providing timing features from signature sequences, and performing feature fusion. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0043] Figure 1 is a flowchart of a first handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0044] Figure 2 is a flowchart of a second handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0045] Figure 3 is a flowchart of a third handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0046] Figure 4 is a flowchart of a fourth handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0047] Figure 5 is a flowchart of a fifth handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0048] Figure 6 is a flowchart of a sixth handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0049] Figure 7 is a flowchart of a seventh handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0050] Figure 8 is a flowchart of an eighth handwritten signature authentication method based on cross-modal feature fusion provided by the embodiments of the present application;

[0051] Figure 9is a schematic diagram of a handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present application;

[0052] Figure 10 is a schematic diagram of another handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present application;

[0053] Figure 11 is a schematic diagram of still another handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present application;

[0054] Figure 12 is a schematic diagram of a handwritten signature authentication device based on cross-modal feature fusion provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0056] Unless otherwise required by the context, the term "comprising" is to be interpreted as open and inclusive, i.e. "including but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" are intended to mean that a particular feature, structure, material or characteristic included in at least one embodiment or example of the present disclosure. The illustrative representation of the above terms does not necessarily mean the same embodiment or example. In addition, the specific features, structures, materials or characteristics described can be included in any one or more embodiments or examples in any appropriate manner, i.e. although they are carried in the embodiments or examples of the above terms due to the order of appearance and location, they are not limited to the combination of one embodiment or example.

[0057] In the description of the present application, the terms "first", "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included in one or more features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, in the description, the same type of nouns can also be described as two independent individuals by adding "A", "B" at the end, in which case the features limited by "A", "B" are only used for the purpose of distinguishing the same type of individual description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0058] In the description of the present application, the expression "A and / or B" (wherein A and B represent specific features) is used in the form of a specific feature, and the corresponding expression includes the following three combinations: only A, only B, and the combination of A and B.

[0059] In the present application, "about", "approximately" or "approximately" includes the stated value and the average value within the acceptable deviation range of the specific value, wherein the acceptable deviation range is determined by the person skilled in the art considering the measurement being discussed and the error related to the measurement of the specific quantity (i.e. the limitation of the measurement system).

[0060] In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as there is no conflict.

[0061] Example 1:

[0062] The signature authentication method in the prior art often focuses on the extraction of single modal features, ignoring the importance of cross-modal feature fusion, resulting in low accuracy of handwritten signature authentication. In order to solve this problem, the embodiment 1 of the present application provides a handwritten signature authentication method based on cross-modal feature fusion, which uses signature image and corresponding signature sequence to jointly perform signature authentication, as shown in Figure 1 The authentication method comprises:

[0063] In step 201, the pre-trained ViT model is used to extract image features from the signature image, and the pre-trained Unet model is used to extract time sequence features from the signature sequence; wherein the pre-trained ViT model and the pre-trained Unet model both refer to the available model which has been trained using the corresponding data set. The step 201 uses a model based on Unet architecture to comprehensively extract the time sequence features of the handwritten signature data. This method makes full use of the encoder-decoder structure of Unet, and in the process of layer-by-layer down-sampling and up-sampling, it captures the rich dynamic information in the handwritten signature in multiple scales.

[0064] In step 202, the image features and the time sequence features are fused to obtain the fusion features.

[0065] In step 203, the fused features are mapped into a plurality of low-dimensional feature vectors, and signature authentication is performed using the low-dimensional feature vectors. The mapping of the fused features into a plurality of low-dimensional feature vectors is achieved by mapping the fused features through a linear layer, and the main goal of this process is to reduce the complexity of high-dimensional features while maintaining their representation ability. Low-dimensional mapping can help reduce computational complexity and memory occupation of vector storage. Through linear mapping, high-dimensional features that may contain redundant information are compressed into more compact low-dimensional representations, thereby significantly reducing computational overhead and storage requirements. This not only improves the efficiency of subsequent similarity calculation, but also makes the model more lightweight and efficient when processing large-scale data.

[0066] The embodiment can effectively improve the accuracy of signature authentication by incorporating signature sequences into authentication, extracting image features from signature images, providing time sequence features from signature sequences, and performing feature fusion.

[0067] In an actual application scenario, the image features are extracted from the signature image using a pre-trained ViT model, as shown in Figure 2 Specifically, the method comprises the following steps:

[0068] In step 301, the signature image is preprocessed to obtain a standard image.

[0069] In step 302, the standard image is divided into a plurality of sub-regions (referred to as patches) through a convolutional layer, and each patch is subjected to feature expansion to obtain a patch feature. Then, a position code is added to each patch feature to obtain an input embedding vector. The position code is obtained according to the spatial position information of the patch feature.

[0070] In step 303, the input embedding vector is input into an N-layer transformer encoder layer to obtain the image feature. Due to the self-attention mechanism of the visual transformer, the model is more likely to learn global contour, inter-stroke position relationship and other information, thereby complementing the dynamic signature sequence features. The image feature can be understood as a static signature, which contains spatial position information such as contour and layout of the signature strokes.

[0071] The preprocessing of the signature image to obtain a standard image comprises the following steps: Figure 3 Specifically, the method comprises the following steps:

[0072] In step 401, the handwritten signature image is converted into a grayscale image, and then the grayscale image is converted into a black-and-white image to remove grayscale noise.

[0073] In step 402, the black and white image is converted into an RGB image, the RGB image is cropped, the cropped sub-image is scaled into a square picture of a preset size, the square picture is normalized, and the standard image is obtained, that is, the i-th pixel in the standard image wherein I μ and I σ is a preset mean value and a preset standard deviation, I μ =(0.5, 0.5, 0.5), I σ =(0.5, 0.5, 0.5), I represents a pixel value in the RGB image, I min represents a minimum value of the pixel in the RGB image, I max represents a maximum value of the pixel in the RGB image. The cropping of the RGB image is randomly implemented, specifically including: the obtained RGB image is randomly cropped, specifically, a sub-image with a length and a width of a certain fixed proportion r of the RGB image is randomly taken on the RGB image, and the proportion r is a random number between 0.5 and 1. The preset size is obtained by empirical analysis of a person skilled in the art. The embodiment compresses the image value to [0, 1] and then normalizes the value to a distribution with a mean value of 0 and a standard deviation of 1.

[0074] In an alternative embodiment, the image features and the time sequence features are fused to obtain fused features, as shown in Figure 4 , specifically including:

[0075] In step 501, the image features and the time sequence features are projected to the same dimensional feature space through a projection layer to obtain the same dimensional image features and the same dimensional time sequence features.

[0076] In step 502, the same dimensional image features and the same dimensional time sequence features are fused to obtain fused features. The projection of the image features and the time sequence features to the same dimensional feature space through the projection layer to obtain the same dimensional image features and the same dimensional time sequence features specifically includes: the image features F img and the time sequence features F temp are projected to the same dimensional feature space through the projection layer to obtain the same dimensional image features F proj_img =P img (F img ) and the same dimensional time sequence features F proj_temp =P temp (F temp ).

[0077] The fusion of the same dimensional image features and the same dimensional time sequence features to obtain the fused features is as shown in​​Figure 5 As shown, specifically comprising:

[0078] In step 601, the same dimension image features and the same dimension time sequence features are cross-attention fused respectively to obtain fused image features F att_img = CrossAtt(F proj_img , F proj_temp , F proj_temp ) and fused time sequence features F att_temp = CrossAtt(F proj_temp , F proj_img , F proj_img ); wherein CrossAtt() represents a cross-attention fusion operation.

[0079] In step 602, the fused image features and the fused time sequence features are passed through an MLP network to predict the weight of the fused image features and the weight of the fused time sequence features using a Sigmoid function, specifically: the weight of the fused image features a = Sigmoid(MLP(F proj_img )), and the weight of the fused time sequence features b = Sigmoid(MLP(F proj_temp )); that is, a lightweight gating mechanism is used to learn the weights of the two.

[0080] In step 603, the same dimension image features and the fused image features are weighted and added using residual connection to obtain the reference image features F fused_img = F proj_img + a x F att_img ; the same dimension time sequence features and the fused time sequence features are weighted and added using residual connection to obtain the reference time sequence features F fused_temp = F proj_temp + b x F att_temp ; wherein the weights used in the weighted addition are the weight of the fused image features a and the weight of the fused time sequence features b in step 602.

[0081] In step 604, the reference image features and the reference time sequence features are spliced to obtain more rich and comprehensive fused features, that is, the fused features F final = [F fused_img ; F fused_temp ].

[0082] In actual use, the low-dimensional feature vector is used for signature authentication, as shown, specifically comprising: Figure 6

[0083] In step 701, the similarity between the low-dimensional feature vector corresponding to the original signature and the low-dimensional feature vector corresponding to the signature to be verified is calculated. ​

[0084] In step 702, according to the similarity, it is judged whether the to-be-verified signature and the original signature are written by the same signer. That is, the above steps 201-203 are performed on the signature image and the signature sequence of the original signature to obtain the low-dimensional feature vector of the original signature; the above steps 201-203 are performed on the signature image and the signature sequence of the to-be-verified signature to obtain the low-dimensional feature vector of the to-be-verified signature, and then the similarity between the low-dimensional feature vectors of the original signature and the to-be-verified signature is calculated. In actual use, when the similarity is greater than a preset value, it is considered that the to-be-verified signature and the original signature are written by the same signer. Alternatively, when there are multiple to-be-verified signatures, the to-be-verified signature with the highest similarity from the to-be-verified signatures with a similarity greater than the preset value is selected as the signature written by the same signer as the original signature, and the preset value is obtained by demand analysis by a person skilled in the art.

[0085] In some embodiments, the pre-trained Unet model is pre-trained, as shown in Figure 7 Specifically, the method comprises the following steps:

[0086] In step 801, random noise and random black window are added to an electronic signature sequence to obtain a signature sequence sample for training; wherein the electronic signature sequence comprises one or more of horizontal coordinates, vertical coordinates, pressure, time stamps and press and release identifiers; and the electronic signature sequence is a signature sequence of a plurality of signatures pre-acquired. The random noise added is mainly random noise obeying uniform distribution added to the horizontal coordinates and the vertical coordinates to form perturbation of handwriting.

[0087] In step 802, a time function is extracted from the signature sequence sample; wherein the time function comprises one or more of a horizontal coordinate difference function with respect to time, a vertical coordinate difference function with respect to time, a speed function, a cosine function of a tangent angle, a sine function of a tangent angle, a speed change difference function, a tangent angle difference function, a logarithmic curvature radius function, a speed change amplitude function, an acceleration function and a pressure function.

[0088] In step 803, a training set is generated using the signature sequence sample and the time function, the Unet model is trained using the training set, and the pre-trained Unet model is obtained.

[0089] Embodiment 2:

[0090] Based on the method described in embodiment 1, the specific application scenario is combined, and the implementation process in the specific scenario of the application is described by the technical description in the related scenario.

[0091] The handwriting signature authentication method provided by the embodiment is based on cross-modal feature fusion, and specifically comprises the following steps. Figure 8 and Figure 9 as shown in the drawings, specifically comprising:

[0092] In step 901, the signature sequence is preprocessed. Random noise and random black window are added to the signature sequence during training, and a time function is extracted for the model, specifically:

[0093] Given an electronic signature sequence containing horizontal coordinates, vertical coordinates, pressure, time stamp and press-up identification information; add random noise conforming to uniform distribution to the horizontal and vertical coordinates to form a disturbance of the handwriting, and then extract 12 time functions, including: horizontal coordinate difference function with respect to time, vertical coordinate difference function with respect to time, velocity function, cosine function of tangent angle, sine function of tangent angle, velocity change difference function, tangent angle difference function, logarithmic curvature radius function, velocity change amplitude function, acceleration function, pressure function.

[0094] In step 902, the time sequence feature is extracted. The Unet model is used to extract the time sequence feature of the signature, and the rich dynamic information contained in the handwritten signature is used in multiple scales to further extract the time sequence feature with recognition degree, specifically:

[0095] The Unet architecture-based model is used to comprehensively extract the time sequence feature of the handwritten signature data. This method fully utilizes the encoder-decoder structure of Unet, and in the process of layer-by-layer downsampling and upsampling, the rich dynamic information in the handwritten signature is captured in multiple scales.

[0096] In step 903, the image is preprocessed. The offline signature is preprocessed, including image preprocessing operations such as black and white, random cropping, size adjustment, normalization, etc., specifically:

[0097] First, the handwritten signature images of different formats are converted into grayscale images, and then the grayscale images are converted into black and white images to remove grayscale noise. Then, the black and white images are converted into RGB images for subsequent processing. The obtained RGB image is randomly cropped, specifically, a sub-image with a length and width of a certain fixed proportion r of the RGB image is randomly taken on the RGB image, and the proportion r is a random number between 0.5 and 1. Then, the cropped sub-image is scaled to a 336x336 square picture. Finally, the image is normalized, which can be expressed by the formula: first,

[0098]

[0099] where I represents the RGB image, I min and I maxThe minimum and maximum values of the image pixels are represented, and this step compresses the image values to [0, 1]. Next, the values are normalized to a distribution with a mean of 0 and a standard deviation of 1, that is, I = I / max(I).

[0100]

[0101] where I μ and I σ are the mean and standard deviation of the image, respectively, (0.5, 0.5, 0.5) and (0.5, 0.5, 0.5).

[0102] In step 904, image feature extraction. The static signature contains spatial position information such as the contour and layout of the signature stroke. The pre-trained ViT is used to extract image features in the signature image, specifically:

[0103] The static signature image features are extracted using the visual transformer. The preprocessed image is divided into patches by a convolution layer, and the feature of each patch is unfolded. The spatial position information of each patch is encoded by adding position encoding to obtain the input embedding vector. The input embedding vector is input into the N-layer transformer encoder layer. Due to the self-attention mechanism of the visual transformer, the model can more easily learn the global contour, inter-stroke position relationship and other information, which complements the dynamic signature sequence features. The process is as follows:

[0104]

[0105] In step 905, feature fusion is performed. First, the image features obtained in step 902 and the time sequence features obtained in step 904 are projected into the same dimensional feature space by a projection layer, and mutual cross-attention fusion is performed on the two. Second, a lightweight gating mechanism is used to learn the weights of the two. Specifically, the projected image and time sequence features are input into an MLP network, and the weights of the image features and the time sequence features are predicted by a Sigmoid function. Then, the image and time sequence features after cross-attention are added to the original image and time sequence features respectively by using residual connection to obtain the fused image and time sequence features. Finally, the fused image and time sequence features are spliced to obtain more comprehensive and comprehensive fused features, as shown in Figure 10 Specifically:

[0106] First, the image features F img obtained in step 902 and the time sequence features F temp obtained in step 904 are projected into the same dimensional feature space by a projection layer, and mutual cross-attention fusion CrossAtt(·) is performed on the two. The process is as follows:

[0107] F proj_img = P img (F img )

[0108] F proj_temp = P temp (F temp )

[0109] F att_img = CrossAtt(F proj_img , F proj_temp , F proj_temp )

[0110] F att_temp = CrossAtt(F proj_temp , F proj_img , F proj_img )

[0111] Secondly, a lightweight gating mechanism is used to learn the adjustment of the weights of the two, specifically, the projected image and the time sequence feature are passed through the MLP network, and the weight of the image feature and the time sequence feature is predicted by the Sigmoid function. The process is as follows:

[0112] a = Sigmoid(MLP(F proj_img ))

[0113] b = Sigmoid(MLP(F proj_temp ))

[0114] Then the residual connection is used to respectively add the image F att_img and the time sequence feature F att_temp after the cross attention to the original image F proj_img and the time sequence feature F proj_temp respectively to obtain the fused image F fused_img and the time sequence feature F fused_temp . Finally, the fused image and the time sequence feature are spliced to obtain more rich and comprehensive fusion features. The process is as follows:

[0115] F fused_img = F proj_img + a * F att_img

[0116] F fused_temp = F proj_temp + b * F att_temp

[0117] F final = [F fused_img ; F fused_temp ]

[0118] In step 906, low-dimensional feature vector mapping. The fused features are mapped through a linear layer into multiple low-dimensional feature vectors. Low-dimensional mapping can help reduce computational complexity and memory occupation of vector storage. Specifically:

[0119] After feature fusion is completed, the fused features are sent to a linear layer for mapping to generate multiple low-dimensional feature vectors. The main goal of this process is to reduce the complexity of high-dimensional features while maintaining their representation ability. Through linear mapping, high-dimensional features that may contain redundant information are compressed into more compact low-dimensional representations, significantly reducing computational overhead and storage requirements. This not only improves the efficiency of subsequent similarity calculation, but also makes the model more lightweight and efficient when processing large-scale data.

[0120] In step 907, similarity calculation. Use the multi-vector late interaction mechanism to calculate the similarity between the original handwriting feature vector and the handwriting feature vector to be verified, and judge the authenticity of the signer's identity by setting a threshold to achieve accurate signature verification. Specifically:

[0121] In step 907, the low-dimensional mapped feature vectors are used for final similarity calculation. Specifically, through the multi-vector interaction mechanism, the feature vectors of the original handwriting (i.e. the low-dimensional feature vectors of the original signature) and the feature vectors of the handwriting to be verified (i.e. the low-dimensional feature vectors of the signature to be verified) are compared one by one to evaluate their similarity. This mechanism can fully exploit the fine-grained similarity between feature vectors to ensure the accuracy of the calculation result. By setting an appropriate similarity threshold, the model can effectively distinguish the authenticity of the signature, i.e. determine whether the handwriting to be verified is written by the same signer. This high-precision similarity calculation provides a reliable basis for signature verification, ensuring the stability and accuracy of the system in practical applications. Wherein, the similarity calculation is calculated by summing the similarity between each feature value in the original handwriting feature vector and each feature value in the handwriting feature vector to be verified, as shown in Figure 11 .

[0122] Example 3:

[0123] As shown in Figure 12 , it is an architecture schematic diagram of the handwriting signature authentication device based on cross-modal feature fusion of the embodiment of the application. The handwriting signature authentication device based on cross-modal feature fusion of the embodiment of the application includes one or more processors 21 and a memory 22. Wherein, Figure 12 take a processor 21 as an example.

[0124] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 12 take the example of connection through the bus.

[0125] The memory 22, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the handwriting signature authentication method based on cross-modal feature fusion in Embodiment 1. The processor 21 executes the handwriting signature authentication method based on cross-modal feature fusion by running the non-volatile software programs and instructions stored in the memory 22.

[0126] The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 can optionally include a memory disposed remotely relative to the processor 21, and these remote memories can be connected to the processor 21 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, perform the handwriting signature authentication method based on cross-modal feature fusion in Embodiment 1 described above.

[0128] It is worth noting that the information interaction, execution process, etc. between the modules and units in the above-mentioned apparatus and system are based on the same concept as the processing method embodiments of the present application, and the specific content can be referred to the description in the method embodiments of the present application, which will not be described here.

[0129] Those of ordinary skill in the art can understand that all or part of the steps in the various embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0130] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A handwritten signature authentication method based on cross-modal feature fusion, characterized in that: Signature authentication is performed using a signature image and the corresponding signature sequence. Authentication methods include: Use the pre-trained ViT model to extract image features from the signature image, and use the pre-trained Unet model to extract temporal features from the signature sequence; Perform feature fusion on image features and time series features to obtain fused features; Mapping the fused features into multiple low-dimensional feature vectors, and using the low-dimensional feature vectors to perform signature authentication; The feature fusion of the image features and the time series features to obtain the fused features specifically includes: Projecting the image features and the time series features into the same-dimensional feature space through a projection layer to obtain image features and time series features of the same dimension; Performing feature fusion on the same-dimensional image features and the same-dimensional time series features to obtain fused features; The projecting the image features and the time series features into the same-dimensional feature space through the projection layer to obtain the same-dimensional image features and the same-dimensional time series features specifically includes: The image features and time series characteristics Projected to the same dimensional feature space through the projection layer, the same dimensional image features are obtained and time series features of the same dimension ; The feature fusion of the same-dimensional image features and the same-dimensional time series features to obtain fused features specifically includes: The same-dimensional image features and the same-dimensional time series features are cross-attended and fused to obtain the fused image features. and fused temporal features ;in, represents the mutual cross-attention fusion operation; The fused image features and fused time series features are passed through the MLP network, and the Sigmoid function is used to predict the weights of the fused image features and the weights of the fused time series features. Specifically: the weights of the fused image features , the weight of the fused temporal features ; Use residual connection to perform weighted addition of the same-dimensional image features and the fused image features to obtain the baseline image features ; Use residual connection to perform weighted addition of the same-dimensional time series features and the fused time series features to obtain the baseline time series features ; The reference image feature and the reference time series feature are spliced ​​together to obtain the fusion feature .

2. The handwritten signature authentication method based on cross-modal feature fusion according to claim 1 is characterized in that: The method of extracting image features from the signature image using the pre-trained ViT model specifically includes: Preprocessing the signature image to obtain a standard image; The standard image is divided into multiple patches through a layer of convolution, each patch is feature expanded to obtain patch features, and then a position code is added to each patch feature to obtain an input embedding vector; wherein the position code is obtained based on the spatial position information of the patch feature; The input embedding vector is passed through layer transformer encoder layer to obtain the image features; that is, .

3. The handwritten signature authentication method based on cross-modal feature fusion according to claim 2 is characterized in that: The preprocessing of the signature image to obtain a standard image specifically includes: Convert the handwritten signature image into a grayscale image, and then convert the grayscale image into a black and white image to remove grayscale noise; Convert the black and white image into an RGB image, crop the RGB image, scale the cropped sub-image into a square image of a preset size, normalize the square image, and obtain the standard image, i.e., the i-th pixel in the standard image. ;in, and is the preset mean and the preset standard deviation, , , , Represents the pixel value in the RGB image, Represents the minimum value of the pixel in the RGB image, Represents the maximum value of a pixel in an RGB image.

4. The handwritten signature authentication method based on cross-modal feature fusion according to claim 1 is characterized in that: Using the low-dimensional feature vector to perform signature authentication specifically includes: Calculate the similarity between the low-dimensional feature vector corresponding to the original signature and the low-dimensional feature vector corresponding to the signature to be verified; Based on the similarity, it is determined whether the signature to be verified and the original signature are written by the same signer.

5. The handwritten signature authentication method based on cross-modal feature fusion according to claim 1 is characterized in that: The pre-trained Unet model is pre-trained, specifically including: Adding random noise and random black windows to an electronic signature sequence to obtain a signature sequence sample for training; wherein the electronic signature sequence includes one or more data of abscissa, ordinate, pressure, timestamp, and press / lift mark; Extracting a time function from the signature sequence sample; wherein the time function includes one or more of a differential function of the abscissa with respect to time, a differential function of the ordinate with respect to time, a velocity function, a cosine function of a tangent angle, a sin function of a tangent angle, a velocity change differential function, a tangent angle differential function, a logarithmic curvature radius function, a velocity change amplitude function, an acceleration function, and a pressure function; A training set is generated using the signature sequence sample and the time function, and a Unet model is trained using the training set to obtain the pre-trained Unet model.

6. A non-volatile computer storage medium, characterized in that The computer storage medium stores computer-executable instructions, which are executed by one or more processors to complete the handwritten signature authentication method based on cross-modal feature fusion as described in any one of claims 1 to 5.

7. A handwritten signature authentication device based on cross-modal feature fusion, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the handwritten signature authentication method based on cross-modal feature fusion as described in any one of claims 1-5.

Images