Handwritten signature authentication method and device based on cross-modal feature fusion

Abstract

The invention relates to the technical field of deep learning, and provides a handwritten signature authentication method and device based on cross-modal feature fusion. The method comprises the following steps: jointly carrying out signature authentication by using a signature image and a corresponding signature sequence, and the authentication method comprises the following steps: extracting image features from the signature image by using a pre-trained ViT model, and extracting time sequence features from the signature sequence by using a pre-trained Unet model; carrying out feature fusion on the image features and the time sequence features to obtain fusion features; and mapping the fusion feature into a plurality of low-dimensional feature vectors, and performing signature authentication by using the low-dimensional feature vectors. According to the method, the signature sequence is included in authentication, the image features are extracted from the signature image, the time sequence features are provided from the signature sequence, and feature fusion is carried out, so that the accuracy of signature authentication can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a handwritten signature authentication method and device based on cross-modal feature fusion. Background Art

[0002] In the field of artificial intelligence and deep learning, handwritten signature authentication technology is an important research topic, which involves the security of extracting features from handwritten signatures and using them to verify personal identity. Handwritten signatures are regarded as a form of biometrics because of their unique personal characteristics. Each person's signature has its own unique stroke, pressure and rhythm pattern, which are difficult to copy. Therefore, handwritten signature authentication has broad application prospects in finance, law, security and other fields.

[0003] With the development of deep learning technology, handwritten signature authentication methods based on deep neural networks have made significant progress. These methods usually rely on large data sets and complex network structures to extract the spatiotemporal and dynamic features of signatures. However, traditional deep learning methods often focus on the extraction of single modality features and ignore the importance of cross-modal feature fusion, resulting in low accuracy of handwritten signature authentication.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a handwritten signature authentication method and device based on cross-modal feature fusion to improve the accuracy of handwritten signature authentication.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a handwritten signature authentication method based on cross-modal feature fusion, using a signature image and a corresponding signature sequence to perform signature authentication, the authentication method comprising:

[0008] Use the pre-trained ViT model to extract image features from the signature image, and use the pre-trained Unet model to extract temporal features from the signature sequence;

[0009] Perform feature fusion on image features and time series features to obtain fusion features;

[0010] The fused features are mapped into a plurality of low-dimensional feature vectors, and the low-dimensional feature vectors are used for signature authentication.

[0011] Preferably, the extracting image features from the signature image using the pre-trained ViT model specifically includes:

[0012] Preprocessing the signature image to obtain a standard image;

[0013] The standard image is divided into multiple patches through a layer of convolution, each patch is feature expanded to obtain patch features, and then a position code is added to each patch feature to obtain an input embedding vector; wherein the position code is obtained according to the spatial position information of the patch feature;

[0014] The input embedding vector is passed through N layers of transformer encoder layers to obtain the image features; that is,

[0015]

[0016] Preferably, the preprocessing of the signature image to obtain a standard image specifically includes:

[0017] Convert the handwritten signature image into a grayscale image, and then convert the grayscale image into a black and white image to remove grayscale noise;

[0018] Convert the black-and-white image into an RGB image, crop the RGB image, scale the cropped sub-image into a square image of a preset size, normalize the square image, and obtain the standard image, that is, the i-th pixel in the standard image Among them, I μ and I σ is the preset mean and preset standard deviation, I μ =(0.5,0.5,0.5), I σ =(0.5,0.5,0.5), I represents the pixel value in the RGB image, I min Represents the minimum value of the pixel in the RGB image, I max Represents the maximum value of a pixel in an RGB image.

[0019] Preferably, the step of fusing the image features and the time series features to obtain the fused features specifically includes:

[0020] Projecting the image features and the time series features to the same-dimensional feature space through a projection layer to obtain image features and time series features of the same dimension;

[0021] The image features with the same dimension and the time series features with the same dimension are fused to obtain fused features.

[0022] Preferably, projecting the image features and the time series features to the same-dimensional feature space through a projection layer to obtain the same-dimensional image features and the same-dimensional time series features specifically includes:

[0023] The image feature Fimg and the time series feature F temp Projected to the same dimensional feature space through the projection layer, the same dimensional image feature F is obtained proj_img =P img (F img ) and the same-dimensional time series feature F proj_temp =P temp (F temp ).

[0024] Preferably, the step of fusing the image features and the time series features of the same dimension to obtain the fused features specifically includes:

[0025] The same-dimensional image features and the same-dimensional time series features are cross-attended and fused to obtain the fused image features F att_img =CrossAtt(F proj_img ,F proj_temp ,F proj_temp ) and the fused temporal features F att_temp =CrossAtt(F proj_temp ,F proj_img ,F proj_img ); where CrossAtt() represents the cross-attention fusion operation;

[0026] The fused image features and the fused time series features are passed through the MLP network, and the weight of the fused image features and the weight of the fused time series features are predicted using the Sigmoid function. Specifically: the weight of the fused image features α = Sigmoid (MLP (F proj_img )), the weight of the fused temporal features β = Sigmoid(MLP(F proj_temp ));

[0027] The residual connection is used to perform weighted addition of the image features of the same dimension and the fused image features to obtain the reference image features F fused_img =F proj_img +α×F att_img ; Use residual connection to weight the same dimension time series features and fused time series features to obtain the benchmark time series features F fused_temp =F proj_temp +β×F att_temp ;

[0028] The reference image feature and the reference time series feature are spliced ​​to obtain the fusion feature F final =[F fused_img ; F fused_temp ].

[0029] Preferably, using the low-dimensional feature vector to perform signature authentication specifically includes:

[0030] Calculate the similarity between the low-dimensional feature vector corresponding to the original signature and the low-dimensional feature vector corresponding to the signature to be verified;

[0031] According to the similarity, it is determined whether the signature to be verified and the original signature are written by the same signer.

[0032] Preferably, the pre-trained Unet model is pre-trained, specifically including:

[0033] Adding random noise and random black windows to the electronic signature sequence to obtain a signature sequence sample for training; wherein the electronic signature sequence includes one or more data of abscissa, ordinate, pressure, timestamp, and press-and-lift mark;

[0034] Extracting a time function from the signature sequence sample; wherein the time function includes one or more of a differential function of the horizontal coordinate to time, a differential function of the vertical coordinate to time, a velocity function, a cosine function of a tangent angle, a sin function of a tangent angle, a velocity change differential function, a tangent angle differential function, a logarithmic curvature radius function, a velocity change amplitude function, an acceleration function, and a pressure function;

[0035] A training set is generated using the signature sequence sample and the time function, and a Unet model is trained using the training set to obtain the pre-trained Unet model.

[0036] In a second aspect, the present invention further provides a handwritten signature authentication device based on cross-modal feature fusion, which is used to implement the handwritten signature authentication method based on cross-modal feature fusion described in the first aspect, and the device includes:

[0037] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the handwritten signature authentication method based on cross-modal feature fusion as described in the first aspect.

[0038] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0039] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect.

[0040] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0041] The present invention can effectively improve the accuracy of signature authentication by incorporating the signature sequence into authentication, extracting image features from the signature image, providing time series features from the signature sequence, and performing feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 It is a flowchart of a first handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0044] Figure 2 4 is a flow chart of a second handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0045] Figure 3 4 is a flow chart of a third method for handwritten signature authentication based on cross-modal feature fusion provided by an embodiment of the present invention;

[0046] Figure 4 4 is a flowchart of a fourth handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0047] Figure 5 4 is a flowchart of a fifth handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0048] Figure 6 4 is a flow chart of a sixth handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0049] Figure 7 is a flowchart of a seventh handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0050] Figure 8 4 is a flow chart of an eighth handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0051] Fig. 9is a schematic diagram of a handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0052] Fig.10 is a schematic diagram of another handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0053] Fig.11 is a schematic diagram of another handwritten signature authentication method based on cross-modal feature fusion provided by an embodiment of the present invention;

[0054] Fig.12 It is a schematic diagram of the architecture of a handwriting signature authentication device based on cross-modal feature fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0057] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0058] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0059] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0060] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0061] Embodiment 1:

[0062] The signature authentication methods in the prior art often focus on the extraction of single modal features and ignore the importance of cross-modal feature fusion, resulting in low accuracy of handwritten signature authentication. In order to solve this problem, Embodiment 1 of the present invention provides a handwritten signature authentication method based on cross-modal feature fusion, which uses a signature image and a corresponding signature sequence to perform signature authentication together. Figure 1 As shown, the authentication methods include:

[0063] In step 201, the pre-trained ViT model is used to extract image features from the signature image, and the pre-trained Unet model is used to extract temporal features from the signature sequence; wherein the pre-trained ViT model and the pre-trained Unet model are both usable models that have been trained using corresponding data sets. The step 201 uses a model based on the Unet architecture to comprehensively extract the temporal features of the handwritten signature data. This method makes full use of the encoder-decoder structure of Unet, and captures the rich dynamic information in the handwritten signature at multiple scales during the layer-by-layer downsampling and upsampling process.

[0064] In step 202, the image features and the time series features are fused to obtain fused features.

[0065] In step 203, the fused features are mapped into multiple low-dimensional feature vectors, and the low-dimensional feature vectors are used for signature authentication. The mapping of the fused features into multiple low-dimensional feature vectors is achieved by mapping the fused features through a linear layer. The main goal of this process is to reduce the complexity of high-dimensional features while maintaining their representation capabilities. Low-dimensional mapping can help reduce computational complexity and memory usage of vector storage. Through linear mapping, high-dimensional features that may contain redundant information are compressed into more compact low-dimensional representations, thereby significantly reducing computational overhead and storage requirements. This not only improves the efficiency of subsequent similarity calculations, but also makes the model more lightweight and efficient when processing large-scale data.

[0066] This embodiment incorporates the signature sequence into the authentication, extracts image features from the signature image, provides time series features from the signature sequence, and performs feature fusion, thereby effectively improving the accuracy of signature authentication.

[0067] In a practical application scenario, the pre-trained ViT model is used to extract image features from the signature image, such as Figure 2 As shown, specifically including:

[0068] In step 301, the signature image is preprocessed to obtain a standard image.

[0069] In step 302, the standard image is divided into multiple sub-regions (referred to as patches) through a layer of convolution, each patch is feature expanded to obtain patch features, and then a position code is added to each patch feature to obtain an input embedding vector; wherein the position code is obtained based on the spatial position information of the patch feature.

[0070] In step 303, the input embedding vector is passed through N layers of transformer encoder layers to obtain the image features; that is, Due to the self-attention mechanism of the visual transformer, the model can more easily learn information such as global contours and the positional relationship between strokes, which complements the dynamic signature sequence features. The image features can be understood as static signatures, including spatial position information such as the contour and layout of the signature strokes.

[0071] Wherein, the signature image is preprocessed to obtain a standard image, such as Figure 3 As shown, specifically including:

[0072] In step 401, the handwritten signature image is converted into a grayscale image, and then the grayscale image is converted into a black and white image to remove grayscale noise.

[0073] In step 402, the black-and-white image is converted into an RGB image, the RGB image is cropped, the cropped sub-image is scaled into a square image of a preset size, and the square image is normalized to obtain the standard image, that is, the i-th pixel in the standard image Among them, I μ and I σ is the preset mean and preset standard deviation, I μ =(0.5,0.5,0.5), I σ =(0.5,0.5,0.5), I represents the pixel value in the RGB image, I min Represents the minimum value of the pixel in the RGB image, I max Represents the maximum value of the pixels in the RGB image. The cropping of the RGB image is implemented randomly, specifically including: randomly cropping the obtained RGB image, specifically, randomly taking a sub-image with a length and width of a fixed ratio r of the RGB image on the RGB image, and the ratio r is a random number between 0.5 and 1. The preset size is obtained by those skilled in the art based on empirical analysis. This embodiment is Compress the image value to between [0,1], and then Normalize the values ​​to have a mean of 0 and a standard deviation of 1.

[0074] In an optional implementation, the image features and the time series features are fused to obtain fused features, such as Figure 4 As shown, specifically including:

[0075] In step 501, the image features and the time series features are projected to the same-dimensional feature space through a projection layer to obtain image features and time series features of the same dimension.

[0076] In step 502, the image features and the time series features of the same dimension are fused to obtain fused features. The image features and the time series features are projected to the feature space of the same dimension through a projection layer to obtain the image features and the time series features of the same dimension, specifically including: img and the time series feature F temp Projected to the same dimensional feature space through the projection layer, the same dimensional image feature F is obtained proj_img =P img (F img ) and the same-dimensional time series feature F proj_temp =P temp (F temp ).

[0077] The image features of the same dimension and the time series features of the same dimension are fused to obtain fused features, such as Figure 5 As shown, specifically including:

[0078] In step 601, the image features and time series features of the same dimension are cross-attended and fused to obtain the fused image features F att_img =CrossAtt(F proj_img ,F proj_temp ,F proj_temp ) and the fused temporal features F att_temp =CrossAtt9F proj_temp ,F proj_img ,F proj_img ), where CrossAtt() represents the cross-attention fusion operation.

[0079] In step 602, the fused image features and the fused time series features are passed through an MLP network, and a Sigmoid function is used to predict the weights of the fused image features and the weights of the fused time series features. Specifically, the weight of the fused image features α = Sigmoid (MLP (F proj_img )), the weight of the fused temporal features β = Sigmoid(MLP(F proj_temp )); that is, a lightweight gating mechanism can be used to learn and adjust the weights of the two.

[0080] In step 603, the same-dimensional image features and the fused image features are weightedly added using residual connections to obtain the reference image features F fused_img =F proj_img +α×F att_img ; Use residual connection to weight the time series features of the same dimension and the fused time series features to obtain the benchmark time series features F fused_temp =F proj_temp +β×F att_temp ; Among them, the weights used for weighted addition are the weight α of the fused image features and the weight β of the fused temporal features in step 602.

[0081] In step 604, the reference image features and the reference time series features are spliced ​​to obtain a richer and more comprehensive fusion feature, that is, the fusion feature F final =[F fused_img ; F fused_temp ].

[0082] In actual use, the low-dimensional feature vector is used for signature authentication, such as Figure 6 As shown, specifically including:

[0083] In step 701, the similarity between the low-dimensional feature vector corresponding to the original signature and the low-dimensional feature vector corresponding to the signature to be verified is calculated.

[0084] In step 702, based on the similarity, it is determined whether the signature to be verified and the original signature are written by the same signer. That is, the above steps 201 to 203 are performed for the signature image and signature sequence of the original signature to obtain the low-dimensional feature vector of the original signature; the above steps 201 to 203 are performed for the signature image and signature sequence of the signature to be verified to obtain the low-dimensional feature vector of the signature to be verified, and then the similarity between the low-dimensional feature vector of the original signature and the low-dimensional feature vector of the signature to be verified is calculated. In actual use, when the similarity is greater than a preset value, it can be considered that the signature to be verified and the original signature are written by the same signer, or when there are multiple signatures to be verified, the signature to be verified with the highest similarity is selected from the signatures to be verified with a similarity greater than a preset value as the signature written by the same signer as the original signature. The preset value is obtained by those skilled in the art according to demand analysis.

[0085] In some embodiments, the pre-trained Unet model is pre-trained, such as Figure 7 As shown, specifically including:

[0086] In step 801, random noise and random black window are added to the electronic signature sequence to obtain a signature sequence sample for training; wherein the electronic signature sequence includes one or more data of abscissa, ordinate, pressure, timestamp and press-up mark; the electronic signature sequence is a signature sequence of multiple signatures collected in advance. The addition of random noise mainly adds random noise that obeys uniform distribution to the abscissa and ordinate to form handwriting disturbance.

[0087] In step 802, a time function is extracted from the signature sequence sample; wherein the time function includes one or more of a differential function of the horizontal coordinate with respect to time, a differential function of the vertical coordinate with respect to time, a velocity function, a cosine function of a tangent angle, a sin function of a tangent angle, a velocity change differential function, a tangent angle differential function, a logarithmic curvature radius function, a velocity change amplitude function, an acceleration function, and a pressure function.

[0088] In step 803, a training set is generated using the signature sequence sample and the time function, and a Unet model is trained using the training set to obtain the pre-trained Unet model.

[0089] Embodiment 2:

[0090] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0091] This embodiment provides a handwritten signature authentication method based on cross-modal feature fusion, such as Figure 8 and Fig. 9 As shown, specifically including:

[0092] In step 901, the signature sequence is preprocessed. During training, random noise and random black windows are added to the signature sequence, and the time function is extracted for model use. Specifically:

[0093] Given an electronic signature sequence containing information such as horizontal coordinate, vertical coordinate, pressure, timestamp, and press-up and lift-down marks; random noise that obeys a uniform distribution is added to its horizontal and vertical coordinates to form a disturbance of the handwriting, and then 12 time functions are extracted, including: the differential function of the horizontal coordinate with respect to time, the differential function of the vertical coordinate with respect to time, the speed function, the cosine function of the tangent angle, the sin function of the tangent angle, the speed change differential function, the tangent angle differential function, the logarithmic curvature radius function, the speed change amplitude function, the acceleration function, and the pressure function.

[0094] In step 902, the temporal features are extracted. The Unet model is used to extract the temporal features of the signature, and the rich dynamic information contained in the handwritten signature is used in multiple scales to extract the recognizable temporal features. Specifically:

[0095] A model based on the Unet architecture is used to comprehensively extract the temporal features of handwritten signature data. This method makes full use of the encoder-decoder structure of Unet to capture the rich dynamic information in handwritten signatures at multiple scales during the process of downsampling and upsampling layer by layer.

[0096] In step 903, image preprocessing is performed on the offline signature, including black and white conversion, random cropping, resizing, normalization and other image preprocessing operations. Specifically:

[0097] First, convert the handwritten signature images of different formats into grayscale images, and then convert the grayscale images into black and white images. Through this process, the grayscale noise is removed, and then the black and white images are converted into RGB images for subsequent processing. The obtained RGB image is randomly cropped. Specifically, a sub-image with a length and width of a fixed ratio r of the RGB image is randomly selected on the RGB image. The ratio r is a random number between 0.5 and 1, and then the cropped sub-image is scaled to a 336×336 square image. Finally, the image is normalized, expressed by the formula, first:

[0098]

[0099] Where I represents the RGB image, I min and I maxRepresents the minimum and maximum values ​​of the image pixels. This step compresses the image values ​​to between [0,1]. Next, the values ​​are normalized to a distribution with a mean of 0 and a standard deviation of 1, that is:

[0100]

[0101] Among them, I μ and I σ are the mean and standard deviation of the image, (0.5, 0.5, 0.5) and (0.5, 0.5, 0.5) respectively.

[0102] In step 904, image features are extracted. The static signature contains spatial position information such as the outline and layout of the signature strokes. The pre-trained ViT is used to extract image features in the signature image. Specifically:

[0103] Use visual transformer to extract static signature image features. The preprocessed image is divided into patches through a layer of convolution, and each patch feature is expanded. Then position encoding is added to encode the spatial position information of each patch to obtain the input embedding vector. The input embedding vector is passed through N layers of transformer encoder layers. Due to the self-attention mechanism of the visual transformer, it is easier for the model to learn information such as global contours and positional relationships between strokes, which complements the dynamic signature sequence features. This process is as follows:

[0104]

[0105] In step 905, feature fusion is performed. First, the image features obtained in step 902 and the time series features obtained in step 904 are projected into the same dimensional feature space through a projection layer, and the two are cross-attention fused. Secondly, a lightweight gating mechanism is used to learn and adjust the weights of the two. Specifically, the projected image and time series features are passed through the MLP network, and the weights of the image features and time series features are predicted using the Sigmoid function. Then, the image and time series features after cross-attention are added to the original image and time series features using residual connections to obtain the fused image and time series features. Finally, the fused image and time series features are spliced ​​to obtain richer and more comprehensive fusion features, such as Fig.10 As shown, specifically:

[0106] First, the image feature F obtained in step 902 img and the time series feature F obtained in step 904 temp Through the projection layer, the two are projected into the same dimensional feature space, and the two are cross-attended and fused CrossAtt(·). This process is as follows:

[0107] F proj_img =P img (F img )

[0108] F proj_temp =P temp (F temp )

[0109] F att_img =CrossAtt(F proj_img ,F proj_temp ,F proj_temp )

[0110] F att_temp =CrossAtt(F proj_temp ,F proj_img ,F proj_img )

[0111] Secondly, a lightweight gating mechanism is used to learn and adjust the weights of the two. Specifically, the projected image and time series features are passed through the MLP network, and the Sigmoid function is used to predict the weights of the image features and time series features. This process is as follows:

[0112] α=Sigmoid(MLP(F proj_img ))

[0113] β=Sigmoid(MLP(F proj_temp ))

[0114] Then, the residual connection is used to convert the cross-attention image F att_img and the time series feature F att_temp Compared with the original image F proj_img and the time series feature F proj_temp Add them separately to get the fused image F fused_img and the time series feature F fused_temp Finally, the fused image and time series features are spliced ​​to obtain richer and more comprehensive fusion features. This process is as follows:

[0115] F fused_img =F proj_img +α*F att_img

[0116] F fused_temp =F proj_temp +β*F att_temp

[0117] F final =[F fused_img ; F fused_temp ]

[0118] In step 906, low-dimensional feature vector mapping is performed. The fused features are mapped into multiple low-dimensional feature vectors through a linear layer. Low-dimensional mapping can help reduce computational complexity and memory usage of vector storage. Specifically:

[0119] After feature fusion is completed, the fused features are sent to the linear layer for mapping to generate multiple low-dimensional feature vectors. The main goal of this process is to reduce the complexity of high-dimensional features while maintaining their representational capabilities. Through linear mapping, high-dimensional features that may originally contain redundant information are compressed into a more compact low-dimensional representation, thereby significantly reducing computational overhead and storage requirements. This not only improves the efficiency of subsequent similarity calculations, but also makes the model more lightweight and efficient when processing large-scale data.

[0120] In step 907, similarity calculation is performed. Using the multi-vector latent interaction mechanism, the original handwriting feature vector and the handwriting feature vector to be verified are similarly calculated, and the authenticity of the signer's identity is determined by setting a threshold, thereby achieving accurate signature verification. Specifically:

[0121] The feature vector that has undergone low-dimensional mapping in step 907 is used for the final similarity calculation. Specifically, through a multi-vector interaction mechanism, the feature vector of the original handwriting (i.e., the low-dimensional feature vector of the original signature) is compared one by one with the feature vector of the handwriting to be verified (i.e., the low-dimensional feature vector of the signature to be verified) to evaluate the degree of similarity between them. This mechanism can fully tap the fine-grained similarity between feature vectors to ensure the accuracy of the calculation results. By setting an appropriate similarity threshold, the model can effectively distinguish the authenticity of the signature, that is, determine whether the handwriting to be verified is written by the same signer. This high-precision similarity calculation provides a reliable basis for judgment for signature verification, ensuring the stability and accuracy of the system in practical applications. Among them, the similarity calculation is obtained by summing the similarities between each eigenvalue in the original handwriting feature vector and each eigenvalue in the handwriting feature vector to be verified, such as Fig.11 shown.

[0122] Embodiment 3:

[0123] like Fig.12 , is a schematic diagram of the architecture of a handwritten signature authentication device based on cross-modal feature fusion according to an embodiment of the present invention. The handwritten signature authentication device based on cross-modal feature fusion according to this embodiment includes one or more processors 21 and a memory 22. Fig.12 A processor 21 is taken as an example.

[0124] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0125] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the handwritten signature authentication method based on cross-modal feature fusion in Example 1. The processor 21 executes the handwritten signature authentication method based on cross-modal feature fusion by running the non-volatile software programs and instructions stored in the memory 22.

[0126] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the handwritten signature authentication method based on cross-modal feature fusion in the above-mentioned embodiment 1 is executed.

[0128] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0129] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A handwritten signature authentication method based on cross-modal feature fusion, characterized in that: The signature image and the corresponding signature sequence are used together for signature authentication. The authentication methods include: Use the pre-trained ViT model to extract image features from the signature image, and use the pre-trained Unet model to extract temporal features from the signature sequence; Perform feature fusion on image features and time series features to obtain fusion features; The fused features are mapped into a plurality of low-dimensional feature vectors, and the low-dimensional feature vectors are used for signature authentication.

2. The handwritten signature authentication method based on cross-modal feature fusion according to claim 1 is characterized in that: The method of extracting image features from the signature image using the pre-trained ViT model specifically includes: Preprocessing the signature image to obtain a standard image; The standard image is divided into multiple patches through a layer of convolution, each patch is feature expanded to obtain patch features, and then a position code is added to each patch feature to obtain an input embedding vector; wherein the position code is obtained according to the spatial position information of the patch feature; The input embedding vector is passed through N layers of transformer encoder layers to obtain the image features; that is, 3. The handwritten signature authentication method based on cross-modal feature fusion according to claim 2 is characterized in that: The preprocessing of the signature image to obtain a standard image specifically includes: Convert the handwritten signature image into a grayscale image, and then convert the grayscale image into a black and white image to remove grayscale noise; Convert the black-and-white image into an RGB image, crop the RGB image, scale the cropped sub-image into a square image of a preset size, normalize the square image, and obtain the standard image, that is, the i-th pixel in the standard image Among them, I μ and I σ is the preset mean and preset standard deviation, I μ =(0.5,0.5,0.5), I σ =(0.5,0.5,0.5), I represents the pixel value in the RGB image, I min Represents the minimum value of the pixel in the RGB image, I max Represents the maximum value of a pixel in an RGB image.

4. The handwritten signature authentication method based on cross-modal feature fusion according to claim 1 is characterized in that: The feature fusion of the image feature and the time series feature to obtain the fusion feature specifically includes: Projecting the image features and the time series features to the same-dimensional feature space through a projection layer to obtain image features and time series features of the same dimension; The image features with the same dimension and the time series features with the same dimension are fused to obtain fused features.

5. The handwritten signature authentication method based on cross-modal feature fusion according to claim 4 is characterized in that: The step of projecting the image features and the time series features to a feature space of the same dimension through a projection layer to obtain image features and time series features of the same dimension specifically includes: The image feature F img and the time series feature F temp Projected to the same dimensional feature space through the projection layer, the same dimensional image feature F is obtained proj_img =P img (F img ) and the same-dimensional time series feature F proj_temp =P temp (F temp ).

6. The handwritten signature authentication method based on cross-modal feature fusion according to claim 4 is characterized in that: The step of fusing the image features with the same dimension and the time series features with the same dimension to obtain the fused features specifically includes: The same-dimensional image features and the same-dimensional time series features are cross-attended and fused to obtain the fused image features F att_img =CrossAtt(F proj_img ,F proj_temp ,F proj_temp ) and the fused temporal features F att_temp =CrossAtt(F proj_temp ,F proj_img ,F proj_img ); where CrossAtt() represents the cross-attention fusion operation; The fused image features and the fused time series features are passed through the MLP network, and the weight of the fused image features and the weight of the fused time series features are predicted using the Sigmoid function. Specifically: the weight of the fused image features α = Sigmoid (MLP (F proj_img )), the weight of the fused temporal features β = Sigmoid(MLP(F proj_temp )); The residual connection is used to perform weighted addition of the image features of the same dimension and the fused image features to obtain the reference image features F fused_img =F proj_img +α×F att_img ; Use residual connection to weight the time series features of the same dimension and the fused time series features to obtain the benchmark time series features F fused_temp =F proj_temp +β×F att_temp ; The reference image feature and the reference time series feature are spliced ​​to obtain the fusion feature F final =[F fused_img ; F fused_temp ].

7. The handwritten signature authentication method based on cross-modal feature fusion according to claim 1 is characterized in that: Using the low-dimensional feature vector to perform signature authentication specifically includes: Calculate the similarity between the low-dimensional feature vector corresponding to the original signature and the low-dimensional feature vector corresponding to the signature to be verified; According to the similarity, it is determined whether the signature to be verified and the original signature are written by the same signer.

8. The handwritten signature authentication method based on cross-modal feature fusion according to claim 1 is characterized in that: The pre-trained Unet model is pre-trained, specifically including: Adding random noise and random black windows to the electronic signature sequence to obtain a signature sequence sample for training; wherein the electronic signature sequence includes one or more data of abscissa, ordinate, pressure, timestamp, and press-and-lift mark; Extracting a time function from the signature sequence sample; wherein the time function includes one or more of a differential function of the horizontal coordinate to time, a differential function of the vertical coordinate to time, a velocity function, a cosine function of a tangent angle, a sin function of a tangent angle, a velocity change differential function, a tangent angle differential function, a logarithmic curvature radius function, a velocity change amplitude function, an acceleration function, and a pressure function; A training set is generated using the signature sequence sample and the time function, and a Unet model is trained using the training set to obtain the pre-trained Unet model.

9. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the handwritten signature authentication method based on cross-modal feature fusion as described in any one of claims 1-8.

10. A handwritten signature authentication device based on cross-modal feature fusion, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the handwriting signature authentication method based on cross-modal feature fusion as described in any one of claims 1-8.

Images