A multi-modal identity verification method based on an attention mechanism

Abstract

The present application relates to a kind of multi-modal identity authentication methods based on attention mechanism, comprising: S1, multi-modal data is collected, including face image data, fingerprint image data and voiceprint audio data;S2, for each modality, corresponding first feature extraction model is constructed, second feature extraction model is obtained by cutting first feature extraction model, first feature extraction model is used to carry out knowledge distillation training to second feature extraction model, and third feature extraction model is obtained;S3, the feature extracted by all third feature extraction model is weighted summation using multi-modal feature fusion model based on attention mechanism, and fusion feature is obtained;S4, the similarity between the fusion feature of the identity to be verified and the corresponding identity in the fusion feature of pre-recorded database is calculated, when similarity is greater than prediction decision threshold, then identity verification is passed, otherwise identity verification is not passed.Compared with prior art, the present application has the advantages of efficient feature extraction and high reliability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of identity verification, and in particular to a multi-modal identity verification method based on an attention mechanism. BACKGROUND

[0002] In today's rapidly developing digital and information age, identity verification technology plays a crucial role in ensuring security, improving efficiency and enhancing user experience. Identity verification technology is widely used in various scenarios such as access control systems, financial transactions, smart device unlocking and public security, greatly facilitating people's life and work.

[0003] Traditional identity verification methods are mostly single-modal identity verification, mainly relying on a single biometric feature such as face, voiceprint or fingerprint. Face verification is a technology that verifies identity by capturing and analyzing feature points in facial images, such as the positions and distances of eyes, nose and mouth. It is widely used in security monitoring, mobile phone unlocking and social media, etc. Through image processing and pattern recognition technology, face verification systems can quickly verify individual identity. Specific technologies include VGGFace, ArcFace, etc. Voiceprint verification is a technology that verifies individual identity by analyzing features in speech signals such as pitch, timbre and speech rate. It is commonly used in telephone banking, voice assistants and smart home devices, etc. Voiceprint verification systems usually rely on signal processing and machine learning techniques, and can complete identity verification in a relatively short time. Specific technologies include TDNN, ECAPA-TDNN, etc. Fingerprint verification is a technology that confirms individual identity by scanning and comparing detailed features in fingerprint images such as ridge and valley distribution and intersection points. Fingerprint verification is widely used in access control systems, attendance punching and mobile payment, etc. Fingerprint verification technology has high uniqueness and stability, and precise verification is achieved through hardware sensors and image processing algorithms.

[0004] Although single-modal verification technology performs well in certain specific scenarios, it has certain limitations when faced with complex and variable real-world environments. For example, face verification may fail in cases of light changes, occlusions and expression changes; voice verification may be affected by background noise, microphone quality and user voice changes; and fingerprint verification may be affected by skin conditions, fingerprint wear and tear, etc. The single feature dependency of these methods limits their robustness and accuracy, and cannot fully meet the needs of modern applications for high security and high reliability.

[0005] To overcome the limitations of single-modal authentication methods, multi-modal authentication methods have emerged. Multi-modal authentication, by utilizing multiple biometric data such as face, voice, and fingerprint, can provide higher accuracy and robustness in a wider range of application scenarios. For example, a common multi-modal authentication system can use both face and voice verification to improve the reliability and accuracy of the overall verification. Specifically, face verification provides visual information, voice verification provides auditory information, and fingerprint verification provides tactile information. These different modalities of data have complementary information and characteristics, and through reasonable fusion, the performance of identity authentication can be significantly improved. However, the fusion and effective use of multi-modal data is still a challenging problem. The characteristics of different modalities of data differ greatly, and direct fusion may result in information loss or increased redundant information, affecting the verification effect. Therefore, researchers are constantly exploring new methods to optimize the fusion of multi-modal data to achieve a more efficient and robust identity authentication system.

[0006] Previous research provides a valuable foundation for identity authentication, but there are still limitations:

[0007] (1) The robustness of single-modal algorithms is relatively poor;

[0008] (2) Current multi-modal fusion methods only use two modalities, and the fusion method is relatively simple;

[0009] (3) The overall architecture is large after multi-modal fusion, and the efficiency is low. SUMMARY

[0010] The purpose of the present application is to overcome the defects of the prior art and provide a multi-modal identity authentication method based on attention mechanism with high robustness and high recognition efficiency.

[0011] The purpose of the present application can be achieved by the following technical solutions:

[0012] A multi-modal identity authentication method based on attention mechanism, comprising:

[0013] S1, collecting multi-modal data, including face image data, fingerprint image data and voiceprint audio data;

[0014] S2, for each modality, constructing a corresponding first feature extraction model, cutting the first feature extraction model to obtain a second feature extraction model, and using the first feature extraction model to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model;

[0015] S3, using a multi-modal feature fusion model based on attention mechanism to perform weighted summation on the features extracted by all third feature extraction models to obtain fused features;

[0016] S4, calculate the similarity between the fusion features of the identity to be verified and the corresponding identity in the pre-recorded database, and when the similarity is greater than the prediction decision threshold, the identity verification is passed, otherwise the identity verification is not passed.

[0017] Preferably, for the face modal, the first feature extraction model is a convolutional neural network, including five consecutive convolutional blocks, and the output of the last convolutional block is connected to a multi-layer perceptron to complete face feature extraction; wherein, the first and second convolutional blocks each include two convolutional layers and a pooling layer, and the third to fifth convolutional blocks each include three convolutional layers and a pooling layer.

[0018] Preferably, for the face modal, the first feature extraction model is cropped to obtain the second feature extraction model, specifically: on the basis of the first feature extraction model, only the first three convolutional blocks and the last multi-layer perceptron are used to obtain the second feature extraction model.

[0019] Preferably, for the fingerprint modal, the first feature extraction model is a convolutional neural network, including five consecutive convolutional blocks, and the output of the last convolutional block is connected to a multi-layer perceptron to complete fingerprint feature extraction; wherein, each convolutional block includes multiple convolutional layers, and a residual connection is arranged before the convolutional layers to extract local features; the first convolutional block only includes one convolutional layer and one pooling layer, and the subsequent four convolutional blocks each include a standard residual block, each residual block includes two convolutional layers, and the number of residual blocks in the subsequent four convolutional blocks is 3, 4, 6 and 3 respectively.

[0020] Preferably, for the fingerprint modal, the first feature extraction model is cropped to obtain the second feature extraction model, specifically: on the basis of the first feature extraction model, the number of residual blocks in the subsequent four convolutional blocks is reduced to 2, and the remaining structure remains unchanged.

[0021] Preferably, for the voiceprint modal, the first feature extraction model is a time delay neural network, specifically:

[0022] The input voiceprint audio data is first passed through a one-dimensional convolutional layer, then sequentially passed through five SE-Res2Block layers for feature extraction, then passed through a one-dimensional convolutional layer and an attention statistical pooling layer for down-sampling, and then passed through a fully connected layer to complete feature extraction; wherein, each SE-Res2Block layer introduces an SE mechanism to apply attention to the channels through compression and excitation, and then through a Res2Net module to divide the output of the convolutional layer into multiple small groups and perform multi-scale feature fusion between the multiple small groups; after passing through the SE-Res2Block, the outputs of all SE-Res2Blocks are spliced and input into the next one-dimensional convolutional layer.

[0023] Preferably, for the voiceprint modal, the first feature extraction model is pruned to obtain the second feature extraction model, specifically:

[0024] On the basis of the first feature extraction model, only three SE-Res2Block layers are used to obtain the second feature extraction model.

[0025] Preferably, the first feature extraction model is used to perform knowledge distillation training on the second feature extraction model to obtain the third feature extraction model, and the knowledge distillation training process is specifically:

[0026] Forward propagation is performed on the second feature extraction model of each modal to obtain the probability score of each class

[0027] The multi-class cross-entropy loss function L of the second feature extraction model is calculated hard and the soft label distillation loss function L soft , the expression is:

[0028]

[0029] In the formula: y represents the label; represents the probability calculated by the second feature extraction model; P t ,P s respectively represent the probability distribution of the first feature extraction model and the second feature extraction model; T is the temperature used to control the smoothness of the softmax function; z t ,z s respectively represent the classification score of the first feature extraction model and the second feature extraction model;

[0030] The total loss function Loss is calculated:

[0031] Loss=L hard +μ·L soft (4)

[0032] In the formula: μ represents a coefficient used to control the size of the soft label loss function;

[0033] Based on the total loss function, the second feature extraction model of each modal is trained by gradient descent to obtain the third feature extraction model.

[0034] Preferably, the multi-modal feature fusion model based on the attention mechanism is used to perform weighted summation on the features extracted by all third feature extraction models to obtain the fused features, specifically including:

[0035] The first face feature e face , the first fingerprint feature e fingerprintand the first voiceprint feature e voice ;

[0036] The features of each modality are processed through the fully connected layer to obtain the second facial features. Second fingerprint feature and the second voiceprint feature

[0037] The first facial feature e face , the first voiceprint feature e voice and the first fingerprint feature e fingerprint The overall features are obtained by splicing, and the attention score a of each modality is calculated through the attention mechanism face ,a voice ,a fingerprint ;

[0038] Use the attention score a of each modality face ,a voice ,a fingerprint The second characteristic of each mode Weighted summation to obtain the fusion feature e feat .

[0039] Preferably, the similarity is cosine similarity.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention uses knowledge distillation technology to allow a more complex first feature extraction model to assist in training a more streamlined second feature extraction model, thereby refining knowledge of single-modal feature extraction technology and making single-modal feature extraction more efficient.

[0042] (2) The present invention fuses the features of the three modalities of face, voiceprint and fingerprint through the attention mechanism, which can achieve more robust identity authentication and higher reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the method of the present invention;

[0044] Figure 2 This is a schematic diagram of the first feature extraction model structure for faces;

[0045] Figure 3 This is a schematic diagram of the first feature extraction model structure for fingerprints;

[0046] Figure 4 This is a schematic diagram of the structure of the first feature extraction model for voiceprint;

[0047] Figure 5 Schematic diagram of feature fusion;

[0048] Figure 6 A feature fusion model architecture based on an attention mechanism. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0050] EMBODIMENT

[0051] As shown in the figure, the present embodiment provides a multi-modal identity verification method based on an attention mechanism, comprising: Figure 1

[0052] S1, collecting multi-modal data, including face image data, fingerprint image data and voiceprint audio data;

[0053] S2, for each modality, constructing a corresponding first feature extraction model, obtaining a second feature extraction model by pruning the first feature extraction model, and performing knowledge distillation training on the second feature extraction model by using the first feature extraction model to obtain a third feature extraction model;

[0054] S3, using a multi-modal feature fusion model based on an attention mechanism to perform weighted summation on the features extracted by all the third feature extraction models to obtain a fusion feature;

[0055] S4, calculating the similarity between the fusion feature of the identity to be verified and the fusion feature of the corresponding identity pre-recorded in the database, and when the similarity is greater than a prediction decision threshold, the identity verification is passed, otherwise the identity verification is not passed.

[0056] Next, the method of the present embodiment will be described in detail.

[0057] S1, collecting multi-modal data, including face image data, fingerprint image data and voiceprint audio data.

[0058] In the present embodiment, the face, voiceprint and fingerprint data are obtained by sensors such as cameras and recorders. Among them, the face and fingerprint exist in the form of images, and the voiceprint exists in the form of audio.

[0059] S2, for each modality, constructing a corresponding first feature extraction model, obtaining a second feature extraction model by pruning the first feature extraction model, and performing knowledge distillation training on the second feature extraction model by using the first feature extraction model to obtain a third feature extraction model.

[0060] As​Figure 2 As shown in the figure, for the face modality, the first feature extraction model is a convolutional neural network, consisting of five consecutive convolutional blocks. The output of the last convolutional block is connected to a multi-layer perceptron to complete facial feature extraction. The first and second convolutional blocks each include two convolutional layers and one pooling layer, while the third to fifth convolutional blocks each include three convolutional layers and one pooling layer. Here, "conv" represents a two-dimensional convolutional layer, "Pool" represents a pooling layer, and "FC" represents a fully connected layer.

[0061] For the face modality, the first feature extraction model is pruned to obtain the second feature extraction model. Specifically, the first feature extraction model is pruned and only the first three convolution blocks and the last multi-layer perceptron are used to obtain the second feature extraction model.

[0062] For fingerprint modality, the first feature extraction model is a convolutional neural network, such as Figure 3 As shown, it includes five consecutive convolutional blocks, and the output of the last convolutional block is connected to a multi-layer perceptron to complete fingerprint feature extraction. Each data block contains multiple convolutional layers, and residual connections are set before the convolutional layers to prevent gradient vanishing and extract local features. In this embodiment, the first convolutional block only contains a 7*7 convolutional layer and a 3*3 pooling layer. The subsequent four convolutional blocks all contain standard residual blocks. Each residual block contains two 3*3 convolutional layers. The number of residual blocks in the subsequent four convolutional blocks is 3, 4, 6, and 3, respectively.

[0063] For the fingerprint modality, the first feature extraction model is pruned to obtain the second feature extraction model. Specifically, the first feature extraction model is pruned, and the number of residual blocks in the subsequent four convolution blocks is reduced to 2, while the rest of the structure remains unchanged.

[0064] For the voiceprint modality, the first feature extraction model is a time-delay neural network, such as Figure 4 As shown, specifically:

[0065] The input voiceprint audio data first passes through the one-dimensional convolution layer Conv1D, then passes through five SE-Res2Block layers in sequence for feature extraction, then passes through the one-dimensional convolution layer and the attention statistical pooling layer for downsampling, and finally passes through the fully connected layer to complete feature extraction; among them, each SE-Res2Block layer introduces the SE mechanism, applies attention to the channel through compression and excitation, and then passes through the Res2Net module to divide the output of the convolution layer into multiple groups, and perform multi-scale feature fusion between multiple groups; after passing through SE-Res2Blocks, the outputs of all SE-Res2Blocks are spliced ​​and input into the next one-dimensional convolution layer Conv1D.

[0066] For the voiceprint modality, the first feature extraction model is tailored to obtain the second feature extraction model, specifically:

[0067] Based on the first feature extraction model, we perform cropping and use only three SE-Res2Block layers to obtain the second feature extraction model.

[0068] The first feature extraction model is used to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model. The specific knowledge distillation training process is as follows:

[0069] The first feature extraction model is used to perform knowledge distillation training on the second feature extraction model. The knowledge of the first feature extraction model is distilled into the second feature extraction model to obtain a third feature extraction model. The specific knowledge distillation training process is as follows:

[0070] 1) Perform forward propagation on the second feature extraction model of each modality to obtain the probability score of each class

[0071] 2) Calculate the multi-classification cross entropy loss function L of the second feature extraction model hard and soft label distillation loss function L soft , the expression is:

[0072]

[0073] Where: y represents the label; represents the probability calculated by the second feature extraction model; P t ,P s Represent the probability distribution of the first feature extraction model and the second feature extraction model respectively; T is the temperature used to control the smoothness of the softmax function; z t ,z s represent the classification scores of the first feature extraction model and the second feature extraction model respectively;

[0074] 3) Calculate the overall loss function Loss:

[0075] Loss = L hard +μ·L soft (4)

[0076] Where: μ represents the coefficient used to control the size of the soft label loss function;

[0077] 4) Based on the overall loss function, the second feature extraction model of each modality is trained with knowledge distillation by gradient descent to obtain the third feature extraction model.

[0078] S3, such as Figure 5 and Figure 6As shown, the multi-modal feature fusion model based on the attention mechanism is used to perform weighted summation on the features extracted by all third feature extraction models to obtain the fusion features, which specifically include:

[0079] The first face feature e face , the first fingerprint feature e fingerprint , and the first voiceprint feature e voice are extracted by the third feature extraction models of the three modalities respectively. face fingerprint voice

[0080] The modal features are processed by the full connection layer to obtain the second face feature e , the second fingerprint feature e , and the second voiceprint feature e

[0081] The first face feature e face , the first voiceprint feature e voice , and the first fingerprint feature e fingerprint are spliced to obtain the overall feature, and the attention scores a face , a voice , and a fingerprint of each modality are calculated by the attention mechanism.

[0082] The attention layer is divided into two steps, first through the linear layer (formula 5), and then through the activation function to calculate the attention score (formula 6). Wherein, z is the score after the linear layer, and a is the final attention score of each modality.

[0083] z {face,voice,fingerprint} = W T [e face ,e voice ,e fingerprint ]+b (5)

[0084]

[0085] The second features of each modality are weighted and summed using the attention scores a face , a voice , and a fingerprint of each modality to obtain the fusion feature e feat , and the expression is:

[0086]

[0087] S4, calculate the similarity between the fusion feature of the identity to be verified and the fusion feature of the corresponding identity pre-recorded in the database. When the similarity is greater than the prediction decision threshold, the identity verification is passed, otherwise the identity verification is not passed. In this embodiment, the similarity is the cosine similarity.

[0088] For the identity feature A entered into the database and the identity feature B to be verified, the cosine similarity sim between the identity feature B to be verified and the identity feature A in the pre-entered database is calculated, which is expressed as:

[0089]

[0090] When the cosine similarity sim is greater than the prediction decision threshold θ, the verification is passed, otherwise the verification is failed.

[0091] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An attention mechanism-based multi-modal identity verification method, characterized in that, Comprise: S1, collect multi-modal data, including face image data, fingerprint image data and voiceprint audio data; S2, for each modality, a corresponding first feature extraction model is constructed, a second feature extraction model is obtained by pruning the first feature extraction model, the first feature extraction model is used for knowledge distillation training of the second feature extraction model, and a third feature extraction model is obtained; S3, using a multi-modal feature fusion model based on attention mechanism, the features extracted by all third feature extraction models are weighted and summed to obtain the fusion features; S4, calculate the similarity between the fusion features of the identity to be verified and the fusion features of the corresponding identity pre-recorded in the database, when the similarity is greater than the prediction decision threshold, the identity verification is passed, otherwise the identity verification is not passed; The knowledge distillation training process of the first feature extraction model to the second feature extraction model to obtain the third feature extraction model is as follows: forward propagate the second feature extraction model of each modality to obtain a probability score of each class ; computing a multi-class cross-entropy loss function for the second feature extraction model and a soft label distillation loss function , expressed as: （1） （2） （3） In the formula: represents a label; represents a probability calculated by the second feature extraction model; respectively represent probability distributions of the first feature extraction model and the second feature extraction model; is a temperature used for controlling a smoothness degree of the function; respectively represent classification scores of the first feature extraction model and the second feature extraction model; computing the overall loss function : （4） In the formula: represents a coefficient for controlling the size of the soft label loss function; The second feature extraction model of each modality is trained by gradient descent based on the overall loss function to obtain the third feature extraction model; The multi-modal feature fusion model based on attention mechanism is used to weight and sum the features extracted by all third feature extraction models to obtain the fusion features, which specifically includes: The first face feature, the first fingerprint feature and the first voiceprint feature are extracted respectively by third feature extraction models of three modalities ​​​ The second face feature, the second fingerprint feature and the second voiceprint feature are obtained by processing the features of each modality through a full connection layer ​​​ The first face feature , the first voiceprint feature , and the first fingerprint feature are spliced to obtain an overall feature, and an attention score of each modality is calculated through an attention mechanism ; Attention scores for each modality are adopted Second features for each modality Weighted sum to get fused features .

2. The multi-modal identity verification method based on attention mechanism according to claim 1, characterized in that, For the face modality, the first feature extraction model is a convolutional neural network, which includes five consecutive convolutional blocks, and the output of the last convolutional block is connected to a multi-layer perceptron to complete face feature extraction; wherein, the first and second convolutional blocks each include two convolutional layers and a pooling layer, and the third to fifth convolutional blocks each include three convolutional layers and a pooling layer. 3.The multi-modal identity verification method based on attention mechanism of claim 2, wherein, For the face modality, the second feature extraction model is obtained by pruning the first feature extraction model, specifically: pruning the first feature extraction model, only using the first three convolutional blocks and the last multi-layer perceptron to obtain the second feature extraction model.

4. The multi-modal identity verification method based on attention mechanism according to claim 1, characterized in that, For the fingerprint modality, the first feature extraction model is a convolutional neural network, which includes five consecutive convolutional blocks, and the output of the last convolutional block is connected to a multi-layer perceptron to complete fingerprint feature extraction; wherein, each convolutional block includes multiple convolutional layers, and a residual connection is provided before the convolutional layers to extract local features; the first convolutional block only includes one convolutional layer and one pooling layer, and the subsequent four convolutional blocks each include a standard residual block, each residual block includes two convolutional layers, and the number of residual blocks in the subsequent four convolutional blocks is 3, 4, 6 and 3, respectively.

5. The multi-modal identity verification method based on attention mechanism according to claim 4, characterized in that, For the fingerprint modality, the second feature extraction model is obtained by pruning the first feature extraction model, specifically: pruning the first feature extraction model, reducing the number of residual blocks in the subsequent four convolutional blocks to 2, and keeping the rest of the structure unchanged.

6. The multi-modal identity verification method based on attention mechanism according to claim 1, characterized in that, For the voiceprint modality, the first feature extraction model is a time delay neural network, specifically: The input voiceprint audio data is first passed through a one-dimensional convolutional layer, then sequentially passed through five SE-Res2Block layers for feature extraction, then passed through a one-dimensional convolutional layer and an attention statistical pooling layer for down-sampling, and then passed through a fully connected layer for feature extraction; wherein each SE-Res2Block layer introduces an SE mechanism to apply attention to the channels through compression and excitation, then through a Res2Net module, the output of the convolutional layer is divided into multiple small groups, and multi-scale feature fusion is performed between the multiple small groups; after passing through the SE-Res2Blocks, the outputs of all the SE-Res2Blocks are spliced and input into the next one-dimensional convolutional layer.

7. The multi-modal identity verification method based on attention mechanism according to claim 6, characterized in that, For the voiceprint modality, a second feature extraction model is obtained by pruning the first feature extraction model, specifically: On the basis of the first feature extraction model, only three SE-Res2Block layers are used to obtain the second feature extraction model. 8.The multi-modal identity verification method based on attention mechanism of claim 1, wherein, The similarity is a cosine similarity.

Images