Multi-modal identity verification method based on attention mechanism

Abstract

The invention relates to a multi-modal identity verification method based on an attention mechanism, and the method comprises the steps: S1, collecting multi-modal data, including face image data, fingerprint image data and voiceprint audio data; s2, for each modal, constructing a corresponding first feature extraction model, cutting the first feature extraction model to obtain a second feature extraction model, and performing knowledge distillation training on the second feature extraction model by using the first feature extraction model to obtain a third feature extraction model; s3, performing weighted summation on all the features extracted by the third feature extraction model by adopting a multi-modal feature fusion model based on an attention mechanism to obtain a fusion feature; and S4, calculating the similarity between the fusion feature of the to-be-verified identity and the fusion feature of the corresponding identity pre-recorded in the database, and when the similarity is greater than a prediction decision threshold, determining that the identity verification is passed, otherwise, determining that the identity verification is not passed. Compared with the prior art, the method has the advantages of efficient feature extraction and high reliability.

Description

Technical Field

[0001] The present invention relates to the field of identity authentication, and in particular to a multimodal identity authentication method based on an attention mechanism. Background Art

[0002] In today's era of rapid digitalization and informatization, identity authentication technology plays a vital role in ensuring security, improving efficiency and improving user experience. Identity authentication technology is widely used in various scenarios, such as access control systems, financial transactions, smart device unlocking and public security, greatly facilitating people's lives and work.

[0003] Most traditional identity authentication methods are single-mode authentication, mainly relying on a single biometric feature, such as face, voiceprint or fingerprint. Face verification is a technology that authenticates identity by capturing and analyzing feature points in face images, such as the position and distance of eyes, nose and mouth. It is widely used in security monitoring, mobile phone unlocking and social media. Through image processing and pattern recognition technology, face verification systems can quickly verify individual identities. Specific technologies include VGGFace, ArcFace, etc. Voiceprint verification is a technology that verifies individual identities by analyzing features in voice signals, such as pitch, timbre and speaking speed. It is often used in scenarios such as telephone banking, voice assistants and smart home devices. Voiceprint verification systems usually rely on signal processing and machine learning technologies to complete identity authentication in a relatively short time. Specific technologies include TDNN, ECAPA-TDNN, etc. Fingerprint verification is a technology that confirms individual identities by scanning and comparing detailed features in fingerprint images, such as the distribution and intersection of ridges and valleys. Fingerprint verification is widely used in access control systems, attendance clocking and mobile payments. Fingerprint verification technology has high uniqueness and stability, and achieves accurate verification through hardware sensors and image processing algorithms.

[0004] Although single-modal verification technology performs well in certain specific scenarios, it often has certain limitations when facing complex and changing real-world environments. For example, face verification may fail in the case of lighting changes, occlusion, and expression changes; voice verification may be affected by background noise, microphone quality, and user voice changes; and fingerprint verification may have verification errors due to factors such as skin conditions and fingerprint wear. The single feature dependence of these methods limits their robustness and accuracy, and they cannot fully meet the needs of modern applications for high security and high reliability.

[0005] To overcome the limitations of single-modal authentication methods, multimodal authentication methods have emerged. Multimodal authentication can provide higher accuracy and robustness in a wider range of application scenarios by comprehensively utilizing multiple biometric data, such as face, voice, and fingerprint. For example, a common multimodal authentication system can use face verification and voice verification at the same time to improve the reliability and accuracy of the overall verification. Specifically, face verification provides visual information, voice verification provides auditory information, and fingerprint verification provides tactile information. These different modal data are complementary in terms of information volume and characteristics, and the performance of authentication can be significantly improved through reasonable fusion. However, the fusion and effective use of multimodal data remains a challenging problem. The data characteristics of different modalities vary greatly, and direct fusion may lead to information loss or increase in redundant information, thus affecting the verification effect. Therefore, researchers are constantly exploring new methods to optimize the fusion of multimodal data to achieve a more efficient and robust authentication system.

[0006] Previous achievements have provided a valuable foundation for identity verification, but there are still limitations:

[0007] (1) The robustness of the single-modal algorithm is relatively poor;

[0008] (2) The current multimodal fusion method only uses two modalities, and the fusion method is relatively simple;

[0009] (3) After adopting multimodal fusion, the overall architecture is huge and the efficiency is low. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multimodal identity authentication method based on attention mechanism with high robustness and high recognition efficiency.

[0011] The purpose of the present invention can be achieved by the following technical solutions:

[0012] A multimodal identity authentication method based on attention mechanism, comprising:

[0013] S1. Collect multimodal data, including face image data, fingerprint image data and voiceprint audio data;

[0014] S2. For each modality, construct a corresponding first feature extraction model, trim the first feature extraction model to obtain a second feature extraction model, and use the first feature extraction model to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model;

[0015] S3, using a multimodal feature fusion model based on an attention mechanism to perform weighted summation on the features extracted by all the third feature extraction models to obtain fused features;

[0016] S4. Calculate the similarity between the fused features of the identity to be verified and the fused features of the corresponding identity pre-entered into the database. When the similarity is greater than the prediction decision threshold, the identity verification is successful, otherwise the identity verification fails.

[0017] Preferably, for the face modality, the first feature extraction model is a convolutional neural network, including five sequentially connected convolution blocks, and the output of the last convolution block is connected to a multi-layer perceptron to complete face feature extraction; wherein, the first and second convolution blocks each include two convolution layers and one pooling layer, and the third to fifth convolution blocks include three convolution layers and one pooling layer.

[0018] Preferably, for the face modality, the first feature extraction model is pruned to obtain the second feature extraction model, specifically: pruned based on the first feature extraction model, only the first three convolution blocks and the last multi-layer perceptron are used to obtain the second feature extraction model.

[0019] Preferably, for the fingerprint modality, the first feature extraction model is a convolutional neural network, including five sequentially connected convolution blocks, and the output of the last convolution block is connected to a multi-layer perceptron to complete the fingerprint feature extraction; wherein each data block contains multiple convolution layers, and a residual connection is set before the convolution layer to extract local features; the first convolution block contains only one convolution layer and one pooling layer, and the subsequent four convolution blocks all contain standard residual blocks, each residual block contains two convolution layers, and the number of residual blocks in the subsequent four convolution blocks are 3, 4, 6 and 3 respectively.

[0020] Preferably, for the fingerprint modality, the first feature extraction model is pruned to obtain the second feature extraction model, specifically: pruned on the basis of the first feature extraction model, the number of residual blocks in the subsequent four convolution blocks is reduced to 2, and the rest of the structure remains unchanged.

[0021] Preferably, for the voiceprint modality, the first feature extraction model is a time-delay neural network, specifically:

[0022] The input voiceprint audio data first passes through a one-dimensional convolution layer, then passes through five SE-Res2Block layers in sequence for feature extraction, then passes through a one-dimensional convolution layer and an attention statistics pooling layer for downsampling, and finally passes through a fully connected layer to complete feature extraction; among them, each SE-Res2Block layer introduces the SE mechanism, which applies attention to the channel through compression and excitation, and then passes through the Res2Net module to divide the output of the convolution layer into multiple groups, and performs multi-scale feature fusion between multiple groups; after passing through SE-Res2Blocks, the outputs of all SE-Res2Blocks are concatenated and input into the next one-dimensional convolution layer.

[0023] Preferably, for the voiceprint modality, the first feature extraction model is tailored to obtain the second feature extraction model, specifically:

[0024] Based on the first feature extraction model, we perform cropping and use only three SE-Res2Block layers to obtain the second feature extraction model.

[0025] Preferably, the first feature extraction model is used to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model. The knowledge distillation training process is specifically as follows:

[0026] Perform forward propagation on the second feature extraction model of each modality to obtain the probability score of each class

[0027] Calculate the multi-classification cross entropy loss function L of the second feature extraction model hard and the soft label distillation loss function L soft , the expression is:

[0028]

[0029] Where: y represents the label; represents the probability calculated by the second feature extraction model; P t ,P s Represent the probability distribution of the first feature extraction model and the second feature extraction model respectively; T is the temperature used to control the smoothness of the softmax function; z t ,z s Represent the classification scores of the first feature extraction model and the second feature extraction model respectively;

[0030] Calculate the overall loss function Loss:

[0031] Loss = L hard +μ·L soft (4)

[0032] Where: μ represents the coefficient used to control the size of the soft label loss function;

[0033] Based on the overall loss function, the second feature extraction model of each modality is trained by gradient descent to obtain the third feature extraction model.

[0034] Preferably, the multimodal feature fusion model based on the attention mechanism is used to perform weighted summation on the features extracted by all the third feature extraction models to obtain the fused features, which specifically includes:

[0035] The first facial features e are extracted respectively through the third feature extraction models of the three modalities face , the first fingerprint feature e fingerprintand the first voiceprint feature e voice ;

[0036] The features of each modality are processed through the fully connected layer to obtain the second face features. Second fingerprint feature and the second voiceprint feature

[0037] The first facial feature e face 、The first voiceprint feature e voice and the first fingerprint feature e fingerprint The overall features are concatenated and the attention score a of each modality is calculated through the attention mechanism face ,a voice ,a fingerprint ;

[0038] Adopt the attention score a of each modality face ,a voice ,a fingerprint The second characteristic of each mode Weighted summation to obtain the fusion feature e feat .

[0039] Preferably, the similarity is cosine similarity.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention uses knowledge distillation technology to allow a more complex first feature extraction model to assist in training a more streamlined second feature extraction model, thereby refining knowledge of the single-modal feature extraction technology and making the single-modal feature extraction more efficient.

[0042] (2) The present invention integrates the features of the three modalities of face, voiceprint and fingerprint through the attention mechanism, which can achieve more robust identity authentication and higher reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the method of the present invention;

[0044] Figure 2 This is a schematic diagram of the first feature extraction model structure for faces;

[0045] Figure 3 It is a schematic diagram of the structure of the first feature extraction model for fingerprints;

[0046] Figure 4 It is a schematic diagram of the structure of the first feature extraction model for voiceprint;

[0047] Figure 5 Schematic diagram of feature fusion;

[0048] Figure 6 This is the architecture diagram of the feature fusion model based on the attention mechanism. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] Example

[0051] like Figure 1 As shown, this embodiment provides a multimodal identity authentication method based on an attention mechanism, including:

[0052] S1. Collect multimodal data, including face image data, fingerprint image data and voiceprint audio data;

[0053] S2. For each modality, construct a corresponding first feature extraction model, trim the first feature extraction model to obtain a second feature extraction model, and use the first feature extraction model to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model;

[0054] S3, using a multimodal feature fusion model based on an attention mechanism to perform weighted summation on the features extracted by all the third feature extraction models to obtain fused features;

[0055] S4. Calculate the similarity between the fused features of the identity to be verified and the fused features of the corresponding identity pre-entered into the database. When the similarity is greater than the prediction decision threshold, the identity verification is successful, otherwise the identity verification fails.

[0056] Next, the method of this embodiment is introduced in detail.

[0057] S1. Collect multimodal data, including face image data, fingerprint image data and voiceprint audio data.

[0058] In this embodiment, the face, voiceprint and fingerprint data are obtained through sensors such as cameras and recorders. Among them, the face and fingerprint exist in the form of images, and the voiceprint exists in the form of audio.

[0059] S2. For each modality, construct a corresponding first feature extraction model, trim the first feature extraction model to obtain a second feature extraction model, use the first feature extraction model to perform knowledge distillation training on the second feature extraction model, and obtain a third feature extraction model.

[0060] like Figure 2 As shown in the figure, for the face modality, the first feature extraction model is a convolutional neural network, including five sequential convolution blocks, and the output of the last convolution block is connected to a multi-layer perceptron to complete the face feature extraction; wherein, the first and second convolution blocks each include two convolution layers and one pooling layer, and the third to fifth convolution blocks include three convolution layers and one pooling layer. Among them, conv represents a two-dimensional convolution layer, Pool represents a pooling layer, and FC represents a fully connected layer.

[0061] For the face modality, the first feature extraction model is pruned to obtain the second feature extraction model. Specifically, the first feature extraction model is pruned to obtain the second feature extraction model by using only the first three convolution blocks and the last multi-layer perceptron.

[0062] For the fingerprint modality, the first feature extraction model is a convolutional neural network, such as Figure 3 As shown, it includes five sequentially connected convolution blocks Conv Block, and the output of the last convolution block is connected to a multi-layer perceptron to complete fingerprint feature extraction; wherein each data block contains multiple convolution layers, and a residual connection is set before the convolution layer to prevent the gradient from disappearing, which is used to extract local features. In this embodiment, the first convolution block only contains one 7*7 convolution layer and one 3*3 pooling layer, and the subsequent four convolution blocks all contain standard residual blocks, each residual block contains two 3*3 convolution layers, and the number of residual blocks in the subsequent four convolution blocks is 3, 4, 6 and 3 respectively.

[0063] For the fingerprint modality, the first feature extraction model is pruned to obtain the second feature extraction model. Specifically, the first feature extraction model is pruned, and the number of residual blocks in the subsequent four convolution blocks is reduced to 2, while the rest of the structure remains unchanged.

[0064] For the voiceprint modality, the first feature extraction model is a time-delay neural network, such as Figure 4 As shown, specifically:

[0065] The input voiceprint audio data first passes through the one-dimensional convolution layer Conv1D, then passes through five SE-Res2Block layers in sequence for feature extraction, then passes through the one-dimensional convolution layer and the attention statistics pooling layer for downsampling, and finally passes through the fully connected layer to complete the feature extraction; among them, each SE-Res2Block layer introduces the SE mechanism, applies attention to the channel through compression and excitation, and then passes through the Res2Net module to divide the output of the convolution layer into multiple groups, and performs multi-scale feature fusion between multiple groups; after passing through SE-Res2Blocks, the outputs of all SE-Res2Blocks are spliced ​​and input into the next one-dimensional convolution layer Conv1D.

[0066] For the voiceprint modality, the first feature extraction model is tailored to obtain the second feature extraction model, specifically:

[0067] Based on the first feature extraction model, we perform cropping and use only three SE-Res2Block layers to obtain the second feature extraction model.

[0068] The first feature extraction model is used to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model. The knowledge distillation training process is specifically as follows:

[0069] The first feature extraction model is used to perform knowledge distillation training on the second feature extraction model, and the knowledge of the first feature extraction model is distilled into the second feature extraction model to obtain a third feature extraction model. The knowledge distillation training process is specifically as follows:

[0070] 1) Perform forward propagation on the second feature extraction model of each modality to obtain the probability score of each class

[0071] 2) Calculate the multi-classification cross entropy loss function L of the second feature extraction model hard and the soft label distillation loss function L soft , the expression is:

[0072]

[0073] Where: y represents the label; represents the probability calculated by the second feature extraction model; P t ,P s Represent the probability distribution of the first feature extraction model and the second feature extraction model respectively; T is the temperature used to control the smoothness of the softmax function; z t ,z s Represent the classification scores of the first feature extraction model and the second feature extraction model respectively;

[0074] 3) Calculate the overall loss function Loss:

[0075] Loss = L hard +μ·L soft (4)

[0076] Where: μ represents the coefficient used to control the size of the soft label loss function;

[0077] 4) Based on the overall loss function, the second feature extraction model of each modality is trained by gradient descent to obtain a third feature extraction model.

[0078] S3, such as Figure 5 and Figure 6As shown, a multimodal feature fusion model based on the attention mechanism is used to perform weighted summation on the features extracted by all the third feature extraction models to obtain fusion features, including:

[0079] The first facial features e are extracted respectively through the third feature extraction models of the three modalities face , the first fingerprint feature e fingerprint and the first voiceprint feature e voice ;

[0080] The features of each modality are processed through the fully connected layer to obtain the second face features. Second fingerprint feature and the second voiceprint feature

[0081] The first facial feature e face 、The first voiceprint feature e voice and the first fingerprint feature e fingerprint The overall features are concatenated and the attention score a of each modality is calculated through the attention mechanism face ,a voice ,a fingerprint ;

[0082] The attention layer is divided into two steps, first through the linear layer (Formula 5), ​​and then through the activation function to calculate the attention score (Formula 6). Among them, z is the score after passing the linear layer, and a is the final attention score of each modality.

[0083] z {face,voice,fingerprint} =W T [e face ,e voice ,e fingerprint ]+b (5)

[0084]

[0085] Adopt the attention score a of each modality face ,a voice ,a fingerprint The second characteristic of each mode Weighted summation to obtain the fusion feature e feat , the expression is:

[0086]

[0087] S4, calculating the similarity between the fused features of the identity to be verified and the fused features of the corresponding identity pre-entered in the database, when the similarity is greater than the prediction decision threshold, the identity verification is passed, otherwise the identity verification fails. In this embodiment, the similarity is cosine similarity.

[0088] For the identity feature A entered into the database and the identity feature B to be verified, the cosine similarity sim between the identity feature B to be verified and the identity feature A in the pre-entered database is calculated, which is expressed as:

[0089]

[0090] When the cosine similarity sim is greater than the prediction decision threshold θ, the verification passes, otherwise the verification fails.

[0091] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal identity authentication method based on attention mechanism, characterized in that: include: S1. Collect multimodal data, including face image data, fingerprint image data and voiceprint audio data; S2. For each modality, construct a corresponding first feature extraction model, trim the first feature extraction model to obtain a second feature extraction model, and use the first feature extraction model to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model; S3, using a multimodal feature fusion model based on an attention mechanism to perform weighted summation on the features extracted by all the third feature extraction models to obtain fused features; S4. Calculate the similarity between the fused features of the identity to be verified and the fused features of the corresponding identity pre-entered into the database. When the similarity is greater than the prediction decision threshold, the identity verification is successful, otherwise the identity verification fails.

2. According to claim 1, a multimodal identity authentication method based on an attention mechanism is characterized in that: For the face modality, the first feature extraction model is a convolutional neural network, which includes five sequential convolution blocks, and the output of the last convolution block is connected to a multi-layer perceptron to complete the face feature extraction; among them, the first and second convolution blocks each include two convolution layers and one pooling layer, and the third to fifth convolution blocks include three convolution layers and one pooling layer.

3. According to claim 2, a multimodal identity authentication method based on an attention mechanism is characterized in that: For the face modality, the first feature extraction model is pruned to obtain the second feature extraction model. Specifically, the first feature extraction model is pruned to obtain the second feature extraction model by using only the first three convolution blocks and the last multi-layer perceptron.

4. According to claim 1, a multimodal identity authentication method based on an attention mechanism is characterized in that: For the fingerprint modality, the first feature extraction model is a convolutional neural network, which includes five consecutive convolution blocks, and the output of the last convolution block is connected to a multi-layer perceptron to complete the fingerprint feature extraction; each data block contains multiple convolution layers, and a residual connection is set before the convolution layer to extract local features; the first convolution block contains only one convolution layer and one pooling layer, and the subsequent four convolution blocks all contain standard residual blocks, each residual block contains two convolution layers, and the number of residual blocks in the subsequent four convolution blocks are 3, 4, 6 and 3 respectively.

5. According to claim 4, a multimodal identity authentication method based on an attention mechanism is characterized in that: For the fingerprint modality, the first feature extraction model is pruned to obtain the second feature extraction model. Specifically, the first feature extraction model is pruned, and the number of residual blocks in the subsequent four convolution blocks is reduced to 2, while the rest of the structure remains unchanged.

6. According to claim 1, a multimodal identity authentication method based on an attention mechanism is characterized in that: For the voiceprint modality, the first feature extraction model is a time-delay neural network, specifically: The input voiceprint audio data first passes through a one-dimensional convolution layer, then passes through five SE-Res2Block layers in sequence for feature extraction, then passes through a one-dimensional convolution layer and an attention statistics pooling layer for downsampling, and finally passes through a fully connected layer to complete feature extraction; among them, each SE-Res2Block layer introduces the SE mechanism, which applies attention to the channel through compression and excitation, and then passes through the Res2Net module to divide the output of the convolution layer into multiple groups, and performs multi-scale feature fusion between multiple groups; after passing through SE-Res2Blocks, the outputs of all SE-Res2Blocks are concatenated and input into the next one-dimensional convolution layer.

7. A multimodal identity authentication method based on attention mechanism according to claim 6, characterized in that: For the voiceprint modality, the first feature extraction model is tailored to obtain the second feature extraction model, specifically: Based on the first feature extraction model, we perform cropping and use only three SE-Res2Block layers to obtain the second feature extraction model.

8. According to claim 1, a multimodal identity authentication method based on an attention mechanism is characterized in that: The first feature extraction model is used to perform knowledge distillation training on the second feature extraction model to obtain a third feature extraction model. The knowledge distillation training process is specifically as follows: Perform forward propagation on the second feature extraction model of each modality to obtain the probability score of each class Calculate the multi-classification cross entropy loss function L of the second feature extraction model hard and the soft label distillation loss function L soft , the expression is: Where: y represents the label; represents the probability calculated by the second feature extraction model; P t ,P s Represent the probability distribution of the first feature extraction model and the second feature extraction model respectively; T is the temperature used to control the smoothness of the softmax function; z t ,z s Represent the classification scores of the first feature extraction model and the second feature extraction model respectively; Calculate the overall loss function Loss: Loss=L hard +μ·L soft (4) Where: μ represents the coefficient used to control the size of the soft label loss function; Based on the overall loss function, the second feature extraction model of each modality is trained by gradient descent to obtain the third feature extraction model.

9. According to claim 1, a multimodal identity authentication method based on an attention mechanism is characterized in that: The multimodal feature fusion model based on the attention mechanism is used to perform weighted summation on the features extracted by all the third feature extraction models to obtain fusion features, which specifically includes: The first facial features e are extracted respectively through the third feature extraction models of the three modalities face , the first fingerprint feature e fingerprint and the first voiceprint feature e voice ; The features of each modality are processed through the fully connected layer to obtain the second face features. Second fingerprint feature and the second voiceprint feature The first facial feature e face 、The first voiceprint feature e voice and the first fingerprint feature e fingerprint The overall features are concatenated and the attention score a of each modality is calculated through the attention mechanism face ,a voice ,a fingerprint ; Adopt the attention score a of each modality face ,a voice ,a fingerprint The second characteristic of each mode Weighted summation to obtain the fusion feature e feat .

10. A multimodal identity authentication method based on attention mechanism according to claim 1, characterized in that: The similarity is cosine similarity.

Images