Multi-mode identity authentication method

Abstract

According to the multi-modal identity authentication method provided by the invention, multiple biological characteristics are fused through multi-modal identity authentication, the defects of single-modal authentication can be made up, a multi-modal system can utilize complementary characteristics among different modals, the authentication accuracy and the anti-interference capability are improved, and multi-layer protection is added in the authentication process, so that the authentication efficiency is improved. And meanwhile, due to multi-modal fusion, the system has higher robustness and flexibility when facing a complex environment, and can adapt to wider application scenes.

Description

Technical Field

[0001] The present invention relates to the field of data privacy and security technology, and in particular to a multi-modal identity authentication method. Background Art

[0002] Single-modal authentication usually relies on only one biometric feature (such as voiceprint or face) for identity verification. Although it can provide a certain degree of security and convenience in simple scenarios, it is easily restricted in complex environments. For example, voiceprint recognition is easily interfered with in noisy environments, and face recognition performs poorly in low light or partially occluded conditions. In addition, single-modal authentication systems are relatively weak in anti-attack capabilities when facing counterfeit attacks. Therefore, single-modal authentication is difficult to meet the needs of high-security and high-robustness scenarios.

[0003] Multimodal identity authentication can make up for the shortcomings of single-modal authentication by integrating multiple biometric features (such as voiceprint and face). Multimodal systems can use the complementary characteristics between different modalities to improve the accuracy and anti-interference ability of authentication, and add multiple layers of protection during the verification process, significantly improving the security of the system. At the same time, multimodal fusion makes the system more robust and flexible in the face of complex environments, and can adapt to a wider range of application scenarios.

[0004] The existing patent application (CN117576763A) discloses an identity recognition method and system based on voiceprint information and face information in a cloud environment, the method includes collecting voiceprint signals and face image information; extracting voiceprint signal features and face image features from the voiceprint signals and face image information; splicing the voiceprint signal features and face image features together in order to form a spliced ​​feature vector; encrypting the spliced ​​feature vector and sending the encrypted vector to a cloud server; the cloud server compares and verifies the received encrypted vector and makes a decision based on the comparison and verification results. The present invention splices the voiceprint signal features and face image features together in order to form a spliced ​​feature vector as an identity recognition standard, providing more biometric features to complement each other, thereby improving the stability and robustness of identity authentication. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a multi-modal identity authentication method, comprising the following steps:

[0006] S1, feature extraction: extract voice and face features respectively;

[0007] S2. Speech and face features are processed by speech or face model modules. Each speech or face model module contains n modules. Each module adopts a mixed expert layer. Each expert is trained with low-quality data in different situations to cope with low-quality situations in actual application scenarios.

[0008] S3, hierarchical fusion, adopts the inter-modal hierarchical fusion method based on attention mechanism;

[0009] S4, splicing, splicing the hierarchical fusion results in the feature dimension so that the subsequent model can process and fuse this information in the same space;

[0010] S5, layer normalization, calculate the mean and standard deviation on the feature dimension of each sample, and normalize the feature to a distribution with a mean of 0 and a variance of 1;

[0011] S6, attention statistical pooling layer, by introducing the attention mechanism, weights are assigned according to the importance of input features to adaptively select more representative features;

[0012] S7, batch normalization, by calculating the mean and standard deviation on the feature dimension of each small batch of data, normalize the features to a distribution with a mean of 0 and a variance of 1;

[0013] S8, linear layer, reduces the dimension of features through linear mapping.

[0014] A multimodal identity authentication method described in the present invention is a preferred method. For voice features, for audio data, a 40-dimensional Fbank feature is extracted using the Kaldi toolkit, with a window of 25 milliseconds and a frame shift of 10 milliseconds, and an energy-based voice activity detector (VAD) is used to eliminate silence, and a cepstral mean operation is performed on the Fbank feature, with a sliding window size of 300 frames; for facial features, for each video clip, one frame of image is extracted per second, and the face in the frame is extracted using the landmarks predicted by MTCNN to align, and then the facial area is mapped to the same shape (3×112×96) using a similarity transform, and the pixel value of each image is standardized to within the range of [-0.5, 0.5].

[0015] A multimodal identity authentication method described in the present invention, as a preferred mode, the speech model module includes 4 experts, corresponding to four types of low-quality data situations: noise, short speech, speaker variability, and speaker overlap; the face model module includes 4 experts, corresponding to four types of low-quality data situations: illumination, occlusion, posture expression, and multiple faces; in addition, in order to reduce the problem of a sharp increase in the number of model parameters due to multiple experts, the LoRA module is used to replace the ordinary dense feedforward layer, thereby greatly reducing the number of model parameters.

[0016] The multimodal identity authentication method described in the present invention is, as a preferred method, the mixed expert layer first receives the input through a router and outputs the score of each expert; then multiplies the expert score with the output of each expert to obtain the mixed output of all experts; finally, the mixed output of all experts is added to the output of the basic model layer to obtain the final output.

[0017] A multimodal identity authentication method described in the present invention, as a preferred method, LoRA uses the product of two low-rank matrices AB to approximate the update of the original weight matrix, thereby significantly reducing the number of parameters.

[0018] The multimodal identity authentication method described in the present invention, as a preferred embodiment, hierarchical fusion includes the following steps:

[0019] S1. The output of each speech model module is recorded as e a , the output of the face model module is recorded as e v ;

[0020] S2, e a and e v Through the attention layer, we get the scores s of speech and face modality a and v ;

[0021] S3, e a and e v Use s respectively a and v The weighted values ​​are added together to get the hierarchical fusion result.

[0022] The present invention provides a multi-modal identity authentication method, which can improve the accuracy and anti-interference ability of authentication, make the system more robust and flexible when facing complex environments, and can adapt to a wider range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an overall framework diagram of a multi-modal identity authentication method of the present invention;

[0024] Figure 2 A schematic diagram of the structure of a voice / face model module of a multimodal identity authentication method of the present invention;

[0025] Figure 3 A schematic diagram of the structure of a LoRA module of a multi-modal identity authentication method of the present invention;

[0026] Figure 4 A schematic diagram of a hierarchical fusion module of a multimodal identity authentication method of the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0028] Example 1

[0029] like Figures 1 to 4 As shown, a multimodal identity authentication method comprises the following steps:

[0030] S1: detect the target’s voice and face;

[0031] S2: Data processing;

[0032] S2.1: For audio data, the Kaldi toolkit is used to extract 40-dimensional Fbank features with a window of 25 milliseconds and a frame shift of 10 milliseconds, and an energy-based voice activity detector (VAD) is used to remove silence. Then, the Fbank features are cepstral averaged with a sliding window size of 300 frames. In addition, all segments less than 400 frames are discarded;

[0033] S2.2: For the video clip, one frame of image is extracted per second. The face in the frame is extracted using the landmarks predicted by MTCNN, and then the face region is mapped to the same shape (3×112×96) using similarity transformation. Finally, the pixel values ​​of each image are normalized to the range of [-0.5, 0.5];

[0034] S3: Intra-modal level fusion based on hybrid expert mechanism. Each expert is trained with low-quality data in different situations to cope with low-quality situations in actual application scenarios;

[0035] S3.1: In terms of speech, four experts are used, corresponding to four types of low-quality data: noise, short speech, speaker variability, and speaker overlap. The router determines which type or types of low-quality data the speech belongs to based on the input speech, and uses the corresponding expert to process the speech, and finally dynamically weights the expert output results according to the routing score;

[0036] S3.2: In terms of faces, four experts are also used, corresponding to four types of low-quality data: illumination, occlusion, posture and expression, and multiple faces. The router determines which type or types of low-quality data the face belongs to based on the input face, and uses the corresponding expert to process the face, and finally dynamically weights the expert output results according to the routing score;

[0037] S4: Inter-modal hierarchical fusion based on attention mechanism. The attention mechanism adjusts the weights of different modal features and adaptively focuses on the most relevant information.

[0038] Step: Concatenate the fusion results between the modalities of each layer, normalize them, and pass them through the attention statistical pooling layer. Finally, use the linear layer feature dimensionality reduction to obtain the identity embedding representation of the detected target.

[0039] S6: Calculate the cosine similarity between the identity embedding representation of the detection target and the identity embedding representation in the model library, and finally determine whether the detection target is allowed based on the threshold.

[0040] The above description is only illustrative rather than restrictive of the present invention. Those skilled in the art will understand that any modification, change or equivalent that can be made without departing from the spirit and scope defined by the claims will fall within the protection scope of the present invention.

Claims

1. A multimodal identity authentication method, characterized in that: The following steps are involved: S1, feature extraction: extract voice and face features respectively; S2, the speech and facial features are processed by a speech or facial model module, wherein the speech or facial model module contains n modules, each of which uses a mixed expert layer, and the mixed expert layer is trained using low-quality data under different circumstances; S3, hierarchical fusion, adopts the inter-modal hierarchical fusion method based on attention mechanism; S4, splicing, splicing the hierarchical fusion results in the feature dimension so that the subsequent model can process and fuse this information in the same space; S5, layer normalization, calculate the mean and standard deviation on the feature dimension of each sample, and normalize the feature to a distribution with a mean of 0 and a variance of 1; S6, attention statistical pooling layer, by introducing the attention mechanism, weights are assigned according to the importance of the input features to adaptively select more representative features; S7, batch normalization, by calculating the mean and standard deviation on the feature dimension of each small batch of data, normalizing the features to a distribution with a mean of 0 and a variance of 1; S8, linear layer, reduces the dimension of the features through linear mapping.

2. A multimodal identity authentication method according to claim 1, characterized in that: For the speech features, for audio data, the Kaldi toolkit is used to extract 40-dimensional Fbank features with a window of 25 milliseconds and a frame shift of 10 milliseconds, and an energy-based voice activity detector is used to eliminate silence, and the Fbank features are subjected to a cepstral mean operation with a sliding window size of 300 frames; for the face features, for each video clip, one frame of image is extracted per second, and the face in the frame is extracted by aligning the landmarks predicted by MTCNN, and then the facial area is mapped to the same shape of 3×112×96 using a similarity transform, and the pixel values ​​of each image are normalized to the range of [-0.5, 0.5].

3. A multimodal identity authentication method according to claim 1, characterized in that: The speech model module includes 4 experts, corresponding to noise, short speech, speaker variability, and speaker overlap respectively; the face model module includes 4 experts, corresponding to illumination, occlusion, posture expression, and multiple faces respectively; the LoRA module is used to replace the ordinary dense feedforward layer.

4. A multimodal identity authentication method according to claim 1, characterized in that: The mixed expert layer first receives input through a router and outputs the score of each expert; then multiplies the expert score with the output of each expert to obtain a mixed output of all the experts; finally, the mixed output of all the experts is added to the output of the basic model layer to obtain the final output.

5. A multimodal identity authentication method according to claim 3, characterized in that: The LoRA uses the product of two low-rank matrices AB to approximate the update of the original weight matrix.

6. A multimodal identity authentication method according to claim 1, characterized in that: The hierarchical fusion comprises the following steps: S1. The output of each speech model module is recorded as e a , the output of the face model module is recorded as e v ; S2, the e a and the v Through the attention layer, we get the scores s of speech and face modality a and v ; S3, the e a and the v Use the s a and the v The weighted values ​​are added together to get the hierarchical fusion result.

Images