Face recognition method based on multi-modal fusion

Abstract

The invention discloses a face recognition method based on multi-modal fusion, and belongs to the technical field of face recognition, in the face recognition method, when visual information is lost due to face partial shielding or strong light irradiation, emotion features in voice signals can automatically compensate for the deficiency of visual information. For example, in a video call scene, when a user shields a chin area with a hand, the system can still accurately recognize anxiety emotion by analyzing intonation trembling characteristics in voice and combining muscle movement of an unshielded eyebrow and eye area, and misjudgment caused by local information loss is avoided. Meanwhile, under the condition that noisy background sound interferes voice, the model can effectively filter interference of voice noise and maintain accuracy of emotion judgment by capturing continuous duration and intensity changes of facial micro-expressions, such as slight twitching of mouth corners lasting for more than 3 seconds.

Description

Technical Field

[0001] The present invention belongs to the field of face recognition technology, and specifically relates to a face recognition method based on multimodal fusion. Background Art

[0002] Facial recognition is a biometric technology based on facial features. It analyzes facial geometry and texture information and compares facial images with a database of known faces to verify or identify a person. This technology is widely used in security surveillance, access control systems, and mobile phone unlocking, and is highly sought after for its contactless nature, convenience, and high accuracy. The advancement of artificial intelligence technologies such as deep learning has significantly improved the accuracy and robustness of facial recognition, sparking discussions about privacy protection and ethical issues.

[0003] However, traditional methods have difficulty establishing deep spatiotemporal correlations between speech features and facial movements, and cannot effectively complement each other when the quality of single-modal data deteriorates. In addition, simple weighted fusion strategies are prone to confusing biological features with facial features, resulting in significant differences in recognition stability for users of different ages and skin colors, and insufficient differentiation of subtle differences in mixed expressions. Summary of the Invention

[0004] The purpose of the present invention is to provide a face recognition method based on multimodal fusion in order to solve the above-mentioned problems.

[0005] The technical solution adopted by the present invention is as follows: a face recognition method based on multimodal fusion, which covers the following process:

[0006] S1: Acquire facial visual data and associated acoustic signals, perform timing alignment and noise filtering, use 3D facial landmark tracking technology to standardize the facial region, and perform segmented denoising and fundamental frequency normalization on the speech waveform.

[0007] S2: The deep residual network after transfer learning is used to extract multi-dimensional facial features, and the expanded convolutional layer is combined to enhance the representation ability of micro-expression areas such as glabellar wrinkles and nasolabial grooves.

[0008] S3: Analyze the rhythmic features and spectrograms of acoustic signals, and use the dynamic time convolution architecture to capture the emotion-related voiceprint fluctuation patterns.

[0009] S4: Build a heterogeneous data interaction engine to perform multi-dimensional mapping between acoustic temporal streams and facial spatial features, and synthesize cross-modal enhanced features through a flexible attention allocation mechanism.

[0010] S5: Establish a parallel feature processing network, implement spatial-temporal modeling for visual and auditory modalities respectively, and use intelligent gated regulators to achieve dynamic integration of feature levels.

[0011] S6: Apply adversarial feature separation technology to eliminate the correlation between individual biological features and essential features of expression through the maximum mutual information loss function.

[0012] S7: Design a hierarchical classification system with the base layer identifying six core emotions, the enhancement layer parsing complex emotions, and the top layer embedding a probabilistic credibility assessment component.

[0013] S8: Implement a progressive training strategy, focusing on unimodal representations in the initial stage, introducing cross-modal adversarial examples in the middle stage, and focusing on optimizing the classification decision boundary in the later stage.

[0014] S9: Integrates a real-time optimization mechanism to dynamically update model parameters based on environmental feedback, combining neural network pruning and low-bit quantization to enable mobile deployment.

[0015] In a preferred embodiment, in step S1, a 68-keypoint detection model is used for real-time positioning, ensuring that the face region is cropped to a uniform size of 256×256 pixels. Voice signals are sampled at a 16kHz sampling rate, with a 25ms frame length and a 10ms overlap between adjacent frames. A Wiener filter algorithm is used to eliminate ambient noise. Frequency normalization utilizes a dynamic time warping algorithm to synchronize the voice signal with the video frame rate to 30fps, with a time alignment error within ±33ms.

[0016] In a preferred embodiment, in step S2, the transfer learning phase uses a ResNet50 network pre-trained on ImageNet as the backbone network, freezing the parameters of the first three residual blocks. During the fine-tuning phase, the learning rate is set to 1e-4. The dilated convolutional layer is configured with three parallel branches, with dilation rates set to 3, 5, and 7, respectively, and a uniform convolution kernel size of 5×5 to capture micro-expression textures at different scales. During the feature fusion phase, a channel attention mechanism is employed, increasing the weight coefficient for the glabella region to 1.5 times its original value to enhance the corrugator muscle movement characteristics.

[0017] In a preferred embodiment, in step S3, the prosodic feature extraction module includes three dimensions: fundamental frequency trajectory, energy envelope, and formant bandwidth. A 25-order Mel filter bank is used to generate the time-frequency spectrum. A dynamic temporal convolutional network employs a four-layer dilated convolutional architecture, with an initial receptive field of 5 frames, which expands to 21 frames layer by layer. The activation threshold of the gated unit is set to 0.65 to suppress noise interference in low-energy regions. Voiceprint fluctuation modeling utilizes a bidirectional LSTM architecture with a hidden layer dimension of 128 and a time step that strictly matches the video frame rate.

[0018] In a preferred embodiment, in step S4, the speech feature sequence and the facial feature map are first mapped to a unified spatiotemporal coding space, and a cross-modal dynamic association is established through a deformable attention mechanism. The speech temporal features are expanded into a spatiotemporal cube through three-dimensional convolution and cross-projected with the channel dimension of the facial feature map. During this process, a dynamic offset generation network is used to calculate the attention offset matrix, so that the speech frame can adaptively align with the sensitive areas of facial expression changes.

[0019] At the same time, spatial attention weights are calculated based on the inter-modal correlation matrix, guiding the feature reconstruction process to focus on key areas for expression discrimination. The reconstructed hybrid features are then transformed into multiple sets of enhanced basis vectors using an affine transformation. A multi-head mechanism is then used to perform cross-modal interactions at different semantic levels in parallel, ultimately fusing them into a highly discriminative joint representation. This module effectively captures the implicit associations between micro-expressions and vocal intonation by simulating the audiovisual neural integration process in human emotional perception.

[0020] The dynamic offset generation formula is:

[0021]

[0022] Among them, P is the speech temporal feature matrix, with dimensions T×D_p; F is the face spatial feature tensor, with dimensions H×W×D_f; represents the tensor outer product operation, which models the cross-modal association between speech frame features and facial spatial position features; W1 and W2 are learnable weight matrices used for nonlinear transformation of feature space; σ is the Sigmoid activation function, which controls the offset amplitude; α is the inter-modal correlation adjustment factor, which is dynamically calculated from the mutual information entropy of the speech energy spectrum and the facial action unit, and is used to enhance the projection accuracy of expression-related areas.

[0023] The multi-head affine transformation formula is:

[0024]

[0025] Among them, A_k is the speech projection matrix corresponding to the kth attention head, B_k is the face projection matrix, represents the Kronecker product operation, which fully interacts the spatial and temporal dimensions of the two modal features. vec(β) is the vectorized form of the modal coupling coefficient, β∈[0,1]^K, generated by a cross-modal gating unit to balance the contributions of speech and facial features in different expression categories. W3 is a learnable parameter of the residual connection to prevent gradient vanishing. The output V_k of each head undergoes layer normalization and is then multi-head aggregated through W4 to generate the final mixed basis vector.

[0026] In a preferred embodiment, in step S5, the face and voice modalities are respectively subjected to intra-modal context modeling through bidirectional LSTM and graph convolutional network to generate spatiotemporal context enhancement features; the gated weight allocation unit adopts a parallel structure of channel attention and spatial attention, first calculates the variance entropy of each modal feature as a confidence indicator, and then generates a fusion weight through a Softmax function with a learnable temperature coefficient.

[0027] In a preferred embodiment, in step S6, a triplet comparative learning sample set is constructed. The expression similarity threshold for positive sample pairs is set to 0.85, and the individual difference requirement for negative sample pairs is greater than 1.2. In the maximum mutual information loss function, the mutual information penalty coefficient between expression features and identity features is set to 0.3. The gradient reversal layer applies a -0.5 scaling factor to identity-related gradients during backpropagation. The illumination interference suppression module uses spectral normalization technology to limit the energy content of high-frequency noise to less than 8% of the total spectrum.

[0028] In a preferred embodiment, in step S7, the base classification layer is configured with seven neuron outputs, corresponding to the six basic emotions plus a neutral state. Focal Loss is used to mitigate class imbalance, with a γ parameter set to 2.0. During the transfer learning phase of the enhancement layer, the base layer parameters are frozen, and the learning rate is reduced to 5e-5 on the composite expression dataset. The probabilistic credibility assessment component uses Monte Carlo dropout sampling 50 times. When the predicted entropy exceeds 1.2, a calibration function is triggered to dynamically adjust the classification boundaries.

[0029] In a preferred embodiment, in step S8, the initial unimodal training cycle is set to 30 rounds. Cross-modal adversarial examples are injected starting at the 15th round in the middle phase, with the noise perturbation amplitude linearly increasing from 5% to 20%. Decision boundary optimization uses a large-margin cosine loss function, with the margin parameter gradually increased from 0.3 to 0.6. The hard example mining module selects samples with prediction confidence levels between 0.4 and 0.6 for triple-weighted training in each round.

[0030] In a preferred embodiment, in step S9, a sliding window cache mechanism retains the last three minutes of multimodal data, the temperature parameter for online distillation is set to 2.5, and the KL divergence loss weight ratio between the teacher model and the student model is 3:1. Neural network pruning uses a global thresholding method, channels with convolution kernel L1 norms below 0.01 are removed, low-bit quantization uses a 4-bit integer format, and dynamic range calibration is performed on the activation values ​​of the feature reconstruction module.

[0031] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0032] 1. In the present invention, when visual information is lost due to partial facial occlusion or strong light exposure, the emotional features in the voice signal can automatically compensate for the lack of visual information. For example, in a video call scenario, when the user covers the chin area with his hand, the system can still accurately identify anxiety by analyzing the trembling characteristics of the voice tone and combining it with the muscle movements of the unobstructed eyebrow and eye area, thus avoiding misjudgment caused by the lack of local information. At the same time, when noisy background sounds interfere with the voice, the model can effectively filter out the interference of voice noise and maintain the accuracy of emotional judgment by capturing the duration and intensity changes of facial micro-expressions, such as slight twitching of the corners of the mouth for more than 3 seconds.

[0033] 2. In this invention, a feature decoupling mechanism is used to separate facial features from biometric features within the representation space. For example, the system will not misinterpret the deep nasolabial folds of the elderly as sad expressions, but instead confirm their true emotional state by synchronously analyzing the gentle changes in the speech formants. Furthermore, for habitual fake smiles, the model can accurately distinguish between social smiles and genuine joy by combining the flat tone of speech, which lacks emotional fluctuations. This dual verification mechanism enhances the credibility of the overall recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process principle of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] Example:

[0037] Reference Figure 1 , a face recognition method based on multimodal fusion, the method covers the following processes:

[0038] S1: Acquire facial visual data and associated acoustic signals, perform timing alignment and noise filtering, use 3D facial landmark tracking technology to standardize the facial region, and perform segmented denoising and fundamental frequency normalization on the speech waveform.

[0039] S2: The deep residual network after transfer learning is used to extract multi-dimensional facial features, and the expanded convolutional layer is combined to enhance the representation ability of micro-expression areas such as glabellar wrinkles and nasolabial grooves.

[0040] S3: Analyze the rhythmic features and spectrograms of acoustic signals, and use the dynamic time convolution architecture to capture the emotion-related voiceprint fluctuation patterns.

[0041] S4: Build a heterogeneous data interaction engine to perform multi-dimensional mapping between acoustic temporal streams and facial spatial features, and synthesize cross-modal enhanced features through a flexible attention allocation mechanism.

[0042] S5: Establish a parallel feature processing network, implement spatial-temporal modeling for visual and auditory modalities respectively, and use intelligent gated regulators to achieve dynamic integration of feature levels.

[0043] S6: Apply adversarial feature separation technology to eliminate the correlation between individual biological features and essential features of expression through the maximum mutual information loss function.

[0044] S7: Design a hierarchical classification system with the base layer identifying six core emotions, the enhancement layer parsing complex emotions, and the top layer embedding a probabilistic credibility assessment component.

[0045] S8: Implement a progressive training strategy, focusing on unimodal representations in the initial stage, introducing cross-modal adversarial examples in the middle stage, and focusing on optimizing the classification decision boundary in the later stage.

[0046] S9: Integrates a real-time optimization mechanism to dynamically update model parameters based on environmental feedback, combining neural network pruning and low-bit quantization to enable mobile deployment.

[0047] In step S1, a 68-keypoint detection model is used for real-time localization, ensuring that the face region is cropped to a uniform size of 256×256 pixels. The speech signal is sampled at a 16kHz sampling rate, with a frame length of 25ms and a 10ms overlap between frames. A Wiener filter algorithm is used to eliminate ambient noise. Frequency normalization uses a dynamic time warping algorithm to synchronize the speech signal with the video frame rate to 30fps, with a time alignment error within ±33ms.

[0048] In step S2, the transfer learning phase uses a ResNet50 network pre-trained on ImageNet as the backbone network, freezing the parameters of the first three residual blocks. During the fine-tuning phase, the learning rate is set to 1e-4. The dilated convolutional layer is configured with three parallel branches, with dilation rates set to 3, 5, and 7, respectively. The convolution kernel size is uniformly 5×5 to capture micro-expression textures at different scales. During the feature fusion phase, a channel-wise attention mechanism is employed, increasing the weight coefficient for the glabella region to 1.5 times its original value to enhance the motor characteristics of the corrugator muscles.

[0049] In step S3, the prosodic feature extraction module includes three dimensions: fundamental frequency trajectory, energy envelope, and formant bandwidth. A 25-order Mel filter bank is used to generate the time-frequency spectrum. The dynamic temporal convolutional network uses a four-layer dilated convolutional architecture, with an initial receptive field of 5 frames, which is gradually expanded to 21 frames. The activation threshold of the gated unit is set to 0.65 to suppress noise interference in low-energy areas. Voiceprint fluctuation modeling uses a bidirectional LSTM structure with a hidden layer dimension of 128 and a time step that strictly matches the video frame rate.

[0050] In step S4, the speech feature sequence and the facial feature map are first mapped to a unified spatiotemporal encoding space, and a cross-modal dynamic association is established through a deformable attention mechanism. The speech temporal features are expanded into a spatiotemporal cube through three-dimensional convolution and cross-projected onto the channel dimension of the facial feature map. During this process, a dynamic offset generation network is used to calculate the attention offset matrix, enabling the speech frame to adaptively align with sensitive areas of facial expression changes.

[0051] At the same time, spatial attention weights are calculated based on the inter-modal correlation matrix, guiding the feature reconstruction process to focus on key areas for expression discrimination. The reconstructed hybrid features are then transformed into multiple sets of enhanced basis vectors using an affine transformation. A multi-head mechanism is then used to perform cross-modal interactions at different semantic levels in parallel, ultimately fusing them into a highly discriminative joint representation. This module effectively captures the implicit associations between micro-expressions and vocal intonation by simulating the audiovisual neural integration process in human emotional perception.

[0052] The dynamic offset generation formula is:

[0053]

[0054] Among them, P is the speech temporal feature matrix, with dimensions T×D_p; F is the face spatial feature tensor, with dimensions H×W×D_f; represents the tensor outer product operation, which models the cross-modal association between speech frame features and facial spatial position features; W1 and W2 are learnable weight matrices used for nonlinear transformation of feature space; σ is the Sigmoid activation function, which controls the offset amplitude; α is the inter-modal correlation adjustment factor, which is dynamically calculated from the mutual information entropy of the speech energy spectrum and the facial action unit, and is used to enhance the projection accuracy of expression-related areas.

[0055] The multi-head affine transformation formula is:

[0056]

[0057] Among them, A_k is the speech projection matrix corresponding to the kth attention head, B_k is the face projection matrix, represents the Kronecker product operation, which fully interacts the spatial and temporal dimensions of the two modal features. vec(β) is the vectorized form of the modal coupling coefficient, β∈[0,1]^K, generated by a cross-modal gating unit to balance the contributions of speech and facial features in different expression categories. W3 is a learnable parameter of the residual connection to prevent gradient vanishing. The output V_k of each head undergoes layer normalization and is then multi-head aggregated through W4 to generate the final mixed basis vector.

[0058] In step S5, the face and voice modalities are respectively subjected to intra-modal context modeling through bidirectional LSTM and graph convolutional network to generate spatiotemporal context enhanced features; the gated weight allocation unit adopts a parallel structure of channel attention and spatial attention, first calculates the variance entropy of each modal feature as a confidence indicator, and then generates the fusion weight through the Softmax function with a learnable temperature coefficient.

[0059] In step S6, a triplet comparative learning sample set is constructed. The expression similarity threshold for positive sample pairs is set to 0.85, and the individual difference requirement for negative sample pairs is greater than 1.2. In the maximum mutual information loss function, the mutual information penalty coefficient between expression features and identity features is set to 0.3. The gradient reversal layer applies a -0.5 scaling factor to identity-related gradients during backpropagation. The illumination interference suppression module uses spectral normalization technology to limit the energy content of high-frequency noise to less than 8% of the total spectrum.

[0060] In step S7, the base classification layer is configured with seven neuron outputs, corresponding to the six basic emotions plus a neutral state. Focal Loss is used to mitigate class imbalance, with the γ parameter set to 2.0. During the transfer learning phase of the enhancement layer, the base layer parameters are frozen, and the learning rate is reduced to 5e-5 on the composite expression dataset. The probabilistic credibility assessment component uses Monte Carlo dropout sampling 50 times. When the predicted entropy exceeds 1.2, a calibration function is triggered to dynamically adjust the classification boundaries.

[0061] In step S8, the initial unimodal training cycle is set to 30 rounds. Cross-modal adversarial examples are injected starting at the 15th round in the middle phase, with the noise perturbation amplitude linearly increasing from 5% to 20%. A large-margin cosine loss function is used for decision boundary optimization, with the margin parameter gradually increased from 0.3 to 0.6. The hard example mining module selects samples with prediction confidence levels between 0.4 and 0.6 for triple-weighted training in each round.

[0062] In step S9, a sliding window cache mechanism retains the last three minutes of multimodal data. The temperature parameter for online distillation is set to 2.5, and the KL divergence loss weight ratio between the teacher model and the student model is 3:1. Neural network pruning uses a global thresholding method, removing channels with a convolution kernel L1 norm below 0.01. Low-bit quantization uses a 4-bit integer format, and dynamic range calibration is performed on the activation values ​​of the feature reconstruction module.

[0063] In the present invention, when visual information is lost due to partial facial occlusion or strong light exposure, the emotional features in the voice signal can automatically compensate for the lack of visual information. For example, in a video call scenario, when the user covers the chin area with his hand, the system can still accurately identify anxiety by analyzing the trembling characteristics of the voice tone and combining it with the muscle movements of the unobstructed eyebrow area, thus avoiding misjudgment caused by the lack of local information. At the same time, when noisy background sounds interfere with the voice, the model can effectively filter out the interference of voice noise and maintain the accuracy of emotional judgment by capturing the duration and intensity changes of facial micro-expressions, such as slight twitching of the corners of the mouth for more than 3 seconds.

[0064] In this invention, a feature decoupling mechanism is used to separate facial features from biometric features within the representational space. For example, the system will not misinterpret the deep nasolabial folds of the elderly as sad expressions, but instead confirm their true emotional state by synchronously analyzing the gentle changes in the speech formants. Furthermore, for habitual fake smiles, the model can accurately distinguish between social smiles and genuine joy by combining the flat tone of voice, which lacks emotional fluctuations. This dual verification mechanism makes the recognition results more reliable.

[0065] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0066] The above description is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A face recognition method based on multimodal fusion, characterized by: The method covers the following processes: S1: Acquire facial visual data and associated acoustic signals, perform timing alignment and noise filtering; use 3D facial landmark tracking technology to standardize the face area, and perform segmented denoising and fundamental frequency normalization on the speech waveform; S2: Using a deep residual network after transfer learning to extract multi-dimensional facial features, combined with dilated convolutional layers to enhance the representation of glabellar folds and nasolabial fold micro-expression areas; S3: Analyze the rhythmic features and spectrograms of acoustic signals and use dynamic temporal convolutional architecture to capture the emotion-related voiceprint fluctuation patterns; S4: Build a heterogeneous data interaction engine to perform multi-dimensional mapping between acoustic temporal streams and facial spatial features, and synthesize cross-modal enhanced features through a flexible attention allocation mechanism; S5: Build a parallel feature processing network to perform spatiotemporal modeling of the visual and auditory modalities, and use an intelligent gated regulator to achieve dynamic integration of feature levels. S6: Apply adversarial feature separation technology to eliminate the correlation between individual biological features and essential features of expression through the maximum mutual information loss function; S7: Design a hierarchical classification system with a base layer identifying six core emotions, an enhanced layer parsing complex emotions, and a top layer embedding a probabilistic credibility assessment component; S8: Implement a progressive training strategy, focusing on unimodal representations in the initial stage, introducing cross-modal adversarial examples in the middle stage, and focusing on optimizing the classification decision boundary in the later stage; S9: Integrates a real-time optimization mechanism to dynamically update model parameters based on environmental feedback, combining neural network pruning and low-bit quantization to enable mobile deployment.

2. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S1, a 68-keypoint detection model is used for real-time positioning to ensure that the face area is cropped to a uniform size of 256×256 pixels; the voice signal is collected at a sampling rate of 16kHz, the frame length is set to 25ms, adjacent frames overlap by 10ms, and the Wiener filter algorithm is used to eliminate environmental noise; the fundamental frequency normalization uses a dynamic time warping algorithm to synchronize the voice signal with the video frame rate to 30fps, and the time alignment error needs to be controlled within ±33ms.

3. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S2, the ImageNet pre-trained ResNet50 is used as the backbone network in the transfer learning stage, the parameters of the first three residual blocks are frozen, and the learning rate in the fine-tuning stage is set to 1e-4; the dilated convolution layer is configured with three parallel branches, the dilation rates are set to 3, 5, and 7 respectively, and the convolution kernel size is unified to 5×5 to capture micro-expression textures of different scales; The channel attention mechanism is used in the feature fusion stage to increase the weight coefficient of the glabellar area to 1.5 times the original value, thereby enhancing the movement characteristics of the corrugator muscles.

4. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S3, the rhythmic feature extraction module includes three dimensions: fundamental frequency trajectory, energy envelope, and resonance peak bandwidth, and uses a 25-order Mel filter bank to generate a time-frequency spectrum; the dynamic temporal convolutional network sets a four-layer dilated convolution structure, with an initial layer receptive field of 5 frames, which is expanded to 21 frames layer by layer, and the gate unit activation threshold is set to 0.65 to suppress noise interference in low-energy areas; the voiceprint fluctuation modeling adopts a bidirectional LSTM structure, the hidden layer dimension is set to 128, and the time step is strictly matched with the video frame rate.

5. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S4, the speech feature sequence and the facial feature map are first mapped to a unified spatiotemporal coding space, and a cross-modal dynamic association is established through a deformable attention mechanism; wherein the speech temporal features are expanded into a spatiotemporal cube through three-dimensional convolution and cross-projected with the channel dimension of the facial feature map. In this process, a dynamic offset generation network is used to calculate the attention offset matrix, so that the speech frame can adaptively align with the sensitive area of ​​facial expression changes; At the same time, the spatial attention weight is calculated based on the inter-modal correlation matrix to guide the feature reconstruction process to focus on the key areas for expression discrimination; The reorganized hybrid features are transformed into multiple sets of enhanced basis vectors through affine transformation. A multi-head mechanism is then used to perform cross-modal interactions at different semantic levels in parallel, ultimately fusing them into a highly discriminative joint representation. This module effectively captures the implicit associations between micro-expressions and changes in voice intonation by simulating the audiovisual neural integration process in human emotional perception. The dynamic offset generation formula is: Among them, P is the speech temporal feature matrix, with dimensions T×D_p; F is the face spatial feature tensor, with dimensions H×W×D_f; represents the tensor outer product operation, which models the cross-modal association between speech frame features and facial spatial position features; W1 and W2 are learnable weight matrices used for nonlinear transformation of feature space; σ is the Sigmoid activation function, which controls the offset amplitude; α is the inter-modal correlation adjustment factor, which is dynamically calculated from the mutual information entropy of the speech energy spectrum and the facial action unit, and is used to enhance the projection accuracy of expression-related areas; The multi-head affine transformation formula is: Among them, A_k is the speech projection matrix corresponding to the kth attention head, B_k is the face projection matrix, Represents the Kronecker product operation, which fully interacts the spatial-temporal dimensions of the two types of modal features; vec(β) is the vectorized form of the modal coupling coefficient, β∈[0,1]^K is generated by the cross-modal gating unit, and is used to balance the contribution weights of speech and facial features in different expression categories; W3 is a learnable parameter of the residual connection to prevent gradient disappearance; V_k output by each head is layer-normalized and then multi-head aggregated through W4 to generate the final mixed basis vector.

6. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S5, the face and voice modalities are respectively subjected to intra-modal context modeling through bidirectional LSTM and graph convolutional network to generate spatiotemporal context enhanced features; The gated weight allocation unit adopts a parallel structure of channel attention and spatial attention. It first calculates the variance entropy of each modal feature as a confidence indicator, and then generates the fusion weight through the Softmax function with a learnable temperature coefficient.

7. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S6, a triplet contrast learning sample group is constructed, the expression similarity threshold of the positive sample pair is set to 0.85, and the individual difference of the negative sample pair is required to be greater than 1.2; in the maximum mutual information loss function, the mutual information penalty coefficient of the expression feature and the identity feature is set to 0.3, and the gradient reversal layer applies a -0.5 times scaling factor to the identity-related gradient during back propagation; The light interference suppression module uses spectrum normalization technology to limit the energy proportion of high-frequency noise to less than 8% of the total spectrum.

8. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S7, the base classification layer is configured with seven neuron outputs, corresponding to six basic emotions plus a neutral state, and Focal Loss is used to alleviate the category imbalance problem, with the γ parameter set to 2.0; the base layer parameters are frozen during the transfer learning phase of the enhancement layer, and the learning rate is reduced to 5e-5 on the composite expression dataset; the probabilistic credibility assessment component is sampled 50 times based on Monte Carlo Dropout, and a calibration function is triggered when the predicted entropy value exceeds 1.2 to dynamically adjust the classification boundary.

9. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S8, the initial single-modal training cycle is set to 30 rounds, cross-modal adversarial samples are injected starting from the 15th round in the middle stage, and the noise perturbation amplitude is linearly increased from 5% to 20%; the decision boundary optimization adopts a large-margin cosine loss function, and the boundary margin parameter is gradually increased from 0.3 to 0.6; the hard example mining module screens samples with prediction confidence in the range of 0.4-0.6 in each round for triple weighted training.

10. The face recognition method based on multimodal fusion according to claim 1, characterized in that: In step S9, the sliding window cache mechanism retains the multimodal data of the last 3 minutes, the temperature parameter of the online distillation is set to 2.5, and the KL divergence loss weight ratio of the teacher model to the student model is 3:1; the neural network pruning adopts the global threshold method, channels with convolution kernel L1 norm lower than 0.01 are removed, low-bit quantization adopts 4-bit integer format, and the activation value of the feature reconstruction module is calibrated for dynamic range.

Images