Face-voice multimodal collaborative recognition method adaptive to complex illumination

Abstract

The application discloses a face-voice multimodal cooperative recognition method suitable for complex illumination, and relates to the technical field of biological feature recognition. The specific steps of the method are as follows: firstly, synchronously collecting a face image and a voice signal and performing abnormality determination; secondly, detecting an illumination type and classifying; thirdly, extracting a voice emotion keynote and a semantic feature; fourthly, combining the illumination type to generate a guide vector to enhance the face feature; fifthly, after consistency verification of lip movement and voice pronunciation, dynamically adjusting a bimodal fusion weight; and finally, generating a fusion feature to input a pre-training model to output a recognition result. The face feature is enhanced by the guide vector generated by combining the voice feature and the illumination type, and is matched with dynamic weight fusion, lip movement-voice consistency verification and low-power cooperative scheduling, so that the recognition distortion problem under complex illumination is solved, the fake attack is resisted, the precision and the power consumption are balanced, and the method is suitable for multiple scenes.

Description

Technical Field

[0001] This invention relates to the field of biometric recognition technology, specifically to a face-voice multimodal collaborative recognition method adapted to complex lighting conditions. Background Technology

[0002] Biometric recognition technology, with its advantages of requiring no additional credentials and offering convenient and efficient identification processes, has been deeply integrated into various scenarios in daily life and industrial production, covering areas such as access control and security, mobile terminal unlocking, online payment verification, and in-vehicle intelligent interaction. Among them, facial recognition is widely favored due to its intuitiveness and fast recognition speed, while voice recognition has unique advantages in scenarios with occlusion and long distances. The multimodal collaborative recognition scheme combining the two can improve the reliability of recognition through feature complementarity, becoming the mainstream development direction in the industry in recent years. With the popularization of various smart devices, the application environment faced by recognition technology is becoming increasingly diverse. As a key environmental factor affecting the effectiveness of facial feature extraction, changes in lighting conditions directly affect the actual application performance of multimodal recognition systems, driving related technologies to continuously optimize for adaptability to complex environments.

[0003] Traditional face-voice multimodal recognition schemes have significant shortcomings in practical applications: When faced with complex lighting conditions such as strong light overexposure, weak light noise, and backlighting with shadows, most schemes only perform general enhancement processing on face images, failing to establish a linkage mechanism between lighting type and feature extraction. This makes it difficult to effectively solve the problem of facial feature distortion, resulting in significant fluctuations in recognition accuracy. Furthermore, most schemes use a fixed-ratio feature fusion method, unable to dynamically adjust the weight ratio of face and voice features according to changes in lighting, further affecting recognition stability in complex environments. In addition, traditional schemes lack targeted anti-spoofing verification designs, making them vulnerable to malicious attacks such as synthesized speech and forged faces. Moreover, when used on low-power devices, it is often difficult to balance recognition accuracy and power consumption; either ensuring accuracy leads to shortened battery life, or reducing power consumption sacrifices recognition performance, limiting the applicability of the technology. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a face-speech multimodal collaborative recognition method adapted to complex lighting conditions. This method deeply integrates face-speech multimodal collaboration with complex lighting adaptation to construct a full-link recognition system, effectively solving the problem of insufficient recognition accuracy under complex lighting conditions. First, face and speech data are collected simultaneously and anomaly detection is performed. Then, lighting type is detected and classified, and speech emotional tone and semantic features are extracted. A guiding vector is generated based on the lighting type to drive the network to adaptively enhance face features. Consistency verification between lip movement and speech pronunciation is used to resist forgery attacks. The dual-modal fusion weights are dynamically adjusted according to lighting conditions, combined with a pre-trained model for comprehensive judgment, and a low-power scheduling strategy is incorporated. The entire method achieves accurate recognition under complex lighting conditions, balancing security and practicality.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a face-speech multimodal collaborative recognition method adapted to complex lighting conditions, the specific steps of which are as follows:

[0006] S1, Data Acquisition and Anomaly Detection: Simultaneously acquire the face image and voice signal of the object to be identified, detect the blurriness of the face image and the signal-to-noise ratio of the voice signal, determine abnormal data and trigger secondary acquisition, and output prompt information if abnormal data is still present.

[0007] S2, Illumination Level Detection: Illumination level detection is performed on the valid face image to determine the type of illumination interference as strong light overexposure, weak light noise, backlight shadow or normal lighting.

[0008] S3, Speech Feature Extraction: Preprocess the effective speech signal and extract the emotional tone features and semantic features of the speech;

[0009] S4, Facial Feature Enhancement: Based on the type of illumination interference and speech features, an illumination compensation guidance vector is generated and input into the facial feature extraction network to drive the network to perform adaptive feature enhancement on the illuminated areas of the face.

[0010] S5, Consistency Check: Extract the temporal features of facial lip movement and speech pronunciation, and determine the authenticity of the data through consistency check. If the check passes, continue; otherwise, output recognition failure.

[0011] S6, Feature Fusion and Recognition: Dynamically adjust the fusion weights of core facial features and core speech features, generate cross-modal fusion features through weighted fusion, input them into the pre-trained recognition model, and output the identity recognition result.

[0012] Furthermore, in step S3, the emotional tone features are extracted by fusing the Mel frequency cepstral coefficients with six prosodic features: speech rate, pitch mean, pitch variance, number of pauses, longest pause time, and average pause time, generating a 12-dimensional static feature and an 18-dimensional vector after normalization of the six-dimensional prosodic features; the semantic features are extracted by using a lightweight DistilBERT model to extract identity-related keyword encoding vectors, which are then normalized by L2 to generate a 32-dimensional vector; together, they constitute a 50-dimensional core speech feature.

[0013] Furthermore, the process of generating the illumination compensation guidance vector in step S4 is as follows: a 4×50-dimensional illumination interference type-speech feature mapping matrix is ​​constructed, with the matrix rows corresponding to four illumination types and the columns corresponding to 50-dimensional core speech features. The matrix elements are feature association weights in the range of 0 to 1. Strong light overexposure is associated with stability and name confidence features, weak light noise is associated with clarity and semantic integrity features, backlight shadow is associated with coherence and key information integrity features, and normal illumination is evenly weighted. The association score is calculated through an attention mechanism, and the 18-dimensional region enhancement weights and 32-dimensional detail repair parameters are split, spliced ​​and normalized to form a 64-dimensional guidance vector.

[0014] Furthermore, in step S4, the face feature extraction network is a CNN architecture with an embedded illumination-invariant feature module. The core structure includes convolutional layers with 3×3 or 5×5 kernels, 2×2 max pooling layers, an attention fusion module, and fully connected layers. The attention fusion module receives a 64-dimensional guiding vector, which is decomposed into a 32-dimensional region weight vector and a 32-dimensional channel weight vector, respectively mapped to a 16×16 spatial attention heatmap and channel weights. For overexposed regions with heatmap values ​​less than 0.3, texture extraction is enhanced, and for shadow regions with heatmap values ​​greater than 0.7, edge contrast is enhanced. Finally, 128-dimensional L2-normalized core face features are output.

[0015] Furthermore, the dynamic weight adjustment in step S6 employs an adaptive algorithm based on the average brightness value, and the calculation formula is as follows:

[0016] When exposed to strong light: ;

[0017] In the case of weak light noise: ;

[0018] Backlighting and shadows: ;

[0019] Under normal lighting conditions: ;

[0020] Wherein, Wf is the face feature weight, Wv is the voice feature weight, and Mean is the average brightness of the effective face image. The Mean range corresponding to the lighting type is as follows: strong light overexposure Mean is greater than 220, weak light noise Mean is less than 50, and backlight shadow and normal lighting Mean is between 50 and 220. The weight values ​​are limited to Wf being between 0.3 and 0.7, and Wv being between 0.3 and 0.7.

[0021] Furthermore, in step S6, the 50-dimensional voice core features are first padded to 128 dimensions, and then fused with the 128-dimensional face core features using weighted averages Wf and Wv. The fusion formula is as follows: If the variance of facial features is less than 0.1 or the confidence of speech semantics is less than 0.5, switch to speech-dominated mode and adjust Wv=0.8 and Wf=0.2; where FusF is the cross-modal fusion feature, Ff is the core facial feature, Wv is the speech feature weight, and Fv is the core speech feature.

[0022] Furthermore, the consistency verification process in step S5 is as follows: extracting a 30-dimensional lip movement temporal feature vector X from 20 key points of the face lips, and a 20-dimensional pronunciation temporal feature vector Y from the speech; calculating the similarity using a dynamic time warping algorithm, the formula is: ;in, 1 is the consistency similarity value between the temporal features of facial lip movement and the temporal features of speech pronunciation; DTWxy is the minimum cumulative distance; maxl is the maximum length of X and Y, and avgd is the average of the pairwise distances between corresponding elements of X and Y; Sim is greater than or equal to 0.75 to be considered as passing the verification, otherwise the recognition fails.

[0023] Furthermore, in step S6, the pre-trained recognition model is a support vector machine model, trained with 5,000 identities and 200,000 sets of face-voice pairing samples. The kernel function is a radial basis function, with a penalty coefficient C=10 and gamma=0.01, and the accuracy is not less than 92.5%. During recognition, the cosine similarity CosS between FusF and template features is calculated, and combined with Sim for comprehensive judgment: if CosS is greater than or equal to 0.85 and Sim is greater than or equal to 0.75, the match is successful; if CosS is less than 0.75 or Sim is less than 0.75, the match fails; if it is between the two, a second recognition is triggered.

[0024] Furthermore, in step S1, the low-power scheduling strategy is as follows: when the device battery level is less than 30%, the camera frame rate is reduced to 15fps and the microphone sampling rate is reduced to 8kHz; in step S3, the semantic feature extraction model is replaced with a combination model of TF-IDF and Naive Bayes, which outputs a 16-dimensional feature vector; in step S4, the face feature extraction network adopts INT8 quantized inference, which reduces power consumption by 40% and reduces the recognition accuracy by no more than 3 percentage points.

[0025] Compared with existing technologies, this face-speech multimodal collaborative recognition method adapted to complex lighting conditions has the following advantages:

[0026] I. This invention generates targeted illumination compensation guidance vectors through deep collaboration between speech features and illumination interference types, driving the facial feature extraction network for adaptive enhancement. This fundamentally solves the core problem of blurred and distorted facial features under complex lighting conditions. First, the emotional tone and semantic features of the speech are extracted. Then, based on different illumination types, feature mapping and attention mechanisms are used to split region enhancement weights and detail repair parameters, allowing the network to accurately enhance textures in overexposed areas and improve contrast in shadow areas. Combined with dynamic weight adjustment based on the average brightness, the dual-modal fusion ratio is flexibly adapted to different scenarios such as strong light and weak light. Regardless of whether it's strong light with overexposure or weak light with noise, it can maintain a stable high recognition accuracy. Compared to traditional single feature enhancement or fixed-ratio fusion schemes, its adaptability and anti-interference effect are significantly improved.

[0027] Second, this invention, through consistency verification of lip movement and speech pronunciation, combined with a low-power collaborative design across the entire chain, not only addresses the anti-spoofing shortcomings of multimodal recognition but also effectively adapts to the usage requirements of mobile terminals. Utilizing a dynamic time warping algorithm to determine the synchronization between lip movement and speech pronunciation, it can effectively intercept malicious attacks such as synthesized speech and fake faces, making the recognition process safer and more reliable. For low-power scenarios, it optimizes layer by layer from hardware acquisition and algorithm models to network inference, reducing the operating load of the acquisition hardware while employing a lightweight feature extraction model and quantized inference method. This maximizes the preservation of recognition accuracy while reducing power consumption, preventing a significant drop in accuracy.

[0028] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0030] Figure 1 This is an overall flowchart of the method of the present invention;

[0031] Figure 2 Flowchart for facial feature enhancement;

[0032] Figure 3 This is a flowchart of the feature fusion and recognition process. Detailed Implementation

[0033] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0034] Example

[0035] This embodiment discloses a specific implementation process of a face-voice multimodal collaborative recognition method adapted to complex lighting conditions. Applied to intelligent access control scenarios, this method simultaneously acquires face images and voice signals, and performs end-to-end processing including anomaly detection, lighting level detection, voice feature extraction, face feature enhancement, consistency verification, feature fusion, and recognition. Combined with dynamic weight adjustment, low-power collaborative scheduling, and anti-spoofing verification mechanisms, it achieves accurate, secure, and low-power recognition under complex lighting environments. This embodiment details the working principles, operational processes, and application logic of each module, fully presenting the implementation process of the technical solution and verifying the adaptability and practicality of the method in complex scenarios.

[0036] The overall process of this method is as follows: Figure 1 As shown, the specific implementation process is as follows:

[0037] S1, Data Acquisition and Anomaly Detection

[0038] This embodiment uses an intelligent access control terminal equipped with a high-definition camera and a high-fidelity microphone. After the recognition function is activated, the terminal simultaneously acquires the facial image and voice signal of the object to be recognized. During the acquisition process, the system detects the blurriness of the facial image and the signal-to-noise ratio of the voice signal in real time: the image sharpness evaluation algorithm is used to analyze the edge contour sharpness of the facial image to determine whether the feature is unusable due to inaccurate focus or motion blur; the voice signal amplitude analysis and noise separation algorithm is used to extract the effective voice components and calculate the signal-to-noise ratio to determine whether the voice feature is distorted due to excessive environmental noise.

[0039] If the detection result is determined to be abnormal data, the system triggers a secondary acquisition mechanism to synchronously acquire the face image and voice signal again. If the data after the secondary acquisition still does not meet the valid standard, it indicates that the current acquisition environment cannot meet the recognition requirements. The system outputs the message "Acquisition environment abnormal, please adjust the environment and try again," terminating the current recognition process. If the acquired data is determined to be valid, it is transmitted to the subsequent processing module to enter the illumination level detection stage.

[0040] For mobile terminals in low-battery scenarios, this embodiment introduces a low-power scheduling strategy: when the remaining battery power obtained by the device's power detection module is lower than a set threshold, the hardware acquisition parameters are automatically adjusted to reduce the camera frame rate and microphone sampling rate. This reduces hardware power consumption while ensuring the basic validity of the acquired data, providing a qualified data source for subsequent feature extraction.

[0041] S2, Illumination Level Detection

[0042] After a valid face image is transmitted to the illumination level detection module, the system first calculates the mean and variance of the image's brightness. The mean brightness is obtained by statistically analyzing the average grayscale value of all pixels in the image, reflecting the overall illumination intensity; the variance is obtained by calculating the degree of deviation of the grayscale value of each pixel from the mean, reflecting the uniformity of illumination distribution.

[0043] Based on the calculation results of the mean and variance of brightness, the system determines the type of light interference: when the mean brightness is greater than the set threshold, it is determined to be strong light overexposure; when the mean brightness is less than the set threshold, it is determined to be weak light noise; when the mean brightness is in the middle range and the variance is greater than the set threshold, it is determined to be backlight shadow; when the mean brightness is in the middle range and the variance is less than the set threshold, it is determined to be normal lighting.

[0044] The illumination level detection results will serve as the core basis for subsequent facial feature enhancement and dynamic weight adjustment, providing basic parameters for targeted processing under different illumination scenarios, and ensuring that the feature extraction and fusion process adapts to the current illumination conditions.

[0045] S3, Speech Feature Extraction

[0046] The valid speech signal enters the preprocessing stage. First, pre-emphasis processing enhances high-frequency signal components. Then, frame segmentation divides the continuous speech signal into short-time frame sequences. Finally, a Hanning window is added to each frame to reduce spectral leakage. The preprocessed speech signal is then used for feature extraction in two ways:

[0047] The process involves extracting emotional tone features: Each frame of speech signal is converted into spectral features using Fourier transform. These spectral features are then mapped to a preset number of Mel filter banks, and the output energy of each filter is calculated. After taking the logarithm of the energy value, a discrete cosine transform is performed to extract the first few dimensions as static Mel frequency cepstral coefficient features. Simultaneously, a speech rhythm analysis algorithm is used to extract prosodic features such as speech rate, pitch mean, pitch variance, number of pauses, longest pause time, and average pause time. The static Mel frequency cepstral coefficient features and prosodic features are normalized separately, and after eliminating dimensional differences, they are fused to generate an emotional tone feature vector.

[0048] Another approach to extract semantic features involves converting speech signals into text information using a speech-to-text tool, filtering out identity-related keywords from the text, and inputting these keywords into a lightweight DistilBERT model. The keywords are then converted into low-dimensional dense vectors through the model's embedding layer, and further processed and enhanced by a multi-layer Transformer encoder. Finally, the model's output CLStoken vector is taken, and after L2 normalization, a semantic feature vector is generated, eliminating the impact of vector length differences on subsequent calculations.

[0049] After fusing the emotional tone feature vector and the semantic feature vector, the core speech feature vector is formed, providing core data support for the generation of illumination-compensated guided vectors and feature fusion. In low-power scenarios, the semantic feature extraction model is replaced by a combination of TF-IDF and Naive Bayes. The feature vectors of text keywords are extracted using the TF-IDF algorithm, and after optimization by the Naive Bayes model, a low-dimensional semantic feature vector is output, reducing computational load while ensuring the effectiveness of semantic features.

[0050] S4, Facial Feature Enhancement

[0051] Based on the illumination interference type obtained in S2 and the core speech features obtained in S3, the system generates an illumination compensation guidance vector as follows: Figure 2 As shown: First, construct the illumination interference type-speech feature mapping matrix. The rows of the matrix correspond to four types: strong light overexposure, weak light noise, backlight shadow, and normal illumination. The columns correspond to the dimensions of the core speech feature vector. The matrix elements are the feature association weights obtained by fitting the training set, with values ​​ranging from 0 to 1.

[0052] The association logic between different lighting types and speech features is as follows: In strong light overexposure scenarios, the focus is on associating the stability feature in the emotional tone feature and the name confidence feature in the semantic feature to ensure that feature enhancement focuses on core identity information; in low light and noisy scenarios, the focus is on associating the clarity feature in the emotional tone feature and the semantic integrity feature in the semantic feature to compensate for the distortion of speech signals under low light; in backlight and shadow scenarios, the focus is on associating the coherence feature in the emotional tone feature and the key information integrity feature in the semantic feature to strengthen the identity features in the shadow area; in normal lighting scenarios, the association weights are evenly distributed to achieve balanced utilization of features in all dimensions.

[0053] The association score between speech features and illumination type is calculated using an attention mechanism. This score is obtained by multiplying the core speech feature vector by the corresponding row of the mapping matrix, then dividing by the sum of the elements in that row, ensuring the score remains within a reasonable range. The association score is then decomposed into region enhancement weights and detail restoration parameters. After dimensionality concatenation and normalization, these are combined to form an illumination compensation guidance vector, providing precise enhancement guidance for the face feature extraction network.

[0054] The face feature extraction network adopts a CNN architecture with embedded illumination-invariant feature modules. The network operation process is as follows: The input layer receives a valid face image. The first convolutional layer uses small-sized convolutional kernels to perform preliminary feature extraction on the image, capturing basic edge and texture information. Then, the first pooling layer performs downsampling to reduce feature dimensions and retain key information. The second and third convolutional layers continue to use small-sized convolutional kernels to gradually deepen feature extraction and enhance identity-related detailed features. The second pooling layer further compresses the feature map size. The fourth and fifth convolutional layers use large-sized convolutional kernels to expand the receptive field of view, fuse local and global features, and form a high-dimensional feature representation.

[0055] After receiving the illumination compensation guidance vector, the attention fusion module decomposes it into a region weight vector and a channel weight vector. The region weight vector is mapped to a spatial attention heatmap with the same size as the output feature map of the fifth convolutional layer. Feature extraction weights are assigned based on the differences in heatmap values. Texture extraction is enhanced in overexposed areas with smaller heatmap values ​​to compensate for the loss of details caused by overexposure. Edge contrast is enhanced in shadow areas with larger heatmap values ​​to highlight identity features under shadow. The channel weight vector is used to adjust the weights of each channel in the fifth convolutional layer, focusing on activating feature channels that are strongly related to identity recognition and suppressing interference from irrelevant channels.

[0056] The feature map after attention fusion is input into the first fully connected layer, where a large number of neurons perform feature integration and dimensionality transformation. The ReLU activation function is used to introduce non-linear features and improve the network's expressive power. Then, it is input into the second fully connected layer to further compress the feature dimension. The Sigmoid activation function is used to map the feature values ​​to a specific interval. Finally, the core facial feature vector is output after L2 normalization to ensure the consistency and comparability of the feature vectors.

[0057] In low-power scenarios, the face feature extraction network adopts a quantization inference method, which quantizes the network parameters from high precision to low precision. While preserving the feature distribution, it reduces the amount of computation and storage requirements, thereby reducing power consumption and ensuring that the recognition accuracy does not drop significantly.

[0058] S5, Consistency Check

[0059] To determine the authenticity of the collected data and avoid malicious attacks such as synthesized speech and fake faces, this embodiment extracts the timing features of facial lip movements and speech pronunciation through a consistency verification module and performs synchronization matching verification.

[0060] First, the key points of the lips of the object to be identified are extracted using a facial lip key point detection algorithm, and the coordinate change sequence of the key points within a set time period is obtained to form the original lip movement time series data. If there are missing values ​​in the original data due to occlusion or collection angle issues, linear interpolation is used to complete them to ensure the continuity of the data. Then, the dimensionality reduction of the completed coordinate sequence is performed using a principal component analysis algorithm to remove redundant information, retain the core lip movement features, and generate the lip movement time series feature vector X.

[0061] Simultaneously, articulation timing features are extracted from the speech signal, including fundamental frequency change rate, vowel duration, and other features strongly correlated with articulation rhythm, forming an articulation timing feature vector Y. A dynamic time warping algorithm is used to calculate the similarity between vector X and vector Y. This algorithm uses dynamic programming to find the optimal matching path between the two timing sequences and calculates the minimum cumulative distance DTWxy to measure the difference in their temporal synchronization.

[0062] The similarity calculation formula is Sim = 1 - DTWxy / (maxl × avgd), where Sim is the consistency similarity value between the temporal features of facial lip movement and the temporal features of speech pronunciation; maxl is the maximum length of vector X and vector Y; and avgd is the average of the pairwise distances between all corresponding elements of vector X and vector Y. This formula normalizes the minimum cumulative distance to the similarity value Sim, which ranges from 0 to 1. When Sim is greater than or equal to the set threshold, it indicates good synchronization between lip movement and pronunciation, and the data is judged to be genuine, passes the verification, and proceeds to the subsequent feature fusion stage. When Sim is less than the set threshold, it indicates that the data is suspected of being forged, fails the verification, the system outputs a recognition failure message, and terminates the recognition process.

[0063] S6, Feature Fusion and Recognition

[0064] After the consistency check passes, the process proceeds to the feature fusion and recognition stage, such as... Figure 3 As shown in the diagram. First, feature dimension adaptation is performed. Since the core feature vectors of the face and the core feature vectors of the speech have different dimensions, zero-padding is performed on the core feature vectors of the speech to ensure the effectiveness of the fusion, so that their dimensions are consistent with those of the core feature vectors of the face.

[0065] Subsequently, an adaptive algorithm based on the mean brightness is used to dynamically adjust the fusion weights. The core logic of the weight adjustment is to allocate the fusion ratio of facial core features and speech core features according to the differences in lighting conditions, so that the fused features are more adapted to the current environment. The specific calculation formula is as follows:

[0066] When overexposed in strong light, Wf = 0.7 - 0.4 × (Mean - strong light threshold) / (maximum brightness value - strong light threshold), Wv = 1 - Wf;

[0067] When there is low light noise, Wf = 0.6 - 0.2 × (low light threshold - Mean) / low light threshold, Wv = 1 - Wf;

[0068] When using backlighting and shadows, Wf = a fixed value, Wv = 1 - Wf;

[0069] Under normal lighting conditions, Wf = 0.6 + 0.1 × (Mean - Normal lighting baseline value) / (Normal lighting upper limit value - Normal lighting baseline value), Wv = 1 - Wf;

[0070] Where Wf is the face feature weight, Wv is the voice feature weight, and Mean is the average brightness of the effective face image. The weight values ​​are limited to a specific range to ensure that both modalities can play a role and avoid recognition bias caused by the dominance of a single modality.

[0071] When the detected facial feature variance is less than the set value, indicating insufficient facial feature discrimination, or when the speech semantic confidence is less than the set value, indicating low speech feature reliability, the system automatically switches to speech-dominated mode, adjusting Wv to a higher value and Wf to a lower value, thereby compensating for the shortcomings of a single modality by strengthening the role of speech features.

[0072] Feature fusion is achieved through a weighted summation formula: FusF = Wf × Ff + Wv × Fv, where FusF is the cross-modal fused feature vector, Ff is the core facial feature vector, and Fv is the adapted core speech feature vector. This formula integrates the two modal features with dynamic weights to generate a fused feature vector that combines facial uniqueness with speech interference resistance, thus improving recognition accuracy.

[0073] A pre-trained recognition model using fusion feature vector inputs is employed. This model is a support vector machine (SVM) model, and the training process utilizes a large number of identity samples covering face-voice pairing data under four different lighting scenarios to ensure the model's adaptability to complex lighting environments. The model kernel function employs a radial basis function (RBF), and the penalty coefficient C balances the model's fitting and generalization abilities. The gamma parameter adjusts the influence range of the kernel function. After multiple iterations of training, the model's accuracy reaches the set standard.

[0074] During the identification and matching process, the cosine similarity (CosS) between the fused feature vector and the identity template feature vector stored in the database is calculated. This is combined with the Sim value obtained from the consistency check for a comprehensive judgment: when CosS is greater than or equal to a set high threshold and Sim is greater than or equal to a check threshold, it indicates a high feature matching degree and reliable data, and a successful match result and corresponding identity identifier are output; when CosS is less than a set low threshold or Sim is less than a check threshold, it indicates insufficient feature matching degree or data risk, and a matching failure message is output; when CosS is between the low and high thresholds and Sim is greater than or equal to the check threshold, it indicates that the feature matching degree is close to the threshold and further verification is required. The system outputs a "Please add voice command and try again" message, triggering a secondary identification process. This secondary identification process is triggered at most once to ensure the accuracy of the identification results.

[0075] In summary, this embodiment achieves multimodal collaborative face-speech recognition adapted to complex lighting conditions through a complete process. Each stage, from data acquisition to final recognition, is tightly integrated to form a closed-loop processing mechanism. By linking illumination level detection with speech features, adaptive enhancement of facial features is achieved, solving the problem of facial feature distortion under complex lighting conditions. Consistency verification between lip movement and pronunciation effectively resists malicious attacks, improving recognition security. Dynamic weight adjustment and end-to-end low-power optimization balance recognition accuracy and power consumption, adapting to the needs of various application scenarios.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A face-speech multimodal collaborative recognition method adapted to complex lighting conditions, characterized in that, The specific steps of this method are as follows: S1, Data Acquisition and Anomaly Detection: Simultaneously acquire the face image and voice signal of the object to be identified, detect the blurriness of the face image and the signal-to-noise ratio of the voice signal, determine abnormal data and trigger secondary acquisition, and output prompt information if abnormal data is still present. S2, Illumination Level Detection: Illumination level detection is performed on the valid face image to determine the type of illumination interference as strong light overexposure, weak light noise, backlight shadow or normal lighting. S3, Speech Feature Extraction: Preprocess the effective speech signal and extract the emotional tone features and semantic features of the speech; S4, Facial Feature Enhancement: Based on the type of illumination interference and speech features, an illumination compensation guidance vector is generated and input into the facial feature extraction network to drive the network to perform adaptive feature enhancement on the illuminated areas of the face. S5, Consistency Check: Extract the temporal features of facial lip movement and speech pronunciation, and determine the authenticity of the data through consistency check. If the check passes, continue; otherwise, output recognition failure. S6, Feature Fusion and Recognition: Dynamically adjust the fusion weights of core facial features and core speech features, generate cross-modal fusion features through weighted fusion, input them into the pre-trained recognition model, and output the identity recognition result.

2. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 1, characterized in that, The emotional tone features in step S3 are extracted by fusing the Mel frequency cepstral coefficients with six prosodic features: speech rate, speech mean, speech variance, number of pauses, longest pause time, and average pause time, generating an 18-dimensional vector after normalizing the 12-dimensional static features and the 6-dimensional prosodic features. Semantic features are extracted by using a lightweight DistilBERT model to extract identity-related keyword encoding vectors, which are then normalized by L2 to generate 32-dimensional vectors; together, they constitute 50-dimensional core speech features.

3. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 2, characterized in that, The process of generating the illumination compensation guidance vector in step S4 is as follows: Construct a 4×50-dimensional illumination interference type-speech feature mapping matrix, where the matrix rows correspond to four illumination types, the columns correspond to 50-dimensional speech core features, and the matrix elements are feature association weights in the range of 0 to 1; strong light overexposure is associated with stability and name confidence features, weak light noise is associated with clarity and semantic integrity features, backlight shadow is associated with coherence and key information integrity features, and normal illumination is evenly weighted. The association score is calculated through an attention mechanism, and the 18-dimensional region enhancement weights and 32-dimensional detail repair parameters are split and concatenated and normalized to form a 64-dimensional guiding vector.

4. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 3, characterized in that, In step S4, the face feature extraction network is a CNN architecture with an embedded illumination-invariant feature module. The core structure includes a convolutional layer with 3×3 or 5×5 convolutional kernels, a 2×2 max pooling layer, an attention fusion module, and a fully connected layer. The attention fusion module receives a 64-dimensional guiding vector, which is decomposed into a 32-dimensional region weight vector and a 32-dimensional channel weight vector. These are mapped to a 16×16 spatial attention heatmap and channel weights, respectively. For overexposed regions with heatmap values ​​less than 0.3, texture extraction is enhanced, and for shadow regions with heatmap values ​​greater than 0.7, edge contrast is enhanced. Finally, 128-dimensional L2-normalized core facial features are output.

5. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 1, characterized in that, In step S6, the dynamic weight adjustment uses an adaptive algorithm based on the average brightness value, and the calculation formula is as follows: When exposed to strong light: ; In the case of weak light noise: ; Backlighting and shadows: ; Under normal lighting conditions: ; Wherein, Wf is the face feature weight, Wv is the voice feature weight, and Mean is the average brightness of the effective face image. The Mean range corresponding to the lighting type is as follows: strong light overexposure Mean is greater than 220, weak light noise Mean is less than 50, and backlight shadow and normal lighting Mean is between 50 and 220. The weight values ​​are limited to Wf being between 0.3 and 0.7, and Wv being between 0.3 and 0.

7.

6. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 5, characterized in that, In step S6, the 50-dimensional speech core features are first padded to 128 dimensions, and then fused with the 128-dimensional face core features using weighted Wf and Wv. The fusion formula is as follows: If the variance of facial features is less than 0.1 or the confidence of speech semantics is less than 0.5, switch to speech-dominated mode and adjust Wv=0.8 and Wf=0.2; where FusF is the cross-modal fusion feature, Ff is the core facial feature, Wv is the speech feature weight, and Fv is the core speech feature.

7. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 1, characterized in that, The consistency verification process in step S5 is as follows: Extract the 30-dimensional lip movement temporal feature vector X from 20 key points of the face lips, and the 20-dimensional pronunciation temporal feature vector Y from the speech; calculate the similarity using the dynamic time warping algorithm, with the formula: ;in, 1 is the consistency similarity value between the temporal features of facial lip movement and the temporal features of speech pronunciation; DTWxy is the minimum cumulative distance; maxl is the maximum length of X and Y, and avgd is the average of the pairwise distances between corresponding elements of X and Y; Sim is greater than or equal to 0.75 to be considered as passing the verification, otherwise the recognition fails.

8. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 7, characterized in that, In step S6, the pre-trained recognition model is a support vector machine model, trained with 5,000 identities and 200,000 sets of face-voice pairing samples. The kernel function is a radial basis function, with a penalty coefficient C=10 and gamma=0.01, and the accuracy is not less than 92.5%. During recognition, the cosine similarity CosS between FusF and template features is calculated, and Sim is used for comprehensive judgment: if CosS is greater than or equal to 0.85 and Sim is greater than or equal to 0.75, the match is successful; if CosS is less than 0.75 or Sim is less than 0.75, the match fails; if it is between the two, a second recognition is triggered.

9. The face-speech multimodal collaborative recognition method adapting to complex lighting conditions according to claim 1, characterized in that, In step S1, the low-power scheduling strategy is as follows: when the device battery is less than 30%, the camera frame rate is reduced to 15fps and the microphone sampling rate is reduced to 8kHz; in step S3, the semantic feature extraction model is replaced with a combination model of TF-IDF and Naive Bayes, which outputs a 16-dimensional feature vector; in step S4, the face feature extraction network adopts INT8 quantization inference, which reduces power consumption by 40% and reduces the recognition accuracy by no more than 3 percentage points.

Images