Face video image processing method

Abstract

The invention discloses a face video image processing method. The method comprises the following steps: acquiring a face video sequence; extracting facial time sequence features from the facial video sequence, wherein the facial time sequence features comprise a heart rate feature, a lip movement feature and a micro-expression feature; performing feature enhancement on the extracted face time sequence features by using a feature enhancement network to obtain enhanced feature representation; compressing the enhanced feature representation into a low-dimensional feature vector through multi-layer nonlinear transformation by using a feature compression network; and outputting the low-dimensional feature vector. According to the method, through extraction and processing of multiple face time sequence features, feature enhancement and feature compression are adopted to improve feature quality and calculation efficiency, and the method has relatively high feature extraction accuracy and processing efficiency.

Description

Technical Field

[0001] This application relates to the field of video image processing, and more particularly to a method for processing human face video images. Background Technology

[0002] With the development of computer vision technology, facial video image processing has been widely applied in fields such as remote authentication, human-computer interaction, and medical diagnosis. Currently, processing methods based on temporal facial features are gradually gaining attention. These methods extract effective information by analyzing facial physiological features and can be used for various downstream tasks.

[0003] However, existing methods for processing facial video images have the following problems: 1. Unstable feature extraction. Existing methods are quite sensitive to environmental factors such as lighting and pose, and are easily disturbed, affecting the quality of features. Especially when extracting subtle features such as heart rate and micro-expressions, environmental noise can cause feature distortion.

[0004] 2. Limited feature representation capabilities. Traditional methods mainly focus on static features or simple dynamic features, making it difficult to fully capture the rich semantics in facial temporal information. For example, they cannot effectively model the correlations between multimodal features such as heart rate, lip movements, and micro-expressions.

[0005] 3. High feature redundancy. Existing methods often extract features with excessively high dimensionality and contain redundant information, increasing storage and computational overhead and hindering practical applications. This is especially true in scenarios with limited computing resources, such as mobile devices, where the processing efficiency of high-dimensional features is low.

[0006] Therefore, designing a robust, expressive, and computationally efficient face video image processing method is crucial for improving the performance of downstream tasks. This can not only enhance the security of remote authentication and improve the naturalness of human-computer interaction, but also provide more reliable feature support for scenarios such as medical diagnosis. Summary of the Invention

[0007] This application proposes a face video image processing method based on facial temporal features. By extracting and analyzing heart rate, lip movement and micro-expression features, and combining feature enhancement network and feature compression network, it achieves face video image processing with strong robustness, high expressive power and good computational efficiency.

[0008] According to one embodiment of this application, a face video image processing method is proposed, comprising: Capture facial video sequences; Facial temporal features, including heart rate features, lip movement features, and micro-expression features, are extracted from the facial video sequence. The extracted facial temporal features are enhanced using a feature enhancement network to obtain the enhanced feature representation; A feature compression network is used to compress the enhanced feature representation into a low-dimensional feature vector through multi-layer nonlinear transformation. Output the low-dimensional feature vector.

[0009] In some implementations, extracting heart rate features includes: Based on the facial video sequence, multiple regions of interest are located, and pixel change sequences of each region are obtained through pixel tracking. An adaptive window filter is applied to the acquired pixel change sequence to eliminate ambient light interference and obtain the filtered sequence. The filtered sequence is passed through a multi-scale feature pyramid to obtain a multi-layer sequence, and the time-domain and frequency-domain features of each layer are extracted. The extracted time-domain and frequency-domain features of each layer are weighted to obtain the heart rate features.

[0010] In some implementations, extracting lip movement features includes: Based on the facial video sequence, the key point coordinate sequence of the lip region is obtained, and the key point coordinate sequence is segmented according to a preset time window. Based on the coordinate change of adjacent time windows, the key point coordinate change sequence is obtained. Extract the audio feature sequence corresponding to the facial video sequence; Unify the key point coordinate change sequence and the audio feature sequence to the same time scale; The cross-correlation coefficient between the key point coordinate change sequence and the audio feature sequence at the same time scale is calculated to obtain the lip movement features.

[0011] In some implementations, the cross-correlation coefficient R(τ) between the keypoint coordinate change sequence and the audio feature sequence at the same time scale is calculated according to the following formula: R(τ)=∑ i ∑ j [E[(Li(t)-μi)(Aj(t+τ)-μj)] / σ i σ j ], Where i is the dimension index of the vector in the keypoint coordinate change sequence, j is the dimension index of the vector in the audio feature sequence, and L i (t) represents the value of the i-th dimension in the keypoint coordinate change sequence at time t, A j (t) represents the value of the j-th dimension of the audio feature sequence at time t, μ i Let σ be the mean of the i-th dimension in the sequence of keypoint coordinate changes. i Let μ be the standard deviation of the i-th dimension in the keypoint coordinate change sequence. jLet σ be the mean of the j-th dimension in the audio feature sequence. j Let be the standard deviation of the j-th dimension in the audio feature sequence, τ be the time delay parameter, and E[] denote the mathematical expectation operator.

[0012] In some implementations, extracting micro-expression features includes: Based on the facial video sequence, facial muscle motion units are analyzed to obtain facial muscle motion parameters; Spatiotemporal feature maps are constructed using the acquired muscle movement parameters; The spatiotemporal feature map is processed by a feature extraction network to obtain spatiotemporal features; Based on the extracted spatiotemporal features, the trend of facial expression changes is analyzed to obtain micro-expression features.

[0013] In some embodiments, the feature enhancement network includes a feature generator and a feature optimizer. The feature enhancement network is used to enhance the extracted facial temporal features to obtain an enhanced feature representation, including: A feature generator is used to generate a corresponding reference feature sequence based on the input facial temporal features. The reference feature sequence has the same dimension as the input facial temporal features and maintains temporal continuity. The feature optimizer is used to enhance features based on the facial temporal features and the reference feature sequence, and the enhanced feature representation is obtained through iterative optimization.

[0014] In some implementations, the method further includes training the feature generator and the feature optimizer based on the following feature enhancement loss function L_adv: L_adv=ln(F(x))+ln(1-F(G(x)))+λ·||G(x)-x||1+μ·∑ k ||G(x)[k+1]-G(x)[k]||2+ω·||FFT(G(x))-FFT(x)||2, Where x represents the input facial temporal feature, F(x) represents the feature optimization score output by the feature optimizer based on the input facial temporal feature x, with a value range of [0,1], G(x) represents the reference feature sequence generated by the feature generator based on the input facial temporal feature x, k represents the time index in the feature sequence, with a value range from 1 to N-1, N is the length of the feature sequence, ln() represents the natural logarithm function, ||·||1 and ||·||2 represent the L1 norm and L2 norm respectively, FFT() represents the fast Fourier transform, and λ, μ and ω are all preset weight coefficients.

[0015] In some implementations, the method further includes training the feature compression network based on the following feature compression loss function L_dist: L_dist=α·KL(σ(s / T),σ(t / T))+β·||W·st||2, Where s is the compressed low-dimensional feature vector, t is the enhanced feature representation of the input, T is the temperature parameter, σ() is the softmax function, KL() is the divergence function, W is the dimension mapping matrix, ||·||2 represents calculating the L2 norm, and α and β are preset weight coefficients.

[0016] The proposed facial video image processing method improves the accuracy, stability, and efficiency of feature extraction through multiple techniques. In heart rate feature extraction, adaptive window filtering effectively eliminates ambient light interference, and a multi-scale feature pyramid is used to extract heart rate information at different scales. In lip movement feature extraction, audio-video cross-correlation analysis based on time-scale alignment improves feature accuracy. In micro-expression feature extraction, spatiotemporal feature maps capture continuous facial expression changes. These techniques significantly enhance the environmental adaptability of feature extraction. This method simultaneously extracts three temporal facial features: heart rate, lip movement, and micro-expression, achieving complete feature representation. Heart rate features reflect blood flow changes, lip movement features reflect speech synchronization, and micro-expression features capture subtle movements; these three features complement each other. The generator-optimizer structure of the feature enhancement network further enhances feature expressiveness. The generator produces a temporally continuous reference feature sequence, and the optimizer enhances the features based on the original and reference features. The feature enhancement loss function maintains feature similarity through L1 norm constraints, avoids feature abrupt changes through temporal continuity constraints, and maintains time-frequency characteristics through frequency domain constraints, effectively improving feature quality.

[0017] In terms of computational efficiency, this method maps high-dimensional features to low-dimensional representations through a feature compression network, significantly reducing storage overhead. The design of the feature compression loss function ensures the preservation of feature distribution and reconstruction performance during compression. The entire scheme requires no dedicated hardware, making it suitable for deployment on mobile terminals, and it maintains good downstream task performance even after feature compression, demonstrating good practicality. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0019] Figure 1 A flowchart of a face video image processing method according to an embodiment of this application is shown.

[0020] Figure 2 This is a schematic diagram of the structure of an electronic device shown in at least one embodiment of this application. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0022] The overall concept of this plan will be introduced here first.

[0023] The face video image processing scheme proposed in this application mainly includes three key parts: feature extraction, feature enhancement, and feature compression.

[0024] The feature extraction section extracts three temporal features from facial video sequences: heart rate features reflect the changes in facial blood flow, lip movement features demonstrate the synchronization between lip shape and speech, and micro-expression features capture subtle facial movements. Furthermore, optimized extraction algorithms for each feature are proposed in several implementations to achieve high-quality feature extraction.

[0025] The feature enhancement network enhances the extracted features to improve their quality. Further, several implementations propose a specially designed feature enhancement network architecture and loss function, which can be based on a generator-optimizer architecture. The generator produces a reference feature sequence with the same dimension as the input features, the optimizer enhances the features based on the original features and the reference features, and the loss function ensures the quality of the enhanced features through feature similarity constraints, temporal continuity constraints, and frequency domain constraints.

[0026] Feature compression networks map high-dimensional features into low-dimensional feature vectors, improving computational efficiency while preserving key feature information. Furthermore, specially designed loss functions are proposed in several implementations to ensure that the compressed features retain key information through feature distribution consistency and reconstruction error constraints.

[0027] In practical applications, these three parts form a complete processing flow, which can efficiently extract and process facial video image features.

[0028] The embodiments of this application can be applied to computer systems / servers that can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0029] Computer systems / servers can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0030] Figure 1 A flowchart of a face video image processing method according to an embodiment of this application is shown. As shown, the method includes steps 1 to 5.

[0031] Step 1: Acquire facial video sequences.

[0032] A high-definition camera can be used to capture facial videos of the training samples. In some implementations, the camera resolution is no less than 1920×1080 pixels, and the frame rate is no less than 30fps. During capture, the subject should be facing the camera directly, the ambient lighting should be sufficient and uniform, and the shooting distance should be maintained at 40-80 cm.

[0033] To ensure data quality, the acquired video can be preprocessed, such as motion blur detection, illumination equalization, and inter-frame stability correction.

[0034] Step 2: Extract facial temporal features from the facial video sequence, including heart rate features, lip movement features, and micro-expression features.

[0035] This step extracts various temporal features from the video sequence to fully reflect the physiological characteristics of the face video from multiple perspectives.

[0036] Heart rate characteristics can reflect subtle color changes in facial skin caused by blood circulation. By analyzing the temporal changes in the RGB channels, the subject's physiological rhythm can be detected. This physiological rhythm reflects the periodic changes caused by facial blood circulation. When the heart pumps blood, the degree of capillary filling in the face changes periodically, causing subtle fluctuations in the skin's RGB values. This change reflects the dynamic characteristics of facial blood flow and is a physiological signal worthy of close attention in video image processing. Accurate extraction of this feature is crucial for subsequent feature processing and applications.

[0037] Lip movement features reflect the patterns of lip movement during speech. By combining video images and audio information, the synchronous changes in lip shape and pronunciation can be analyzed, capturing dynamic facial features.

[0038] Micro-expression features capture subtle movements of facial muscles. Even when the face is relatively still, there are minute changes in expression that contain rich temporal information about the face.

[0039] This embodiment extracts multiple features to reflect the dynamic changes of the face from different perspectives, providing a foundation for subsequent feature enhancement and compression processing.

[0040] In some possible implementations, extracting the heart rate features includes: Based on the facial video sequence, multiple regions of interest are located, and pixel change sequences of each region are obtained through pixel tracking. An adaptive window filter is applied to the acquired pixel change sequence to eliminate ambient light interference and obtain the filtered sequence. A multi-scale feature pyramid is constructed for the filtered sequence to obtain multiple layers of sequences. The time-domain and frequency-domain features of each layer are extracted, and the extracted time-domain and frequency-domain features of each layer are weighted to obtain the heart rate features.

[0041] For example, a standard facial landmark detection algorithm can be used to detect 68 key points on the face. Twelve regions rich in capillaries, such as the forehead, cheeks, and nose, are selected as regions of interest. Pixel tracking is used to obtain the RGB value change sequence for each region. Adaptive window filtering is applied to the obtained RGB value change sequence; a bandpass filter of 0.7Hz-4Hz can be used to eliminate ambient light interference.

[0042] The filtered sequence can be decomposed into three levels using a Gaussian pyramid: Level 1, the original sequence N1(t); Level 2, the downsampled and smoothed sequence N2(t); and Level 3, the downsampled and smoothed sequence N3(t). Time-domain and frequency-domain features are extracted from each level, and the extracted features are fused using a weighted method. The fused feature vector N can be calculated using a learnable weight matrix W: N = ∑ i (W i ·N i ), i=1,2,3,N i Let W be the feature vector of the i-th layer. i This is the corresponding weight matrix. Finally, the periodic intensity of the fused feature vector N can be calculated as the score of heart rate variation amplitude, i.e., the heart rate feature. Specifically, a Fourier transform can be performed on the feature vector N to find the frequency component fp with the largest amplitude in the range of 0.7Hz-4Hz, and the ratio of its energy to the total energy can be calculated as the periodic intensity score, i.e., the heart rate feature.

[0043] In some implementations, extracting lip movement features includes: Based on the facial video sequence, the key point coordinate sequence of the lip region is obtained, and the key point coordinate sequence is segmented according to a preset time window. Based on the coordinate change of adjacent time windows, the key point coordinate change sequence is obtained. Extract the audio feature sequence corresponding to the facial video sequence; Unify the key point coordinate change sequence and the audio feature sequence to the same time scale; The cross-correlation coefficient between the key point coordinate change sequence and the audio feature sequence at the same time scale is calculated to obtain the lip movement features.

[0044] In some examples, 9 lip keypoints out of 68 facial keypoints can be extracted. Assuming each time window is 30ms (corresponding to the standard window for speech processing), the displacement of lip keypoints between adjacent windows is calculated, i.e., Δx and Δy for each keypoint. Then, the 18-dimensional change vector (9 points × 2 coordinates) constitutes the feature of a window, and the feature vectors of continuous windows constitute the coordinate change sequence.

[0045] The audio corresponding to the facial video sequence can be segmented into 30ms windows, and the 13-dimensional MFCC coefficients of each window can be extracted to form an audio feature sequence.

[0046] By unifying the audio sampling rate and video frame rate to the same time scale, dynamic time warping algorithms can be used to handle different speech rates.

[0047] Calculate the cross-correlation coefficient of the two sequences after time-scale alignment, and output the cross-correlation coefficient as the lip movement feature.

[0048] According to this embodiment, the dynamic characteristics of lip movement are reflected by the changes in the coordinates of key lip points, which better reflects the movement patterns than simple positional features. Maintaining synchronization between audio and video sequences through a unified time scale improves the accuracy of feature extraction. Based on the cross-correlation analysis of lip movement and speech, the temporal features of the face can be represented more completely.

[0049] In some implementations, the cross-correlation coefficient R(τ) between the keypoint coordinate change sequence and the audio feature sequence at the same time scale is calculated according to the following formula: R(τ)=∑ i ∑ j [E[(Li(t)-μi)(Aj(t+τ)-μj)] / σ i σ j ], Where i is the dimension index of the vector in the keypoint coordinate change sequence, j is the dimension index of the vector in the audio feature sequence, and L i (t) represents the value of the i-th dimension in the keypoint coordinate change sequence at time t, A j(t) represents the value of the j-th dimension of the audio feature sequence at time t, μ i Let σ be the mean of the i-th dimension in the sequence of keypoint coordinate changes. i Let μ be the standard deviation of the i-th dimension in the keypoint coordinate change sequence. j Let σ be the mean of the j-th dimension in the audio feature sequence. j Let be the standard deviation of the j-th dimension in the audio feature sequence, τ be the time delay parameter, and E[] denote the mathematical expectation operator.

[0050] The cross-correlation coefficient calculation method proposed in this embodiment achieves a comprehensive comparison of all dimensions of the coordinate transformation sequence and the audio feature sequence through a dual summation mechanism, significantly improving the accuracy of feature correlation analysis. By introducing a time delay parameter τ, the method can tolerate slight audio-video asynchrony, enhancing processing performance. The above embodiment also employs a standard deviation normalization method, effectively solving the problem of differences in numerical ranges across different dimensions and ensuring the comparability of calculation results.

[0051] This implementation can accurately capture fine-grained correlation features between lip movements and speech, while maintaining temporal consistency across multiple feature dimensions. Furthermore, this method exhibits good adaptability to different speaking speeds and audio / video capture devices, improving the robustness of feature extraction.

[0052] In some implementations, extracting micro-expression features includes: Based on the facial video sequence, facial muscle motion units are analyzed to obtain facial muscle motion parameters; Spatiotemporal feature maps are constructed using the acquired muscle movement parameters; The spatiotemporal feature map is processed by a feature extraction network to obtain spatiotemporal features; Based on the extracted spatiotemporal features, the trend of facial expression changes is analyzed to obtain micro-expression features.

[0053] In one example, 12 key facial muscle regions can be located first, including the area between the eyebrows, the corners of the eyes, and the corners of the mouth. Displacement and deformation parameters are extracted for each region, and the changes of these parameters over time are recorded to obtain facial muscle motion parameters.

[0054] Based on the acquired muscle movement parameters, the parameter data of 12 regions are organized into a spatial matrix, and N frames of data are continuously stacked along the time dimension to form a spatiotemporal feature map. The size of this feature map is T×H×W, where T represents the length of the time window, and H and W represent the height and width of the feature map, respectively.

[0055] The constructed spatiotemporal feature map is input into the feature extraction network, where common convolution operations can be used to extract features. The network can output a 256-dimensional feature vector containing temporal information about facial motion.

[0056] The rate and amplitude of facial expression changes can be calculated based on the extracted spatiotemporal features, and their patterns of change can be analyzed to obtain the final micro-expression features. The analysis of these patterns primarily considers the continuity and coordination of muscle movements. For example, the trend score V of the extracted spatiotemporal features can be calculated using the following formula as a micro-expression feature: V=∑ i (w i ·||F i –F i-1 ||2) / N, Among them, F i Let w be the spatiotemporal feature of the i-th frame. i Here, represents the weighting coefficient, and N represents the number of frames. By calculating the weighted L2 norm of the spatiotemporal features of adjacent frames, the amplitude and rate of facial expression changes can be quantified.

[0057] This implementation effectively captures subtle facial expression changes by analyzing motion parameters of key muscle regions. The method of constructing the spatiotemporal feature map preserves the temporal information of facial expression changes, improving the accuracy of feature extraction. Furthermore, this implementation has a moderate computational load, making it suitable for real-time processing applications.

[0058] Step 3: Use a feature enhancement network to enhance the extracted facial temporal features to obtain the enhanced feature representation.

[0059] In some implementations, the feature enhancement network may employ a generator-optimizer architecture. The feature generator can generate a corresponding reference feature sequence based on the input facial temporal features. The reference feature sequence has the same dimension as the input facial temporal features and maintains temporal continuity. The feature optimizer can then perform feature enhancement based on the facial temporal features and the reference feature sequence, and obtain the enhanced feature representation through iterative optimization.

[0060] In some examples, the feature generator can employ a fully connected neural network structure with three hidden layers, each using the ReLU activation function. The input layer receives a sequence of temporal facial features, and intermediate layers process these features through non-linear transformations. The output layer generates a reference feature sequence. To ensure the quality of the generated sequence, a batch normalization layer is added after each hidden layer, and a residual connection is added to the last layer. The feature optimizer can also employ a fully connected network structure with four hidden layers. The input layer connects the original sequence and the reference feature sequence. Intermediate layers use the ReLU activation function and batch normalization layers, with skip connections added between each pair of layers to enhance feature transfer. The output layer maps the feature optimization score to the [0,1] interval using the sigmoid function.

[0061] Through iterative training, the two networks in the generator-optimizer architecture can work together to improve feature quality.

[0062] In some implementations, the feature generator and the feature optimizer can be trained based on the following feature enhancement loss function: L_adv=ln(F(x))+ln(1-F(G(x)))+λ·||G(x)-x||1+μ·∑ k ||G(x)[k+1]-G(x)[k]||2+ω·||FFT(G(x))-FFT(x)||2, Where x represents the input facial temporal feature, F(x) represents the feature optimization score output by the feature optimizer based on the input facial temporal feature x, with a value range of [0,1], G(x) represents the reference feature sequence generated by the feature generator based on the input facial temporal feature x, k represents the time index in the feature sequence, with a value range from 1 to N-1, N is the length of the feature sequence, ln() represents the natural logarithm function, ||·||1 and ||·||2 represent the L1 norm and L2 norm respectively, FFT() represents the fast Fourier transform, and λ, μ and ω are all preset weight coefficients.

[0063] According to the feature enhancement loss function L_adv designed in this embodiment, the first two terms, ln(F(x)) and ln(1-F(G(x))), constitute the basic feature enhancement objective, enabling the optimizer F(·) to learn to distinguish the differences between the original features and the generated features; the third term, the L1 norm ||G(x)-x||1, constrains the similarity between the generated features and the original features; the fourth term, the L2 norm ∑ k ||G(x)[k+1]-G(x)[k]||2 constrains the continuity of features at adjacent time points; the fifth term ||FFT(G(x))-FFT(x)||2 is used to preserve the frequency characteristics of the features.

[0064] In one example, during training, the Adam optimizer can be used with a learning rate of 0.0001 and a batch size of 32. The weight coefficients λ, μ, and ω are set to 0.1, 0.05, and 0.1, respectively, to balance the contributions of each loss term while maintaining feature quality. Network convergence is gradually achieved by alternately optimizing the feature generator and feature optimizer.

[0065] Step 4: Use a feature compression network to compress the enhanced feature representation into a low-dimensional feature vector through multi-layer nonlinear transformation.

[0066] Feature compression networks can employ an encoder structure, achieving dimensionality reduction through multiple fully connected layers, with each layer using the ReLU activation function and a batch normalization layer. To avoid information loss, skip connections can be added between key layers. The compression ratio can be set according to actual needs, such as compressing the feature dimension to 1 / 3 of the original dimension.

[0067] In some implementations, the feature compression network can be trained based on the following feature compression loss function: L_dist=α·KL(σ(s / T),σ(t / T))+β·||W·st||2, Where s is the compressed low-dimensional feature vector, t is the enhanced feature representation of the input, T is the temperature parameter, σ() is the softmax function, KL() is the divergence function, W is the dimension mapping matrix, ||·||2 represents calculating the L2 norm, and α and β are preset weight coefficients.

[0068] The loss function L_dist according to this embodiment includes two terms: the KL divergence term maintains the consistency of the feature distribution before and after compression, and adjusts the smoothness of the distribution through the temperature parameter T; the L2 norm term constrains the reconstruction error, ensuring that the compressed features can be mapped back to the original space well. The weighting coefficients α and β are used to balance the importance of the two losses, and can be set to 0.5 and 0.5 respectively.

[0069] By combining the aforementioned feature compression network and loss function, effective feature compression can be achieved. The network structure ensures the non-linear expressiveness of the compression process, while the loss function ensures the preservation of key information during compression. This design reduces storage and computational overhead while maintaining the expressive power of the features.

[0070] Step 5: Output the low-dimensional feature vector.

[0071] Obtaining low-dimensional feature vectors significantly reduces the computational complexity of subsequent processing. For example, in real-time processing on mobile devices, low-dimensional features can reduce computational resource consumption. From a storage efficiency perspective, low-dimensional feature vectors also greatly reduce storage space requirements, especially in scenarios requiring the storage of large amounts of video features; this compression can save significant storage costs. Moreover, the low-dimensional feature vectors processed by the above feature enhancement and compression retain the key temporal information of facial videos with high quality, and can be used as input for various downstream tasks. This compact feature representation is highly beneficial for the development and deployment of subsequent applications.

[0072] The proposed facial video image processing method improves the stability of feature extraction through multiple techniques. In heart rate feature extraction, adaptive window filtering effectively eliminates ambient light interference, and a multi-scale feature pyramid is used to extract heart rate information at different scales. In lip movement feature extraction, audio-video cross-correlation analysis based on time-scale alignment improves feature accuracy. In micro-expression feature extraction, spatiotemporal feature maps capture continuous facial expression changes. These techniques significantly enhance the environmental adaptability of feature extraction. This method simultaneously extracts three temporal facial features: heart rate, lip movement, and micro-expression, achieving complete feature representation. Heart rate features reflect blood flow changes, lip movement features reflect speech synchronization, and micro-expression features capture subtle movements; these three features complement each other. The generator-optimizer structure of the feature enhancement network further enhances feature expressiveness. The generator produces a temporally continuous reference feature sequence, and the optimizer enhances the features based on the original and reference features. The feature enhancement loss function maintains feature similarity through L1 norm constraints, avoids feature abrupt changes through temporal continuity constraints, and maintains time-frequency characteristics through frequency domain constraints, effectively improving feature quality.

[0073] In terms of computational efficiency, this method maps high-dimensional features to low-dimensional representations through a feature compression network, significantly reducing storage overhead. The design of the feature compression loss function ensures the preservation of feature distribution and reconstruction performance during compression. The entire scheme requires no dedicated hardware, making it suitable for deployment on mobile terminals, and it maintains good downstream task performance even after feature compression, demonstrating good practicality.

[0074] In remote medical monitoring, physiological features need to be extracted from patients' facial videos for health status assessment. This solution can extract and process information such as heart rate changes, speech patterns, and emotional changes from facial video images, and then efficiently compress and store these features.

[0075] In a specific application example, a high-definition camera can be used to capture facial video sequences at a resolution of 1920×1080 pixels, a frame rate of 30fps, and continuous capture for 60 seconds. During capture, the subject should face the camera, the ambient lighting should be sufficient and uniform, and the shooting distance should be within 40-80 centimeters. To ensure capture quality, the video can be pre-processed, including inter-frame stability correction and illumination equalization.

[0076] Facial temporal feature extraction is divided into three parts. For heart rate feature extraction, 12 regions of interest (ROIs) on the face are first located, including areas rich in capillaries such as the forehead and cheeks. A 0.7-4Hz bandpass filter is used to eliminate ambient light interference, and the filtered sequence is then subjected to multi-scale analysis through a three-layer feature pyramid to obtain a score for heart rate variation amplitude. For lip movement feature extraction, nine lip keypoints are located, and coordinate changes are calculated in 30ms windows. Simultaneously, 13-dimensional MFCC audio features are extracted, and cross-correlation coefficients are calculated to characterize lip movement features. For micro-expression feature extraction, motion parameters of 12 facial muscle regions are analyzed to construct a spatiotemporal feature map. After processing by a feature extraction network, an expression change trend score is obtained.

[0077] Feature enhancement employs a generator-optimizer structure. The feature generator consists of a 3-layer fully connected network, each layer equipped with a ReLU activation function and a batch normalization layer, with a residual connection added in the last layer. The feature optimizer uses a 4-layer fully connected network, also employing a ReLU activation function and a batch normalization layer, with skip connections added between layers to enhance feature transfer. During training, the batch size is set to 32, the learning rate to 0.0001, the maximum number of iterations to 50, and the loss function weights λ, μ, and ω to 0.1, 0.05, and 0.1, respectively.

[0078] The feature compression network employs an encoder structure and achieves feature dimensionality reduction through a multi-layer fully connected network. During training, the weight coefficients α and β of the loss function are both set to 0.5, and the temperature parameter T is set to 10 to ensure that key feature information is preserved during compression.

[0079] The output low-dimensional feature vector has approximately one-third the dimension of the original features, significantly reducing storage overhead. The latency of the entire processing flow is controlled within 100ms, meeting real-time requirements. These features can be directly used for subsequent health status analysis, providing important data for remote medical monitoring.

[0080] Figure 2 An electronic device provided in at least one embodiment of this application includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the face video image processing method described in any embodiment or implementation of this application when executing the computer instructions.

[0081] At least one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the face video image processing method described in any embodiment or implementation of this application.

[0082] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the data processing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0084] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0086] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0087] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0088] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A method for processing facial video images, characterized in that, include: Capture facial video sequences; Facial temporal features, including heart rate features, lip movement features, and micro-expression features, are extracted from the facial video sequence. The extracted facial temporal features are enhanced using a feature enhancement network to obtain the enhanced feature representation; A feature compression network is used to compress the enhanced feature representation into a low-dimensional feature vector through multi-layer nonlinear transformation. Output the low-dimensional feature vector.

2. The method according to claim 1, characterized in that, Heart rate features extracted include: Based on the facial video sequence, multiple regions of interest are located, and pixel change sequences of each region are obtained through pixel tracking. An adaptive window filter is applied to the acquired pixel change sequence to eliminate ambient light interference and obtain the filtered sequence. The filtered sequence is passed through a multi-scale feature pyramid to obtain a multi-layer sequence, and the time-domain and frequency-domain features of each layer are extracted. The extracted time-domain and frequency-domain features of each layer are weighted to obtain the heart rate features.

3. The method according to claim 1, characterized in that, Extracting lip movement features includes: Based on the facial video sequence, the key point coordinate sequence of the lip region is obtained, and the key point coordinate sequence is segmented according to a preset time window. Based on the coordinate change of adjacent time windows, the key point coordinate change sequence is obtained. Extract the audio feature sequence corresponding to the facial video sequence; Unify the key point coordinate change sequence and the audio feature sequence to the same time scale; The cross-correlation coefficient between the key point coordinate change sequence and the audio feature sequence at the same time scale is calculated to obtain the lip movement features.

4. The method according to claim 3, characterized in that, The cross-correlation coefficient R(τ) between the keypoint coordinate change sequence and the audio feature sequence at the same time scale is calculated using the following formula: R(τ)=∑ i ∑ j [E[(Li(t)-μi)(Aj(t+τ)-μj)] / s i s j ]; Where i is the dimension index of the vector in the keypoint coordinate change sequence, j is the dimension index of the vector in the audio feature sequence, and L i (t) represents the value of the i-th dimension in the keypoint coordinate change sequence at time t, A j (t) represents the value of the j-th dimension of the audio feature sequence at time t, μ i Let σ be the mean of the i-th dimension in the sequence of keypoint coordinate changes. i Let μ be the standard deviation of the i-th dimension in the keypoint coordinate change sequence. j Let σ be the mean of the j-th dimension in the audio feature sequence. j Let be the standard deviation of the j-th dimension in the audio feature sequence, τ be the time delay parameter, and E[] denote the mathematical expectation operator.

5. The method according to claim 1, characterized in that, Extracting micro-expression features includes: Based on the facial video sequence, facial muscle motion units are analyzed to obtain facial muscle motion parameters; Spatiotemporal feature maps are constructed using the acquired muscle movement parameters; The spatiotemporal feature map is processed by a feature extraction network to obtain spatiotemporal features; Based on the extracted spatiotemporal features, the trend of facial expression changes is analyzed to obtain micro-expression features.

6. The method according to claim 1, characterized in that, The feature enhancement network includes a feature generator and a feature optimizer. It enhances the extracted temporal facial features to obtain an enhanced feature representation, including: A feature generator is used to generate a corresponding reference feature sequence based on the input facial temporal features. The reference feature sequence has the same dimension as the input facial temporal features and maintains temporal continuity. The feature optimizer is used to enhance features based on the facial temporal features and the reference feature sequence, and the enhanced feature representation is obtained through iterative optimization.

7. The method according to claim 6, characterized in that, The method further includes training the feature generator and the feature optimizer based on the following feature enhancement loss function L_adv: L_adv=ln(F(x))+ln(1-F(G(x)))+λ·||G(x)-x||1+μ·∑ k ||G(x)[k+1]-G(x)[k]||2+ω·||FFT(G(x))-FFT(x)||2; Where x represents the input facial temporal feature, F(x) represents the feature optimization score output by the feature optimizer based on the input facial temporal feature x, with a value range of [0,1], G(x) represents the reference feature sequence generated by the feature generator based on the input facial temporal feature x, k represents the time index in the feature sequence, with a value range from 1 to N-1, N is the length of the feature sequence, ln() represents the natural logarithm function, ||·||1 and ||·||2 represent the L1 norm and L2 norm respectively, FFT() represents the fast Fourier transform, and λ, μ and ω are all preset weight coefficients.

8. The method according to claim 1, characterized in that, The method further includes training the feature compression network based on the following feature compression loss function L_dist: L_dist=α·KL(σ(s / T),σ(t / T))+β·||W·st||2; Where s is the compressed low-dimensional feature vector, t is the enhanced feature representation of the input, T is the temperature parameter, σ() is the softmax function, KL() is the divergence function, W is the dimension mapping matrix, ||·||2 represents calculating the L2 norm, and α and β are preset weight coefficients.

Images