A non-contact multi-modal mental health evaluation method based on domain adaptation meta-learning

Abstract

The application provides a non-contact multi-modal mental health evaluation method based on domain adaptive meta-learning. Aiming at the multi-modal characteristics, the whole process from data preprocessing, feature extraction and fusion to meta-training and meta-testing is designed. First, the facial video and speech data of the subjects are collected and preprocessed, and facial, speech and physiological features are extracted. Then, a DSConv-BiLSTM-iAFF multi-modal feature extraction and fusion model is constructed, and through the deep separable convolution, bidirectional LSTM and iterative attention fusion module, the adaptive weighted fusion of multi-modal features is realized. In view of the cross-dataset domain offset problem, the DA-MAML model is constructed, the domain adversarial training mechanism and the gradient reversal layer are fused, and the source domain and target domain feature distribution alignment is realized. The generalization ability of the model on the new unknown domain task is verified through meta-training and meta-testing. The application uses multi-modal fusion and domain adaptive meta-learning technology to provide a new method for early screening and daily monitoring of mental health.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and mental health assessment technology, specifically involving a non-contact mental state assessment method based on multimodal feature fusion and domain adaptive meta-learning. Background Technology

[0002] Mental health is a crucial component of human physical and mental well-being, impacting individual quality of life, family happiness, and social harmony and stability. Mental illnesses such as depression and anxiety not only cause physical and mental suffering for patients, affecting their normal work and life, but also trigger a series of social problems, creating a significant socioeconomic burden. Conducting efficient and objective mental health assessments is of significant practical and social value for timely identification of abnormal states, assisting clinical intervention, and optimizing the allocation of medical resources.

[0003] Currently, mental health assessment technologies still have many shortcomings and cannot meet the needs of practical applications. On the one hand, traditional assessment methods rely heavily on the subjective judgment of professional medical personnel, requiring face-to-face interviews and questionnaire completion. This is not only inefficient and costly, but also easily affected by factors such as the assessor's experience level, subjective preferences, assessment environment, and patient condition. The objectivity and consistency of the assessment results are difficult to guarantee, making it impossible to achieve large-scale, routine early screening and daily monitoring. On the other hand, while existing non-contact intelligent assessment technologies have made some progress, they still face core technical bottlenecks: the multimodal feature fusion methods are relatively simple, making it difficult to fully explore the complementary correlations between facial, voice, and physiological signals; the model's generalization ability is insufficient due to the scarcity of labeled samples and limited sample scenarios, making it difficult to quickly adapt to different populations and scenarios; at the same time, it has not effectively handled the feature distribution shift caused by cross-dataset and cross-device issues, resulting in significant performance fluctuations in practical applications, insufficient stability and robustness, and difficulty in meeting the reliable use requirements in real-world scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a non-contact, multimodal mental health assessment method based on domain-adaptive meta-learning. This method, through multimodal feature extraction and fusion, combined with meta-learning and domain adaptation techniques, solves the problems of high annotation costs, difficulty in adapting to limited samples, and poor cross-scenario generalization in existing assessment methods, achieving an objective, efficient, and robust assessment of mental health status. The method includes the following steps: S01, Input non-contact acquisition of facial video and voice data, extract facial signals, voice signals and pulse wave physiological signals from them respectively, perform standardized preprocessing on the three types of modal data, and extract three-dimensional facial key point features, Mel spectrogram features and rPPG temporal physiological features in sequence; S02, construct the DSConv-BiLSTM-iAFF multimodal feature extraction and fusion model. Input the preprocessed three-modal features into the model, use depthwise separable convolution (DSConv) to extract the local discriminative features of each modality, use bidirectional LSTM (BiLSTM) to capture the temporal variation of features, and use the iterative attention fusion (iAFF) module to automatically focus on the feature information that is more important for the discrimination of mental health status, realize the adaptive weighted fusion of multimodal features, and output the fused feature vector. S03, Construct the DA-MAML model. Based on the feature fusion model constructed in step S02, embed a gradient inversion layer between the feature fusion module and the classification output layer. Through domain adversarial training, the feature distribution of the source domain and the target domain is aligned, alleviating the domain offset problem across datasets and populations. S04, the meta-training phase, learns cross-task general meta-parameters and extracts domain-invariant features. Randomly sampled task batches are divided into support sets and query sets. The support set is used for small-step fine-tuning, achieving task adaptation and domain separation through dual-path gradient propagation. The query set calculates the joint loss, backpropagates the gradients to update the meta-parameters, and iterates to a preset number of rounds to obtain the optimized meta-parameters.

[0005] S05, Meta-testing phase: Select cross-domain tasks that were not included in the training and divide them into support sets and query sets. Load meta-parameters, fine-tune the model using the support set to adapt to the new task and new domain, use the fine-tuned parameters to predict the query set, calculate multiple evaluation metrics, and verify the model's cross-domain generalization and fast adaptation capabilities. Attached Figure Description

[0006] Figure 1 : This is a model framework diagram of the non-contact multimodal mental health assessment method based on domain adaptive meta-learning of the present invention. Figure 2 : This is a flowchart of the dataset preprocessing method of the present invention. Figure 3 : This is a diagram illustrating the multimodal feature extraction and fusion model of the non-contact multimodal mental health assessment method based on domain adaptive meta-learning, as presented in this invention. Figure 4 : This is a flowchart illustrating the meta-training process of the non-contact multimodal mental health assessment method based on domain adaptive meta-learning of this invention. Figure 5 : This is a flowchart illustrating the meta-testing process of the non-contact multimodal mental health assessment method based on domain adaptive meta-learning of this invention. Detailed Implementation

[0007] This invention proposes a non-contact multimodal mental health assessment method based on domain adaptive meta-learning. The overall process includes five parts: multimodal data acquisition and preprocessing, multimodal feature extraction and adaptive fusion, meta-learning general knowledge learning, domain adaptive distribution alignment, and rapid adaptation assessment with few samples. The specific implementation steps are as follows: Step 1: Multimodal data preprocessing and feature extraction.

[0008] Multimodal data preprocessing and feature extraction, such as Figure 2 As shown, firstly, FFmpeg is used to unify the format of the original video, converting videos with different encodings, resolutions, and frame rates into MP4 videos with a unified format, fixed frame rate, and resolution. This eliminates differences in acquisition devices and recording parameters, providing consistent input for subsequent feature extraction.

[0009] For facial signals, the OpenFace tool was used to extract three-dimensional facial features, including 68 three-dimensional facial key points and three-dimensional gaze vectors, which were mapped to the 0-1 range using min-max normalization. The length of all facial feature sequences was unified to 900 frames, with zero padding for insufficient length and truncation for excessive length. Finally, the 68 three-dimensional key points (900×68×3) and the binocular three-dimensional gaze vectors (900×2×3) were concatenated as the facial feature output.

[0010] For the speech signal, FFmpeg is used to extract the audio stream from the video, which is then uniformly converted to mono WAV format with a fixed sampling rate. The audio duration is standardized to 30 seconds and aligned with 900 frames of facial features. The Mel spectrogram is extracted using Librosa, with a non-linear Mel scale mapping, as shown in the formula: The final output is a Mel-spectral feature with a size of 900×80.

[0011] For physiological signals, the RGB average signal of the skin region was extracted from the face region of the video using the convex hull method; a 5th-order bandpass filter (0.65Hz-4.0Hz) was used to remove noise and motion artifacts; and the CHROM algorithm was used to calculate the chromaticity components. , The formula is: in, , , It consists of three channel vectors: red, green, and blue. , which is the standard skin vector. The relationship between the two is: , , .

[0012] The proportionality coefficient is obtained from the standard deviation. The formula is: in, yes standard deviation , yes and The output after bandpass filtering This is the proportionality coefficient.

[0013] The final blood volume pulse (BVP) signal is obtained using the following formula: Where S is the output BVP signal. The length of the BVP signal is unified to 900 frames and normalized to finally output the standardized rPPG timing features.

[0014] Finally, the standardized 3D facial key points, 3D gaze features, Mel spectrograms, and rPPG pulse signals are saved in NumPy array format as input for subsequent models.

[0015] Step 2: Multimodal feature extraction and fusion model construction.

[0016] Targeting the characteristics of facial, voice, and rPPG multimodal temporal signals, a DSConv-BiLSTM-iAFF multimodal mental health assessment model was constructed. The model is divided into three modules: intra-modal feature extraction, inter-modal feature fusion, and final classification. The overall process is as follows: Figure 3 As shown.

[0017] The intra-modal feature extraction module sequentially feeds the preprocessed features from the three modalities into a depthwise separable convolutional module to complete intra-modal feature extraction. Corresponding convolutional structures and parameters are set for different modal characteristics. Depthwise convolution is used to extract independent spatial features for each channel, followed by pointwise convolution to achieve cross-channel information fusion and channel dimension unification. Batch normalization and ReLU activation are then used to complete nonlinear transformation and feature standardization, effectively reducing model computation and improving training stability. The features output from each modality are downsampled using one-dimensional max pooling, compressing the sequence length while retaining key feature information and reducing subsequent computational overhead. The pooled temporal features are then fed into a bidirectional LSTM module to extract historical and future context information from the temporal data. After temporal modeling, a fully connected layer performs global weighted fusion and dimension mapping on the features, transforming high-dimensional temporal features into a unified representation suitable for cross-modal fusion.

[0018] The intermodal feature fusion module feeds the three types of modal features into the iAAFResBlock module for adaptive fusion. First, it performs multi-level dimensionality reduction through a bottleneck layer and retains the original feature information by combining residual connections. Then, it fuses the local channel context and the global channel context through an iterative attention mechanism. After two attention iterations, it optimizes the feature weights, strengthens the feature information that plays a key role in mental state discrimination, and suppresses redundant interference. Finally, it restores the features to the input dimension through the dimensionality upscaling module to ensure that the input and output dimensions of the module are consistent, thereby realizing the deep adaptive fusion of multimodal features and outputting the final fused feature vector.

[0019] The final classification module uses a multilayer perceptron constructed with four fully connected layers in series. Each intermediate layer uses the ReLU activation function to enhance the nonlinear expression capability of the network. The last fully connected layer outputs the classification result and generates the probability distribution of normal and abnormal categories through the Softmax activation function, thus completing the final classification of mental health status.

[0020] Step 3: DA-MAML model construction.

[0021] To address the degradation of generalization ability caused by domain bias in classical meta-learning for cross-task and cross-dataset mental health assessment, a DA-MAML model is constructed. This model uses classical MAML as its core framework and incorporates a domain adversarial training mechanism. While retaining the ability to quickly adapt to small datasets, it achieves feature distribution alignment across datasets. The model architecture is as follows: Figure 1 As shown.

[0022] The DA-MAML model consists of four core modules: a multimodal feature extraction backbone network, a label prediction head, a domain discriminator, and a gradient inversion layer.

[0023] The multimodal feature extraction backbone network reuses the previously designed DSConv-BiLSTM multimodal feature extraction network, which is responsible for mapping the original features of three heterogeneous modalities—facial, audio, and physiological—into a unified 768-dimensional shared feature vector. This vector serves as the input to the label prediction head for mental health status classification and as the input to the domain discriminator for domain source discrimination. Its parameters, as the core meta-parameters of DA-MAML, participate in the joint iteration of bilayer optimization and domain adversarial training.

[0024] The label prediction head adopts a lightweight fully connected network structure, consisting of two fully connected layers. Its parameters are decoupled from the backbone network parameters but are optimized collaboratively, adapting to the dual-layer optimization logic of MAML.

[0025] The domain discriminator adopts a lightweight multilayer perceptron structure, consisting of three fully connected layers, which are updated collaboratively only in the outer loop of the meta-training.

[0026] The gradient inversion layer is embedded between the output of the backbone network and the input of the domain discriminator. During forward propagation, it acts as an identity mapping and does not change the features. During backward propagation, it inverts and scales the gradients returned by the domain discriminator, thereby achieving adversarial optimization between the backbone network and the domain discriminator.

[0027] The model's total loss function is a weighted average of the MAML meta-loss and the domain adversarial loss. The MAML meta-loss uses cross-entropy loss to optimize fast adaptation with few samples, and its formula is: in, The meta-loss for MAML; This refers to the task batch size, which is the number of tasks sampled in a single training iteration. The number of samples in the query set for each task; For the first A query set for each task; For the model to adapt parameters and Below, predict the sample The probability of belonging to an abnormal class.

[0028] Domain adversarial loss also adopts the form of cross-entropy loss, which is transformed by a gradient inversion layer to achieve cross-dataset feature distribution alignment. The formula is: in, Losses due to domain confrontation; This represents the total number of samples participating in domain adversarial training. The sample set for all tasks; For a single sample and its corresponding domain label, This is a multimodal fusion feature vector. One-hot encoding of the domain label ( For the first The label value of the domain, if the sample comes from the domain. Each domain ,otherwise ); The number of fields; For the domain discriminator in parameters (Backbone Network) and Predicting samples under (domain discriminator) Belongs to the The probability of each domain; It is the natural logarithm function.

[0029] The overall loss function achieves joint optimization of the dual objectives of fast adaptation with few samples and cross-domain feature alignment, through the balancing coefficients. The optimization weights of the two are controlled by the following formula: in, The total loss of the DA-MAML model, For MAML meta loss, Losses due to domain confrontation; This is the loss balance coefficient, and its value range is... , used to optimize weights to balance the losses of the main task and the auxiliary task.

[0030] Step 4: Meta-training of the DA-MAML model.

[0031] The DA-MAML meta-training phase integrates domain adversarial mechanisms within a two-layer optimization framework, as follows: Figure 4 As shown.

[0032] At the start of meta-training, a batch of tasks is randomly sampled from the meta-training task distribution composed of multi-source datasets. Each task corresponds to a subset of a single-domain or cross-domain dataset, and the data is divided into a support set and a query set according to a unified partitioning rule. The support set is used for intra-task adaptive fine-tuning, while the query set is used to evaluate the cross-task and cross-domain generalization performance of meta-parameters, and at the same time provides a distribution alignment basis for domain adversarial training.

[0033] During the in-task fine-tuning phase, each task is initialized from shared meta-parameters and performs a few-step gradient descent using support set data. During fine-tuning, the classification loss and domain adversarial loss are optimized simultaneously. Gradient propagation adopts a dual-path collaborative mode: during forward propagation, the feature vector output by the backbone network is directly fed into the label prediction head and then fed into the domain discriminator after passing through the gradient inversion layer; during backward propagation, the gradient of the classification loss is directly propagated back to the backbone network and the label prediction head to enhance task discriminability, and the gradient of the domain classification loss is propagated forward to the domain discriminator to improve domain differentiation ability, and then inverted through the gradient inversion layer and propagated back to the backbone network to weaken domain specificity.

[0034] In the gradient update phase, the classification loss and domain adversarial loss are calculated on the query sets of each task, and the joint gradient is backpropagated to the initial meta-parameters. This simultaneously optimizes the model's few-shot adaptation ability and cross-domain feature alignment ability. The gradient of the total loss is a weighted sum of the gradient of the classification loss and the gradient of the reversed domain adversarial loss. This gradient is used to update the meta-parameters, achieving coordinated optimization of task adaptation and domain alignment. This cross-task and cross-domain update is performed sequentially on all batches of tasks until a preset number of rounds are completed, ultimately yielding optimized meta-parameters with strong generalization ability, providing high-quality initial values ​​for the subsequent meta-testing phase.

[0035] Step 5: Meta-testing of the DA-MAML model.

[0036] The DA-MAML meta-testing phase tests the model's ability to quickly adapt and generalize to novel, unknown domain tasks using the optimal meta-parameters obtained through domain-adaptive meta-learning. The process is as follows: Figure 5 As shown.

[0037] During the in-task fine-tuning phase, the model loads the optimal meta-parameters obtained from meta-training as initial weights and performs a few-step gradient descent fine-tuning on the support set of the new task. During the fine-tuning process, the dual-path mode of domain adversarial mechanism and gradient propagation is maintained: during forward propagation, the backbone network features output classification probabilities through the label prediction head, and after passing through the gradient inversion layer, they are passed into the domain discriminator to output the domain classification probabilities; during backpropagation, the classification loss gradient is forward propagated to optimize the discrimination accuracy, and the domain classification loss gradient is inverted through the gradient inversion layer and then propagated back to optimize the domain alignment effect. This enables the model to quickly learn the features of the new task while automatically aligning the feature distributions of the source and target domains, reducing the negative impact of domain differences.

[0038] Subsequently, the query set is predicted using the fine-tuned domain adaptive parameters. During the prediction process, the gradient propagation mechanism remains consistent to ensure the domain invariance of feature extraction. Multiple evaluation metrics, such as accuracy, precision, recall, and F1 score, are calculated to measure the overall performance of the model under conditions of few samples and cross-domain conditions.

[0039] This invention compares the performance of three models—traditional MAML, Reptile, and DA-MAML—on a cross-dataset test set for mental health classification. The experimental results are shown in Table 1.

[0040] Table 1. Overall performance comparison of each model on the cross-dataset test set.

Claims

1. A non-contact, multimodal mental health assessment method based on domain adaptive meta-learning, characterized by: Includes the following steps: S01, Input non-contact acquisition of facial video and voice data, extract facial signals, voice signals and pulse wave physiological signals from them respectively, perform standardized preprocessing on the three types of modal data, and extract three-dimensional facial key point features, Mel spectrogram features and rPPG temporal physiological features in sequence; S02, construct the DSConv-BiLSTM-iAFF multimodal feature extraction and fusion model. Input the preprocessed three-modal features into the model, use depthwise separable convolution (DSConv) to extract the local discriminative features of each modality, use bidirectional LSTM (BiLSTM) to capture the temporal variation of features, and use the iterative attention fusion (iAFF) module to automatically focus on the feature information that is more important for the discrimination of mental health status, realize the adaptive weighted fusion of multimodal features, and output the fused feature vector. S03, Construct the DA-MAML model. Based on the feature fusion model constructed in step S02, embed a gradient inversion layer between the feature fusion module and the classification output layer. Through domain adversarial training, the feature distribution of the source domain and the target domain is aligned, alleviating the domain offset problem across datasets and populations. S04, the meta-training phase, learns cross-task general meta-parameters and extracts domain-invariant features, randomly samples task batches and divides them into support set and query set. The support set is used for small-step fine-tuning, and task adaptation and domain separation are achieved through dual-path gradient propagation. The query set calculates the joint loss, backpropagates the gradient to update the meta-parameters, and iterates to the preset number of rounds to obtain the optimized meta-parameters. S05, Meta-testing phase: Select cross-domain tasks that were not included in the training and divide them into support sets and query sets. Load meta-parameters, fine-tune the new tasks and new domains using the support sets in a few steps, use the fine-tuned parameters to predict the query sets, calculate multiple evaluation metrics, and verify the model's cross-domain generalization and fast adaptation capabilities.

2. The method according to claim 1, wherein step S01 is characterized in that: The original video was format-unified using FFmpeg and converted into an MP4 video with a uniform format, fixed frame rate, and resolution. Facial signals were extracted using OpenFace to obtain 68-point 3D facial key points and 3D gaze vectors, which were then spliced ​​together after min-max normalization and length unification. Speech signals were extracted from the video and converted into mono WAV format with a fixed sampling rate and a uniform duration of 30 seconds. Mel spectrogram features with a size of 900×80 were extracted using Librosa. Physiological signals are extracted from the face region of the video using the convex hull method to extract the average RGB signal of the skin region. The signal is then denoised by a 5th-order bandpass filter (0.65Hz-4.0Hz). The chromaticity components and scaling factors are calculated using the CHROM algorithm to obtain the BVP signal. After length unification and normalization, the standardized rPPG temporal features are output.

3. The method according to claim 1, wherein the step S02 is characterized in that: The DSConv-BiLSTM-iAFF model consists of three modules: intra-modal feature extraction, inter-modal feature fusion, and final classification. The intra-modal feature encoding module extracts local features through DSConv, unifies the channel dimension through pointwise convolution, and then performs batch normalization, ReLU activation, and one-dimensional max pooling downsampling before inputting it into BiLSTM to capture temporal context information. Finally, it completes feature mapping through fully connected layers. The inter-modal feature fusion module uses the iAAFResBlock module to achieve deep adaptive fusion of multi-modal features through bottleneck layer dimensionality reduction, iterative attention fusion, and dimensionality upscaling module restoration. The final classification module uses four cascaded fully connected layers to construct a multilayer perceptron. The middle layers use the ReLU activation function, and the last layer uses the Softmax activation function to output a binary classification probability distribution.

4. The method according to claim 1, wherein step S03 is characterized in that: The DA-MAML model consists of four core modules: a multimodal feature extraction backbone network, a label prediction head, a domain discriminator, and a gradient inversion layer. The backbone network reuses the DSConv-BiLSTM multimodal feature extraction network, outputting a 768-dimensional shared feature vector. The label prediction head is a lightweight two-layer fully connected network whose parameters are decoupled from and co-optimized with the backbone network parameters. The domain discriminator is a lightweight three-layer fully connected perceptron, which is co-updated only in the outer loop of the meta-training. The gradient inversion layer is embedded between the output of the backbone network and the input of the domain discriminator; forward propagation is an identity mapping, and backward propagation inverts and scales the gradient back from the domain discriminator. The total loss function of the DA-MAML model is: in For MAML meta-loss, cross-entropy loss is used to optimize the ability to quickly adapt to a small number of samples; For domain adversarial loss, cross-entropy loss is adopted and transformed by gradient inversion layer to achieve cross-dataset feature distribution alignment. This is the loss balancing coefficient, with a value range of [0,1], used to balance the optimization weights of the main task loss and the auxiliary task loss.

5. The method according to claim 1, wherein step S04 is characterized in that: During meta-training, task batches are randomly sampled from the meta-training task distribution composed of multi-source datasets; intra-task fine-tuning adopts few-step gradient descent, simultaneously optimizing classification loss and domain adversarial loss, and gradient propagation adopts a dual-path collaborative mode; in the gradient update phase, the joint loss is calculated through the query set, the gradient is back-propagated to update the meta-parameters, and the optimized meta-parameters are obtained by iterating to a preset number of rounds.

6. The method according to claim 1, wherein step S05 is characterized in that: The meta-test uses a novel cross-domain task that was not involved in the meta-training. The task is divided into a support set and a query set. The model loads the optimal meta-parameters obtained from the meta-training and fine-tunes them in a few steps through the support set to adapt to the new task and new domain. The fine-tuning process maintains the domain adversarial mechanism and the dual-path gradient propagation mode. The fine-tuned parameters are used to predict the query set, and the accuracy, precision, recall and F1 score are calculated as evaluation metrics to measure the model's performance in generalizing with few samples and across domains.

Images