Depression degree detection method and system based on bimodal emotion purification and alignment

Abstract

The application is suitable for the technical field of deep learning, and provides a depression degree detection method and system based on bimodal emotion purification and alignment, which comprises the following steps: obtaining original video data, extracting video frames for preprocessing to obtain a visual image sequence; extracting a face heat map and visual emotion features from the visual image sequence; extracting speech signals corresponding to the video frames from the original video data to obtain speech emotion features; performing emotion alignment on the visual emotion features and the speech emotion features; fusing the visual emotion features and the speech emotion features after emotion alignment to obtain fused features; and inputting the fused features into a final regression predictor to output a depression degree prediction value. Through the synergistic optimization of emotion purification, emotion alignment and multi-modal fusion, the application realizes a depression prediction effect with higher robustness and stronger generalization capability, and is suitable for the fields of mental health assessment and intelligent emotion analysis.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and in particular relates to a method and system for detecting depression levels based on bimodal emotion purification and alignment. Background Technology

[0002] Traditional clinical assessment methods for depression rely on questionnaires and expert interviews, which are limited by their high subjectivity, time-consuming nature, and heavy dependence on the experience of the assessors. With the development of deep learning and multimodal perception technologies, the automatic identification of depression using facial expression dynamics, vocal acoustic cues, and temporal behavioral patterns observed in daily interviews has gradually become a research hotspot in academia and industry. The visual modality primarily focuses on periorbital movements, facial micro-expressions, and temporal changes; while the audio modality captures depression-related emotional and behavioral characteristics from voiceprint stability, speech rate, and energy variations.

[0003] However, differences in individual identities, expression habits, and shooting conditions can significantly interfere with the extraction of emotional features, making it difficult for models to accurately learn effective cues related to depression levels from natural dialogue. Therefore, there is an urgent need for a depression detection algorithm that can effectively refine emotional features, suppress identity interference, and achieve cross-modal deep emotion alignment. This would allow for the accurate learning of multimodal representations highly correlated with depression levels from complex and diverse natural data, thus providing a reliable technical foundation for building an objective, real-time, and personalized mental health assessment system. Summary of the Invention

[0004] The purpose of this invention is to provide a method for detecting depression levels based on bimodal emotion purification and alignment, in order to solve the above-mentioned technical problems.

[0005] This invention is implemented as follows: a method for detecting depression levels based on bimodal emotion purification and alignment, comprising the following steps:

[0006] The raw video data is acquired, and video frames are extracted and preprocessed to obtain a visual image sequence.

[0007] Facial heatmaps and visual emotional features are extracted from visual image sequences, and heatmap loss is calculated based on facial heatmaps. Identity classification and depression prediction are performed based on visual emotional features to purify the visual emotional features and calculate the corresponding losses, resulting in identity loss and visual regression loss. The first-stage loss function is determined based on identity loss, visual regression loss, and heatmap loss.

[0008] Speech signals corresponding to video frames are extracted from the raw video data to obtain speech emotion features;

[0009] The visual emotion features after emotion purification are aligned with the speech emotion features, and the cross-modal alignment loss is calculated as the second-stage loss function.

[0010] The emotionally aligned visual emotional features are fused with the phonological emotional features to obtain fused features. The fused features are then input into the final regression predictor to output the predicted depression level, and the fused regression loss is calculated as the third-stage loss function.

[0011] A depression severity prediction model is trained based on the first-stage loss function, the second-stage loss function, and the third-stage loss function to predict the degree of depression.

[0012] Furthermore, the preprocessing method includes one or more of arithmetic frame extraction, center clipping, and normalization.

[0013] Furthermore, the method for extracting face heatmaps and visual emotion features from visual image sequences is as follows: the visual image sequence is input into a visual encoder including an SE module and an HourGlass module, and after passing through multiple convolutional layers, basic blocks containing the SE module, and the HourGlass module, the face heatmap and visual emotion features are output; the SE module is used to adaptively adjust the channel weights of the intermediate feature maps to enhance emotion-related features; the HourGlass module consists of basic blocks with SE, used to achieve multi-scale convolution effects, capture facial features at different scales, and generate face heatmaps.

[0014] Furthermore, the heatmap loss adopts the Awing loss function; the identity loss adopts the CE loss function; and the visual regression loss adopts the Huber loss function.

[0015] Furthermore, the speech emotion features include 40-dimensional MFCC features and 88-dimensional eGeMAPS features, which are concatenated into 128-dimensional speech features.

[0016] Furthermore, the emotion alignment method includes the following steps:

[0017] Based on the cross-modal attention mechanism, the similarity matrix between the visual emotion features and the speech emotion features after emotion purification is calculated;

[0018] Based on the similarity matrix, cross-modal emotion alignment is performed by maximizing the similarity between two modal features in the emotion representation space at the same time step, and cross-modal alignment loss is calculated to constrain the process, so that visual emotion features and speech emotion features are close to each other in a unified emotion representation space.

[0019] Furthermore, the cross-modal alignment loss employs the MSE loss function and the JSD loss function.

[0020] Furthermore, the fusion regression loss employs the CE loss function and / or the Huber loss function.

[0021] Another objective of this invention is to provide a depression severity detection system based on bimodal emotion purification and alignment, for implementing the aforementioned depression severity detection method, comprising:

[0022] The data acquisition and preprocessing module is used to acquire raw video data and extract video frames for preprocessing to obtain a visual image sequence;

[0023] The sentiment purification module is used to extract face heatmaps and visual sentiment features from visual image sequences, and calculate heatmap loss based on face heatmaps; and to perform identity classification and depression prediction based on visual sentiment features to purify the visual sentiment features and calculate the corresponding losses to obtain identity loss and visual regression loss; and to determine the first-stage loss function based on identity loss, visual regression loss and heatmap loss.

[0024] The speech feature extraction module is used to extract speech signals corresponding to video frames from the raw video data to obtain speech emotion features;

[0025] The emotion alignment module is used to align the emotion-purified visual emotion features with the speech emotion features and calculate the cross-modal alignment loss as the second-stage loss function.

[0026] The multimodal feature fusion module is used to fuse the emotion-aligned visual emotion features with the speech emotion features to obtain fused features; the fused features are input into the final regression predictor, outputting the depression level prediction value, and the fusion regression loss is calculated as the third-stage loss function;

[0027] The depression severity prediction module is used to train a depression severity prediction model based on the first-stage loss function, the second-stage loss function, and the third-stage loss function to predict the degree of depression.

[0028] This invention provides a depression severity detection method based on bimodal sentiment purification and alignment. Through the synergistic optimization of sentiment purification, sentiment alignment, and multimodal fusion, it achieves higher robustness and stronger generalization ability in depression prediction, making it applicable to fields such as mental health assessment and intelligent emotion analysis. Furthermore, this invention uses a three-stage loss function for model training, enabling better training results with limited training data in real-world scenarios. It is particularly effective in environments with poor data quality and incomplete modal features, allowing for better feature purification and fusion. The model trained by this invention can also be integrated into real-world applications, achieving high-efficiency and high-accuracy depression detection through simple data uploads. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the first stage of the depression level detection method based on bimodal emotion purification and alignment provided in an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the second stage of the depression level detection method based on bimodal emotion purification and alignment provided in an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of the third stage of the depression level detection method based on bimodal emotion purification and alignment provided in an embodiment of the present invention.

[0032] In the diagram: y h -Heatmap labels; - Face heatmap; y id -Identity tags; -Identity prediction result; y dep -Visual depression score label; - Visual prediction of depression level; λ1 - Weight of identity loss; λ2 - Weight of depression level; L1 - First-stage loss function; f v -Visual emotional characteristics; f a - Voice emotion features; Mv - Video modality feature similarity matrix; Ma - Audio modality feature similarity matrix; L2 - Second-stage loss function; y - True label of depression level; - Predicted final depression level; L3 - Third-stage loss function. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0034] The collaborative alignment of visual and audio data not only enhances sensitivity to emotional fluctuations but also establishes a more stable emotional association structure at the frame level. Simultaneously, by employing identity feature suppression strategies, the model's generalization ability among subjects can be further improved, reducing the impact of individual differences on emotion prediction. Therefore, this invention proposes a bimodal emotion purification and alignment depression detection algorithm, applicable to multimodal scenarios, reducing training costs, and improving detection accuracy, laying a technical foundation for building a more objective, real-time, and personalized mental health assessment system.

[0035] Specifically, such as Figures 1-3As shown, in one embodiment of the present invention, a method for detecting depression level based on bimodal emotion purification and alignment is provided. The detection process can employ a multi-stage training approach, wherein the first stage trains the visual encoder, the second stage incorporates the speech modality into the input for bimodal alignment training, and the third stage uses a fusion algorithm to fuse features and obtain a predicted value or emotion score for depression level. The method specifically includes the following steps:

[0036] Phase 1: Acquire raw video data and extract video frames for preprocessing to obtain a visual image sequence; extract face heatmaps from the visual image sequence. and visual emotional features f v And based on facial heatmaps Calculate the heatmap loss; and based on visual emotional features f v Identity classification and depression prediction were performed separately to assess visual emotional features f v Sentiment purification is performed, and the corresponding losses are calculated to obtain identity loss and visual regression loss; the first-stage loss function L1 is determined based on identity loss, visual regression loss, and heatmap loss.

[0037] The second stage involves extracting the speech signals corresponding to the video frames from the raw video data to obtain the speech emotion features f. a Visual emotional features after emotional refinement f v With speech emotion features f a Sentiment alignment is performed, and cross-modal alignment loss is calculated as the second-stage loss function L2;

[0038] Phase 3: Aligning visual emotional features with emotions v With speech emotion features f a The features are fused to obtain fused features; the fused features are input into the final regression predictor to output the predicted depression level, and the fused regression loss is calculated as the third-stage loss function L3;

[0039] Based on the first-stage loss function L1, the second-stage loss function L2, and the third-stage loss function L3 mentioned above, a depression level prediction model is trained to predict the degree of depression.

[0040] Correspondingly, in another embodiment of the present invention, a depression level detection system based on bimodal emotion purification and alignment is also provided to implement the above-mentioned depression level detection method, comprising:

[0041] The data acquisition and preprocessing module is used to acquire raw video data and extract video frames for preprocessing to obtain a visual image sequence;

[0042] The sentiment extraction module is used to extract face heatmaps from visual image sequences. and visual emotional features f v And based on facial heatmaps Calculate heatmap loss; and use it to calculate the loss based on visual emotion features f v Identity classification and depression prediction were performed separately to assess visual emotional features f v Sentiment purification is performed, and the corresponding losses are calculated to obtain identity loss and visual regression loss; the first-stage loss function L1 is determined based on identity loss, visual regression loss, and heatmap loss.

[0043] The speech feature extraction module is used to extract speech signals corresponding to video frames from the raw video data to obtain speech emotion features f. a ;

[0044] The emotion alignment module is used to refine the visual emotion features f. v With speech emotion features f a Sentiment alignment is performed, and cross-modal alignment loss is calculated as the second-stage loss function L2;

[0045] The multimodal feature fusion module is used to integrate the sentiment-aligned visual sentiment features f v With speech emotion features f a The features are fused to obtain fused features; the fused features are input into the final regression predictor to output the predicted depression level, and the fused regression loss is calculated as the third-stage loss function L3;

[0046] The depression severity prediction module is used to train a depression severity prediction model based on the first-stage loss function L1, the second-stage loss function L2, and the third-stage loss function L3, so as to predict the degree of depression.

[0047] In a preferred embodiment of the present invention, in the first stage, consecutive video frames are read from the original video as visual input, and preprocessing operations such as arithmetic frame extraction, center cropping, and image normalization are performed on them to obtain a visual image sequence that is temporally uniform, spatially aligned, and has a uniform numerical range, laying the foundation for subsequent stable feature extraction. In practical applications, arithmetic frame extraction is used to sample the original video sequence at fixed intervals to obtain a frame sequence, ensuring the temporal uniformity of the visual image input; center cropping can be automatically processed using the OpenFace library.

[0048] As a preferred embodiment of the present invention, such as Figure 1 As shown, face heatmaps are extracted from visual image sequences. and visual emotional features f vThe method is as follows: input the visual image sequence into a visual encoder that includes an SE module and an HourGlass module, pass through multiple convolutional layers, basic blocks containing SE modules, and HourGlass modules, and output a face heatmap. and visual emotional features f v The SE module adaptively adjusts the channel weights of intermediate feature maps to enhance emotion-related features. The visual encoder's basic structure incorporates the SE module to enhance its channel attention modeling capability in multi-channel features. The HourGlass module, composed of basic blocks with SE, achieves multi-scale convolution effects, capturing facial features at different scales and extracting 64 key facial points. One-stage training of this module strengthens its ability to extract facial depression features while removing some identity-related interference information from the features, generating a facial heatmap. And compared with the deep features extracted by the visual encoder (i.e., visual emotion features f) v They were then fed into the emotion purification module and entered the first stage of training.

[0049] During the first phase of training, the sentiment refinement module is constructed based on the adversarial learning approach, extracting the visual sentiment features f from the visual encoder output. v Input the data into the identity classifier and depression predictor respectively for identity classification and depression prediction, and output the identity prediction results. and visual predictive value of depression level And calculate the corresponding loss, whereby the loss is based on the identity prediction result. With identity tag y id Identity loss can be calculated based on visual predictive values ​​of depression levels. and visual depression value label y dep It can calculate the visual regression loss; and simultaneously calculate the face heatmap output by the HourGlass module. With the corresponding heatmap label y h Calculate the heatmap loss. Based on the preset weights for identity loss and the weight λ2 for depression level, the identity loss, visual regression loss, and heatmap loss can be combined with hyperparameters to form the first-stage loss function L1. Based on the first-stage loss function L1, the visual encoder parameters for this stage can be trained through gradient backpropagation. During joint optimization and backpropagation, this adversarial mechanism prompts the visual encoder to weaken the influence of identity-related information, while further enhancing the HourGlass module's ability to extract features from key facial regions. This embodiment of the invention, through the above-mentioned emotion purification and adversarial learning strategies, enables the visual encoder to learn visual representations that are more identity-independent and more emotion-related.

[0050] It should be noted that the heatmap loss can use the Adaptive Wing loss function; the identity loss can use the Cross Entropy (CE) loss function; and the visual regression loss can use the Huber loss function.

[0051] In a preferred embodiment of the present invention, after completing the first stage of training, the obtained emotionally refined visual emotion features f v Simultaneously, audio modal data is loaded as subsequent input; the raw speech signal is processed by a feature extractor to obtain speech emotion features f aligned with the video frames. a The sequence represents the speech emotion features f corresponding to each time slice. a Vector. The speech emotion feature f a It contains 40-dimensional MFCC features and 88-dimensional eGeMAPS features, which are concatenated to form 128-dimensional speech features.

[0052] As a preferred embodiment of the present invention, such as Figure 2 As shown, in the second stage, the extracted speech emotion features f a As the second input, it is combined with the refined visual emotional features f obtained in the previous stage. v The data are jointly sent to the sentiment alignment module for sentiment alignment; specifically, the sentiment alignment method includes the following steps:

[0053] Based on a cross-modal attention mechanism, the visual emotional features f after emotional purification are analyzed. v With speech emotion features f a These two modal features are linearly transformed and mapped to the emotion space to obtain the projected two modal features, and the emotionally refined visual emotion feature f is calculated. v With speech emotion features f a The similarity matrices are used to obtain the video modal feature similarity matrix Mv and the audio modal feature similarity matrix Ma. Based on the video modal feature similarity matrix Mv and the audio modal feature similarity matrix Ma, cross-modal emotion alignment is performed by maximizing the similarity of the two modal features in the emotion representation space at the same time step. The cross-modal alignment loss is calculated to constrain the process, so that the visual emotion feature f v With speech emotion features f a They move closer to each other within a unified sentiment representation space. The cross-modal alignment loss can be a combination of the mean squared error (MSE) loss function and the Jensen-Shannon divergence (JSD) loss function.

[0054] In this embodiment of the invention, the emotion alignment module aims to establish emotional consistency between images and speech at a cross-modal level. Specifically, the emotion alignment module utilizes a cross-modal attention mechanism and feature transformation structure to integrate visual modality (i.e., visual emotional features f) with the speech. v ) and audio modality (i.e., speech emotion features f) a The visual and audio features are mapped to a unified sentiment representation space, and a frame-level alignment strategy is used to capture the synchronous relationship between visual facial expression changes and speech acoustic cues in the sentiment dimension. Within the sentiment alignment module, a similarity enhancement method is employed to achieve higher matching degrees between sentiment cues from both modalities at the same temporal position, effectively improving the stability of cross-modal fusion. At this stage, by jointly optimizing visual and audio features and performing cross-modal sentiment alignment, the model's ability to understand the user's true emotional state is further strengthened, reducing the impact of missing information or noise from a single modality. Simultaneously, this stage suppresses inconsistencies between modalities, making the final output sentiment features more unified, robust, and discriminative, providing a more consistent and high-quality multimodal representation for subsequent sentiment regression prediction.

[0055] As a preferred embodiment of the present invention, such as Figure 3 As shown, in the third stage, after completing cross-modal emotion alignment, the aligned visual emotion features f are... v With speech emotion features f a The input is processed by a multimodal feature fusion module, which generates stable and consistent multimodal fused features based on the weights and contributions of the two modal features in the current sample. This stage employs mutual guidance learning, feature concatenation, and attention-based dynamic fusion strategies. This allows the multimodal feature fusion module to automatically adjust the fusion weights according to the importance of different modalities, thereby highlighting the modality signal with higher emotional relevance in the final prediction. The fused unified emotional feature is then input into the final regression predictor to output the user's emotional score or the predicted value of the final depression level. The fusion regression loss is calculated based on the true label y of depression severity. This stage is specifically implemented through the following modules:

[0056] Audio guidance module: The average of the audio features of k segments is taken to obtain the global audio features. Then, the global audio features are added together with the features of each frame of the video, which is equivalent to integrating the global audio features into the features of each frame of the video. Then, it is multiplied by w on the left. This w is a trainable weight matrix. Then, after passing through the normalized exponential (softmax) function, a k-dimensional attention score can be obtained. Then, this score is multiplied by the source video features. This module can be understood as using audio features to perform an attention operation on video features.

[0057] Video guidance module: It simply swaps the input order of audio and video, and the process is the same as the audio guidance module;

[0058] The co-attention module uses the transformer's QKV as attention. First, the video provides Q, and the audio provides K and V. The attention is calculated and weighted onto the audio. Then, the order is reversed, with the audio as Q and the video as K and V.

[0059] Frame selection module: Concatenate the features of the two modalities along the feature dimension to obtain the fused feature, multiply it by the weight matrix w2, w2 can be trained, and then use the softmax function to calculate the score. Then multiply the score by the fused feature to assign higher weights to more critical frames.

[0060] Fully connected layer: This layer processes features through a weight matrix to obtain a numerical value, which is the predicted value. This stage is optimized using a unified fusion regression loss, enabling end-to-end training of the multimodal feature fusion module and improving the accuracy and stability of sentiment prediction. In practical applications, the fusion regression loss can employ the CE loss function and / or the Huber loss function.

[0061] In this embodiment of the invention, the depression level prediction model finally trained based on the first-stage loss function L1, the second-stage loss function L2, and the third-stage loss function L3 includes components such as a visual encoder, an identity classifier, a depression predictor, an emotion alignment module, a multimodal feature fusion module, and a final regression predictor. The first-stage loss function L1 is used to train the parameters of the visual encoder, identity classifier, and depression predictor; the second-stage loss function L2 is used to train the parameters of the emotion alignment module; and the third-stage loss function L3 is used to train the parameters of the multimodal feature fusion module and the final regression predictor. In practical applications, the training phase strictly follows the three-stage steps. When using the above depression level prediction model, only the required bimodal data (such as video data including audio) needs to be input to obtain the depression level detection value in one step.

[0062] It should be noted that each of the above modules can be implemented as a computer program, which can run on a computer device. The computer device's memory can store the computer program that makes up each module, enabling the processor to execute each step of the above method.

[0063] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0064] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0065] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method for detecting depression levels based on bimodal emotion purification and alignment, characterized in that, Includes the following steps: The raw video data is acquired, and video frames are extracted and preprocessed to obtain a visual image sequence. Extract face heatmaps and visual emotion features from visual image sequences, and calculate heatmap loss based on face heatmaps; Based on visual emotional features, identity classification and depression prediction are performed to purify the visual emotional features and calculate the corresponding losses, resulting in identity loss and visual regression loss; the first-stage loss function is determined based on identity loss, visual regression loss and heatmap loss. Speech signals corresponding to video frames are extracted from the raw video data to obtain speech emotion features; The visual emotion features after emotion purification are aligned with the speech emotion features, and the cross-modal alignment loss is calculated as the second-stage loss function. The emotionally aligned visual emotional features are fused with the speech emotional features to obtain the fused features; The fused features are input into the final regression predictor, which outputs the predicted depression level and calculates the fused regression loss as the third-stage loss function. A depression severity prediction model is trained based on the first-stage loss function, the second-stage loss function, and the third-stage loss function to predict the degree of depression.

2. The method for detecting depression based on bimodal emotion purification and alignment according to claim 1, characterized in that, The preprocessing methods include one or more of arithmetic frame extraction, center cropping, and normalization.

3. The method for detecting depression based on bimodal emotion purification and alignment according to claim 1, characterized in that, The method for extracting face heatmaps and visual emotion features from visual image sequences is as follows: The visual image sequence is input into a visual encoder including an SE module and an HourGlass module. After passing through multiple convolutional layers, basic blocks containing the SE module, and the HourGlass module, the face heatmap and visual emotion features are output. The SE module is used to adaptively adjust the channel weights of the intermediate feature maps to enhance emotion-related features. The HourGlass module consists of basic blocks with SE and is used to achieve multi-scale convolution effects, capture facial features at different scales, and generate face heatmaps.

4. The method for detecting depression based on bimodal emotion purification and alignment according to claim 1 or 3, characterized in that, The heatmap loss uses the Awing loss function; the identity loss uses the CE loss function; and the visual regression loss uses the Huber loss function.

5. The method for detecting depression based on bimodal emotion purification and alignment according to claim 1, characterized in that, The speech emotion features include 40-dimensional MFCC features and 88-dimensional eGeMAPS features, which are concatenated to form 128-dimensional speech features.

6. The method for detecting depression based on bimodal emotion purification and alignment according to claim 1, characterized in that, The emotional alignment method includes the following steps: Based on the cross-modal attention mechanism, the similarity matrix between the visual emotion features and the speech emotion features after emotion purification is calculated; Based on the similarity matrix, cross-modal emotion alignment is performed by maximizing the similarity between two modal features in the emotion representation space at the same time step, and cross-modal alignment loss is calculated to constrain the process, so that visual emotion features and speech emotion features are close to each other in a unified emotion representation space.

7. The method for detecting depression based on bimodal emotion purification and alignment according to claim 1 or 6, characterized in that, The cross-modal alignment loss employs the MSE loss function and the JSD loss function.

8. The method for detecting depression based on bimodal emotion purification and alignment according to claim 1, characterized in that, The fusion regression loss adopts the CE loss function and / or the Huber loss function.

9. A depression severity detection system based on bimodal emotion purification and alignment, used to implement the depression severity detection method according to any one of claims 1-8, characterized in that, include: The data acquisition and preprocessing module is used to acquire raw video data and extract video frames for preprocessing to obtain a visual image sequence; The emotion extraction module is used to extract face heatmaps and visual emotion features from visual image sequences, and calculate heatmap loss based on the face heatmaps. And it is used to classify identity and predict depression based on visual emotional features, to purify the visual emotional features and calculate the corresponding loss, to obtain the identity loss and visual regression loss; and to determine the first-stage loss function based on the identity loss, visual regression loss and heatmap loss. The speech feature extraction module is used to extract speech signals corresponding to video frames from the raw video data to obtain speech emotion features; The emotion alignment module is used to align the emotion-purified visual emotion features with the speech emotion features and calculate the cross-modal alignment loss as the second-stage loss function. The multimodal feature fusion module is used to fuse the emotion-aligned visual emotion features with the speech emotion features to obtain fused features; the fused features are input into the final regression predictor, outputting the depression level prediction value, and the fusion regression loss is calculated as the third-stage loss function; The depression severity prediction module is used to train a depression severity prediction model based on the first-stage loss function, the second-stage loss function, and the third-stage loss function to predict the degree of depression.

Images