Automatic depression identification and analysis method based on multi-modal data

Abstract

The invention relates to an automatic depression identification and analysis method based on multi-modal data. Technically, an efficient depression recognition and analysis model is constructed by integrating analysis of facial expressions, voices and text data. According to the invention, deep learning and multi-mode technologies are utilized to improve the early recognition and treatment effects of depression. The multi-modal emotion recognition method has the innovation points that compared with a traditional single modal, the multi-modal emotion recognition method has greater advantages in the aspect of emotion recognition, communication and perception can be carried out by utilizing multiple sensory modals at the same time, and comprehensive analysis can be carried out by combining multiple data sources and feature extraction models so as to provide more comprehensive and accurate emotion information. The method comprises the following steps: firstly, collecting a facial image, voice data and related text information of a patient through a data acquisition module, then respectively extracting and analyzing data features from different modalities by using a deep learning model, and finally integrating the features through a multi-modal fusion module, extracting relevance among the different modalities, and finally, carrying out multi-modal fusion. And the depression identification model is further optimized and improved. The multi-modal fusion method can significantly improve the accuracy and reliability of depression identification.

Description

Technical Field

[0001] This invention belongs to the field of image processing and deep learning research, specifically involving an automatic identification and analysis method for depression based on multimodal data. It is applicable to the field of mental health and aims to automatically identify and analyze depression by analyzing an individual's facial expressions, voice and text information. Background Technology

[0002] Depression is a common mental health problem that has a profound impact on an individual's social life, work, and overall quality of life. Traditional diagnosis of depression relies primarily on face-to-face interviews by clinical psychologists and patient self-reports. However, these methods are susceptible to subjective judgment and are not suitable for large-scale screening. With technological advancements, automated detection of depression using multimodal data has become a new research direction. While current technologies have made progress in single modalities such as facial expression analysis, speech processing, and text analysis, they still face the challenge of fully integrating data from various modalities to improve diagnostic accuracy.

[0003] Existing multimodal depression identification methods commonly suffer from problems including insufficient information fusion between modalities and a lack of an effective mechanism to simultaneously process and analyze different types of data (such as images, speech, and text). Furthermore, existing methods fail to effectively utilize advanced techniques in deep learning, such as convolutional networks, to extract deep-level features from each modality's data. Therefore, how to effectively fuse data from different modalities and use this data for accurate prediction of depression states is an urgent technical problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide an automatic identification and analysis method for depression based on multimodal data. This method aims to achieve more accurate and effective identification of depressive states by comprehensively analyzing multidimensional information from facial images, speech, and text data. This method integrates advanced computer vision, acoustic analysis, and natural language processing technologies, enabling automatic detection of depression to go beyond single emotional or facial expression features. It allows for a comprehensive assessment of the patient's emotional state, improving diagnostic accuracy and accelerating the diagnostic process, thus providing timely and appropriate intervention and treatment plans for the patient.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An automatic identification and analysis method for depression based on multimodal data includes the following steps:

[0007] S1. Collect patients' facial images, voice data, and text information;

[0008] S2. Use deep learning models to extract features from facial expression data;

[0009] S3. Apply Long Short-Term Memory (LSTM) network to extract emotional features from speech data;

[0010] S4. Apply natural language processing models to perform sentiment analysis on text data;

[0011] S5. The data features of the above three modalities are fused, and a hypergraph convolutional layer is used to perform deep feature interaction fusion.

[0012] S6. Input the global features of the three modalities after interactive fusion into the multilayer perceptron to obtain the prediction results of each modality;

[0013] S7. The average value of the prediction results of each modality is used as the total prediction and identification result. Based on the result, the patient's depression status is determined.

[0014] Preferably, the specific process for extracting feature information from facial image data in step S2 is as follows:

[0015] A11. Use a high-resolution camera to capture patient facial data;

[0016] A12. Preprocess facial images using noise reduction and image enhancement techniques to ensure image quality and consistency;

[0017] A13. Use the sliding window strategy to segment the key expression regions in a facial image into non-overlapping 256*256 slice images.

[0018] A14. Use pre-trained convolutional neural networks (CNNs), such as ResNet50, to perform deep learning extraction of facial expression features.

[0019] Preferably, the specific process of feature extraction of speech data in step S3 is as follows:

[0020] B11: Use a high-sensitivity microphone to capture the patient's voice;

[0021] B12: Preprocessing speech signals using digital signal processing technology, including noise reduction and sound enhancement;

[0022] B13: Use Long Short-Term Memory (LSTM) networks to analyze speech data and extract emotional features from the speech.

[0023] Preferably, the specific process of sentiment analysis of the text data in step S4 is as follows:

[0024] C11: Collect text data entered by patients via digital devices;

[0025] C12: Use text preprocessing techniques, such as standardization and stop word removal;

[0026] C13: Apply natural language processing models, such as ChineseBERT, to perform sentiment analysis and extract key emotional features.

[0027] Preferably, the patient facial data in step A11 includes videos, images, etc.

[0028] Preferably, the deep learning model described in step S2 is a convolutional neural network (ResNet50) with residual connections, and its feature extraction formula is as follows:

[0029] F face =ReLU(W·(X+F(X))+b)

[0030] Where X is the input facial image data, F(X) is the feature map processed by multiple convolutional layers, W and b are network parameters, and ReLU is the activation function. Residual connections allow the input X to be directly added to the feature F(X) processed by several convolutional network layers, thereby strengthening feature transfer, solving the gradient vanishing problem that may occur during deep neural network training, and improving the efficiency and accuracy of feature extraction.

[0031] Preferably, the expression for the hypergraph convolutional layer fusion method in step S5 is:

[0032]

[0033] H inter :=H Τ +MLP2((LayerNorm(H)) Τ )

[0034] H inter :=(H inter ) Τ +MLP3(LayerNorm((H inter ) Τ ))

[0035]

[0036] Where V n m It refers to features; MLP is a multilayer perceptron used to compute V. m The attention score. Therefore, H m ∈R B ×1×C H m It is the information carrier of intermodal interactions. The primordial hyperedge set is represented as H∈R B×M×C Each hyperedge H interm ∈R B ×1×C Further disperse and add to the corresponding graph G intra m ∈R B×N×C In order to obtain the output multimodal graph G out The symbol ⊕ indicates a broadcast addition, and G out m Includes modality-specific information and higher-order interaction information with other modalities.

[0037] By adopting the above technical solution, the present invention has the following beneficial effects:

[0038] 1. This invention uses a multimodal model to flexibly model medical data of various modalities, uses convolutional networks for intramodal information interaction, uses hypergraph convolutional layers for intermodal information interaction, and uses an online masking autoencoder to process missing data in modalities, thereby improving the survival prediction effect under multimodal data.

[0039] 2. This invention processes multimodal data separately and inputs them into a convolutional neural network to extract features, and then performs feature fusion to improve the correlation between multimodal data.

[0040] 3. This invention uses a novel hypergraph convolutional network to improve the fusion and expansion capabilities of multimodal data.

[0041] 4. This invention uses an online masking autoencoder to handle scenarios with missing modalities. By leveraging the inherent dependencies between multimodal data, missing data is generated during model inference, thereby improving the accuracy of prediction. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a flowchart of the present invention;

[0044] Figure 3 This is a user-tested diagram of the present invention;

[0045] Figure 4 This is a diagram illustrating the facial image data processing of the present invention.

[0046] Figure 5 This is a diagram illustrating the voice audio data processing of the present invention;

[0047] Figure 6 This is a diagram illustrating the text information data processing of the present invention.

[0048] Figure 7 This is a diagram of the hyperedge hybrid structure in the (hypergraph convolutional layer) hybrid graph convolutional layer of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] Example

[0051] like Figures 1 to 7 As shown, a lung cancer survival prediction method based on hybrid convolutional networks and multimodal data includes the following steps:

[0052] S1. Collect patients' facial images, voice data, and text information;

[0053] S2. Use deep learning models to extract features from facial expression data;

[0054] The specific process for extracting feature information from facial image data in step S2 is as follows:

[0055] A11. Use a high-resolution camera to capture patient facial data;

[0056] A12. Preprocess facial images using noise reduction and image enhancement techniques to ensure image quality and consistency;

[0057] A13. Use the sliding window strategy to segment the key expression regions in a facial image into non-overlapping 256*256 slice images.

[0058] A14. Use pre-trained convolutional neural networks (CNNs), such as ResNet50, to perform deep learning extraction of facial expression features.

[0059] The deep learning model described in step S2 is a convolutional neural network (ResNet50) with residual connections, and its feature extraction formula is as follows:

[0060] F face =ReLU(W·(X+F(X))+b)

[0061] S3. Apply Long Short-Term Memory (LSTM) network to extract emotional features from speech data;

[0062] The specific process of feature extraction of speech data in step S3 is as follows:

[0063] B11: Use a high-sensitivity microphone to capture the patient's voice;

[0064] B12: Preprocessing speech signals using digital signal processing technology, including noise reduction and sound enhancement;

[0065] B13: Use Long Short-Term Memory (LSTM) networks to analyze speech data and extract emotional features from the speech.

[0066] S4. Apply natural language processing models to perform sentiment analysis on text data;

[0067] The specific process of sentiment analysis of the text data in step S4 is as follows:

[0068] C11: Collect text data entered by patients via digital devices;

[0069] C12: Use text preprocessing techniques, such as standardization and stop word removal;

[0070] C13: Apply natural language processing models, such as ChineseBERT, to perform sentiment analysis and extract key emotional features.

[0071] S5. The data features of the above three modalities are fused, and a hypergraph convolutional layer is used to perform deep feature interaction fusion.

[0072] The expression for the hypergraph convolutional layer fusion method described in step S5 is:

[0073]

[0074] H inter :=H Τ +MLP2((LayerNorm(H)) Τ )

[0075] H inter :=(H inter ) Τ +MLP3(LayerNorm((H inter ) Τ ))

[0076]

[0077] Where V n m It refers to features; MLP is a multilayer perceptron used to compute V. m The attention score. Therefore, H m ∈R B ×1×C H m It is the information carrier of intermodal interactions. The primordial hyperedge set is represented as H∈R B×M×C Each hyperedge H inter m ∈R B ×1×C Further disperse and add to the corresponding graph G intram ∈R B×N×C In order to obtain the output multimodal graph G out The symbol ⊕ indicates a broadcast addition, and G out m Includes modality-specific information and higher-order interaction information with other modalities.

[0078] S6. Input the global features of the three modalities after interactive fusion into the multilayer perceptron to obtain the prediction results of each modality;

[0079] S7. The average value of the prediction results of each modality is used as the total prediction and identification result. Based on the result, the patient's depression status is determined.

[0080] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatic identification and analysis of depression based on multimodal data, the method comprising: S1. Collect patients' facial images, voice data, and text information; S2. Use deep learning models to extract features from facial expression data; S3. Apply Long Short-Term Memory (LSTM) network to extract emotional features from speech data; S4. Apply natural language processing models to perform sentiment analysis on text data; S5. The data features of the above three modalities are fused, and a hypergraph convolutional layer is used to perform deep feature interaction fusion. S6. Input the global features of the three modalities after interactive fusion into the multilayer perceptron to obtain the prediction results of each modality; S7. The average value of the prediction results of each modality is used as the total prediction and identification result. Based on the result, the patient's depression status is determined.

2. The method for automatic identification and analysis of depression based on multimodal data as described in claim 1, characterized in that... The specific process of feature extraction from facial expression data in step S2 is as follows: A11. Use a high-resolution camera to capture patient facial data; A12. Preprocess facial images using noise reduction and image enhancement techniques to ensure image quality and consistency; A13. Use the sliding window strategy to segment the key expression regions in a facial image into non-overlapping 256*256 slice images. A14. Use a pre-trained convolutional neural network (CNN), such as ResNet50, to perform deep learning extraction of facial expression features. The feature extraction formula is as follows: F face =CNN(I face (i) (1) Where F face It is the extracted facial features, I face The input is the facial image data, and θ represents the parameters of the CNN model.

3. The method for automatic identification and analysis of depression based on multimodal data as described in claim 1, characterized in that... The specific process of feature extraction of speech data in step S3 is as follows: B11: Use a high-sensitivity microphone to capture the patient's voice; B12: Preprocessing speech signals using digital signal processing technology, including noise reduction and sound enhancement; B13: Use a Long Short-Term Memory (LSTM) network to analyze speech data and extract emotional features from the speech. The feature extraction formula is as follows: F voice =LSTM(I voice ;φ) (2) Where F voice It is the extracted speech emotion features, I voice It is the input speech audio data, and φ represents the parameters of the LSTM model.

4. The method for automatic identification and analysis of depression based on multimodal data as described in claim 1, characterized in that... The specific process of sentiment analysis of the text data in step S4 is as follows: C11: Collect text data entered by patients via digital devices; C12: Use text preprocessing techniques, such as standardization and stop word removal; C13: Apply natural language processing models, such as ChineseBERT, to perform sentiment analysis and extract key emotional features. The sentiment analysis formula is as follows: Where F text It is the extracted text sentiment features, I text It is the input text data. This represents the parameters of the BERT model.

5. The method for automatic identification and analysis of depression based on multimodal data as described in claim 2, characterized in that... The patient facial data mentioned in step A11 includes videos, images, etc.

6. The depression identification and analysis method as described in claim 1, characterized in that, The deep learning model described in step S2 is a convolutional neural network (ResNet50) with residual connections, and its feature extraction formula is as follows: F face =ReLU(W·(X+F(X))+b) (4) Where X is the input facial image data, F(X) is the feature map processed by multiple convolutional layers, W and b are network parameters, and ReLU is the activation function. Residual connections allow the input X to be directly added to the feature F(X) processed by several convolutional network layers, thereby strengthening feature transfer, solving the gradient vanishing problem that may occur during deep neural network training, and improving the efficiency and accuracy of feature extraction.

7. The lung cancer survival prediction method based on hybrid graph convolutional networks and multimodal data as described in claim 1, characterized in that... The expression for the hypergraph convolutional layer fusion method in step S5 is: H inter :=H Τ +MLP2((LayerNorm(H)) Τ ) (6) H inter :=(H inter ) Τ +MLP3(LayerNorm((H inter ) Τ )) (7) Where V n m It refers to features; MLP is a multilayer perceptron used to compute V. m The attention score. Therefore, H m ∈R B×1×C H m It is the information carrier of intermodal interactions. The primordial hyperedge set is represented as H∈R B×M×C Each hyperedge H inter m ∈R B×1×C Further disperse and add to the corresponding graph G intra m ∈R B×N×C In order to obtain the output multimodal graph G out ,symbol This indicates that the broadcast has been added, and G out m It contains specific information about mode m and higher-order interaction information with other modes.

Images