Depression identification method based on feature fusion and multichannel neural network

Abstract

The invention discloses a depression identification method based on feature fusion and a multichannel neural network, and relates to the technical field of artificial intelligence. The method aims at solving the problems that an existing method lacks advanced semantic features containing more complex and rich facial attribute information, and a traditional deep learning algorithm lacks complementarity and feature fusion of different source features. The method specifically comprises the following steps: establishing a video data set of male and female depression patients of different age groups; a multi-channel neural network integrating a cross-multi-branch channel intelligent attention mechanism and comprehensive feature fusion is established, and the multi-channel neural network firstly uses an open source computer vision library to extract frames from a video data set at corresponding time intervals. According to the method, global and local features of the whole face, eye and mouth areas are combined, advanced semantic features including more complex and rich face attribute information are included, and the accuracy of depression recognition is improved through complementarity and feature fusion of different source features.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for identifying depression based on feature fusion and multi-channel neural networks. Background Technology

[0002] Depression is a very serious mental illness characterized by high incidence, frequent relapses, and a trend towards affecting younger people. However, there are currently no unique and effective clinical features for depression. In most cases, the diagnosis of depression requires methods such as structured clinical interviews and depression scales, and it is highly dependent on the experience of clinicians. The entire process lacks objective and effective evidence and is time-consuming. Since the facial expressions of depressed patients often have obvious features related to sadness, researchers in the field of affective computing are attempting to use knowledge from psychology and computer vision to automatically identify depression through facial visual features.

[0003] A search revealed a Chinese patent application (CN202311037340.7) that discloses an automatic depression identification method based on facial video. The method includes the following steps: acquiring facial video data to be identified and preprocessing it to obtain facial video subsamples (Vc); constructing a deep learning model (LDSML) that jointly learns the depression label distribution and label metric. However, a limitation of existing methods is the lack of high-level semantic features containing more complex and rich facial attribute information. Furthermore, traditional deep learning algorithms lack the complementarity of features from different sources and the importance of feature fusion. Therefore, this paper designs a depression identification method based on feature fusion and multi-channel neural networks to compensate for the shortcomings of existing identification methods. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a depression recognition method based on feature fusion and multi-channel neural networks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for identifying depression based on feature fusion and multi-channel neural networks includes the following steps:

[0007] S1: Establish a video dataset including male and female patients with depression of different ages;

[0008] S2: Establish a multi-channel neural network that integrates channel-wise attention mechanisms across multiple branches and comprehensive feature fusion;

[0009] S3: The multi-channel neural network first uses an open-source computer vision library to extract frames from the video dataset at corresponding time intervals;

[0010] S4: Utilize a multi-task cascaded convolutional neural network to detect and locate facial key points in each frame;

[0011] S5: Using the coordinates obtained from the identified key facial points, selectively divide the area corresponding to the entire face, eyes, and mouth;

[0012] S6: Use the entire face image as input to a ResNet50 with a dedicated CABottleneck;

[0013] S7: Extract global facial features from facial images, and use different ResNets to process the corresponding images for the eye and mouth regions to extract local features;

[0014] S8: The extracted global and local features are fused through a cascade of feature fusion layers. The fully connected layer generates an output score based on the fused information, providing a comprehensive representation. Combining global and local facial features, it can accurately identify depression.

[0015] Preferably, in step S1, video datasets are formed by collecting video data that can reflect the facial expressions and body postures of patients with depression.

[0016] Furthermore, in S2, the multi-channel neural network combined with the CA mechanism transforms the 50-layer residual network into a feature extraction module to mine global and local features. The global features are the patient's facial features, and the local features are the patient's mouth and eye features.

[0017] A further preferred embodiment: In S3, the open-source computer vision library extracts frames from the video dataset at 5-second intervals.

[0018] As a preferred embodiment of the present invention: in S4, the key facial points are the patient's entire face, eyes, and mouth.

[0019] As a further preferred embodiment of the present invention: in S5, the size of the overall facial area is set to 256x256x3, representing the height, width, and red, green, and blue channels, respectively, and the sizes of the trimmed areas for the eyes and lips are 96x192x3 and 96x128x3, respectively.

[0020] As a further aspect of the present invention: in S6, CABotleneck is used to integrate the attention mechanism on the channel side, and the fully connected layer in ResNet50 is replaced by an empty sequence layer, removing the classification head.

[0021] Based on the aforementioned scheme: In S7, different ResNets are used for specific facial components to extract the patient's overall face, mouth, and eyes in a targeted manner to obtain healthy local features.

[0022] The beneficial effects of this invention are as follows:

[0023] 1. The multi-channel neural network disclosed in this invention integrates a channel intelligent attention mechanism across multiple branches to extract different visual features, which helps in the automatic identification of depression. This method extracts and fuses global features of the entire face image and local features of the eye and mouth images. By combining global and local features of the entire face, eye and mouth regions, it contains high-level semantic features with more complex and rich facial attribute information. Through the complementarity of features from different sources and feature fusion, the accuracy of depression identification is improved.

[0024] 2. This method establishes a database for AI-based depression detection by collecting data such as patients' facial expressions and body postures. It preprocesses videos by extracting frames and detects and locates facial key points using MTCNN. Based on the coordinates of the identified key points, it selectively crops facial regions, eyes, and mouth images. Deep analysis is performed on the collected database to construct a deep learning classification model. The AI-based depression detection technology is tested and validated by introducing an improved ResNet50 architecture with a custom CABottleneck and channel-based intelligent attention mechanism. This design aims to extract and fuse the global and local features necessary for depression recognition. A comprehensive evaluation of the MTCNN system's performance in depression recognition is conducted to improve the accuracy of the MTCNN system in identifying depression through facial features.

[0025] 3. The overall facial area size is set to 256x256x3, representing the height, width, and red, green, and blue (RGB) channels, respectively. The trimmed areas for the eyes and lips are 96x192x3 and 96x128x3, respectively. This standardized implantation process helps to separate key facial components, maintain size uniformity, facilitate subsequent processing steps, and improve the accuracy of facial feature recognition.

[0026] 4. CABotleneck is used to integrate channel-side attention mechanisms. The fully connected layers in ResNet50 are replaced by an empty sequence layer, and the classification head is removed. These transformations are implemented to convert the model into a dedicated feature extraction module, focusing on extracting robust global features from facial images. The integrated CABotleneck embeds a channel-aware mechanism, which can successfully guide the model's attention and improve the accuracy of facial and local feature extraction.

[0027] 5. Channel attention mechanism plays a significant role in improving the accuracy of MCNN in identifying depression. Although individual facial features provide a basic understanding, the fusion of multiple features promotes a more detailed and comprehensive recognition method. The fusion of facial, mouth, and eye data is consistently superior to other configurations, enabling better analysis and recognition of facial features, thereby accurately diagnosing depression. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the main view structure of a depression recognition method based on feature fusion and multi-channel neural network proposed in this invention;

[0029] Figure 2 This is a schematic diagram of the multi-channel neural network system structure of a depression recognition method based on feature fusion and multi-channel neural network proposed in this invention;

[0030] Figure 3 This is a bar chart illustrating the relationship between the AVEC2014 dataset and whether or not CABottleneck is used in the depression recognition method based on feature fusion and multi-channel neural networks proposed in this invention.

[0031] Figure 4 This is a bar chart illustrating the relationship between the corresponding hospital dataset and whether or not CABottleneck is used in the depression identification method based on feature fusion and multi-channel neural network proposed in this invention.

[0032] Figure 5 This is a bar graph illustrating the relationship between the AVEC2014 dataset and whether or not feature fusion is performed in the depression recognition method based on feature fusion and multi-channel neural networks proposed in this invention.

[0033] Figure 6 This is a bar graph illustrating the relationship between the corresponding hospital dataset and whether or not feature fusion is performed in the depression identification method based on feature fusion and multi-channel neural networks proposed in this invention. Detailed Implementation

[0034] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0035] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0036] Example:

[0037] A depression identification method based on feature fusion and multi-channel neural networks, such as Figure 1 As shown, it includes the following steps:

[0038] S1: Establish a video dataset including male and female patients with depression of different ages;

[0039] S2: Establish a multichannel neural network (MCNN) that integrates channel-wise attention (CA) mechanisms across multiple branches and comprehensive feature fusion;

[0040] S3: Multichannel Neural Network (MCNN) first uses an open-source computer vision library (OpenCV) to extract frames from the video dataset at corresponding time intervals;

[0041] S4: Detect and locate facial key points in each frame using a multi-task cascaded convolutional neural network (MTCNN);

[0042] S5: Using the coordinates obtained from the identified key facial points, selectively divide the area corresponding to the entire face, eyes, and mouth;

[0043] S6: Use the entire face image as input to a ResNet50 with a dedicated CABottleneck;

[0044] S7: Extract global facial features from facial images, and use different ResNets to process the corresponding images for the eye and mouth regions to extract local features;

[0045] S8: The extracted global and local features are fused through a cascade of feature fusion layers. The fully connected layer generates an output score based on the fused information, providing a comprehensive representation. Combining global and local facial features, it can accurately identify depression.

[0046] In S1, video datasets are collected that can reflect the facial expressions and physical characteristics of patients with depression.

[0047] In S2, the multichannel neural network (MCNN) combined with the CA mechanism transforms the 50-layer residual network into a feature extraction module to mine global and local features. The global features are the patient's facial features, and the local features are the patient's mouth and eye features.

[0048] In S3, the open-source computer vision library (OpenCV) extracts frames from the video dataset at 5-second intervals.

[0049] In S4, the key facial points are the patient's overall face, eyes, and mouth;

[0050] In S5, the overall facial area size is set to 256x256x3, representing the height, width, and red, green, and blue (RGB) channels, respectively. The sizes of the trimmed areas for the eyes and lips are 96x192x3 and 96x128x3, respectively. This standardized implantation process helps to separate key facial components, maintain size uniformity, and facilitate subsequent processing steps.

[0051] In S6, CABotleneck is used to integrate the channel-side attention mechanism. The fully connected layer in ResNet50 is replaced by an empty sequence layer, and the classification head is removed. These transformations are implemented to convert the model into a dedicated feature extraction module, focusing on extracting robust global features from facial images. The integrated CABotleneck embeds a channel-aware mechanism, which can successfully guide the model's attention and improve the accuracy of facial and local feature extraction.

[0052] In S7, different ResNets are used for specific facial components to extract the patient's overall face, mouth, and eyes in a targeted manner to obtain healthy local features;

[0053] In S8, this integration of features at different spatial scales enhances the system’s ability to capture subtle patterns associated with depressive expressions.

[0054] Verification Example 1:

[0055] To verify the performance of MCNN in identifying depression, two configurations were evaluated: MCNN without channel attention (CA) and MCNN with channel attention (CA).

[0056] Figure 3-4 This demonstrates the significant impact of the CA mechanism on the recognition capabilities of MCNN. Figure 3 Display the results of the AVEC2014 dataset. Figure 4 This displays the results of the hospital dataset obtained for the corresponding hospital.

[0057] For the AVEC2014 dataset, the MAE of MCNN without CA is 8.23 ​​and the RMSE is 9.94. In contrast, after adding the CA mechanism, the MAE is improved, dropping to 8.04 and the RMSE drops to 9.65. Therefore, the MAE decreases by 0.19 and the RMSE decreases by 0.29, highlighting the impact of the CA mechanism on the recognition ability of MCNN.

[0058] When evaluating the model on datasets obtained from corresponding hospitals, the CA mechanism showed even more significant results:

[0059] Without CA, MCNN achieved a MAE of 7.61 and an RMSE of 9.55. After adding the CA mechanism, the MAE and RMSE of these metrics decreased significantly by 0.77 and 0.78, respectively, with the final MAE at 6.84 and RMSE at 8.77. This significant improvement highlights the key role of the CA mechanism in improving the recognition performance of MCNN, especially for datasets with different features.

[0060] The above validation results illustrate the key role of channel attention in improving the recognition accuracy of MCNN. The improvements on the two datasets affirm the importance of incorporating this attention mechanism into neural network architectures, especially when dealing with nuanced tasks such as depression recognition.

[0061] Verification Example 2:

[0062] To investigate the impact of feature fusion on the depression recognition performance of MCNN, Figure 5-6 The impact of feature fusion on the recognition performance of MCNN is illustrated graphically.

[0063] For the AVEC2014 dataset, using facial features alone yields a MAE of 8.22 and an RMSE of 9.80. Combining mouth data with facial features slightly increases the MAE to 8.24, but decreases the RMSE to 9.77.

[0064] A similar trend was observed when integrating eye data, with MAE slightly increasing to 8.28 and RMSE to 9.99. However, after fusing facial, mouth, and eye features, the MAE was 8.04 and the RMSE was 9.65; compared to using only facial features, the MAE decreased by 0.18 and the RMSE decreased by 0.15. This result highlights the synergistic advantages of fusing multiple facial features.

[0065] Using only facial features in the corresponding hospital dataset, the MAE was 7.03 and RMSE was 9.04. Adding mouth data slightly decreased the MAE to 6.98, while the RMSE decreased to 8.88. Adding eye data further improved the MAE to 7.03 and the RMSE to 8.96. Notably, the comprehensive fusion of face, mouth, and eye features achieved the optimal results, with an MAE of 6.84 and an RMSE of 8.77. Compared to using only face features, the MAE and RMSE decreased by 0.19 and 0.27 respectively. These results demonstrate the importance of fusing face, mouth, and eye features to enhance depression recognition and effectively improve the accuracy of MCNN in depression identification.

[0066] The two evaluation methods above illustrate the importance of feature fusion in improving the recognition accuracy of MCNN. Although individual facial features provide a basic understanding, the fusion of multiple features promotes a more detailed and comprehensive recognition method. The fusion of facial, mouth, and eye data is consistently superior to other configurations, highlighting the advantages of overall feature integration.

[0067] This method collects data on patients' facial expressions and body features to establish a database for AI-based depression detection. It preprocesses videos by extracting frames and uses MTCNN to detect and locate facial key points. Based on the coordinates of these key points, it selectively crops facial regions, eyes, and mouth images. Deep analysis is performed on the collected database to construct a deep learning classification model. The method then tests and validates the AI-based depression detection solution by introducing an improved ResNet50 architecture with a custom CABottleneck and channel-based intelligent attention mechanism. This design aims to extract and fuse the global and local features necessary for depression identification. A comprehensive evaluation of the MTCNN system's performance in depression identification is conducted to improve the accuracy of the MTCNN system in identifying depression through facial features.

[0068] The multichannel neural network (MCNN) disclosed in this invention integrates a channel-based intelligent attention mechanism across multiple branches to extract different visual features, which helps in the automatic identification of depression. This method extracts and fuses global features of the entire face image and local features of the eye and mouth images. By combining global and local features of the entire face, eye and mouth regions, it contains high-level semantic features with more complex and rich facial attribute information. Through the complementarity of features from different sources and feature fusion, the accuracy of depression identification is improved.

[0069] The workflow of this method is summarized as follows: First, frames are extracted from the video every 5 seconds, and facial key points are detected and located using a multi-task cascaded convolutional neural network (MTCNN).

[0070] Based on the coordinates of the identified key points, selective cropping is performed on the entire face, the areas corresponding to the eyes and mouth;

[0071] Then, the entire face image is fed as input into an improved residual network (ResNet) with 50 layers, which has a custom cabottlenneck with a coordinate attention mechanism to extract global features;

[0072] Similarly, images of the eyes and mouth are fed into two other ResNets to extract local features. The extracted global and local features are concatenated and fused in a feature fusion layer to produce an output score. Combining global and local facial features, depression can be accurately identified.

[0073] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1.A method for depression recognition based on feature fusion and multi-channel neural network, characterized in that, It comprises the following steps: S1: establishing a video dataset including male and female patients with depression of different ages; S2: establishing a multi-channel neural network integrating a cross-multi-branch channel-wise attention mechanism and comprehensive feature fusion; S3: the multi-channel neural network first uses an open-source computer vision library to extract frames from the video dataset at a corresponding time interval; S4: a multi-task cascaded convolutional neural network is used to detect and locate facial key points in each frame of picture; S5: using the coordinates obtained from the determined key facial points, the regions corresponding to the whole face, eyes and mouth are selectively divided; S6: the whole face image is used as the input of ResNet50 with a dedicated CABottleneck; S7: global facial features are extracted from the face image, and different ResNet is used to process the corresponding images to extract local features; S8: the extracted global and local features are fused through the cascade of feature fusion layers, and the fully connected layer generates output scores according to the fusion information, providing comprehensive representation, combining global and local facial features to accurately identify depression. 2.The depression recognition method based on feature fusion and multi-channel neural network according to claim 1, characterized in that, In S1, video data reflecting the facial expression and body feature of patients with depression is collected to form a video dataset. 3.The depression recognition method based on feature fusion and multi-channel neural network according to claim 1, characterized in that, In S2, the multi-channel neural network combines the CA mechanism, converts the 50-layer residual network into a feature extraction module, and excavates global and local features. The global feature is the patient's facial feature, and the local feature is the patient's mouth and eye feature. 4.The depression recognition method based on feature fusion and multi-channel neural network according to claim 1, characterized in that, In S3, the open-source computer vision library extracts frames from the video dataset at a time interval of 5 seconds. 5.The depression recognition method based on feature fusion and multi-channel neural network according to claim 1, characterized in that, In S4, the facial key points are the overall face, eyes and mouth of the patient. 6.The depression recognition method based on feature fusion and multi-channel neural network according to claim 1, characterized in that, In S5, the size of the overall face region is set to 256x256x3, representing height, width and red, green and blue channels respectively, and the size of the cropped region of eyes and lips is 96x192x3 and 96x128x3 respectively. 7.The depression recognition method based on feature fusion and multi-channel neural network according to claim 1, characterized in that, In S6, CABotleneck is used to integrate the channel-side attention mechanism, and the fully connected layer in ResNet50 is replaced by an empty sequence layer, removing the classification head. 8.The depression recognition method based on feature fusion and multi-channel neural network according to claim 1, characterized in that, In S7, different ResNet is used for specific facial components to extract the overall face, mouth and eyes of the patient, and obtain complete local features.

Images