Depression degree identification method based on residual network transfer learning

Abstract

The invention belongs to the technical field of depression level evaluation, and particularly relates to a depression degree identification method based on residual network transfer learning, which comprises the following steps: S1, loading a pre-training model (downloading and loading the weight of the pre-training model); s2, selecting a to-be-frozen parameter layer (reserving a full-connection layer and a plurality of later layers), and performing parameter fine adjustment on the pre-training model; s3, preprocessing the data set; s4, introducing an attention mechanism; and S5, outputting the depression score and depression grade (no, mild, moderate and severe) of the face of the depression patient. According to the method, transfer learning and an improved attention mechanism are used, the accuracy of depression intelligent rating is higher, the result is more accurate, the more accurate result is generated when a doctor is assisted in judgment, the time cost of the doctor is saved, doctor inquiry is replaced with machine diagnosis, the emotion of a patient can be more stable, and detection is simpler and more efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of depression level assessment, and in particular to a method for identifying depression levels based on residual network transfer learning. Background Art

[0002] Since the causes of depression may be related to multiple factors such as genetics, physiology, and psychosocial factors. At present, the diagnosis of depression usually needs to be conducted by professional doctors, using clinical interviews and assessment tools to confirm the presence and extent of symptoms. Compared with manual evaluation, the use of deep learning detection has the advantages of objectivity, efficiency, and standardization. Studies on mental health assessments have shown that there is a significant correlation between patients' facial expressions and their depressive symptoms, indicating that facial expression analysis can assist in the diagnosis of depression. Although the efficiency and objectivity of previous methods of diagnosing depression through facial analysis have surpassed traditional methods, the challenges posed by the complex visual representation patterns of depression still exist, which to some extent hinders the universality and widespread application of automated diagnosis technology for depression. Summary of the invention

[0003] 1. Technical issues to be solved

[0004] In view of the deficiencies in the prior art, the present invention provides a method for identifying depression levels based on residual network transfer learning, which solves the problems raised in the above background technology.

[0005] (II) Technical solution

[0006] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0007] A method for identifying depression level based on residual network transfer learning includes the following steps:

[0008] S1: Load the pre-trained model (download and load its weights);

[0009] S2: Select the parameter layers to be frozen (keep the fully connected layer and the following layers) and fine-tune the parameters of the pre-trained model;

[0010] S3: dataset preprocessing;

[0011] S4: Introducing the attention mechanism;

[0012] S5: Output the depression score and depression level (none, mild, moderate and severe) of the depressed patient's face.

[0013] Furthermore, the specific content of S1 is: first, a deep learning model trained with ResNet50 is selected. The model has a good performance in processing images. The key to the ResNet network lies in the residual unit in its structure. Among them, ConvBlock is used to change the dimension of the network, first reducing the dimension through 1×1 convolution, then performing 3×3 convolution, and then increasing the dimension through 1×1 convolution. The dotted line uses 1×1 convolution to reduce the dimension. IdentityBlock is used to deepen the network, without reducing the dimension through the convolution network, and directly adding the input to the 1×1 convolution output.

[0014] Furthermore, the specific content of S2 is as follows: the ResNet50 network is divided into six parts, stage 1 is the input module, stage 2 to stage 5 are residual modules, stage 6 is the output module, which is composed of 49 convolutional layers and 1 fully connected layer; after the image is passed into the network, it is passed into 16 residual blocks after the first convolution pooling, and in each subsequent Stage, it must undergo convolution (Conv) and BN operations, namely ConvBlock, and then pass through multiple IdentityBlocks with consistent input and output dimensions. After the convolution of Stage 2 to 5, the data is compressed into a one-dimensional array through the global average pooling layer (GAP), and then connected with the fully connected layer (FC) and the softmax layer with an output feature of 64, and the label value of the depression degree of the image to be evaluated is output.

[0015] Furthermore, the S3 specifically includes the following steps: S3.1: preliminary preprocessing of the video data set; S3.2. using MTCNN to perform face alignment and cropping; S3.3. constructing an automatic depression assessment data set and dividing it into a training set and a test set;

[0016] Among them, S3.1: The specific content of the preliminary preprocessing of the video dataset is as follows: First, video data containing patients with depression and normal controls are collected from multiple channels. These videos cover different scenes, lighting conditions, and patient age groups to ensure the diversity and representativeness of the dataset; then, professional video processing software or programming scripts are used to periodically or randomly extract a certain number of frame images from each video, and these frame images will be used as input data for subsequent deep learning models; while extracting video frames, strictly correspond each frame image to its original video label information to ensure that each picture in the dataset can correctly reflect its category (such as patients with depression or normal control group);

[0017] The specific content of S3.2. Using MTCNN for face alignment and cropping is as follows: MTCNN (Multi-task Cascaded Convolutional Networks) is an efficient face detection and alignment tool that can simultaneously detect the position of the face and the coordinates of the key points of the facial features; based on the face boundaries and key point information detected by MTCNN, each image is accurately aligned and cropped. This process ensures that all facial images are displayed in the same position and angle, thereby eliminating interference caused by changes in head posture; after cropping, all facial images are uniformly adjusted to the same size to facilitate the input and processing of subsequent deep learning models.

[0018] The specific contents of S3.3. Constructing an automatic depression assessment dataset and dividing it into training and test sets are as follows: the preprocessed facial images and their corresponding labels are sorted and archived to form a complete automatic depression assessment dataset, which not only contains the facial images of the patients, but also records the detailed information and diagnostic labels of each image; in order to evaluate the generalization ability and prediction accuracy of the model, we randomly divide the dataset into training and test sets in a ratio of 7:3, where the training set is used for the training and learning process of the model; the test set is used to independently verify the performance of the model on unseen data.

[0019] Furthermore, the specific content of S4 is: introducing the improved CBAM into the model; the facial image depression degree recognition task requires the model to not only capture the overall features of the face, but also pay attention to subtle changes in the face, such as eyes, smiles, wrinkles, etc. The traditional ResNet may not be able to fully capture these subtle local features. In order to further improve the performance of the model in such tasks, an attention mechanism module is inserted on the basis of the ResNet network.

[0020] The attention mechanism module is mainly composed of two sub-modules: channel attention module (CAM) and spatial attention module (SAM); the channel attention module CAM aims to adaptively learn the importance between different channels and weightedly fuse the features of these important channels; its principle is as follows: given the input feature map X, first use the maximum pooling and average pooling operations to aggregate the spatial information of the feature map to generate two different spatial descriptors: and Represent the average pooling feature and the maximum pooling feature respectively; then the two descriptors are sent to the shared fully connected layer to compress the spatial dimension of the input feature map, sum and merge them element by element, and generate the channel attention map M C .

[0021] In order to avoid the mutual interference problem that may occur when the channel attention module (CAM) and the spatial attention module (SAM) are placed in parallel; the new network structure places the channel attention module (CAM) and the spatial attention module (SAM) in parallel, thereby avoiding the problem that the latter is affected by the former.

[0022] When processing feature maps, traditional multi-layer perceptrons (MLPs) usually reduce their dimensions to 1×1×C / r first, and then increase their dimensions to 1×1×C. However, this dimensionality reduction process will lead to the loss of key information to a certain extent, thus affecting the extraction of detailed features. In view of the high requirements for detailed feature extraction in order to accurately and quickly identify the degree of depression of patients, the MLP in CAM has been improved.

[0023] In the improved MLP, the idea of ​​first increasing the dimension and then reducing the dimension is adopted; specifically, the 1×1×C feature map after average pooling and global pooling is first increased in dimension to 1×1×rC, and then reduced in dimension to 1×1×C; this processing method helps to improve the CBAM mechanism’s ability to extract image detail features.

[0024] Through this improvement, the network structure can better capture the detailed features in the image while maintaining high computational efficiency, which is particularly important for depression identification tasks because such tasks usually require sensitive capture of subtle changes in the patient's facial expressions, eyes, etc.

[0025] The Dropout regularization method and its limitations are discussed, and AlphaDropout is proposed as an alternative. The traditional Dropout regularization method improves the generalization ability of the model by randomly shutting down some neurons during the training process; however, this method also has certain defects, that is, only a part of the parameters are updated in each iteration, which may slow down the gradient descent process and thus slow down the convergence of the model.

[0026] To solve this problem, AlphaDropout is used. AlphaDropout is a special Dropout method that randomly inactivates neurons while keeping the input mean and variance unchanged. This means that AlphaDropout can achieve effective regularization while maintaining the self-regulation of the data. Compared with traditional Dropout, AlphaDropout has better convergence performance and helps improve the training efficiency of the model.

[0027] In addition, the SeLU activation function is also used. The SeLU activation function has the following characteristics: when the input is negative, it has an exponential growth characteristic; when the input is positive, it is approximately linear. The characteristics of this activation function enable the model to perform better when processing different types of input data. Specifically, the SeLU activation function can reduce the problem of gradient disappearance or explosion while maintaining the nonlinearity of the model, thereby helping to improve the stability and convergence speed of the model. As shown in the formula:

[0028]

[0029] Where: λ is approximately 1.0507, a is approximately 1.67326.

[0030] First, the SeLU activation function introduces a self-standardization mechanism, which keeps the mean and variance of the output of the neural network close to 1 during the training process. This self-standardization feature helps to avoid the gradient explosion and gradient vanishing problems in the neural network to a certain extent, thereby improving the stability and convergence speed of the model.

[0031] Combined with AlphaDropout regularization and SeLU activation function, the input distribution remains unchanged, reducing the occurrence of overfitting. Finally, the Softmax function is used to convert the original linear score into a probability distribution to achieve accurate recognition of the degree of depression in facial images of patients with depression. The formula is:

[0032]

[0033] Where: For all input signals x i The exponential function and .

[0034] (III) Beneficial effects

[0035] Compared with the prior art, the present invention provides a method for identifying depression levels based on residual network transfer learning, which has the following beneficial effects:

[0036] The present invention uses transfer learning and an improved attention mechanism to make the intelligent rating of depression more accurate and the results more precise. It produces more accurate results when assisting doctors in making judgments, saves doctors' time costs, and replaces doctors' consultations with machine diagnosis, which can make patients' emotions more stable and detection simpler and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the backbone network model structure of the present invention;

[0038] Figure 2 Schematic diagram of the channel attention module structure of the present invention;

[0039] Figure 3 This is a schematic diagram of the improved CBAM attention mechanism structure of the present invention;

[0040] Figure 4 This is a schematic diagram of the Dep-CA-ResNet50 structure of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Example

[0043] like Figure 1-4 As shown, a method for identifying depression level based on residual network transfer learning proposed in one embodiment of the present invention includes the following steps:

[0044] S1: Load the pre-trained model (download and load its weights);

[0045] First, we select a deep learning model trained with ResNet50. The model performs well in processing images. The key to the ResNet network lies in the residual unit in its structure. ConvBlock is used to change the dimension of the network. First, 1×1 convolution is used to reduce the dimension, then 3×3 convolution is performed, and then 1×1 convolution is used to increase the dimension. The dotted line uses 1×1 convolution to reduce the dimension, such as Figure 1 (b) IdentityBlock is used to deepen the network, without going through the convolutional network to reduce the dimension, and directly add the input to the 1×1 convolution output, as shown in Figure 1 (c)

[0046] S2: Select the parameter layers to be frozen (keep the fully connected layer and the following layers) and fine-tune the parameters of the pre-trained model;

[0047] The ResNet50 network is divided into six parts. Stage 1 is the input module, stage 2 to stage 5 are residual modules, and stage 6 is the output module, which consists of 49 convolutional layers and 1 fully connected layer.

[0048] After loading the pre-trained parameters, the model can now recognize the basic features in the image.

[0049] After the image is passed into the network, it is passed into 16 residual blocks after the first convolution and pooling. In each subsequent stage, it undergoes convolution (Conv) and BN operations, namely ConvBlock, and then passes through multiple IdentityBlocks with consistent input and output dimensions. After the convolution of Stage 2 to 5, the data is compressed into a one-dimensional array through the global average pooling layer (GAP), and then connected with the fully connected layer (FC) and softmax layer with an output feature of 64, and the label value of the depression level of the image to be evaluated is output, such as Figure 1 (a) shows that the fully connected layer is replaced by a fully connected layer with an output feature of 64; the weights of some layers are frozen by setting requires-grad = False.

[0050] S3: dataset preprocessing;

[0051] Specifically, the following steps are included: S3.1: Preliminary preprocessing of the video dataset; S3.2. Using MTCNN for face alignment and cropping; S3.3. Constructing an automatic depression assessment dataset and dividing it into training and test sets;

[0052] Among them, S3.1: The specific content of the preliminary preprocessing of the video dataset is as follows: First, video data containing patients with depression and normal controls are collected from multiple channels. These videos cover different scenes, lighting conditions, and patient age groups to ensure the diversity and representativeness of the dataset; then, professional video processing software or programming scripts are used to periodically or randomly extract a certain number of frame images from each video, and these frame images will be used as input data for subsequent deep learning models; while extracting video frames, strictly correspond each frame image to its original video label information to ensure that each picture in the dataset can correctly reflect its category (such as patients with depression or normal control group);

[0053] The specific content of S3.2. Using MTCNN for face alignment and cropping is as follows: MTCNN (Multi-task Cascaded Convolutional Networks) is an efficient face detection and alignment tool that can simultaneously detect the position of the face and the coordinates of the key points of the facial features; based on the face boundaries and key point information detected by MTCNN, each image is accurately aligned and cropped. This process ensures that all facial images are displayed in the same position and angle, thereby eliminating interference caused by changes in head posture; after cropping, all facial images are uniformly adjusted to the same size (such as 224x224 pixels) to facilitate the input and processing of subsequent deep learning models.

[0054] The specific contents of S3.3. Constructing an automatic depression assessment dataset and dividing it into training and test sets are as follows: the preprocessed facial images and their corresponding labels are sorted and archived to form a complete automatic depression assessment dataset, which not only contains the facial images of the patients, but also records the detailed information and diagnostic labels of each image; in order to evaluate the generalization ability and prediction accuracy of the model, we randomly divide the dataset into training and test sets in a ratio of 7:3, where the training set is used for the training and learning process of the model; the test set is used to independently verify the performance of the model on unseen data.

[0055] S4: Introducing the attention mechanism;

[0056] like Figure 2 As shown in the figure, the improved CBAM is introduced into the model; the facial image depression degree recognition task requires the model to not only capture the overall features of the face, but also pay attention to subtle changes in the face, such as eyes, smiles, wrinkles, etc. The traditional ResNet may not be able to fully capture these subtle local features. In order to further improve the performance of the model in such tasks, an attention mechanism module is inserted based on the ResNet network.

[0057] The attention mechanism module is mainly composed of two sub-modules: channel attention module (CAM) and spatial attention module (SAM); the channel attention module CAM aims to adaptively learn the importance between different channels and weightedly fuse the features of these important channels; its principle is as follows: given the input feature map X, first use the maximum pooling and average pooling operations to aggregate the spatial information of the feature map to generate two different spatial descriptors: and Represent the average pooling feature and the maximum pooling feature respectively; then the two descriptors are sent to the shared fully connected layer to compress the spatial dimension of the input feature map, sum and merge them element by element, and generate the channel attention map M C .

[0058] like Figure 3 As shown in Figure 1, an improved attention mechanism is proposed to solve the problem of mutual interference that may occur when the traditional channel attention module (CAM) and spatial attention module (SAM) are placed side by side. Our new network structure places the spatial and channel attention mechanisms in parallel, thus avoiding the problem that the latter is affected by the former.

[0059] When processing feature maps, traditional multi-layer perceptrons (MLPs) usually reduce their dimensions to 1×1×C / r first, and then increase their dimensions to 1×1×C. However, this dimensionality reduction process will lead to the loss of key information to a certain extent, thus affecting the extraction of detailed features. Given that accurate and rapid identification of the patient's depression level has high requirements for detailed feature extraction, the MLP in CAM has been improved.

[0060] In the improved MLP, the idea of ​​first increasing the dimension and then reducing the dimension is adopted. Specifically, the 1×1×C feature map after average pooling and global pooling is first increased in dimension to 1×1×rC, and then reduced in dimension to 1×1×C. This processing method helps to improve the CBAM mechanism's ability to extract image detail features.

[0061] Through this improvement, the network structure can better capture the detailed features in the image while maintaining high computational efficiency. This is particularly important for depression recognition tasks, because such tasks usually require sensitive capture of subtle changes in the patient's facial expressions, eyes, etc.

[0062] The Dropout regularization method and its limitations are discussed, and AlphaDropout is proposed as an alternative. The traditional Dropout regularization method improves the generalization ability of the model by randomly shutting down a portion of neurons during training. However, this method also has certain defects, that is, only a portion of the parameters are updated in each iteration, which may slow down the gradient descent process and thus slow down the convergence of the model.

[0063] To solve this problem, AlphaDropout is used. AlphaDropout is a special Dropout method that randomly inactivates neurons while keeping the input mean and variance unchanged. This means that AlphaDropout can achieve effective regularization while maintaining the self-regulation of the data. Compared with traditional Dropout, AlphaDropout has better convergence performance and helps improve the training efficiency of the model.

[0064] In addition, the SeLU activation function is also used. The SeLU activation function has the following characteristics: when the input is negative, it has an exponential growth characteristic; when the input is positive, it is approximately linear. The characteristics of this activation function enable the model to perform better when processing different types of input data. Specifically, the SeLU activation function can reduce the problem of gradient disappearance or explosion while maintaining the nonlinearity of the model, thereby helping to improve the stability and convergence speed of the model. As shown in the formula.

[0065]

[0066] Where: λ is approximately 1.0507, a is approximately 1.67326.

[0067] like Figure 4 As shown in Figure 1, the CBAM mechanism is improved and integrated into the ResNet50 network. The improved CBAM mechanism can better extract the key features of facial images while filtering out irrelevant information. This makes the Dep-CA-ResNet50 network more accurate and robust when processing facial images.

[0068] Combined with AlphaDropout regularization and SeLU activation function, the input distribution remains unchanged, reducing the occurrence of overfitting. Finally, the Softmax function is used to convert the original linear score into a probability distribution to achieve accurate recognition of the degree of depression in facial images of patients with depression. The formula is:

[0069]

[0070] Where: For all input signals x i The exponential function and .

[0071] S5: Output the depression score and depression level (none, mild, moderate and severe) of the depressed patient's face.

[0072] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A depression degree recognition method based on residual network transfer learning, characterized by: The following steps are involved: S1: Load the pre-trained model (download and load its weights); S2: Select the parameter layers to be frozen (keep the fully connected layer and the following layers) and fine-tune the parameters of the pre-trained model; S3: dataset preprocessing; S4: Introducing the attention mechanism; S5: Output the depression score and depression level (none, mild, moderate and severe) of the depressed patient's face.

2. The method for identifying depression level based on residual network transfer learning according to claim 1, characterized in that: The specific content of S1 is: select a deep learning model trained with ResNet50. The key to the ResNet network lies in the residual unit in its structure. ConvBlock is used to change the dimension of the network. First, the dimension is reduced by 1×1 convolution, then 3×3 convolution is performed, and then the dimension is increased by 1×1 convolution. The dotted line uses 1×1 convolution to reduce the dimension. IdentityBlock is used to deepen the network. The input is directly added to the 1×1 convolution output without reducing the dimension through the convolution network.

3. The method for identifying depression level based on residual network transfer learning according to claim 2, characterized in that: The specific content of S2 is as follows: the ResNet50 network is divided into six parts, stage 1 is the input module, stage 2 to stage 5 are residual modules, stage 6 is the output module, which consists of 49 convolutional layers and 1 fully connected layer; after the image is passed into the network, it is passed into 16 residual blocks after the first convolution pooling, and in each subsequent Stage, it must undergo convolution (Conv) and BN operations, namely ConvBlock, and then pass through multiple IdentityBlocks with consistent input and output dimensions. After the convolution of Stage 2 to 5, the data is compressed into a one-dimensional array through the global average pooling layer (GAP), and then connected with the fully connected layer (FC) and the softmax layer with an output feature of 64, and the label value of the depression level of the image to be evaluated is output.

4. The method for identifying depression level based on residual network transfer learning according to claim 1, characterized in that: The S3 specifically includes the following steps: S3.1: preliminary preprocessing of the video data set; S3.

2. using MTCNN to perform face alignment and cropping; S3.

3. constructing an automatic depression assessment data set and dividing it into a training set and a test set; S3.1: The specific contents of the preliminary preprocessing of the video dataset are as follows: first, video data of patients with depression and normal controls are collected from multiple channels. These videos cover different scenes, lighting conditions and patient age groups to ensure the diversity and representativeness of the dataset; then, a certain number of frame images are extracted from each video at regular intervals or randomly using professional video processing software or programming scripts. These frame images will be used as input data for subsequent deep learning models; and while extracting video frames, the label information of each frame image is strictly matched to its original video to ensure that each image in the dataset can correctly reflect its category (such as patients with depression or normal control group); The specific content of S3.

2. Using MTCNN for face alignment and cropping is as follows: MTCNN (Multi-task Cascaded Convolutional Networks) is an efficient face detection and alignment tool that can simultaneously detect the position of the face and the coordinates of the key points of the facial features. Based on the face boundaries and key point information detected by MTCNN, each image is accurately aligned and cropped; after cropping, all facial images are uniformly adjusted to the same size; The specific contents of S3.

3. Constructing an automatic depression assessment dataset and dividing it into training and test sets are as follows: the preprocessed facial images and their corresponding labels are sorted and archived to form a complete automatic depression assessment dataset, which not only contains the facial images of the patients, but also records the detailed information and diagnostic labels of each image; in order to evaluate the generalization ability and prediction accuracy of the model, the dataset is randomly divided into training and test sets in a ratio of 7:3, where the training set is used for the training and learning process of the model, and the test set is used to independently verify the performance of the model on unseen data.

5. The method for identifying depression level based on residual network transfer learning according to claim 1, characterized in that: The specific content of S4 is as follows: introducing the improved CBAM into the model, CBAM is an attention mechanism module, and the attention mechanism module consists of a channel attention module (CAM) and a spatial attention module (SAM); the channel attention module CAM aims to adaptively learn the importance between different channels and weightedly fuse the features of these important channels; the channel attention module (CAM) and the spatial attention module (SAM) are placed in parallel; and the multi-layer perceptron (MLP) in the channel attention module (CAM) is improved. In the improved MLP, the processing idea of ​​first increasing the dimension and then reducing the dimension is adopted. Specifically, firstly, the 1×1×C feature map after average pooling and global pooling is increased in dimension to 1×1×rC, and then reduced in dimension to 1×1×C; Use AlphaDropout instead of the traditional Dropout regularization method. AlphaDropout is a special Dropout method that randomly inactivates neurons while keeping the input mean and variance unchanged. The SeLU activation function is also used, and a self-normalization mechanism is introduced, so that the output of the neural network keeps the mean and variance close to 1 during the training process, as shown in the formula: Where: λ is about 1.0507, a is about 1.67326; Combined with AlphaDropout regularization and SeLU activation function, the input distribution remains unchanged, reducing the occurrence of overfitting. Finally, the Softmax function is used to convert the original linear score into a probability distribution to achieve accurate recognition of the degree of depression in facial images of patients with depression. The formula is: Where: For all input signals x i The exponential function and .

Images