A method and device for constructing a depression recognition model based on deep network feature fusion

Abstract

A kind of construction method and device of depression recognition model based on deep network feature fusion, its method includes: data acquisition;Build the video dataset containing video regional level label;Respectively using C3D neural network and channel attention network to the whole video stream is reinforced feature extraction;The reinforced feature is used RPN to generate region proposal and produces region start mark and end mark and region expression class;All video frames are extracted to multiple scale features including local eccentricity features and global features;Weakly supervised classification is carried out to multiple scale features;The operation result of video stream level feature and picture level feature is classified by neural network, and depression recognition model is obtained.The present application effectively utilizes video label, takes into account global and local information, and combines the advantages of high accuracy and reduced label workload, can become a potential depression recognition auxiliary means, and can be widely applied in the field of automatic processing of mental illness.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology and discloses a method and apparatus for constructing a depression recognition model based on deep network feature fusion. Background Technology

[0002] Detecting and identifying depression is a complex and challenging task, requiring the overcoming of several difficulties to achieve accurate results. First, video data is massive, consuming significant time and computational resources to process. Furthermore, unlike ordinary image recognition, the dynamic information and temporal features in videos are crucial for depression identification. Therefore, efficiently extracting and analyzing key features from video data is a problem that needs to be solved. Second, label creation is also a complex and highly subjective process. The diagnosis of depression relies on the assessment and judgment of doctors or professionals. However, different professionals may have different standards and subjective preferences, leading to uncertainty and inconsistency in label creation. Therefore, ensuring the accuracy and consistency of labels to improve the reliability of the model is a problem that needs to be addressed. In addition, human facial expressions are very rich and varied, and the differences in facial expressions between individuals in different emotional states are also significant. This poses a challenge to depression identification, as a single facial expression feature may not be able to fully capture the complexity of depression. Therefore, combining multiple facial expression features and integrating them with other clinical and psychological indicators to improve the accuracy and reliability of depression identification is a direction that requires further research.

[0003] Definitions:

[0004] OpenCV: An open-source computer vision library that implements many common algorithms in image processing and computer vision, and is often used for image processing and analysis.

[0005] C3D: In the field of video understanding, in order to preserve temporal information at the same time, it is necessary to learn spatiotemporal features simultaneously. C3D convolutional networks can effectively capture dynamic information in the time dimension, thereby better modeling facial expression changes in video sequences.

[0006] RPN: Region Proposal Network, used to generate candidate regions. The input is an image of arbitrary scale, and the output is a series of candidate regions.

[0007] Bilinear interpolation, also known as bilinear interpolation, is based on the idea of ​​performing linear interpolation in two directions. As an interpolation algorithm in numerical analysis, bilinear interpolation is widely used in signal processing, digital image and video processing, and other fields.

[0008] Eccentricity: It is the ratio of the distance between the two foci to the length of the major axis. Eccentricity reflects the degree of deviation of an elliptical orbit from an ideal circular ring. Long elliptical orbits have high eccentricity, while orbits that are close to a circle have low eccentricity.

[0009] K-means is a clustering algorithm. Due to its simple principle, strong interpretability, convenient implementation, and fast convergence speed, it has been widely used in data mining, cluster analysis, data clustering, pattern recognition and other fields.

[0010] Convolutional Neural Network (CNN): A type of deep neural network commonly used to process grid-like data, primarily in image processing, including image classification, image feature extraction, and image synthesis.

[0011] VGG is a deep convolutional neural network architecture proposed by the Visual Geometry Group in 2014. It has been widely used in computer vision tasks such as image classification, object detection, and semantic segmentation, and its simple network structure and ease of implementation have made VGG one of the classic models in the field of deep learning. Summary of the Invention

[0012] This invention aims to overcome the aforementioned problems of the prior art by proposing a method and apparatus for constructing a depression recognition model based on deep network feature fusion, using patient consultation videos as a foundation.

[0013] This invention employs a multi-region labeling method based on a large number of patient consultation videos. First, facial expressions at different stages of the video are labeled to generate multi-region tags. Then, features are extracted from the video stream using both deep feature networks and photo multi-scale structures.

[0014] In the first direction, the deep feature map is processed by a spatial channel model to obtain an enhanced feature map. Next, the concept of RPN (Representational Processing Network) is used to generate proposal regions, which are then fed into a classifier to obtain a region expression classification matrix, as well as the start and end frames of each region. This processing method yields results that include global information.

[0015] In the second approach, multi-scale facial expression features are acquired from the video stream frame by frame, including both global and local scales of the images. Since the data labels are based on global annotations, local labels may be ambiguous. To address this, a weakly supervised classification method is used to re-annotate each frame, ultimately obtaining an expression classification vector for each image. Twenty-two expressions are used (including no expression, six basic expressions, and 15 composite expressions). The final classification result incorporates local information from each image.

[0016] Finally, the two results are processed and fed into a neural network for classification. This method overcomes the inaccuracy of sample facial expressions and fuses global and local information from the video stream to obtain video-level multi-scale feature results. Therefore, the model has high accuracy and ensures the robustness of the depression recognition method.

[0017] The present invention discloses a method for constructing a depression recognition model based on deep network feature fusion, comprising the following steps:

[0018] S1. Batch collection of required video data of patients with depression;

[0019] S2. In the preprocessing stage, a series of facial information missing operations need to be performed on the collected video data to improve the quality of the video data required for subsequent processing steps.

[0020] S3. Based on the data obtained in step S2, use the C3D convolutional network to extract the facial expression transformation features of each keyframe in the video data.

[0021] S4. Based on step S3, perform enhanced feature extraction and proposal generation;

[0022] S5. Generate expression categories and time periods. Input the generated proposal regions and enhanced feature maps into the classification module to generate the expression category for each region, as well as the labels for the start and end frames of the region. These classification results and time period information can be used for subsequent expression analysis and recognition.

[0023] S6. Multi-scale facial expression feature extraction: Global and local features are extracted from each keyframe in the video to obtain the facial expression feature vector for each frame. Global features can capture overall facial expression information, while local features can better describe facial details and changes in facial expression regions;

[0024] S7. Weakly supervised classification: In step S6, the multi-scale facial expression features are weakly supervised to classify and generate a weakly supervised classification matrix.

[0025] S8. Integrate video stream-level features with image frame-level features and perform neural network classification.

[0026] Furthermore, in step S2, to provide the required video quality data, data preprocessing is required according to the following steps:

[0027] S21. Use bilinear interpolation algorithm to reduce the video resolution to the specified pixel size;

[0028] S22. Use a face detection algorithm to detect and delete images lacking clear faces to ensure the accuracy and reliability of data in subsequent processing steps;

[0029] S23. In this step, labels need to be added to each keyframe interval of the entire video sequence in the video data. These labels are usually annotated by professional psychological counselors based on their understanding and experience of facial expressions. The labels include emotion categories (such as happiness, sadness, anger, etc., a total of 22 types of expressions) and interval start and end labels;

[0030] S24. Use the obtained data to construct training and testing sets.

[0031] Furthermore, in step S4, feature enhancement and proposal generation need to be performed according to the following steps:

[0032] S41. Enhanced Feature Acquisition: Feature maps are enhanced by applying a spatial attention model to further extract useful spatial information. The spatial attention model automatically adjusts the weights of the feature maps based on the importance of different regions, enabling the model to better focus on expression-related areas and ultimately obtain enhanced feature maps.

[0033] S42. Generate proposals: Utilizing enhanced feature maps, a proposal model is used to generate proposal regions that may contain facial expressions. Proposal models are typically based on candidate region generation algorithms; this invention uses RPN (Region Proposal Networks) models, which can effectively generate potential facial expression regions.

[0034] Furthermore, in step S6, global feature extraction and local feature extraction need to be performed according to the following steps:

[0035] S61. Global feature extraction: Global feature extraction is performed by using the convolutional layer of the VGG16 neural network as the global facial expression feature extraction model.

[0036] S62. Local feature extraction: Obtain facial key points and construct curvature set features for eyebrows, eyes, mouth, bridge of nose and cheeks respectively. These features are described by using the eccentricity of the fitted ellipse.

[0037] S63. Multi-scale facial expression feature normalization: In this step, the multi-scale facial expression features are normalized to ensure scale consistency between different features. The normalization method used in this invention is zero-mean normalization.

[0038] Furthermore, in step S7, the multi-scale facial expression features need to be weakly supervised for classification according to the following steps:

[0039] S71. A clustering algorithm is used to classify the feature vectors of each frame into one of 22 expression categories. The clustering algorithm used in this invention is K-means.

[0040] S72. Perform frame-by-frame label annotation on the classification results, that is, assign expression labels to each frame based on the clustering results;

[0041] S73. Facial expression classification is performed on the obtained annotated data frames using a deep neural network. The deep neural network used in this invention is a combination model of convolutional neural network (CNN) and recurrent neural network (RNN);

[0042] S74. Combine the region start frame marker and region end frame marker obtained in step S5 with the result obtained in step S73 to form a proposal region frame marker. Then normalize each proposal region to obtain a weakly supervised classification matrix.

[0043] Furthermore, in step S8, feature integration and neural network classification need to be performed according to the following steps:

[0044] S81. Calculate the features of facial expression relationships in the deep network, and perform AB transformation on matrices A and B. T Calculate and obtain matrix F, which is the relationship matrix between the expressions in these regions;

[0045] S82. Construct a Ghost neural network, and feed the relation matrix F as input into the convolutional neural network (Ghost) to obtain the final classification result.

[0046] The second invention relates to a device for constructing a depression recognition model based on deep network feature fusion, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the method for constructing the depression recognition model based on deep network feature fusion of the present invention.

[0047] The third invention relates to a computer-readable storage medium, characterized in that it stores a program thereon, which, when executed by a processor, implements the method for constructing a depression recognition model based on deep network feature fusion according to the present invention.

[0048] The present invention has the following advantages:

[0049] High accuracy: This invention achieves high accuracy even under conditions of imperfect precision based on video tags. Even with a certain degree of error in the video tags, this method can still effectively perform facial expression recognition and obtain relatively accurate results.

[0050] Reduced labeling workload: Compared to traditional methods that require labeling large amounts of video data, this invention significantly reduces the amount of video labeling work required for model training. Good recognition results can be achieved by ensuring a certain level of label accuracy, saving considerable time and human resources.

[0051] Balancing Global and Local Information: This invention can simultaneously consider global information from video-level data streams and global and local information from photo-level data. By comprehensively considering the facial expression features of different frames in the video, as well as the facial expression changes in different regions within each frame, it ensures the diversity and richness of features, thereby improving the accuracy and robustness of facial expression recognition.

[0052] An auxiliary means for identifying depression: Because this invention can effectively recognize facial expressions, it can be used as an auxiliary means for identifying depression. By analyzing and recognizing changes in a patient's facial expressions, a more accurate diagnosis and assessment of depression can be provided, offering important reference information for clinicians.

[0053] In summary, this invention, by effectively utilizing video tags, taking into account both global and local information, and combining the advantages of high accuracy and reduced tagging workload, has become a promising auxiliary means for the identification of depression. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a method for constructing a depression recognition model based on deep network feature fusion according to the present invention.

[0055] Figure 2 For the deep network feature extraction of this invention, after generating a region proposal, the expression type, start frame, and end frame of that expression are determined for that region.

[0056] Figure 3 This invention relates to an expression recognition module based on weakly supervised multi-scale classification.

[0057] Figure 4 This is the process of fusing the weakly supervised multi-scale classification results and the regional proposal results in this invention;

[0058] Figure 5 This is a schematic diagram of the neural network model used after feature calculation in a specific embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0061] Example 1

[0062] Reference Figure 1 A method for constructing a depression recognition model based on deep network feature fusion includes the following steps:

[0063] S1. Collect the required video data of patients with depression in batches. The video should capture the frontal head image of the patient and include a label indicating whether the patient has depression: 1 indicates depression and 0 indicates no depression.

[0064] S2. In the preprocessing stage, a series of facial information missing operations need to be performed on the collected video data to improve the quality of the video data required for subsequent processing steps.

[0065] S3. Reference Figure 2 Based on the data obtained in step S2, the feature extraction backbone structure of the C3D convolutional network is used to extract the facial expression transformation features of each key frame in the video data.

[0066] S4. Based on step S3, perform enhanced feature extraction and proposal generation;

[0067] S5. Generate expression categories and time periods. Input the generated proposal regions and enhanced feature maps into the classification module to generate an expression category matrix for each region. Figure 4 Matrix A in the data, and the markers for the start and end frames of the region, i.e. Figure 4 T1 in the data. These classification results and time period information can be used for subsequent facial expression analysis and recognition;

[0068] S6. Multi-scale facial expression feature extraction: Global and local features are extracted from each keyframe in the video to obtain the facial expression feature vector for each frame. Global features can capture overall facial expression information, while local features can better describe facial details and changes in facial expression regions.

[0069] S7. Weakly supervised classification: In step 6), the multi-scale facial expression features are weakly supervised to generate a weakly supervised classification matrix;

[0070] S8. Combine video stream-level features with image frame-level features and classify them using a neural network.

[0071] Furthermore, in step S2, to provide the required video quality data, data preprocessing is required according to the following steps:

[0072] S21. Use bilinear interpolation algorithm to reduce the video resolution to the specified pixel size, with a target pixel size of 600*800;

[0073] S22. Use the face detection algorithm provided by OpenCV to detect and delete images that lack clear faces to ensure the accuracy and reliability of data in subsequent processing steps;

[0074] S23. In this step, it is necessary to add markers to the keyframe intervals of the entire video sequence in the video data. These markers are usually added by professional psychological counselors based on their understanding and experience of facial expressions. The markers include emotion categories (such as happiness, sadness, anger, etc., a total of 22 types of expressions) and frame interval start and end markers;

[0075] S24. Use the obtained data to construct training and testing sets.

[0076] In step S4, refer to Figure 2 Feature enhancement and proposal generation need to be performed according to the following steps:

[0077] S41. Enhanced Feature Acquisition: After obtaining the original feature map based on the C3D neural network, the feature map is enhanced by applying a spatial attention model. Figure 2 In the middle section, SA is used to further extract useful spatial information. The spatial attention model can automatically adjust the weights of the feature map according to the importance of different regions, enabling the model to better focus on expression-related regions and finally obtain enhanced feature maps;

[0078] S42. Generate proposals: The enhanced feature map is used to generate proposal regions that may contain facial expressions using a proposal model. Proposal models are typically based on candidate region generation algorithms; this invention uses RPN models, which can effectively generate potential facial expression regions.

[0079] Furthermore, in step S6, global feature extraction and local feature extraction need to be performed according to the following steps:

[0080] S61. Global feature extraction: The VGG16 neural network convolutional layer is used as the global expression feature extraction model to extract global features and obtain the final 19-dimensional feature vector.

[0081] S62. Local feature extraction: This feature is described by using the eccentricity of the fitted ellipse. A total of 17 eccentricities were selected for the local features, namely 2 for eyebrows, 4 for eyes, 1 for nose bridge, 2 for mouth, and 8 for cheeks.

[0082] S63. After concatenating the feature vectors obtained in S61 and S62, multi-scale facial expression feature normalization is performed. In this step, the multi-scale facial expression features are normalized to ensure scale consistency between different features. The normalization method used in this invention is zero-mean normalization.

[0083] Furthermore, in step S7, referring to Figure 3 Weakly supervised classification of multi-scale facial expression features is required by following these steps:

[0084] S71. A clustering algorithm is used to classify the feature vectors of each frame into one of 22 expression categories. The clustering algorithm used in this invention is K-means.

[0085] S72. Perform frame-by-frame label annotation on the classification results, that is, reassign expression labels to each frame based on the clustering results;

[0086] S73. Facial expression classification is performed on the obtained annotated data frames using a deep neural network. The deep neural network used in this invention is a combination model of convolutional neural network (CNN) and recurrent neural network (RNN);

[0087] S74, reference Figure 4 The region start frame label and region end frame label obtained in step S5 are labeled separately with the result obtained in step S73, and then they are fused and normalized to obtain the weakly supervised classification matrix, i.e. Figure 4 Matrix B in the matrix.

[0088] Furthermore, in step S8, feature integration and neural network classification need to be performed according to the following steps:

[0089] S81. Calculate the features of facial expression relationships in the deep network, and perform AB transformation on matrices A and B. T Calculate and obtain matrix F, which is the relationship matrix between the expressions in these regions;

[0090] S82. Construct a Ghost neural network, and feed the relation matrix F as input into the convolutional neural network (Ghost) to obtain the final classification result.

[0091] This invention effectively utilizes video tags, takes into account both global and local information, and combines the advantages of high accuracy and reduced tagging workload. It has the potential to become a helpful tool for depression identification and can be widely applied in the field of automated processing of mental illnesses.

[0092] Example 2

[0093] This embodiment relates to a device for constructing a depression recognition model based on deep network feature fusion, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for constructing a depression recognition model based on deep network feature fusion in Embodiment 1.

[0094] Example 3

[0095] This embodiment relates to a computer-readable storage medium, characterized in that it stores a program thereon, which, when executed by a processor, implements the method for constructing a depression recognition model based on deep network feature fusion as described in Embodiment 1.

Claims

1. A method for constructing a depression recognition model based on deep network feature fusion, comprising the following steps: S1, batch collecting video data of patients with depression required; S2, in the preprocessing stage, a series of facial information missing operations are required to be performed on the collected video data and the quality of the video data required for the subsequent processing steps; S3, based on the data obtained in step S2, a C3D convolutional network is used to extract the expression transformation features of each key frame in the video data; The 3D convolutional network effectively captures dynamic information in the time dimension, thereby modeling the expression changes in the video sequence; S4, based on step S3, enhanced feature extraction and proposal generation are performed; S5, generate expression classification and time period, input the generated proposal region and enhanced feature map into the classification module to generate the expression category of each region and the start frame and end frame label of the region, and the expression category of each region is recorded as matrix A; these classification results and time period information are used for subsequent expression analysis and recognition; S6, multi-scale expression feature extraction, global and local feature extraction is performed on each key frame in the video to obtain the expression feature vector of each frame; Global features capture overall expression information, while local features describe facial details and regional expression changes; S7, weakly supervised classification: in step S6, the multi-scale expression features are weakly supervised classified to generate matrix B; S8, integrate the video stream level features with the picture frame level features and perform neural network classification; specifically including: S81, calculate the features of the deep network expression relationship, AB T Calculate to obtain matrix F, matrix F is the relationship matrix between these regional expressions; S82, construct a Ghost neural network, input the relationship matrix F into the convolutional neural network Ghost, and obtain the final classification result. 2.The method of claim 1, wherein, In step S2, to provide the required video quality data, data preprocessing needs to be performed according to the following steps: S21, use the bilinear interpolation algorithm to reduce the definition of the video to a specified pixel size; S22, use a face detection algorithm to detect and delete images lacking clear faces to ensure the accuracy and reliability of the data in the subsequent processing steps; S23, in this step, labels need to be added for each key frame interval of the full video sequence in the video data; these labels are usually annotated by professional psychological instructors according to their understanding and experience of expressions; the labels include emotion categories and interval start and end labels; S24, use the obtained data to construct training and test sets. 3.The method of claim 1, wherein, In step S4, feature enhancement and proposal generation need to be performed according to the following steps: S41, enhanced feature acquisition, enhance the feature map by applying a spatial attention model to further extract useful spatial information; the spatial attention model automatically adjusts the weights of the feature map according to the importance of different regions, so that the model can focus on expression-related regions, and finally obtain an enhanced feature map; S42, generate proposals, use the enhanced feature map to generate proposal regions that may contain expressions using a proposal model. 4.The method of claim 1, wherein, In step S6, global feature extraction and local feature extraction need to be performed according to the following steps: S61, global feature extraction, global feature extraction is performed by using the convolutional layer of the VGG16 neural network as a global expression feature extraction model to obtain a final 19-dimensional feature vector; S62, local feature extraction, the feature is described by using the eccentricity of the fitted ellipse, and 17 eccentricities of local features are selected, including 2 eyebrows, 4 eyes, 1 nose bridge, 2 mouths and 8 cheeks; S63, the feature vectors obtained in S61 and S62 are spliced and then subjected to multi-scale expression feature normalization, in which step, the multi-scale expression features are subjected to normalization processing to ensure the scale consistency between different features. 5.The method of claim 1, wherein, In step S7, the multi-scale expression features need to be subjected to weak supervision classification in the following steps: S71, using a clustering algorithm to classify the feature vectors of each frame into one of 22 categories of expressions; the clustering algorithm is K-means; S72, performing frame-by-frame label annotation on the classification results, i.e., assigning an expression label to each frame according to the classification results; S73, using a deep neural network to classify the obtained annotated data frames; the deep neural network is a combined model of a convolutional neural network CNN and a recurrent neural network RNN; S74, performing proposal region frame labeling on the results obtained in step S73 by using the region start frame label and the region end frame label obtained in step S5, and then normalizing each proposal region to obtain a weak supervision classification matrix.

6. A depression recognition model construction device based on deep network feature fusion, characterized in that, The memory stores executable code, and the one or more processors execute the executable code to implement the construction method of the depression recognition model based on deep network feature fusion.

7. A computer-readable storage medium, characterized in that, The program is stored on the memory and is executed by the processor to implement the construction method of the depression recognition model based on deep network feature fusion.

Images