COPD anxiety and depression screening method and system based on deep convolutional neural network

Abstract

The invention discloses a COPD anxiety and depression screening method and system based on a deep convolutional neural network, and relates to the technical field of disease screening, and the method comprises the steps: obtaining a sampling video of a patient, carrying out the facial feature extraction and voice feature extraction of the sampling video of the patient, and obtaining the text feature of the patient and the audio feature of the patient; fusing the text features of the patient and the audio features of the patient in a channel dimension to obtain multi-dimensional fusion features; and inputting the multi-dimensional fusion features into a pre-established regression and classification anxiety-depression screening model, and outputting to obtain an anxiety-depression screening result of the patient.

Description

Technical Field

[0001] The present invention relates to the technical field of disease screening, and specifically to a COPD anxiety and depression screening method and system based on a deep convolutional neural network. Background Art

[0002] Chronic obstructive pulmonary disease (COPD) is a common respiratory disease that seriously endangers human health. In recent years, due to the increase in the number of smokers, environmental pollution, and population aging, the incidence of COPD in my country has increased year by year. The prevalence of COPD in people aged 40 and above has increased from 8.2% in 2007 to 13.7% in 2018. COPD is more common in the middle-aged and elderly population, with clinical manifestations such as chronic cough, sputum, wheezing, and dyspnea, and is often accompanied by a variety of extrapulmonary diseases, such as cardiovascular disease, osteoporosis, anxiety, and depression, which bring a heavy burden to patients, their families, and society. Anxiety and depression are very common in COPD patients, with a higher incidence than in the general population and higher than in patients with other chronic diseases such as hypertension, diabetes, and tumors. The presence of anxiety and depression directly leads to the adverse consequences of COPD, increases the number of acute exacerbations and hospitalizations, and reduces the quality of life of COPD patients. Despite this, anxiety and depression are often overlooked in COPD patients.

[0003] At present, the scales used in clinical evaluation of anxiety and depression are mainly divided into two categories: self-rating scales and other-rating scales. Commonly used self-rating scales include: Self-rating Depression Scale, Self-rating Anxiety Scale, Hospital Depression Scale, Hospital Anxiety Scale, Beck Anxiety Inventory, Beck Depression Inventory; other-rating scales include: Hamilton Depression Rating Scale, Hamilton Anxiety Scale, etc. The evaluation of anxiety and depression varies according to the selected scale, and there is no unified diagnostic standard yet. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a COPD anxiety and depression screening method and system based on deep convolutional neural network.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a COPD anxiety and depression screening method based on a deep convolutional neural network, the method comprising the following steps:

[0006] Obtain a sample video of the patient, extract facial features and voice features from the sample video of the patient, and obtain text features and audio features of the patient;

[0007] The text features and audio features of the patient are fused in the channel dimension to obtain multi-dimensional fusion features;

[0008] The multi-dimensional fusion features are input into the pre-established regression and classification anxiety and depression screening model, and the patient's anxiety and depression screening results are output.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the sampling video of the patient is a collection of multiple frames of images, and the acquisition range of the sampling video of the patient is the upper body of the patient in a sitting state.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: a preprocessing process of the sampled video of the patient performs key feature extraction through an openface framework.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of extracting key features through the openface framework is as follows:

[0012] Face extraction: Detect and crop faces in the video frames of the patient's sampled video to obtain face images;

[0013] Facial unit AU extraction: For each frame of face image, facial units are extracted, and the data format is a time series vector;

[0014] Face key point extraction: For each frame of face image, extract the face key points;

[0015] Gaze angle heat map: The gaze angle of a patient in a video is expressed in the form of a heat map. Each pixel of the heat map represents the length of time the patient gazes at a spatial position.

[0016] Among them, the video classification model Temporal shift module is used as the basic feature extraction module, and the video classification model expression is as follows:

[0017] Using TSM for temporal modeling, for the video understanding framework, the shape of the activation is [N, C, T, H, W], the activation A∈RN×C×T×H×W, where N is the batch size, C is the number of channels, T is the temporal dimension, H and W are the spatial resolutions, and the weights of the convolution are assumed to be W=(w1,w2,w3). The input X is an infinitely long one-dimensional vector, and the convolution operator Y=Conv(W,X) is written as:

[0018] Y i =w1X i-1 +w2X i +w3X i+1 .

[0019] Decouple the convolution operation into two steps:

[0020] Shift and multiply and accumulate: shift the input X by -1, 0, +1, and then multiply it by w1, w2, w3 respectively, and their sum is Y;

[0021] The shift operation is: X i -1 =X i-1 , X i 0 =X i , X i +1 =x i+1

[0022] And the multiplication-accumulation operation is:

[0023] Y=w1X -1 +w2X 0 +w3X +1 .

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further includes: firstly transmitting the multi-dimensional fusion feature to a regression module, transmitting the output of the regression module to a classification module at a training node, and outputting a regression result according to the classification module;

[0025] In the testing phase, the scores of each question based on the regression results are added together to get the total score, and then the patients are classified according to the set rules and the total score.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the overall training loss function of the pre-established regression and classification anxiety and depression screening model is designed as

[0027]

[0028] in are the loss functions for regression and classification, respectively, score is the actual score of each question for each patient, y label is the true classification label of each patient, λ is the hyperparameter that balances the two loss functions, and y reg and cls They are the regression and classification results of the model respectively.

[0029] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: optimizing the pre-established regression and classification anxiety and depression screening model using stochastic gradient descent SGD in deep learning, and fixing the weights of the 2D convolutional neural network ResNet in the feature extraction module TSM during the training phase, and optimizing the remaining module parameters.

[0030] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a COPD anxiety and depression screening system based on a deep convolutional neural network, comprising:

[0031] A feature extraction module is used to obtain a sample video of the patient, extract facial features and voice features from the sample video of the patient, and obtain text features and audio features of the patient;

[0032] A feature fusion module is used to fuse the patient's text features and the patient's audio features in the channel dimension to obtain multi-dimensional fusion features;

[0033] The model screening module is used to input multi-dimensional fusion features into the pre-established regression and classification anxiety and depression screening model, and output the patient's anxiety and depression screening results.

[0034] In conjunction with the second aspect, in certain implementations of the second aspect, the system further includes: the sampling video of the patient of the feature extraction module is a collection of multiple frames of images, and the acquisition range of the sampling video of the patient is the upper body of the patient in a sitting state;

[0035] Feature extraction module The preprocessing process of the patient's sample video is performed through the openface framework to extract key features;

[0036] The process of feature extraction module extracting key features through the openface framework is as follows:

[0037] Face extraction: Detect and crop faces in the video frames of the patient's sampled video to obtain face images;

[0038] Facial unit AU extraction: For each frame of face image, facial units are extracted, and the data format is a time series vector;

[0039] Face key point extraction: For each frame of face image, extract the face key points;

[0040] Gaze angle heat map: The gaze angle of a patient in a video is expressed in the form of a heat map. Each pixel of the heat map represents the length of time the patient gazes at a spatial position.

[0041] The multi-dimensional fusion features of the feature fusion module are first transmitted to the regression module. At the training node, the output of the regression module is transmitted to the classification module, and the regression result is output according to the classification module;

[0042] In the test phase, the scores of each question based on the regression results are added together to get the total score, and then the patients are classified according to the set rules and the total score;

[0043] Among them, the video classification model Temporal shift module is used as the basic feature extraction module, and the video classification model expression is as follows:

[0044] Using TSM for temporal modeling, for the video understanding framework, the shape of the activation is [N, C, T, H, W], the activation A∈RN×C×T×H×W, where N is the batch size, C is the number of channels, T is the temporal dimension, H and W are the spatial resolutions, and the weights of the convolution are assumed to be W=(w1,w2,w3). The input X is an infinitely long one-dimensional vector, and the convolution operator Y=Conv(W,X) is written as:

[0045] Y i =w1X i-1 +w2X i +w3X i+1 .

[0046] Decouple the convolution operation into two steps:

[0047] Shift and multiply and accumulate: shift the input X by -1, 0, +1, and then multiply it by w1, w2, w3 respectively, and their sum is Y;

[0048] The shift operation is: X i -1 =X i-1 , X i 0 =X i , X i +1 =x i+1

[0049] And the multiplication-accumulation operation is:

[0050] Y=w1X -1 +w2X 0 +w3X +1 .

[0051] The overall training loss function of the pre-established regression and classification anxiety and depression screening model in the model screening module is designed as

[0052]

[0053] in are the loss functions for regression and classification, respectively, score is the actual score of each question for each patient, y label is the true classification label of each patient, λ is the hyperparameter that balances the two loss functions; y reg and cls They are the regression and classification results of the model respectively.

[0054] The optimization of the regression and classification anxiety and depression screening model pre-established in the model screening module adopts the stochastic gradient descent SGD in deep learning, and the weights of the 2D convolutional neural network ResNet in the feature extraction module TSM are fixed during the training phase, and the parameters of the remaining modules are optimized.

[0055] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the COPD anxiety and depression screening method based on a deep convolutional neural network as described above is adopted.

[0056] Beneficial effects of the present invention:

[0057] This invention uses the currently popular AI technology to make the screening of anxiety and depression in COPD patients more intelligent. This technology is expected to realize remote preliminary diagnosis and treatment, non-contact rapid diagnosis and treatment screening, and interactive monitoring and intervention of physical and mental health on the basis of promoting the Internet of Things medicine, improve patient compliance and mental health, further reduce the problem of delayed illness or poor prognosis caused by insufficient medical resources or insufficient diagnostic accuracy, and improve the quality of life of patients.

[0058] The present invention integrates the facial expression features of COPD patients into the modality, and evaluates the emotional state of patients by analyzing the facial expressions of patients and extracting facial expression features. The technology can be implemented based on deep learning algorithms, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0059] The present invention uses the conversation content in the collected video data as one of the modalities to extract sound features and speech emotion features to evaluate the patient's emotional state. This technology can be implemented based on speech signal processing algorithms, such as Fourier transform, wavelet transform and speech signal enhancement.

[0060] The present invention performs multimodal data fusion on the basis of single-modal training, which can make full use of information of different modalities and improve the reliability, generalization ability and scalability of the model.

[0061] The loss function based on regression and classification proposed in the present invention can better utilize the rule constraints of anxiety and depression scores and degrees, so that the output of the regression module is more in line with the actual situation.

[0062] This paper establishes a screening model based on deep convolutional neural network technology, and proposes a non-contact data collection method for COPD combined with anxiety and depression. It uses deep learning methods to more comprehensively and accurately assess the anxiety and depression of COPD patients. Compared with traditional methods, it has the advantages of non-contact, rapid, non-invasive, and automated. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a schematic flow chart of the method of the present invention;

[0065] Figure 2 It is a schematic diagram of the time dimension and channel dimension in the tensor of the present invention;

[0066] Figure 3 It is a schematic diagram of the time delay under different ratio channels of the present invention;

[0067] Figure 4 It is a schematic diagram of the multimodal model framework of the present invention;

[0068] Figure 5 This is a schematic diagram of anxiety and depression in COPD patients of the present invention;

[0069] Figure 6 is a schematic diagram of the distribution of anxiety and depression in patients with different lung function groups of the present invention;

[0070] Figure 7 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Embodiment 1:

[0073] like Figure 1 As shown, a COPD anxiety and depression screening method based on a deep convolutional neural network comprises the following steps:

[0074] S101: Obtain a sample video of a patient, extract facial features and voice features from the sample video of the patient, and obtain text features and audio features of the patient;

[0075] The sampled video of the patient is a collection of multiple frames of images, and the acquisition range of the sampled video of the patient is the upper body of the patient in a sitting state;

[0076] In order to effectively extract key features, detect and evaluate depression in patients based on their faces, and avoid converting the task into human identity recognition, we extracted the following key features through the OpenFace framework:

[0077] Face extraction: detect and crop faces from video frames;

[0078] Facial unit (AU) extraction: For each frame of the image, extract the facial unit, the data form is a time series vector;

[0079] Face key point extraction: For each frame of image, extract the face key points;

[0080] Gaze angle heat map: The gaze angle of a patient in a video is expressed in the form of a heat map. Each pixel of the heat map represents the duration (intensity) that the patient gazes at that spatial position.

[0081] Unimodal model design and training

[0082] (1) Key feature extraction

[0083] Since the main input data modality is video, we use the current more advanced video classification model Temporal shift module (TSM) as the basic feature extraction module. The way Temporal Shift Module (TSM) models time series is shift. The figure below shows the time dimension and channel dimension in the tensor. Figure 2 As shown, different colors in each row represent different moments. In the time dimension, when we move forward and backward, there will be other colors in a row of the channel dimension. This means that for each frame, we exchange information with nearby frames.

[0084] The simplest movement strategy is to move each channel (naive shift), but this has two problems. Low efficiency. Although the movement operation does not increase the parameters and calculation amount, for large-scale data such as video (5D activation), the delay caused by data movement cannot be ignored. Poor accuracy. Obviously, when we move in time sequence, we move the information of many adjacent frames to the current frame, and the spatial structure of the current frame itself is disrupted, which affects the spatial modeling ability of the model, resulting in poor results. In order to solve these two problems, TSM has made two designs - partial shift and residual shift.

[0085] Artial shift. Only part of the channel is moved, such as 1 / 4 or 1 / 8. As for the specific ratio, it is necessary to consider the latency and accuracy under this ratio. Residual shift. The TSM used in this article is based on ResNet. The most direct way to add shift to ResNet is to do a shift before each convolution or each residual block. We call it In-place TSM. When doing convolution in this way, the moving operation loses the spatial information at the moment. Therefore, we take different approaches, such as Figure 3 As shown, the shift operation is placed in the residual value, and the other one remains unchanged to retain the original spatial information of a certain frame. This can avoid damaging the spatial feature learning ability of the original 2D CNN model. We call it Residual TSM. Here is an explanation of why each convolutional layer or block must be shifted, rather than just shifting it at the beginning and not using it anymore. This is because the receptive field in the time domain will double with each +-1 shift operation. It is known that as the ResNet network deepens, the spatial receptive field will become larger and larger, and more and more information will be seen. To simulate this situation in time series, the receptive field must be expanded as the network deepens, which is equivalent to seeing more and more information in the time domain to simulate the process of time series convolution.

[0086] Among them, the video classification model Temporal shift module is used as the basic feature extraction module, and the video classification model expression is as follows:

[0087] Using TSM for temporal modeling can efficiently understand temporal videos in the long term. For the video understanding framework, the shape of the activation is usually [N, C, T, H, W] (activation A∈RN×C×T×H×W), where N is the batch size, C is the number of channels, T is the temporal dimension, and H and W are the spatial resolutions. First consider an ordinary convolution operation. For simplicity, we take a one-dimensional convolution with a kernel size of 3 as an example. Assume that the weights of the convolution are W=(w1,w2,w3) and the input X is an infinitely long one-dimensional vector. The convolution operator Y=Conv(W,X) can be written as: Y i =w1X i-1 +w2X i +w3X i+1 .

[0088] We can decouple the convolution operation into two steps: shift and multiply-accumulate: we shift the input X by -1, 0, +1, and then multiply it by w1, w2, w3 respectively, and their sum is Y.

[0089] The shift operation is: X i -1 =X i-1 , X i 0 =X i , X i +1 =x i+1(1)

[0090] And the multiplication-accumulation operation is:

[0091] Y=w1X -1 +w2X 0 +w3X +1 (2)

[0092] The first shift step can be performed for free since it only requires an offset address pointer. Although the second step is more computationally expensive, our temporal shift module merges the multiplication-accumulation into the following 2D convolution, so it has no additional cost compared to 2DCNN based models.

[0093] S102: Fusing the patient's text features and the patient's audio features in the channel dimension to obtain a multi-dimensional fusion feature;

[0094] The multi-dimensional fusion features are first transmitted to the regression module. At the training node, the output of the regression module is transmitted to the classification module, and the regression result is output according to the classification module.

[0095] In the testing phase, the scores of each question based on the regression results are added together to get the total score, and then the patients are classified according to the set rules and the total score.

[0096] We adopted a multimodal data fusion method that fuses at the feature level, that is, using different feature extraction networks to extract features from different modal data, and then fuse the extracted features in the channel dimension and transmit them to the subsequent regression and classification modules. Figure 4 shown.

[0097] S103: Input the multi-dimensional fusion features into a pre-established regression and classification anxiety and depression screening model, and output the patient's anxiety and depression screening results.

[0098] The overall training loss function of the pre-established regression and classification anxiety and depression screening model was designed as

[0099]

[0100] in are the loss functions for regression and classification, respectively, score is the actual score of each question for each patient, y label is the true classification label for each patient, and λ is the hyperparameter y that balances the two loss functions. reg and cls They are the regression and classification results of the model respectively.

[0101] The model is optimized using stochastic gradient descent (SGD), which is commonly used in deep learning. During the training phase, the weights of the 2D convolutional neural network ResNet (pretrained on the large image classification dataset ImageNet) in the feature extraction module TSM are fixed, and the parameters of the remaining modules are optimized.

[0102] Specifically, the present invention is further described below by way of embodiments:

[0103] After the model training is completed, the depression screening system is installed in the relevant departments. While the patients are undergoing COPD disease diagnosis and treatment in the relevant departments, they are also assessed for anxiety and depression, so as to timely detect the psychological problems of the patients and intervene and treat them as soon as possible.

[0104] In the early stage, high-quality video collection was performed on 94 COPD patients in stable stage. The psychological state was evaluated through contactless and multimodal affective computing (hereinafter referred to as AI technology). At the same time, the traditional HAMA and HAMD scales were evaluated and the data of the two were compared. The study showed that 4 COPD patients in stable stage failed to complete the HAMA or HAMD scale assessment due to poor respiratory function and other factors. All COPD patients received video information collection and obtained the preliminary psychological assessment results of the subjects. The results are shown as follows:

[0105] 1) The prevalence of anxiety and depression in COPD patients (e.g. Figure 5 (shown)

[0106] 2) AI technology and scales for evaluating anxiety and depression in patients with stable COPD (Table 1)

[0107]

[0108] 3) Distribution of anxiety and depression in patients with different lung function groups (e.g. Figure 6 (shown)

[0109] The above preliminary results show that AI technology is more convenient to implement in the assessment of anxiety and depression disorders, and has good consistency with various scales. For patients in the stable phase of COPD, especially those with poor lung function, the more cumbersome assessment scale is difficult to adhere to, which may lead to data loss.

[0110] Embodiment 2: In the second aspect, as Figure 7 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a COPD anxiety and depression screening system based on a deep convolutional neural network, comprising:

[0111] The feature extraction module 11 is used to obtain a sample video of the patient, extract facial features and voice features from the sample video of the patient, and obtain text features and audio features of the patient;

[0112] A feature fusion module 12 is used to fuse the patient's text features and the patient's audio features in the channel dimension to obtain a multi-dimensional fusion feature;

[0113] The model screening module 13 is used to input the multi-dimensional fusion features into the pre-established regression and classification anxiety and depression screening model, and output the patient's anxiety and depression screening results.

[0114] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: the sampling video of the patient of the feature extraction module 11 is a collection of multiple frames of images, and the acquisition range of the sampling video of the patient is the upper body of the patient in a sitting state;

[0115] Feature extraction module 11 The preprocessing process of the patient's sampled video is performed through the openface framework to extract key features;

[0116] The process of feature extraction module 11 extracting key features through the openface framework is as follows:

[0117] Face extraction: Detect and crop faces in the video frames of the patient's sampled video to obtain face images;

[0118] Facial unit AU extraction: For each frame of face image, facial units are extracted, and the data format is a time series vector;

[0119] Face key point extraction: For each frame of face image, extract the face key points;

[0120] Gaze angle heat map: The gaze angle of a patient in a video is expressed in the form of a heat map. Each pixel of the heat map represents the length of time the patient gazes at a spatial position.

[0121] Among them, the video classification model Temporal shift module is used as the basic feature extraction module, and the video classification model expression is as follows:

[0122] Using TSM for temporal modeling, for the video understanding framework, the shape of the activation is [N, C, T, H, W], the activation A∈RN×C×T×H×W, where N is the batch size, C is the number of channels, T is the temporal dimension, H and W are the spatial resolutions, and the weights of the convolution are assumed to be W=(w1,w2,w3). The input X is an infinitely long one-dimensional vector, and the convolution operator Y=Conv(W,X) is written as:

[0123] Y i =w1X i-1 +w2X i +w3X i+1 .

[0124] Decouple the convolution operation into two steps:

[0125] Shift and multiply and accumulate: shift the input X by -1, 0, +1, and then multiply it by w1, w2, w3 respectively, and their sum is Y;

[0126] The shift operation is: X i -1 =X i-1 , X i 0 =X i , X i +1 =x i+1

[0127] And the multiplication-accumulation operation is:

[0128] Y=w1X -1 +w2X 0 +w3X +1

[0129] The multi-dimensional fusion features of the feature fusion module 12 are first transmitted to the regression module. At the training node, the output of the regression module is transmitted to the classification module, and the regression result is output according to the classification module;

[0130] In the test phase, the scores of each question based on the regression results are added together to get the total score, and then the patients are classified according to the set rules and the total score;

[0131] The overall training loss function of the pre-established regression and classification anxiety and depression screening model in model screening module 13 is designed as

[0132]

[0133] in are the loss functions for regression and classification, respectively, score is the actual score of each question for each patient, y label is the true classification label of each patient, λ is the hyperparameter that balances the two loss functions; y reg and cls They are the regression and classification results of the model respectively.

[0134] The optimization of the regression and classification anxiety and depression screening model pre-established in the model screening module 13 adopts the stochastic gradient descent SGD in deep learning, and fixes the weights of the 2D convolutional neural network ResNet in the feature extraction module TSM during the training phase, and optimizes the parameters of the remaining modules.

[0135] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0136] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0137] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0138] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A COPD anxiety and depression screening method based on deep convolutional neural network, characterized in that: The method comprises the following steps: Obtain a sample video of the patient, extract facial features and voice features from the sample video of the patient, and obtain text features and audio features of the patient; The text features and audio features of the patient are fused in the channel dimension to obtain multi-dimensional fusion features; The multi-dimensional fusion features are input into the pre-established regression and classification anxiety and depression screening model, and the patient's anxiety and depression screening results are output.

2. The COPD anxiety and depression screening method based on deep convolutional neural network according to claim 1, characterized in that: The sampled video of the patient is a collection of multiple frames of images, and the acquisition range of the sampled video of the patient is the upper body of the patient in a sitting state.

3. The COPD anxiety and depression screening method based on deep convolutional neural network according to claim 2, characterized in that: The preprocessing process of the patient's sample video is performed by extracting key features through the openface framework.

4. The COPD anxiety and depression screening method based on deep convolutional neural network according to claim 3, characterized in that: The process of extracting key features through the openface framework is as follows: Face extraction: Detect and crop faces in the video frames of the patient's sampled video to obtain face images; Facial unit AU extraction: For each frame of face image, facial units are extracted, and the data format is a time series vector; Face key point extraction: For each frame of face image, extract the face key points; Gaze angle heat map: The gaze angle of a patient in a video is expressed in the form of a heat map. Each pixel of the heat map represents the length of time the patient gazes at a spatial position. Among them, the video classification model Temporal shift module is used as the basic feature extraction module, and the video classification model expression is as follows: Using TSM for temporal modeling, for the video understanding framework, the shape of the activation is [N, C, T, H, W], the activation A∈RN×C×T×H×W, where N is the batch size, C is the number of channels, T is the temporal dimension, H and W are the spatial resolutions, and the weights of the convolution are assumed to be W=(w1,w2,w3). The input X is an infinitely long one-dimensional vector, and the convolution operator Y=Conv(W,X) is written as: Y i =w1X i-1 +w2X i +w3X i+1 Decouple the convolution operation into two steps: Shift and multiply and accumulate: shift the input X by -1, 0, +1, and then multiply it by w1, w2, w3 respectively, and their sum is Y; The shift operation is: X i -1 =X i-1 , X i 0 =X i , X i +1 =x i+1 And the multiplication-accumulation operation is: Y=w1X -1 +w2X 0 +w3X +1 。 5. The COPD anxiety and depression screening method based on deep convolutional neural network according to claim 1, characterized in that: The multi-dimensional fusion features are first transmitted to the regression module. At the training node, the output of the regression module is transmitted to the classification module, and the regression result is output according to the classification module. In the testing phase, the scores of each question based on the regression results are added together to get the total score, and then the patients are classified according to the set rules and the total score.

6. The COPD anxiety and depression screening method based on deep convolutional neural network according to claim 1, characterized in that: The overall training loss function of the pre-established regression and classification anxiety and depression screening model was designed as in are the loss functions for regression and classification, respectively, score is the actual score of each question for each patient, y label is the true classification label of each patient, λ is the hyperparameter that balances the two loss functions, and y reg and cls They are the regression and classification results of the model respectively.

7. The COPD anxiety and depression screening method based on deep convolutional neural network according to claim 6, characterized in that: The optimization of the pre-established regression and classification anxiety and depression screening model adopts stochastic gradient descent SGD in deep learning, and fixes the weights of the 2D convolutional neural network ResNet in the feature extraction module TSM during the training phase, and optimizes the remaining module parameters.

8. A COPD anxiety and depression screening system based on deep convolutional neural network, characterized in that: include: A feature extraction module is used to obtain a sample video of the patient, extract facial features and voice features from the sample video of the patient, and obtain text features and audio features of the patient; A feature fusion module is used to fuse the patient's text features and the patient's audio features in the channel dimension to obtain multi-dimensional fusion features; The model screening module is used to input multi-dimensional fusion features into the pre-established regression and classification anxiety and depression screening model, and output the patient's anxiety and depression screening results.

9. The COPD anxiety and depression screening system based on deep convolutional neural network according to claim 8, characterized in that: The sampled video of the patient in the feature extraction module is a collection of multiple frames of images, and the acquisition range of the sampled video of the patient is the upper body of the patient in a sitting state; Feature extraction module The preprocessing process of the patient's sample video is performed through the openface framework to extract key features; The process of feature extraction module extracting key features through the openface framework is as follows: Face extraction: Detect and crop faces in the video frames of the patient's sampled video to obtain face images; Facial unit AU extraction: For each frame of face image, facial units are extracted, and the data format is a time series vector; Face key point extraction: For each frame of face image, extract the face key points; Gaze angle heat map: The gaze angle of a patient in a video is expressed in the form of a heat map. Each pixel of the heat map represents the length of time the patient gazes at a spatial position. The multi-dimensional fusion features of the feature fusion module are first transmitted to the regression module. At the training node, the output of the regression module is transmitted to the classification module, and the regression result is output according to the classification module; In the test phase, the scores of each question based on the regression results are added together to get the total score, and then the patients are classified according to the set rules and the total score; Among them, the video classification model Temporal shift module is used as the basic feature extraction module, and the video classification model expression is as follows: Using TSM for temporal modeling, for the video understanding framework, the shape of the activation is [N, C, T, H, W], the activation A∈RN×C×T×H×W, where N is the batch size, C is the number of channels, T is the temporal dimension, H and W are the spatial resolutions, and the weights of the convolution are assumed to be W=(w1,w2,w3). The input X is an infinitely long one-dimensional vector, and the convolution operator Y=Conv(W,X) is written as: Y i =w1X i-1 +w2X i +w3X i+1 Decouple the convolution operation into two steps: Shift and multiply and accumulate: shift the input X by -1, 0, +1, and then multiply it by w1, w2, w3 respectively, and their sum is Y; The shift operation is: X i -1 =X i-1 , X i 0 =X i , X i +1 =x i+1 And the multiplication-accumulation operation is: Y=w1X -1 +w2X 0 +w3X +1 The overall training loss function of the pre-established regression and classification anxiety and depression screening model in the model screening module is designed as in are the loss functions for regression and classification, respectively, score is the actual score of each question for each patient, y label is the true classification label of each patient, λ is the hyperparameter that balances the two loss functions; y reg and cls They are the regression and classification results of the model respectively; The optimization of the regression and classification anxiety and depression screening model pre-established in the model screening module adopts the stochastic gradient descent SGD in deep learning, and the weights of the 2D convolutional neural network ResNet in the feature extraction module TSM are fixed during the training phase, and the parameters of the remaining modules are optimized.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on a processor. When the processor loads and executes the computer program, the COPD anxiety and depression screening method based on a deep convolutional neural network described in any one of claims 1 to 7 is adopted.

Images