Depression auxiliary diagnosis method based on multi-scale facial information integration

Abstract

The application discloses a depression auxiliary diagnosis method based on multi-scale face information integration. Firstly, video segment feature extraction is carried out based on a space-time attention mechanism; important video regions and video segments are adaptively weighted by a model; then, important video segments are sampled at a smaller interval, more effective information is obtained therefrom, and the information is used as input data for model learning in the next round; finally, all model results are fused by using a reasonable integration scheme to obtain a prediction regression value. The method of the application obtains more effective information, and solves the problems of a large amount of redundant information, loss of key data information and poor performance of a classification model in the existing model learning process.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of face recognition, and particularly relates to a depression auxiliary diagnosis method based on multi-scale face information integration. BACKGROUND

[0002] Facial expression is one of the most powerful and universal means of human communication, which is highly related to human psychological state, attitude and intention. In addition to the ordinary facial expressions we see in daily life, emotions can also be expressed in special formats of micro-expressions under certain conditions. When people try to hide their true feelings, they will reveal their hidden feelings under certain stimuli. Compared with ordinary people, the most obvious difference of the facial expression of the depression patients is that the change range of the facial expression is small, and the patients will show obvious characteristics of emotional depression and anxiety. By using the facial movements made by the patients subconsciously, the depression patients can be effectively identified, and the detection results can be prevented from being affected by the subjective consciousness of the patients. In the actual outpatient stage, one of the main ways for doctors to judge the mental state of patients is to observe the change of the facial expression of the patients. However, artificial diagnosis needs the field knowledge of professional persons, and part of the micro-expression changes that are difficult to detect will affect the judgment of the doctors. Therefore, there is an urgent need for a facial expression recognition method of the depression patients which can completely capture the change of the facial expression of the patients and is not affected by the micro-expression changes in the artificial judgment.

[0003] In the prior art, the facial expression-based depression expression recognition mainly processes video stream data. Compared with the picture classification field, the video understanding increases the time dimension, and it is crucial to obtain dynamic information based on the time dimension. Al Jazaery (Ieee Transactions on Affective Computing, 2021.12 (1)), He (Ieee Transactions on Multimedia, 2019.21 (6)) and others disclose that the light flow, HOG and hand-made features are used to generate descriptors with appearance and dynamic information coding as the time dimension information extracted from the video. However, the model of this non-end-to-end way lacks the automatic learning of the feature hierarchy from large-scale raw data, thereby having the characteristics of generating discriminative features with high abstraction and invariance, and it is difficult to be applied to practice. Deep learning can solve this problem, and currently there are two ways to learn the dynamic information of the video in space and time. One is to use CNN to encode and extract features of each picture of the video, and then use LSTM to learn the context relationship between pictures in sequence as the dynamic information of the time dimension. The other is to add a time dimension on the basis of 2D CNN, and use 3D CNN to consider the nearness and remoteness of the context association by controlling the size of the step in the time dimension. The artificial and deep learning methods mentioned above follow the same assumption that all the time features of the entire video contribute uniformly to the classification task. However, the facial expression of the person in the video is changeable, and does not contain useful information at all times, resulting in that the video contains redundant information; if all frames are taken as the training data of the model, serious overfitting problem will be caused, and the large amount of data will cause training difficulty. Although the current research obtains a picture sequence set by taking one frame every 6 frames or every 10 frames of video data, this interval sampling can cause the problem of information loss in the time dimension (the phenomenon of discontinuous facial expression change), resulting in that some key video frames are discarded. In addition, in the field of depression recognition based on expression change, the schemes used at present mostly use single classifiers to generate results by fusing feature information. However, the single classifier cannot comprehensively consider the diversity information of the data, and the classification performance is poor. SUMMARY

[0004] The present application is a depression auxiliary diagnosis method based on multi-scale facial information integration. In the prior art, a large amount of redundant information and noise is introduced in the model learning process, which seriously affects the recognition accuracy and increases the calculation difficulty. If one frame is selected every several frames to obtain a picture sequence, information loss will occur, and key frames may be missed. In addition, the single classifier used in the prior art cannot comprehensively consider the diversity information of the data, and the classification performance is poor. The present application is proposed.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is a depression auxiliary diagnosis method based on multi-scale face information integration, comprising the following steps:

[0006] S1, attention mechanism-based image sequence spatio-temporal feature extraction;

[0007] S2, multi-scale information integration-based video stream data processing;

[0008] S3, integrated learning of face features extracted under different sampling rate scales based on a weighting mechanism.

[0009] Further, the specific steps of S1, attention mechanism-based spatio-temporal feature extraction, are as follows:

[0010] S11, g(x) represents generating attention values based on input data x, and f(g(x), x) represents processing input data x based on attention, establishing an image sequence attention mechanism model:

[0011] Attention=f(g(x),x)

[0012] S12, spatial feature extraction

[0013] Global average pooling and maximum pooling are performed on the input face video data respectively; then the attention values M c :

[0014]

[0015] The reference from the channel is converted to spatial content information, and the spatial attention value M S :

[0016]

[0017] wherein σ() is an activation function, represents average pooling on the channel dimension, represents maximum pooling on the channel dimension, W0, W1, W2, W3, and W4 represent learnable parameters; represents average pooling on the spatial dimension, represents maximum pooling on the spatial dimension; average pooling and maximum pooling are performed on the channel dimension respectively to obtain two HxWx1 channel descriptions, which are spliced together according to the channel, and then a convolution operation is performed to obtain the weight coefficient M S ; finally, multiplying the weight coefficient by the feature can obtain the scaled new feature;

[0018] S13, time feature extraction

[0019] First, the time attention value is calculated:

[0020] M T (X t )=δ(W3X t +W4h t-1 +b)

[0021] wherein, h t-1 represents the influence calculated based on the last time period, b represents a constant, X t represents the tth video segment, and the attention weight coefficient is finally obtained through a convolution operation.

[0022] Further, the specific steps of S2 based on multi-scale information integration video stream data processing are as follows:

[0023] In the first input, 5fps frame rate is used as the input, so that the total model learns the change of time information on the basis of image sequence space features; the average integration strategy is used, and the final prediction result y of the total model is as follows:

[0024]

[0025] After the key time period is obtained by the attention mechanism, the sampling rate of the key time period is multiplied, and the sampling rate relationship is:

[0026] rate(n)=2 n-1 rate(1)

[0027] wherein, f represents a network model, W() represents parameters of the network model, represents the input of the ith round after processing at the sampling rate of rate(i), and n represents the iteration round; the average integration strategy can also be replaced by any one of the following: voting integration, stacking integration, Boosting integration, and Bagging integration.

[0028] Further, the specific steps of S3 based on the weighted mechanism integration learning module are as follows:

[0029] The weight calculation of each iteration is as shown in the following formula:

[0030] q k =MAX(softmax(M T (X k )))(1-q n )(1-q n-1 )...(1-q k+1 )

[0031] q k is the weight value in the kth iteration of calculation, X k is the input of the kth iteration, and q nThe weight of the next round iteration is higher than that of the current round iteration; and the integrated result of each base learner is calculated as shown in the following formula:

[0032]

[0033] The weight of the next round iteration is higher than that of the current round iteration; and the integrated result of each base learner is calculated as shown in the following formula:

[0034]

[0035] y is the final prediction result of the total model, M T is the calculation time attention value, Model i indicates a learner network, and softmax is an activation function used to calculate the probability distribution of the result, and before the model training, 50 epochs of training need to be experienced in each iteration before entering the next iteration.

[0036] The application has the beneficial effects that: the application extracts key content information in the video through the spatial attention and channel attention mechanisms, extracts key video segments of the video through the time attention mechanism, and extracts effective features in combination with the three attention mechanisms; different frame rates are used for sampling, so that discarded information in previous research is utilized, more effective information is obtained, and excessive redundant information is not introduced; the facial expression features of the depression patients can be accurately extracted, and the application has guiding significance for clinical depression diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only show some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0038] Figure 1 is a structure diagram of a depression auxiliary diagnosis method based on multi-scale facial information integration;

[0039] Figure 2 is a facial expression recognition model diagram of a depression patient based on facial visual clues;

[0040] Figure 3 is a spatial attention mechanism schematic diagram;

[0041] Figure 4 is a time attention mechanism schematic diagram;

[0042] Figure 5 is a multi-scale based data processing schematic diagram in the model;

[0043] Figure 6 is an integrated learning schematic diagram in the model. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0045] The present application provides a depression auxiliary diagnosis method based on multi-scale facial information integration, as shown in the figure, the method described in the present application comprises the following steps: Figure 1

[0046] S1, image sequence spatio-temporal feature extraction based on attention mechanism;

[0047] S2, video stream data processing based on multi-scale information integration;

[0048] S3, integrated learning of facial features extracted under different sampling rate scales based on weighting mechanism.

[0049] First, the key regions and key sequence segments containing depression facial expression features are detected by the attention mechanism, and the key sequence segments are taken as the input data of the next iteration. Then, higher sampling frequency is used to process the input data of the next iteration to obtain more video information. Finally, the prediction regression value is obtained by effectively combining the results of multiple iterations. The algorithm framework is shown in the figure Figure 2 The specific algorithm process will be described below.

[0050] S1, image sequence spatio-temporal feature extraction based on attention mechanism

[0051] S11, g(x) represents generating attention value based on input data x, f(g(x), x) represents processing input data x based on attention, and an image sequence attention mechanism model is established:

[0052] Attention=f(g(x),x) (1)

[0053] g(x) represents generating attention value based on input data x, and the attention corresponds to the attention degree to the judgment area. f(g(x), x) represents processing input data x based on attention. The present application uses the attention mechanism to find the effective area and time period in the video, and extracts the spatio-temporal features.​

[0054] S12, spatial feature extraction

[0055] A video frame contains a lot of information, but the information related to the identification task mainly appears in part of the image area, and these key areas have the most important influence on the video identification task.

[0056] The channel attention and the spatial attention of the like Figure 3 are calculated in series. The channel attention solves the problem of what the model should pay attention to, and the spatial attention solves the problem of where the model should pay attention to. The attention value is calculated by introducing global pooling. For channel attention, global average pooling and maximum pooling are performed on the input data, respectively, and then the weight coefficient is calculated using a convolutional neural network and an activation function, that is, the attention value corresponding to different channels. As shown in equation (2):

[0057]

[0058] wherein, represents using an average pooling operation on the channel dimension, represents using a maximum pooling operation on the channel dimension, and W0 and W1 are learnable parameters. The calculation of spatial attention is the same as that of channel attention, and the calculation basis is changed from channel to spatial content information, as shown in equation (3):

[0059]

[0060] Average pooling and maximum pooling on the channel dimension are performed to obtain two HxWx1 channel descriptions. These two descriptions are spliced together according to the channel, and a convolution operation is performed to obtain the weight coefficient M s . Finally, the weight coefficient and the feature are multiplied to obtain the scaled new feature.

[0061] S13, time feature extraction

[0062] For a video, key information is not always uniformly distributed in each frame of image, and key frames often have a greater impact on the final identification result.

[0063] As shown in Figure 4 , the calculation of time attention needs to consider the correlation between different time periods, and the influence of local and global and adjacent segments to calculate the final attention value. The calculation of the attention value is shown in equation (4):

[0064] M T (X t )=σ(W3X t +W4h t-1 +b) (4)

[0065] h t-1 represents the influence calculated based on the previous time period, b represents a constant, X t represents the tth video segment, the influence is introduced to consider the correlation between adjacent time periods and between the first t-1 time periods and the tth time period. Finally, the attention weight coefficient is obtained through convolution operation.

[0066] S2, video stream data processing based on multi-scale information integration

[0067] Multi-scale information integration is sampling data at different granularities. The model learns different feature information for input data at different scales. In the processing of video stream data, the model has different attention degrees for video information at different frame rates. Studies on retinal ganglion cells in primate visual systems in biology show that the cells that acquire visual information from the outside world are composed of small cells and large cells. Large cells work at high refresh frequency and mainly capture information on position changes in vision, and are not sensitive to spatial details and color. Small cells are the opposite, and mainly collect spatial information. This result shows that humans collect visual information based on multi-scale frame rate in practice. Slowfast is a video understanding model designed based on this. It consists of two branches of Resnet. Different branches use video data at different frame rates as input. In this invention, the following strategies are adopted to utilize video stream data at multi-scale frame rate.

[0068] As Figure 5 In the first input, 5fps frame rate is used as input. The video may contain a large amount of invalid information. Direct use of high frame rate sampling not only cannot efficiently obtain valid information, but also increases the calculation difficulty, but the idea of multi-scale frame rate can still be used to make the model learn the change of time information on the basis of spatial features.

[0069]

[0070] f represents the network model, W() represents the parameters of the network model, represents the input at rate(i) after processing at the sampling rate of rate(i) in the ith round. The integration method can be voting integration, average integration, stacking integration, Boosting integration, and Bagging integration. In this embodiment, the average integration method is adopted. Therefore, as shown in formula (6), the results obtained at different sampling rates are integrated. After the attention mechanism obtains the key time period, the sampling rate of the key time period is multiplied to obtain more time change information. That is, the sampling rate relationship is:

[0071] rate(n)=2 n-1 rate(1) (6)

[0072] n represents the iteration round, the amount of data obtained will also multiply due to the multiple growth of the sampling rate, so the video segments are screened based on the attention weight, which can effectively control the data amount, and prevent the spatial information redundancy from increasing by multiple after the sampling rate is improved. The model learning process is carried out in this iterative way, and finally the most spatio-temporal information is learned under the multi-scale frame rate.

[0073] S3, integrated learning module based on weighting mechanism

[0074] As Figure 6 shown, integrated learning completes the learning task by considering the results of multiple single classifiers, which can effectively improve the prediction accuracy while ensuring the diversity of weak classifiers.

[0075] As shown in S1 and S2, the number of iterations of the model is proportional to the proportion of effective data, and the attention mechanism will gradually select the segment with the highest prediction accuracy as the input for the next iteration. Since the input data is not completely the same each time, and the input each time is a subset of the last iteration, each learner has a certain independence. In the integration of multiple learners, the influence of data effectiveness on the result should be considered, so the weight should show an increasing effect with the number of iterations. Then the weight calculation of each iteration should be as shown in formula (7):

[0076] q k = MAX(softmax(M T (X k )))(1-q n )(1-q n-1 )...(1-q k+1 ) (7)

[0077] q k is the weight value in the kth iteration of calculation, X k is the input of the kth iteration, and the value of q n is the weight proportion sum of the video segment with the maximum attention weight in the nth round and the adjacent m segments. First, calculate the proportion in this round, then multiply it by the weight proportion sum of the video segments not selected in the next round, which means the more iterations, the less effective information proportion in the previous iteration, and the less influence.

[0078] The ratio of two adjacent iterations satisfies:

[0079]

[0080] The weight of the next iteration is higher than that of this iteration, which meets the principle of effective information influence in the attention mechanism. Finally, the integrated result of each base learner is calculated as shown in formula (9):

[0081]

[0082] Model i The representation learner network, softmax is an activation function for calculating the probability distribution of the result, the value of the selected adjacent video segment number m needs to be considered in combination with the actual situation of the experiment when calculating q, the integration scheme meets the effect that different influences in the attention mechanism affect the result, and also increases the influence of effective information on the final prediction result. In the early stage of model training, it needs to experience 50 epochs of training in each iteration before entering the next iteration, the purpose is to make the attention mechanism weight reliable.

[0083] Experimental example:

[0084] The prediction effect of the attention mechanism module of the present application on the data set AVEC2013 is shown in Table 1:

[0085] Table 1 Effect on data set AVEC2013

[0086] Methods MAE RMSE Baseline 10.88 13.61 Brunel-Beihang 9.14 11.19 Wen et al. 8.22 10.27 Kaya et al. 7.86 9.72 Zhu et al. 7.58 9.82 RNN-C3D 7.37 9.28 LGA-CNN 6.59 8.39 He et al. 6.83 8.46 Zhou et al. 6.63 8.37 Invention 6.41 7.68

[0087] It can be seen that the mean absolute error MAE and the root mean square error RMSE on the data set are the lowest.

[0088] Each embodiment in the specification is described in a related manner, and the same and similar parts between each embodiment can be referred to each other, and each embodiment mainly explains the difference from other embodiments. Especially, for the system embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and the related parts can refer to the part of the method embodiment.

[0089] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application is included in the protection scope of the present application.

Claims

1.A method for auxiliary diagnosis of depression based on multi-scale facial information integration, characterized in that, The method comprises the following steps: S1, image sequence space-time feature extraction based on attention mechanism; S2, video stream data processing based on multi-scale information integration; S3, integrated learning of facial features extracted under different sampling rate scales based on a weighting mechanism; The specific steps of the S1, image sequence space-time feature extraction based on attention mechanism, are as follows: S11、 representing processing input data based on attention generate attention values, representing processing input data based on attention , establish an image sequence attention mechanism model: S12, spatial feature extraction respectively, and then calculate the attention value of different channels Calculate the reference from the channel to the spatial content information, and calculate the spatial attention value wherein, is an activation function, indicates average pooling on the channel dimension, indicates maximum pooling on the channel dimension, indicates parameters that can be learned; indicates average pooling on the spatial dimension, indicates maximum pooling on the spatial dimension; average pooling and maximum pooling on the channel dimension are performed respectively to obtain two channel descriptions, the two descriptions are spliced together according to the channel, and a convolution operation is performed to obtain weight coefficients Finally, the weight coefficients and the features are multiplied to obtain the scaled new features; S13, time feature extraction First, the time attention value is calculated: wherein, represents the influence based on the previous time period calculation, represents a constant, represents the first video segment, and finally the attention weight coefficient is obtained through a convolution operation; The specific steps of the S2, video stream data processing based on multi-scale information integration, are as follows: In the first input, 5fps frame rate is used as input, so that the total model learns the change of time information on the basis of image sequence space features; the average integration strategy is used, and the final prediction result obtained by integrating the results under different sampling rates is obtained As follows: After the key time period is obtained by the attention mechanism, the sampling rate of the key time period is multiplied, and the sampling rate relationship is: Wherein, The network model is represented by, The parameters of the network model are represented by, The sampling rate is represented by, The input of the first round is processed at the sampling rate of as input data, The iteration round is represented by; The specific steps of the S3, integrated learning of facial features extracted under different sampling rate scales based on a weighting mechanism, are as follows: The weight calculation of each iteration is shown in the following formula: is the weight value in the th iteration of calculation, is the input of the th iteration, is the weight value of the video segment with the maximum attention weight in the th round and the weight proportion sum of adjacent segments, the weight proportion sum in the current round is first calculated in other rounds, then multiplied by the weight proportion sum of the video segment not selected in the next round, which indicates that the more the number of iterations is, the less the effective information proportion in the previous iteration is, and the proportion of two adjacent iterations satisfies: The weight of the next round iteration is higher than that of the current round iteration; the final ensemble result of each base learner is calculated as shown in the following formula: is the final prediction result of the total model, is the calculation time attention value, is the representation learner network, is the activation function for calculating the probability distribution of the result, and before the model training, it needs to go through 50 epochs of training in each iteration before entering the next iteration. 2.The method of claim 1, wherein, The average integration strategy can also be replaced by any one of the following: voting integration, stacking integration, Boosting integration, and Bagging integration.

Images