Intelligent depression emotion recognition method and system based on stress micro-expression

Abstract

The invention discloses an intelligent depression emotion recognition method and system based on stress micro expressions, and relates to the technical field of computer vision and deep learning, and the method comprises the steps: based on a self-evaluation result of a picture stimulation test questionnaire, calculating a depression awakening degree and a depression association degree, establishing a stress micro-expression image sequence according to the test result of the picture stimulation of the high depression exhalation degree and the high depression correlation degree; obtaining a time dimension attention sequence through sequence difference feature extraction and time clustering boundary sequence subscript processing; utilizing a spatial feature extraction module to obtain depressive emotion spatial feature extraction result vectors of the stress micro-expression image subsequences; and extracting a time positive-sequence feature extraction vector and a time reverse-sequence feature extraction vector by using a time feature extraction module, and obtaining a depressive emotion recognition classification result of the stress micro-expression image sequence through vector mapping. According to the invention, the depressive emotion features of the long-time sequence image can be extracted, and a more accurate depressive emotion recognition result can be output.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a method and system for intelligently identifying depressive emotions based on stress micro-expressions. Background Art

[0002] Depression is one of the most common mental illnesses, with a high incidence rate worldwide. According to data from the World Health Organization, there are as many as 322 million people suffering from depression worldwide. Many documents have shown that there are significant differences between the micro-expressions of patients with depression and normal people. By observing the micro-expressions of patients, experts can more accurately distinguish between patients with depression and non-depressed people, providing a reference for accurate diagnosis. In general, micro-expression recognition is an indispensable part of modern depression diagnosis. By accurately distinguishing the micro-expressions of depression, the patient's depression can be more objectively evaluated, providing a scientific basis for treatment decisions, and further promoting the development of psychiatry in the direction of precision and objectivity, which is of great significance to improving the accuracy of screening patients with depression.

[0003] In the diagnosis of depression, micro-expressions are an important indicator for identifying depression and further diagnosing depression. For people who are in a depressed mood, their micro-expressions have many characteristics, such as drooping eyebrows, high degree of eye closure, drooping corners of the eyes, drooping corners of the mouth, etc. Compared with people with depression but not depression, the above micro-expression characteristics of patients with depression are more significant and have significant differences in intensity; in addition, there are also certain differences in the intensity of micro-expression characteristics of patients with different levels of depression. Accurately obtaining differentiated stress micro-expressions is of great significance for guiding the diagnosis of depression. Therefore, when judging whether a patient has depression, doctors need to fully consider this information so as to diagnose and grade depression more objectively and accurately.

[0004] In recent years, artificial intelligence has been used to identify depression based on stress micro-expressions, mainly in two methods: direct feature extraction and attention-enhanced feature extraction. Micro-expression emotion recognition based on direct feature extraction often generates a large number of redundant parameters, which not only increases the training cost, but also over-extracts many useless features. The accuracy of the method is not ideal, and the time complexity of the algorithm is also high. In contrast, the micro-expression feature processing based on attention enhancement uses adaptive time-enhanced attention processing, which can increase the proportion of effective information in the input data, while improving the accuracy of the model and reducing the number of model parameters.

[0005] Although machine learning and deep learning have achieved many results in computer vision, there are still some shortcomings: the existing technology is affected by factors such as low effectiveness of data collection methods, imprecise data feature enhancement methods, and insufficient extraction of heterogeneous features, which often lead to inaccurate model prediction results or unstable model prediction accuracy; secondly, the traditional data collection method is direct collection under general conditions, which has the problem of insufficient depressive emotion arousal and cannot fully obtain effective data; thirdly, general feature enhancement methods fail to fully consider the important influence of micro-expression time changes in depressive emotion recognition, and fail to accurately enhance feature image sequences. At the same time, the commonly used feature extraction methods have the defects of only extracting regional features or too many parameters. These problems lead to reduced model accuracy or excessive recognition time complexity.

[0006] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In view of the deficiencies in the prior art, the present invention provides an intelligent recognition method and system for depression based on stress micro-expressions. The method and system have the advantages of utilizing a double-layer nested spatial state model to extract depression features of long-time series images through a spatial feature extraction module and a temporal feature extraction module, and outputting more accurate depression recognition results, thereby providing more accurate auxiliary suggestions for clinical diagnosis, thereby solving the problems that the prior art fails to fully consider the important influence of micro-expression temporal changes in depression recognition, fails to accurately enhance feature image sequences, and the commonly used feature extraction methods have the defects of only extracting regional features or having too many parameters, resulting in reduced model accuracy or excessive recognition time complexity.

[0009] (II) Technical solution

[0010] In order to achieve the advantages of utilizing the double-layer nested spatial state model, extracting the depressive emotion features of long-time series images through the spatial feature extraction module and the temporal feature extraction module, and outputting more accurate depressive emotion recognition results, and providing more accurate auxiliary suggestions for clinical diagnosis, the specific technical solutions adopted by the present invention are as follows:

[0011] According to one aspect of the present invention, a method for intelligently identifying depression emotions based on stress micro-expressions is provided, and the method for intelligently identifying depression emotions based on stress micro-expressions comprises the following steps:

[0012] S1. Based on the self-assessment results of the picture stimulation test questionnaire, calculate the depression arousal and depression correlation of different picture stimuli, screen the picture stimuli with high depression arousal and high depression correlation, and establish the stress micro-expression image sequence according to the test results of the picture stimuli with high depression arousal and high depression correlation;

[0013] S2. According to the stress micro-expression image sequence, the time dimension attention sequence of the stress micro-expression image sequence is obtained by extracting sequence difference features and processing the temporal clustering boundary sequence subscript, combining the global pooling operation and the one-dimensional convolution operation;

[0014] S3, performing temporal clustering based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of the subsequence spatial unit, and using the spatial feature extraction module to perform hidden state calculation to obtain the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence;

[0015] S4. Based on the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, a feature unit group with time information is generated, and the time feature extraction module is used to extract the time positive feature extraction vector and the time reverse feature extraction vector. The depressive emotion recognition and classification result of the stress micro-expression image sequence is obtained through vector mapping.

[0016] Furthermore, based on the self-assessment results of the picture stimulation test questionnaire, the depression arousal and depression correlation of different picture stimuli are calculated, and picture stimuli with high depression arousal and high depression correlation are screened. According to the test results of the picture stimuli with high depression arousal and high depression correlation, the stress micro-expression image sequence is established, including the following steps:

[0017] S11. According to the self-assessment results of the picture stimulation test questionnaire, the scores of different picture stimulations and the scores of the PHQ-9 self-assessment depression scale are counted, and a score set of picture stimulations is established;

[0018] S12, based on the score set of picture stimuli, calculating the depression arousal degree of different picture stimuli, and screening using the depression arousal degree threshold to obtain a score set of high depression arousal degree picture stimuli;

[0019] S13, based on the score set of high depression arousal picture stimuli, combined with the PHQ-9 self-rating depression scale score, bivariate analysis was performed to calculate the depression relevance of the picture stimuli, and the depression relevance threshold was used for screening to obtain picture stimuli with high depression arousal and high depression relevance;

[0020] S14. Collecting facial micro-expression image data of the test subject under stimulation of pictures with high depression arousal and high depression correlation, and pre-processing to generate a stress micro-expression image sequence.

[0021] Furthermore, the expression of the score set of high depression arousal picture stimulation is:

[0022]

[0023] Where S″ n is the score set of the nth high depression arousal picture stimulus, S′ n is the score set of the nth image stimulus; S′ n,k is the score of the kth picture stimulus tester in the score set of the nth picture stimulus, and m is the number of valid picture stimulus test questionnaires;

[0024] The expression of depression correlation is:

[0025]

[0026] In the formula, DR n is the depression relevance of the picture stimulus, S″ n,i is the i-th valid score in the score set of the n-th high depression arousal picture stimulus, is the average value of the score set of the nth high depression arousal picture stimulus, X i is the PHQ-9 self-rating depression scale score of the test subject for the i-th picture stimulus, For X i The average value of .

[0027] Furthermore, according to the stress micro-expression image sequence, through sequence difference feature extraction and time clustering boundary sequence subscript processing, combined with global pooling operation and one-dimensional convolution operation, the time dimension attention sequence of the stress micro-expression image sequence is obtained, which includes the following steps:

[0028] S21, based on the stress micro-expression image sequence, using sequence difference feature map calculation and gray value remapping to obtain a feature map sequence after difference processing;

[0029] S22, according to the feature map sequence after difference processing, by calculating the sum of pixel differences between the feature map and the difference feature map, obtaining a time clustering boundary sequence subscript set that is less than the effective threshold and greater than the spatial consistency threshold;

[0030] S23, setting the corresponding values ​​in the global pooling sequence corresponding to the image sequence subscripts that are less than the effective threshold to zero, and constructing a difference data distribution sequence of the overall time series;

[0031] S24. Based on the difference data distribution sequence of the overall time series, a time dimension attention sequence of the stress micro-expression image sequence is obtained through one-dimensional convolution operation and weight multiplication.

[0032] Further, the temporal clustering boundary sequence subscript set includes a sequence subscript set of spatial information repeated images and non-spatial consistency images;

[0033] The spatial information repetitive image includes a feature map in which the sum of the absolute values ​​of the difference between the original feature map in the stress micro-expression image sequence and the corresponding sequence difference feature map after pixel-by-pixel subtraction is less than the effective threshold;

[0034] The non-spatially consistent image includes a feature map in which the sum of the absolute values ​​of the difference between the original feature map and the corresponding sequence difference feature map in the stress micro-expression image sequence after pixel-by-pixel subtraction is greater than the spatial consistency threshold.

[0035] Furthermore, the expression for calculating the sequence difference feature map is:

[0036]

[0037] Where SDFM C,N is the sequence difference feature map of stress micro-expression image sequence with a difference span of N, f C is the feature map of the sequence with the subscript C, N is the total number of feature maps of the stress micro-expression image sequence;

[0038] The expression of the feature map sequence after difference processing is:

[0039]

[0040] Where f′ n is the feature map sequence after difference processing, f n is the original feature map of the stress micro-expression image sequence, gray is the gray value of the image in the stress micro-expression image sequence, max and min are the maximum gray value and the minimum gray value of the image in the stress micro-expression image sequence, respectively.

[0041] Furthermore, temporal clustering is performed based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of the subsequence spatial unit, and a spatial feature extraction module is used to perform hidden state calculation to obtain a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, including the following steps:

[0042] S31, according to the stress micro-expression image sequence and the time dimension attention sequence, using the sequence subscript set of the non-spatially consistent image to perform time clustering, obtain a stress micro-expression image subsequence after time attention is enhanced, and based on the subsequence traversal order set, obtain a subsequence space unit traversal order vector group;

[0043] S32, traversing the sequential vector group based on the subsequence spatial unit, obtaining several feature unit groups through dimension mapping and position information binding, and using the spatial feature extraction module to perform hidden state calculation to obtain the final output feature vector of each feature unit group;

[0044] S33, obtaining a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence by summing and averaging the final output feature vectors of all feature unit groups;

[0045] The stress micro-expression image subsequence after the temporal attention is strengthened includes a number of subsequence space units, and the subsequence traversal sequence set includes a forward traversal sequence set and a reverse traversal sequence set;

[0046] The forward traversal sequence set includes a horizontal traversal sequence, a vertical traversal sequence, a first diagonal oblique traversal sequence, and a second diagonal oblique traversal sequence;

[0047] The reverse traversal order set includes a reverse horizontal traversal order, a reverse vertical traversal order, a reverse first diagonal oblique cutting traversal order, and a reverse second diagonal oblique cutting traversal order; and the reverse horizontal traversal order, the reverse vertical traversal order, the reverse first diagonal oblique cutting traversal order, and the reverse second diagonal oblique cutting traversal order are mirror-symmetrical with the horizontal traversal order, the vertical traversal order, the first diagonal oblique cutting traversal order, and the second diagonal oblique cutting traversal order in spatial position.

[0048] Furthermore, the expression for hidden state calculation is:

[0049] Γ i ∈{Γ1,Γ2,…,Γ9}

[0050]

[0051] y t =C×h t

[0052] In the formula, Γ i The information matrix of the direction conversion from the feature unit corresponding to the previous input vector to the subsequence space unit corresponding to the current input vector, is the direction conversion information matrix obtained by discretizing the i-th input vector through the discretization parameter step size Δ, h t is the hidden state of the current input vector, h t-1 is the hidden state of the previous input vector, is the state transfer matrix obtained by matrix discretization, is the mapping matrix that represents the influence of the current input on the hidden state after discretization, and C is the mapping matrix from the hidden state to the output.

[0053] Furthermore, according to the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, a feature unit group with time information is generated, and a time feature extraction module is used to extract a time positive feature extraction vector and a time reverse feature extraction vector, and the depressive emotion recognition and classification result of the stress micro-expression image sequence is obtained by vector mapping, which includes the following steps:

[0054] S41, extracting the result vector of the depressive emotion spatial feature based on the stress micro-expression image subsequence, binding the position information according to the time sequence of the stress micro-expression image subsequence, and obtaining a feature unit group with time information, wherein the feature unit group with time information includes a time positive sequence feature unit group and a time reverse sequence feature unit group;

[0055] S42, using a time feature extraction module, inputting a time positive sequence feature unit group and a time reverse sequence feature unit group, and outputting a time positive sequence feature extraction vector and a time reverse sequence feature extraction vector;

[0056] S43. Sum and average all the time-forward feature extraction vectors and time-reverse feature extraction vectors, and use a fully connected layer to perform vector mapping to obtain the depressive emotion recognition and classification results of the stress micro-expression image sequence.

[0057] According to another aspect of the present invention, there is also provided a depression emotion intelligent recognition system based on stress micro-expressions, the depression emotion intelligent recognition system based on stress micro-expressions comprising:

[0058] The stress image screening and sequence construction unit is used to calculate the depression arousal and depression relevance of different picture stimuli based on the self-assessment results of the picture stimulation test questionnaire, screen the picture stimuli with high depression arousal and high depression relevance, and establish the stress micro-expression image sequence according to the test results of the picture stimuli with high depression arousal and high depression relevance;

[0059] A time dimension attention extraction unit is used to obtain a time dimension attention sequence of the stress micro-expression image sequence through sequence difference feature extraction and time clustering boundary sequence subscript processing, combined with a global pooling operation and a one-dimensional convolution operation according to the stress micro-expression image sequence;

[0060] A spatial feature extraction unit is used to perform temporal clustering based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of a subsequence spatial unit, and use a spatial feature extraction module to perform hidden state calculation to obtain a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence;

[0061] The temporal feature fusion and classification unit is used to generate a feature unit group with time information based on the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, and use the temporal feature extraction module to extract the time positive feature extraction vector and the time reverse feature extraction vector, and obtain the depressive emotion recognition and classification results of the stress micro-expression image sequence through vector mapping.

[0062] (III) Beneficial effects

[0063] Compared with the prior art, the present invention provides a method and system for intelligently identifying depression emotions based on stress micro-expressions, which has the following beneficial effects:

[0064] (1) The present invention proposes a technology that automatically collects depressive micro-expressions by applying picture stimulations of high depressive arousal and high depression correlation to the subjects, and extracts features through a multi-adaptive attention fusion spatial feature and temporal feature module; the present invention can accurately screen picture stimulations of high depressive arousal and high depression correlation, and collect micro-expression image data of high depression correlation shown by the test subjects under picture stimulation. Compared with the traditional micro-expression extraction method, it integrates the time dimension proximity and global time series attention extraction algorithms, and can better reduce the impact of images with too low information content on the model recognition results; finally, by using a double-layer nested spatial state model to extract spatial features and temporal features of the stress micro-expression image sequence after feature enhancement, the depressive emotion recognition and classification results can be obtained, which can efficiently extract features from long time series with insufficient effective information content, providing doctors with more information and decision support.

[0065] (2) In the process of deep neural network learning, optimizing the proportion of effective information in data is still a challenge, especially in deep convolutional neural networks and spatial state models. The optimization problem of the proportion of effective information in data often leads to inaccurate results. In order to further improve the accuracy of the model, it is often necessary to further increase the depth of the model, thereby increasing the number of model parameters and reducing the recognition and classification efficiency of the model. In order to solve this problem, the present invention proposes a feature enhancement technology that combines neighboring and global time series attention, which can enhance the information of the model input feature map sequence data to ensure more accurate recognition and classification results. This method can effectively improve the performance and robustness of deep learning models.

[0066] (3) When the time series attention is strengthened, only the influence between the time adjacent sequence images or the influence of all the images in the overall time series is often considered, and the joint influence of the local adjacent images and the overall time series images is not considered, which may lead to certain limitations in the time series feature extraction of stress micro-expression image data. To solve this problem, the present invention proposes a time series neighbor and global time series attention fusion algorithm, which aims to consider the joint influence of the overall time series information weight and the neighbor information relationship. Through the time series neighbor and global time series attention fusion algorithm, the proportion of effective information in the time series information is further improved to obtain a more optimized time series feature of the stress micro-expression image sequence. The application of this method is expected to reduce the overall size of the model, reduce the model parameters, and further improve the recognition accuracy of depressive emotions, providing a more efficient and reliable basis for clinical diagnosis.

[0067] (4) When extracting features from long time series images, three-dimensional convolution or multi-head self-attention that ignores long time series information is usually used for feature extraction. This will cause the model to be unable to correctly extract the spatial position change information on the time series, thereby reducing the model's recognition accuracy of depressive emotions. To solve this problem, the present invention proposes a spatial state model based on a double-layer nested module, which can extract the depressive emotion features of long time series images by using a spatial feature extraction module and a temporal feature extraction module, and output more accurate depressive emotion recognition results, providing more accurate auxiliary suggestions for clinical diagnosis.

[0068] (5) The present invention realizes accurate screening based on picture stimulation of high depression arousal and high depression correlation, and collects depression-related stress micro-expression image sequence data with greater differences through picture stimulation, solving the problem of too high proportion of invalid information in traditional data collection; at the same time, based on the difference changes in adjacent image information in the time dimension, it realizes information fusion of the global weight sequence of adjacent time series in the time dimension and time clustering of non-spatially consistent image sequences, effectively reducing the problem of low proportion of valid data and excessive number of model parameters caused by too long time series.

[0069] (6) The present invention designs and implements a Mamba-based multi-directional spatial continuous spatial feature extraction module Spatial Mamba nested in a bidirectional temporal feature extraction module Temporal Mamba for stress micro-expression image subsequences after temporal attention feature enhancement and temporal clustering, successfully avoiding the mutual interference between spatial position information binding and temporal position information binding, and solving the problem of one-sided and insufficient information extraction of the visual Mamba network in the task of depressive emotion recognition in long-term micro-expression image sequences, effectively improving the accuracy of depressive emotion recognition and depression classification prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 is a flow chart of a method for intelligently identifying depression emotions based on stress micro-expressions according to an embodiment of the present invention;

[0072] Figure 2 It is a flow chart of an algorithm for screening picture stimulations of high depression arousal and high depression correlation in a method for intelligently identifying depression emotions based on stress micro-expressions according to an embodiment of the present invention;

[0073] Figure 3 It is a flowchart of an attention sequence extraction algorithm in the time dimension and a non-spatial consistency image sequence subscript screening in a method for intelligent recognition of depression emotions based on stress micro-expressions according to an embodiment of the present invention;

[0074] Figure 4 It is a flow chart of extracting the final vector of spatial features based on Spatial Mamba in the method for intelligently identifying depression emotions based on stress micro-expressions according to an embodiment of the present invention;

[0075] Figure 5 is a flow chart of a depression emotion recognition algorithm based on Temporal Mamba in a depression emotion intelligent recognition method based on stress micro-expressions according to an embodiment of the present invention;

[0076] Figure 6 is a specific implementation diagram of a method for intelligently identifying depression emotions based on stress micro-expressions according to an embodiment of the present invention;

[0077] Figure 7 It is a principle block diagram of a depression emotion intelligent recognition system based on stress micro-expressions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0078] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0079] According to an embodiment of the present invention, a method and system for intelligently identifying depression emotions based on stress micro-expressions are provided.

[0080] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 and Figure 6 As shown, the method for intelligently identifying depression emotions based on stress micro-expressions according to an embodiment of the present invention comprises the following steps:

[0081] S1. Based on the self-assessment results of the picture stimulation test questionnaire, calculate the depression arousal and depression correlation of different picture stimuli, screen the picture stimuli with high depression arousal and high depression correlation, and establish the stress micro-expression image sequence according to the test results of the picture stimuli with high depression arousal and high depression correlation;

[0082] S2. According to the stress micro-expression image sequence, the time dimension attention sequence of the stress micro-expression image sequence is obtained by extracting sequence difference features and processing the temporal clustering boundary sequence subscript, combining the global pooling operation and the one-dimensional convolution operation;

[0083] S3, performing temporal clustering based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of the subsequence spatial unit, and using the spatial feature extraction module to perform hidden state calculation to obtain the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence;

[0084] S4. Based on the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, a feature unit group with time information is generated, and the time feature extraction module is used to extract the time positive feature extraction vector and the time reverse feature extraction vector. The depressive emotion recognition and classification result of the stress micro-expression image sequence is obtained through vector mapping.

[0085] Specifically, the present invention designs a method for screening picture stimulation based on high depression arousal and high depression correlation. Considering that patients with depression will have more negative micro-expressions than normal people when facing negative picture stimulation, they cannot show positive micro-expressions like normal people when facing positive picture stimulation. Through picture stimulation, highly differentiated stress micro-expression image data of depressed patients and non-depressed patients after awakening are collected to achieve the effect of improving data differentiation, and ultimately improve the accuracy of depressive emotion screening of the method. Through experimental simulation, it is shown that this method has been improved to a certain extent in the collection of effective stress micro-expression data.

[0086] Specifically, in the traditional micro-expression depression recognition work, invalid micro-expression data and low depression-related micro-expression data may be collected, resulting in a low upper limit of the accuracy of the final model depression recognition result. In response to the above problems, first, a large number of pictures that can stimulate the expression of depression are collected, and the scores of the picture stimulation testers on the depression stimulation pictures and the scores of the testers' depression self-rating scale PHQ-9 are counted. Then, for each picture stimulus, the stimulus type and the depression arousal degree of the stimulus are calculated, and the picture stimulus with low depression arousal is discarded. Subsequently, for all picture stimuli with high depression arousal, the depression association of the picture stimulus is calculated by combining the stimulus scores given by all testers with the tester's PHQ-9 score. Finally, the picture stimulus with high depression arousal and high depression association is applied to the depression tester, and the facial expression of the depression tester after watching the picture stimulus is recorded to obtain the tester's stress micro-expression image sequence.

[0087] Specifically, in the process of enhancing the features of traditional micro-expression image sequences, there are problems such as too long time series, low proportion of effective data in the sequence length, and uneven distribution of effective data. To address the above problems, the neighboring and global time series attention extraction methods are used. For the stress micro-expression image sequence, the sequence difference feature map is first obtained by interpolating the stress micro-expression images of the neighboring time series, and then the corresponding difference feature map is subtracted from each stress micro-expression feature map, and the gray value mapping is re-performed to obtain the stress micro-expression feature map sequence after difference processing. Next, the global maximum pooling sequence with the normalized and low effective information content feature map corresponding value set to zero is used to obtain the difference data distribution sequence weight of the overall time series, and then the attention sequence of the stress micro-expression time dimension that fully enhances the image information of the adjacent time series and the overall time series is extracted through the squeeze operation and one-dimensional convolution.

[0088] Specifically, in the traditional long time series image feature extraction task, there are problems such as the time series is too long and the image position information between sequences is transformed. In view of the above problems, the present invention proposes a time clustering algorithm, according to the global gray value sum of the difference feature map in step S2, the stress micro-expression image corresponding to the difference feature map whose sum result is greater than the spatial consistency threshold is used as the starting image of the new time clustering, and different time clustered stress micro-expression image subsequences with unified spatial information are obtained.

[0089] Specifically, in the traditional micro-expression feature extraction, since the visual Mamba model cannot consider the long-time sequence image information, and the stress micro-expression image sequence has the problem of spatial position change in the long-time sequence, this will cause the model to be unable to correctly extract the stress micro-expression information, and ultimately lead to inaccurate recognition results of the model for depression. In view of the above problems, the present invention proposes a spatial state model based on a double-layer nested module, wherein: the first layer is a spatial feature extraction module (Spatial Mamba), and the second layer is a temporal feature extraction module (Temporal Mamba). The temporal features are strengthened and the stress micro-expression sub-sequence obtained by temporal clustering is extracted using the Spatial Mamba module. The above feature extraction result vector group is then sent to the temporal feature extraction module Temporal Mamba to extract the temporal feature result vector, and finally the depression recognition and classification results that take into account the spatial features and temporal features of the stress micro-expression image sequence are output.

[0090] In one embodiment, based on the self-assessment results of the picture stimulation test questionnaire, calculating the depression arousal and depression correlation of different picture stimuli, screening picture stimuli with high depression arousal and high depression correlation, and establishing a stress micro-expression image sequence according to the test results of the picture stimuli with high depression arousal and high depression correlation includes the following steps:

[0091] S11. According to the self-assessment results of the picture stimulation test questionnaire, the scores of different picture stimulations and the scores of the PHQ-9 self-assessment depression scale are counted, and a score set of picture stimulations is established;

[0092] S12, based on the score set of picture stimuli, calculating the depression arousal degree of different picture stimuli, and screening using the depression arousal degree threshold to obtain a score set of high depression arousal degree picture stimuli;

[0093] S13, based on the score set of high depression arousal picture stimuli, combined with the PHQ-9 self-rating depression scale score, bivariate analysis was performed to calculate the depression relevance of the picture stimuli, and the depression relevance threshold was used for screening to obtain picture stimuli with high depression arousal and high depression relevance;

[0094] S14. Collecting facial micro-expression image data of the test subject under stimulation of pictures with high depression arousal and high depression correlation, and pre-processing to generate a stress micro-expression image sequence.

[0095] Specifically, in step S1, depressive picture stimuli with high depression arousal and high depression correlation are screened out with the help of medical statistical methods and the self-rating depression scale, and stress micro-expression image sequence data of the test subject under the picture stimulation are collected, including: collecting a large number of pictures with depressive emotional stimulation; removing pictures with psychological crisis stimulation, embedding the remaining pictures with depressive emotional stimulation into the scoring questionnaire, collecting the test subject's rating levels for all picture stimuli and the test subject's PHQ-9 depression self-rating scale score; for each picture stimulus, calculating the picture stimulus type and the depression arousal of the picture stimulus, and discarding the picture stimulus with low depression arousal; for all picture stimuli with high depression arousal, calculating the depression correlation of the picture stimulus by performing bivariate analysis on the questionnaire score of the picture stimulus and the PHQ-9 score of the picture stimulus test subject; collecting the facial micro-expression image data of the test subject under the picture stimulation of high depression arousal and high depression correlation, and pre-processing the collected micro-expression image sequence.

[0096] Specifically, Figure 2 As shown, first, a large number of pictures with depressive emotional stimulation are collected, and psychological experts are organized to conduct preliminary evaluations; picture stimulations with high-intensity stimulation ratings are selected, and the questionnaire is embedded with the depression self-rating scale PHQ-9 to obtain the picture stimulation test questionnaire; the self-evaluation results of the picture stimulation test questionnaire of the tester are collected, and the picture stimulation tester's score on the picture stimulation and the PHQ-9 depression self-rating scale score in the questionnaire self-evaluation results are counted. Among them, the stimulation intensity scoring component of the picture stimulation is set to a strip drag-type scoring component from -4 to 4; it should be noted that the PHQ-9 self-rating scale is a mental health assessment tool containing 9 assessment items, and this application uses the assessment data of this scale as a reference standard for emotional state.

[0097] Specifically, after removing invalid data from the questionnaire results, the score set of the nth picture stimulus is recorded as S n ={S n,1 ,S n,2 ,S n,3 ,…,S n,m}, for each picture stimulus, take the average of all the scores of the test takers. If the average score is greater than or equal to 0, add 5 to all the scores. If the average score is less than 0, take the inverse of all the scores and add 5. The score set of the processed nth picture stimulus is denoted as S′ n ;

[0098]

[0099] Specifically, then, for each processed picture stimulus score set S′ n Take the average value of all elements to calculate its depression wakefulness Recall n, only retain the depression picture stimuli whose depression arousal degree is greater than or equal to the preset depression arousal degree threshold (2 in this embodiment), and obtain the score set S″ of picture stimuli with high depression arousal degree n , the expression of the score set of high depression arousal picture stimulus is:

[0100]

[0101] Where S″ n is the score set of the nth high depression arousal picture stimulus, S′ n is the score set of the nth image stimulus; S′ n,k is the score of the kth picture stimulus tester in the score set of the nth picture stimulus, and m is the number of valid picture stimulus test questionnaires.

[0102] Specifically, for the picture stimulation with high depression arousal, the PHQ-9 scores of all stimulus picture testers were combined, and then a bivariate analysis (Pearson correlation coefficient analysis was used in this embodiment) was performed to obtain the depression association DR of each high depression arousal picture stimulation. n The PHQ-9 score of the person who received the picture stimulus test is X. i , where i = 1, 2, 3…m, m is the number of valid questionnaire results, For X i Average value. S″ n,i represents the i-th valid score of the n-th high depression arousal picture stimulus score set, is the average value of the nth set of picture stimulus scores with high depression arousal. Then the depression relevance DR of the picture stimulus is n It can be expressed as X and S″ n The quotient of the covariance and standard deviation of , the expression of depression correlation is:

[0103]

[0104] In the formula, DR n is the depression relevance of the picture stimulus, S″ n,i is the i-th valid score in the score set of the n-th high depression arousal picture stimulus, is the average value of the score set of the nth high depression arousal picture stimulus, X i is the PHQ-9 self-rating depression scale score of the test subject for the i-th picture stimulus, For X i The average value of .

[0105] Specifically, the depression correlation DR nPicture stimuli that are higher than a preset depression arousal threshold (0.7 in this embodiment) are retained, thereby obtaining picture stimuli with high depression arousal and high depression association.

[0106] Specifically, after applying picture stimulation with high depression arousal and high depression association to the depression emotion recognition test subjects, the original stress micro-expression image sequence data displayed by the test subjects are collected.

[0107] Specifically, the collected stress micro-expression image sequence data with certain depressive emotional expressions are preprocessed. The size of the collected images is unified to 256×256, and the color images are converted to grayscale images. We record the pixel value of the red channel as R, the pixel value of the green channel as G, and the pixel value of the blue channel as B. The grayscale value Gray calculation formula of each image frame of the video is as follows:

[0108] Gray=0.229×R+0.587×G+0.144×B

[0109] Specifically, through the above process, the original stress micro-expression image sequence data is converted into a 256×256×T stress micro-expression image sequence with high depressive emotion expression.

[0110] In one embodiment, according to the stress micro-expression image sequence, by extracting sequence difference features and processing time clustering boundary sequence subscripts, combined with global pooling operation and one-dimensional convolution operation, obtaining the time dimension attention sequence of the stress micro-expression image sequence includes the following steps:

[0111] S21, based on the stress micro-expression image sequence, using sequence difference feature map calculation and gray value remapping to obtain a feature map sequence after difference processing;

[0112] S22, according to the feature map sequence after difference processing, by calculating the sum of pixel differences between the feature map and the difference feature map, obtaining a time clustering boundary sequence subscript set that is less than the effective threshold and greater than the spatial consistency threshold;

[0113] S23, setting the corresponding values ​​in the global pooling sequence corresponding to the image sequence subscripts that are less than the effective threshold to zero, and constructing a difference data distribution sequence of the overall time series;

[0114] S24. Based on the difference data distribution sequence of the overall time series, a time dimension attention sequence of the stress micro-expression image sequence is obtained through one-dimensional convolution operation and weight multiplication.

[0115] In one embodiment, the temporal cluster boundary sequence subscript set includes a sequence subscript set of spatial information repetitive images and non-spatial consistency images;

[0116] The spatial information repetitive image includes a feature map in which the sum of the absolute values ​​of the difference between the original feature map in the stress micro-expression image sequence and the corresponding sequence difference feature map after pixel-by-pixel subtraction is less than the effective threshold;

[0117] The non-spatially consistent image includes a feature map in which the sum of the absolute values ​​of the difference between the original feature map and the corresponding sequence difference feature map in the stress micro-expression image sequence after pixel-by-pixel subtraction is greater than the spatial consistency threshold.

[0118] Specifically, in step S2, the stress micro-expression image sequence is subjected to neighboring and global time series attention extraction, and the sequence subscripts of the images that do not meet the spatial consistency are recorded to perform time clustering of the image sequence, including: calculating the difference feature map of the micro-expression grayscale image sequence by sequence subscript, and calculating the feature map sequence after difference processing according to the difference feature map corresponding to the feature map sequence; calculating the sum of the pixel difference between each feature map and its corresponding difference feature map, and recording the sequence subscripts whose summation results are less than the effective threshold or greater than the spatial consistency threshold; extracting the global maximum pooling sequence and the global average pooling sequence of the feature map sequence after difference processing. Average pooling sequence, set the corresponding value of the sequence subscript in the global maximum pooling sequence to zero when the sum of the pixel points of the difference feature map is less than the effective threshold; multiply the global average pooling sequence with the corresponding value of the global maximum pooling sequence normalized after threshold judgment and set to zero to obtain the difference data distribution sequence weight of the overall time series; use a one-dimensional convolution kernel to extract the difference data distribution sequence weight of the overall time series to obtain the associated attention weight sequence of adjacent sequence images; multiply the difference data distribution weight sequence with the image attention weight sequence of the adjacent sequence by the corresponding value to obtain the time series attention sequence of stress micro-expressions.

[0119] Specifically, Figure 3 As shown, for the stress micro-expression time series feature enhancement, first, from the stress micro-expression image sequence with high depressive emotion expression information content obtained in step S1, the five adjacent feature map sequences are subtracted from each other and then summed and averaged to obtain the sequence difference feature map SDFM with a difference span of N. C,N ; The expression for calculating the sequence difference feature map is:

[0120]

[0121] Where SDFM C,N is the sequence difference feature map of stress micro-expression image sequence with a difference span of N, f C is the feature map of the sequence with subscript C, and N is the total number of feature maps in the stress micro-expression image sequence.

[0122] Specifically, in the above formula, coefficient C is the sequence subscript of the first feature graph in the sequence of sequence difference feature graphs involved in the calculation, coefficient N is the total number of feature graphs involved in the calculation of sequence difference feature graphs, and symbol f C Refers to the feature map with the sequence subscript C. Among them, the coefficient 1≤C≤S-N+1, the coefficient 12≤N≤18. When the actual model is used, the adjacent SDFM is calculated C,N When and Need for guarantee

[0123] Specifically, subsequently, the sequence subscript n is between C and Subtract SDFM from the feature map between C,N Then remap the grayscale value to between 0 and 255. Note that when C+N-1=S, the sequence subscript n is between Subtract SDFM from the feature map between C+N-1 C,N And do gray value mapping, separate the gray value difference of stress micro-expression neighboring images and remap the gray value process can be expressed as follows:

[0124]

[0125] Where f′ n is the feature map sequence after difference processing, f n is the original feature map of the stress micro-expression image sequence, gray is the gray value of the image in the stress micro-expression image sequence, max and min are the maximum gray value and the minimum gray value of the image in the stress micro-expression image sequence, respectively.

[0126] Specifically, in the above formula, the original feature map f n And the feature map f′ after difference processing n The subscripts are all n, gray represents the gray value of the image, max and min represent the maximum gray value and the minimum gray value of the image respectively, and when max-min = 0, f′ n Set all grayscale values ​​to zero.

[0127] Specifically, in addition, the present invention records the characteristic diagram f n The corresponding sequence difference feature map SDFM C,N The feature map subscript n whose sum of the absolute values ​​of the difference after pixel-by-pixel subtraction is less than the effective threshold (2048 in this embodiment) is recorded as the set Dropf (sequence subscript set of spatial information repeated images), and the feature map subscript n whose sum of the absolute values ​​after subtraction is greater than the spatial consistency threshold is recorded as the set Dividef (sequence subscript set of non-spatially consistent images) as the temporal clustering boundary sequence subscript set.

[0128] Specifically, then, the obtained difference feature map sequence is subjected to global maximum pooling and global average pooling to obtain the global maximum pooling sequence and global average pooling sequence of the feature sequence respectively. The corresponding values ​​of the vector value subscript belonging to the set Dropf in the S-dimensional vector used to represent the global maximum pooling feature sequence are set to zero and normalized, and multiplied by the corresponding number of the global average pooling sequence to obtain the difference data distribution sequence SV of the overall time series S . Let the nth dimension value of the global maximum pooling vector be gMSeq n If n belongs to the set Dropf, gMSeq n Set to zero to obtain the effective global average pooling sequence gMSEq′ n , the nth dimension value of the global average pooling vector is gASeq n , then the difference data distribution sequence of the overall time series is diffSeq S It can be expressed as:

[0129]

[0130] Specifically, the superscript T in the numerator bracket in the above formula represents matrix transposition, and the remaining subscripts T in the above formula and T appearing in the context represent the number of feature maps of the stress micro-expression image sequence in the time dimension.

[0131] Specifically, after this, the difference data distribution sequence diffSeq of the overall time series is obtained T Mapping to a dimension less than But greater than , and then mapped to a high-dimensional vector of dimension T, so as to obtain the final difference data distribution sequence diffSeq′ of the overall time series T . Then, we vector diffSeq′ T Unitize to get the difference data distribution sequence weight diffSV of the overall time series T .

[0132] Specifically, then, diffSeq is calibrated by a one-dimensional convolution kernel of size six. T Convolution is performed to become a dimension of T, and the adjacent sequence data of the adjacent feature graphs of the adjacent six time series are extracted, and the associated attention weight sequence nearSV is obtained. T .

[0133] Specifically, finally, the difference data distribution sequence weight diffSV of the overall time series T Associate the attention weight sequence nearSV with the data of adjacent frames T Multiply the corresponding values ​​to obtain the temporal dimension attention sequence temporalAtt of the stress micro-expression image sequencen .

[0134] In one embodiment, temporal clustering is performed based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of the subsequence spatial unit, and a spatial feature extraction module is used to perform hidden state calculation to obtain a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, including the following steps:

[0135] S31, according to the stress micro-expression image sequence and the time dimension attention sequence, using the sequence subscript set of the non-spatially consistent image to perform time clustering, obtain a stress micro-expression image subsequence after time attention is enhanced, and based on the subsequence traversal order set, obtain a subsequence space unit traversal order vector group;

[0136] S32, traversing the sequential vector group based on the subsequence spatial unit, obtaining several feature unit groups through dimension mapping and position information binding, and using the spatial feature extraction module to perform hidden state calculation to obtain the final output feature vector of each feature unit group;

[0137] S33, by summing and averaging the final output feature vectors of all feature unit groups, a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence is obtained.

[0138] In one embodiment, the stress micro-expression image subsequence after temporal attention enhancement includes a plurality of subsequence spatial units;

[0139] The subsequence traversal order set includes a forward traversal order set and a reverse traversal order set;

[0140] The forward traversal sequence set includes a horizontal traversal sequence, a vertical traversal sequence, a first diagonal oblique traversal sequence, and a second diagonal oblique traversal sequence;

[0141] The reverse traversal order set includes a reverse horizontal traversal order, a reverse vertical traversal order, a reverse first diagonal oblique cutting traversal order, and a reverse second diagonal oblique cutting traversal order; and the reverse horizontal traversal order, the reverse vertical traversal order, the reverse first diagonal oblique cutting traversal order, and the reverse second diagonal oblique cutting traversal order are mirror-symmetrical with the horizontal traversal order, the vertical traversal order, the first diagonal oblique cutting traversal order, and the second diagonal oblique cutting traversal order in spatial position.

[0142] It should be noted that the spatial feature extraction module (Spatial Mamba) is a deep learning architecture based on the state space model, which can effectively capture the spatial feature information in the image sequence through multi-directional traversal and state transfer calculation. The present invention uses this module to extract spatial features from the stress micro-expression image subsequence to obtain the spatial feature representation of depressive emotions.

[0143] Specifically, in step S3, the stress micro-expression image sequence is temporally clustered, and the spatial features of each stress micro-expression image subsequence are extracted using the Spatial Mamba module, including: the stress micro-expression neighbors and the global time series obtained in step S2 are focused on the stress micro-expression image sequence; the stress micro-expression image sequence after the time feature enhancement is temporally clustered according to the sequence subscript of the non-spatial consistency image in step 2, and a plurality of stress micro-expression image subsequences after the time feature enhancement are obtained; the stress micro-expression image subsequences after the time feature enhancement are cut into patches with a width and height of 16, and all patches are flattened into high-dimensional vectors; the high-dimensional vector obtained by flattening the patch is mapped to a vector with a dimension of D, and according to eight patch traversal orders, the position information is respectively bound to it, and eight groups of stress micro-expression image subsequence tokens with patch image information and original image position information are obtained; the eight groups of stress micro-expression image subsequence tokens are respectively placed in the Spatial Mamba extracts spatial feature vectors; the eight spatial feature extraction vectors of the same stress micro-expression image subsequence are averaged to obtain the spatial feature extraction result vector of the subsequence.

[0144] Specifically, Figure 4 As shown, first, the temporal dimension attention sequence temporalAtt of the stress micro-expression image sequence obtained in step S2 is n Enhanced with stress micro-expression image sequences.

[0145] Specifically, then, the feature-enhanced stress micro-expression image sequence is temporally clustered according to the sequence subscript set Dividef of the non-spatially consistent image obtained in step S2 to obtain a number of num stress micro-expression image subsequences after temporal attention enhancement.

[0146] Specifically, then, the image width is W, the height is H, and the time dimension is T sub The stress micro-expression image subsequences after the time attention enhancement are divided into Subsequence space unit patch, flatten all patches into dimensions A vector of time dimensions T sub The subscript sub∈{1,2,3,…}.

[0147] Specifically, subsequently, for a stress micro-expression image subsequence after enhanced temporal attention, the vectors obtained by cutting and flattening are obtained according to four continuous traversals from the upper left patch to the lower right patch, namely horizontal, vertical, and diagonal traversals, to obtain four patch traversal order vector groups.

[0148] Specifically, the four traversal methods from the upper left patch to the lower right patch are as follows: The horizontal traversal order is to traverse all patches in this row in the horizontal direction, and then traverse in the opposite direction of the previous row, and repeat this process until all patches are traversed. The vertical traversal order is to traverse all patches in this column in the vertical direction, and then traverse in the opposite direction of the previous column, and repeat this process until all patches are traversed. The first diagonal oblique traversal order is to vertically build a new patch from the upper left corner patch and then traverse obliquely to the upper right to the boundary, then traverse a new patch horizontally and then traverse in the opposite direction of the previous oblique traversal to the boundary, and repeat this process alternately until all patches are traversed. The second diagonal oblique traversal order is to traverse a new patch horizontally from the upper left corner patch and then traverse obliquely downward to the boundary, then traverse a new patch vertically and then traverse to the boundary in the opposite direction of the previous oblique traversal, and repeat this process alternately until all patches are traversed.

[0149] Specifically, four other patch traversal order vector groups are obtained by the same four continuous traversals from the lower right patch to the upper left patch, horizontally, vertically, and diagonally. Each vector group consists of 512 dimensions. Vector composition.

[0150] Specifically, secondly, the above eight vector groups are respectively used with sizes The mapping matrix transforms the vector dimension from Mapped to D and bound to the position information, eight groups of feature units (tokens) with a number of 512 and a dimension of D are obtained. The eight token groups in the traversal order are input into independent spatial state extraction modules (SpatialMamba) to extract the features of the stress micro-expression image subsequence after attention enhancement.

[0151] Specifically, Figure 4 As shown in the Spatial Mamba module, the input vector is first normalized by the normalization layer, and then the input vector group x and the residual vector group x are obtained by two parallel linear mappings. res , then the input vector x is passed through a one-dimensional convolutional layer (Conv1d) to extract the dimensional information in the vector, and then the vector with enhanced dimensional information is input into the spatial state model (Spatial SSM). After completing the processing of the spatial state model, the residual vector group will be processed through the activation function (SiLU).

[0152] Specifically, during the Spatial SSM model training process, the hidden state h corresponding to the t-th input vector tand output y t The calculation formula is as follows:

[0153] Γ i ∈{Γ1,Γ2,…,Γ9}

[0154]

[0155] y t =Ch t

[0156] In the formula, Γ i The information matrix of the direction conversion from the feature unit corresponding to the previous input vector to the subsequence space unit corresponding to the current input vector, is the direction conversion information matrix obtained by discretizing the i-th input vector through the discretization parameter step size Δ, h t is the hidden state of the current input vector, h t-1 is the hidden state of the previous input vector, is the state transfer matrix obtained by matrix discretization, is the mapping matrix that represents the influence of the current input on the hidden state after discretization, and C is the mapping matrix from the hidden state to the output.

[0157] Specifically, in the above formula, h t-1 is the hidden state of the previous input vector. HIPPO matrix discretization is a commonly used discretization method for numerically solving differential equations. is the state transfer matrix obtained by discretizing the HIPPO matrix, It is a mapping matrix that can represent the influence of the current input on the hidden state after discretization. When i is 1 to 8, Γ i Indicates the direction conversion information matrix of the patch corresponding to the previous input vector by traversing in one of the eight different directions to the patch corresponding to this input vector. is the direction conversion information matrix obtained by discretizing the discretization parameter step size Δ, Γ9 represents the direction conversion information matrix of the initial input vector, Δ is the discretization parameter step size, and C is the mapping matrix from the hidden state to the output. and They are obtained by discretizing the parameter step size Δ, and the unit matrix is ​​denoted as I. Then their calculation formulas are as follows:

[0158]

[0159] Specifically, during the test of the Spatial SSM model, the calculation formula for the output y vector group corresponding to the input vector group with M input vectors is as follows:

[0160]

[0161] Specifically, after the input vector group x is obtained through Spatial SSM to obtain the output vector group y, we combine the output vector group y with the residual vector group x after activation by the activation function (SiLU) layer res The corresponding vectors are dot-multiplied to obtain the final output feature vector. The feature unit (token) groups obtained by the eight traversal methods are respectively extracted through the above process to extract the final output feature vector, and then added and averaged to obtain the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence.

[0162] In one embodiment, according to the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, a feature unit group with time information is generated, and a time feature extraction module is used to extract a time positive feature extraction vector and a time reverse feature extraction vector, and the depressive emotion recognition and classification result of the stress micro-expression image sequence is obtained by vector mapping, which includes the following steps:

[0163] S41, extracting the result vector of the depressive emotion spatial feature based on the stress micro-expression image subsequence, binding the position information according to the time sequence of the stress micro-expression image subsequence, and obtaining a feature unit group with time information, wherein the feature unit group with time information includes a time positive sequence feature unit group and a time reverse sequence feature unit group;

[0164] S42, using a time feature extraction module, inputting a time positive sequence feature unit group and a time reverse sequence feature unit group, and outputting a time positive sequence feature extraction vector and a time reverse sequence feature extraction vector;

[0165] S43. Sum and average all the time-forward feature extraction vectors and time-reverse feature extraction vectors, and use a fully connected layer to perform vector mapping to obtain the depressive emotion recognition and classification results of the stress micro-expression image sequence.

[0166] It should be noted that the temporal feature extraction module (Temporal Mamba) is a model variant optimized for the characteristics of time series data based on Spatial Mamba. It is specially used to process data with time series dependency by simplifying state transfer calculation and superimposing multi-layer structures. The present invention uses this module to extract features of spatial feature sequences in the time dimension, thereby effectively capturing the temporal change characteristics of stress micro-expressions.

[0167] Specifically, in step S4, the subsequence feature vector extracted by Spatial Mamba is used to extract the time feature vector through the Temporal Mamba module (time feature extraction module) to obtain the depressive emotion recognition and classification results of the stress micro-expression image sequence, including: the spatial feature extraction result vector of the stress micro-expression image subsequence extracted in step S3 is sorted in positive or reverse order according to the order of its subsequence in the time series, and the time position information is bound to obtain a number of tokens equal to the number of subsequences; the tokens traversed in positive order and reverse order with the time series are respectively input into two parallel Temporal Mamba modules to extract the positive and reverse time feature extraction feature vectors; the positive and reverse time feature extraction vectors extracted by the Temporal Mamba module are added and averaged to obtain the feature result vector of the stress micro-expression image sequence through spatial feature extraction and time feature extraction; the feature result vector extracted by the Temporal Mamba module is mapped to the classification two-dimensional vector through a fully connected layer to obtain the depressive emotion recognition and classification results of the stress micro-expression image sequence.

[0168] Specifically, Figure 5 As shown, first, all stress micro-expression image subsequences are extracted through step 3. The num depressive emotion spatial feature extraction result vectors are bound to the position information in the positive or reverse order of the time sequence of the subsequences in the original stress micro-expression image sequence, and tokens with a number of num, a dimension of D and time information are obtained, and these tokens are input into the Temporal Mamba module.

[0169] Specifically, in the Temporal Mamba module, the input token group is first normalized by the Norm layer, and then is also subjected to two parallel linear mappings to obtain the input vector group x′ and the residual vector group x′. res , and then the input vector group x′ is input into the spatial state model composed of four layers of Temporal Mamba modules for feature extraction.

[0170] Specifically, the testing process in the Temporal Mamba module is no different from the testing process in the Spatial SSM module of the Spatial Mamba module, but the mapping vector used to represent the position conversion relationship between the previous input vector and this input vector is deleted during the training process. Therefore, the training process formula of the Temporal Mamba module is as follows:

[0171]

[0172] yt =Ch t

[0173]

[0174] Specifically, in addition, the meaning of each parameter in the formula is the same as the meaning of the parameters in the Spatial SSM module training process of the Spatial Mamba module.

[0175] Specifically, the output vector group y′ after four-layer Temporal SSM feature extraction is combined with the residual vector group x′ after activation by the SiLU layer res Perform point multiplication of the corresponding vectors to obtain the stress micro-expression feature vector that completes the time feature extraction. Then, the stress micro-expression feature vectors extracted from the token groups input in the positive and reverse order of time are averaged and summed, and finally the vector is vector-mapped through the fully connected layer to obtain the final depression emotion recognition and classification results.

[0176] like Figure 7 As shown, according to another embodiment of the present invention, a depression emotion intelligent recognition system based on stress micro-expressions is also provided, and the depression emotion intelligent recognition system based on stress micro-expressions includes:

[0177] Stress image screening and sequence construction unit 1, used to calculate the depression arousal and depression relevance of different picture stimuli based on the self-assessment results of the picture stimulation test questionnaire, screen picture stimuli with high depression arousal and high depression relevance, and establish a stress micro-expression image sequence based on the test results of picture stimuli with high depression arousal and high depression relevance;

[0178] The time dimension attention extraction unit 2 is used to obtain the time dimension attention sequence of the stress micro-expression image sequence through sequence difference feature extraction and time clustering boundary sequence subscript processing, combined with global pooling operation and one-dimensional convolution operation according to the stress micro-expression image sequence;

[0179] The spatial feature extraction unit 3 is used to perform time clustering based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of the subsequence spatial unit, and use the spatial feature extraction module to perform hidden state calculation to obtain the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence;

[0180] The temporal feature fusion and classification unit 4 is used to generate a feature unit group with time information based on the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, and use the time feature extraction module to extract the time positive feature extraction vector and the time reverse feature extraction vector, and obtain the depressive emotion recognition and classification result of the stress micro-expression image sequence through vector mapping.

[0181] In summary, with the help of the above technical solution of the present invention, accurate screening based on high depression arousal and high depression correlation picture stimulation is achieved, and depression-related stress micro-expression image sequence data with greater differences are collected through picture stimulation, solving the problem of too high invalid information proportion in traditional data collection; at the same time, based on the difference changes in adjacent image information in the time dimension, the information fusion of the global weight sequence of adjacent time series in the time dimension and the time clustering of non-spatial consistency image sequences are realized, effectively reducing the problems of low effective data proportion and excessive model parameters caused by too long time series. The present invention designs and implements the spatial feature extraction module Spatial Mamba based on Mamba's multi-directional spatial continuity and the nested bidirectional temporal feature extraction module Temporal Mamba for the stress micro-expression image subsequence after temporal attention feature enhancement and temporal clustering, successfully avoiding the mutual interference between spatial position information binding and temporal position information binding, solving the problem of one-sided and insufficient information extraction of the visual Mamba network in the depression emotion recognition task of long-term micro-expression image sequence, and effectively improving the accuracy of depression emotion recognition and depression classification prediction results.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent method for identifying depression based on stress micro-expressions, characterized in that: The method for intelligently identifying depression emotions based on stress micro-expressions comprises the following steps: S1. Based on the self-assessment results of the picture stimulation test questionnaire, calculate the depression arousal and depression correlation of different picture stimuli, screen the picture stimuli with high depression arousal and high depression correlation, and establish the stress micro-expression image sequence according to the test results of the picture stimuli with high depression arousal and high depression correlation; S2. According to the stress micro-expression image sequence, the time dimension attention sequence of the stress micro-expression image sequence is obtained by extracting sequence difference features and processing the temporal clustering boundary sequence subscript, combining the global pooling operation and the one-dimensional convolution operation; S3, performing temporal clustering based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of the subsequence spatial unit, and using the spatial feature extraction module to perform hidden state calculation to obtain the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence; S4. Based on the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, a feature unit group with time information is generated, and the time feature extraction module is used to extract the time positive feature extraction vector and the time reverse feature extraction vector. The depressive emotion recognition and classification result of the stress micro-expression image sequence is obtained through vector mapping.

2. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 1, characterized in that: The method of calculating the depression arousal and depression correlation of different picture stimuli based on the self-assessment results of the picture stimulation test questionnaire, screening picture stimuli with high depression arousal and high depression correlation, and establishing a stress micro-expression image sequence according to the test results of the picture stimuli with high depression arousal and high depression correlation includes the following steps: S11. According to the self-assessment results of the picture stimulation test questionnaire, the scores of different picture stimulations and the scores of the PHQ-9 self-assessment depression scale are counted, and a score set of picture stimulations is established; S12, based on the score set of picture stimuli, calculating the depression arousal degree of different picture stimuli, and screening using the depression arousal degree threshold to obtain a score set of high depression arousal degree picture stimuli; S13, based on the score set of high depression arousal picture stimuli, combined with the PHQ-9 self-rating depression scale score, bivariate analysis was performed to calculate the depression relevance of the picture stimuli, and the depression relevance threshold was used for screening to obtain picture stimuli with high depression arousal and high depression relevance; S14. Collecting facial micro-expression image data of the test subject under stimulation of pictures with high depression arousal and high depression correlation, and pre-processing to generate a stress micro-expression image sequence.

3. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 2, characterized in that: The expression of the score set of the high depression arousal picture stimulus is: In the formula, S″ n is the score set of the nth high depression arousal picture stimulus, S′ n is the score set of the nth image stimulus; S′ n,k is the score of the kth picture stimulus tester in the score set of the nth picture stimulus, and m is the number of valid picture stimulus test questionnaires; The expression of the depression correlation is: In the formula, DR n is the depression relevance of the picture stimulus, S″ n,i is the i-th valid score in the score set of the n-th high depression arousal picture stimulus, is the average value of the score set of the nth high depression arousal picture stimulus, X i is the PHQ-9 self-rating depression scale score of the test subject for the i-th picture stimulus, For X i The average value of .

4. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 1, characterized in that: The method of obtaining the time dimension attention sequence of the stress micro-expression image sequence according to the stress micro-expression image sequence by extracting sequence difference features and performing time clustering boundary sequence subscript processing, combining global pooling operation and one-dimensional convolution operation, comprises the following steps: S21, based on the stress micro-expression image sequence, using sequence difference feature map calculation and gray value remapping to obtain a feature map sequence after difference processing; S22, according to the feature map sequence after difference processing, by calculating the sum of pixel differences between the feature map and the difference feature map, obtaining a time clustering boundary sequence subscript set that is less than the effective threshold and greater than the spatial consistency threshold; S23, setting the corresponding values ​​in the global pooling sequence corresponding to the image sequence subscripts that are smaller than the effective threshold to zero to construct a difference data distribution sequence of the overall time series; S24. Based on the difference data distribution sequence of the overall time series, a time dimension attention sequence of the stress micro-expression image sequence is obtained through one-dimensional convolution operation and weight multiplication.

5. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 4, characterized in that: The temporal clustering boundary sequence subscript set includes a sequence subscript set of spatial information repetitive images and non-spatial consistency images; The spatial information repetitive image includes a feature map in which the sum of the absolute values ​​of the differences between the original feature map and the corresponding sequence difference feature map in the stress micro-expression image sequence after pixel-by-pixel subtraction is less than the effective threshold; The non-spatial consistency image includes a feature map in which the sum of the absolute values ​​of the differences between the original feature map and the corresponding sequence difference feature map in the stress micro-expression image sequence after pixel-by-pixel subtraction is greater than the spatial consistency threshold.

6. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 5, characterized in that: The expression for calculating the sequence difference characteristic graph is: Where SDFM C,N is the sequence difference feature map of stress micro-expression image sequence with a difference span of N, f C is the feature map of the sequence with the subscript C, N is the total number of feature maps of the stress micro-expression image sequence; The expression of the feature map sequence after the difference processing is: Where f′ n is the feature map sequence after difference processing, f n is the original feature map of the stress micro-expression image sequence, gray is the gray value of the image in the stress micro-expression image sequence, max and min are the maximum gray value and the minimum gray value of the image in the stress micro-expression image sequence, respectively.

7. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 5, characterized in that: The method of performing temporal clustering based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of a subsequence spatial unit, and using a spatial feature extraction module to perform hidden state calculation to obtain a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence includes the following steps: S31, according to the stress micro-expression image sequence and the time dimension attention sequence, using the sequence subscript set of the non-spatially consistent image to perform time clustering, obtain a stress micro-expression image subsequence after time attention is enhanced, and based on the subsequence traversal order set, obtain a subsequence space unit traversal order vector group; S32, traversing the sequential vector group based on the subsequence spatial unit, obtaining several feature unit groups through dimension mapping and position information binding, and using the spatial feature extraction module to perform hidden state calculation to obtain the final output feature vector of each feature unit group; S33, by summing and averaging the final output feature vectors of all feature unit groups, a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence is obtained. The stress micro-expression image subsequence after the temporal attention enhancement includes a plurality of subsequence space units, and the subsequence traversal sequence set includes a forward traversal sequence set and a reverse traversal sequence set; The forward traversal sequence set includes a horizontal traversal sequence, a vertical traversal sequence, a first diagonal oblique traversal sequence, and a second diagonal oblique traversal sequence; The reverse traversal order set includes a reverse horizontal traversal order, a reverse vertical traversal order, a reverse first diagonal oblique cutting traversal order and a reverse second diagonal oblique cutting traversal order; and the reverse horizontal traversal order, the reverse vertical traversal order, the reverse first diagonal oblique cutting traversal order and the reverse second diagonal oblique cutting traversal order are mirror-symmetrical with the horizontal traversal order, the vertical traversal order, the first diagonal oblique cutting traversal order and the second diagonal oblique cutting traversal order in spatial position.

8. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 7, characterized in that: The expression for hidden state calculation is: C i ∈{Γ1,Γ2,…,Γ9} and t =C×h t In the formula, Γ i The information matrix of the direction conversion from the feature unit corresponding to the previous input vector to the subsequence space unit corresponding to the current input vector, is the direction conversion information matrix obtained by discretizing the i-th input vector through the discretization parameter step size Δ, h t is the hidden state of the current input vector, h t-1 is the hidden state of the previous input vector, is the state transfer matrix obtained by matrix discretization, is the mapping matrix that represents the influence of the current input on the hidden state after discretization, and C is the mapping matrix from the hidden state to the output.

9. The method for intelligently identifying depression emotions based on stress micro-expressions according to claim 1, characterized in that: The method of extracting the result vector of the depressive emotion spatial feature of the stress micro-expression image subsequence to generate a feature unit group with time information, extracting the time positive feature extraction vector and the time reverse feature extraction vector by using the time feature extraction module, and obtaining the depressive emotion recognition and classification result of the stress micro-expression image sequence by vector mapping comprises the following steps: S41, based on the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, binding the position information according to the time sequence of the stress micro-expression image subsequence to obtain a feature unit group with time information, wherein the feature unit group with time information includes a time positive sequence feature unit group and a time reverse sequence feature unit group; S42, using a time feature extraction module, inputting a time positive sequence feature unit group and a time reverse sequence feature unit group, and outputting a time positive sequence feature extraction vector and a time reverse sequence feature extraction vector; S43. Sum and average all the time-forward feature extraction vectors and time-reverse feature extraction vectors, and use a fully connected layer to perform vector mapping to obtain the depressive emotion recognition and classification results of the stress micro-expression image sequence.

10. A system for intelligently identifying depression emotions based on stress micro-expressions, used to implement the method for intelligently identifying depression emotions based on stress micro-expressions as claimed in any one of claims 1 to 9, characterized in that: The depression emotion intelligent recognition system based on stress micro-expressions includes: The stress image screening and sequence construction unit is used to calculate the depression arousal and depression relevance of different picture stimuli based on the self-assessment results of the picture stimulation test questionnaire, screen the picture stimuli with high depression arousal and high depression relevance, and establish the stress micro-expression image sequence according to the test results of the picture stimuli with high depression arousal and high depression relevance; A time dimension attention extraction unit is used to obtain a time dimension attention sequence of the stress micro-expression image sequence through sequence difference feature extraction and time clustering boundary sequence subscript processing, combined with a global pooling operation and a one-dimensional convolution operation according to the stress micro-expression image sequence; A spatial feature extraction unit is used to perform temporal clustering based on the stress micro-expression image sequence and the time dimension attention sequence to obtain a feature unit group of a subsequence spatial unit, and use a spatial feature extraction module to perform hidden state calculation to obtain a depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence; The temporal feature fusion and classification unit is used to generate a feature unit group with time information based on the depressive emotion spatial feature extraction result vector of the stress micro-expression image subsequence, and use the temporal feature extraction module to extract the time positive feature extraction vector and the time reverse feature extraction vector, and obtain the depressive emotion recognition and classification results of the stress micro-expression image sequence through vector mapping.

Images