A method and system for identifying depression disorders based on facial video sequences

Abstract

The present application proposes a method for identifying depression disorders based on facial video sequences, including: segmenting the video sequence and performing facial image detection to obtain the key points and action units of the facial images contained in each video; constructing graph structures for the key points and action units of the facial images contained in each video to obtain the key point graph and action unit graph of each video; convolving the constructed graph structures of each video respectively, and enhancing the graph structure features obtained by convolution based on the attention mechanism, and fusing the enhanced key point feature graph and facial unit feature graph of each video to obtain the fused feature graph of each video; obtaining the segment-level representation of each video based on the fused feature graph of each video, establishing the video-level representation based on all the segment-level representations, and performing depression disorder recognition on the video-level representation to obtain the recognition result. The present invention using the above scheme can recognize depression based on facial videos.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for identifying depression disorders based on facial video sequences. Background Art

[0002] Depression is one of the most common mental illnesses in modern society. Its clinical symptoms are long-term and obvious emotional loss. Moreover, people often find it difficult to talk about their mental health problems; mental disorders are more difficult to detect than physical diseases. In addition, factors such as social opinion, economic costs and lack of treatment have exacerbated the burden of mental health.

[0003] In clinical practice, doctors mainly judge and identify the severity of depression through face-to-face clinical interviews, which is effective but highly dependent on clinical experience. On the other hand, existing psychiatrists cannot meet the actual needs, and many people cannot get an accurate diagnosis of depression and miss the best treatment in the early stages.

[0004] Facial expressions play a very important role in people's communication. They can convey subtle information that is difficult to express in words. By observing the slight changes in facial expressions, people's moods can be judged. Studies have shown that the state of patients with depression can be expressed through some visual features, such as facial expressions and body postures, and facial expressions are an effective external manifestation of depression. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first purpose of the present application is to propose a depressive disorder identification method based on facial video sequences, which can identify depressive emotions based on facial videos.

[0007] The second objective of this application is to propose a depressive disorder recognition system based on facial video sequences.

[0008] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a method for identifying depressive disorders based on facial video sequences, including: segmenting the video sequence and performing face image detection to obtain the key points and action units of the face images contained in each video; constructing graph structures for the key points and action units of the face images contained in each video to obtain the key point graph and action unit graph of each video; convolving the key point graph and action unit graph of each video, respectively, and enhancing the graph structure features obtained by the convolution based on the attention mechanism, and fusing the enhanced key point feature graph and facial unit feature graph of each video to obtain a fused feature graph of each video; obtaining a segment-level representation of each video based on the fused feature graph of each video, establishing a video-level representation based on all segment-level representations, and performing depressive disorder recognition on the video-level representation to obtain a recognition result.

[0009] The method for identifying depressive disorders based on facial video sequences in an embodiment of the present application first segments the entire video sequence, constructs a key point coordinate map and an action unit map around the facial image; then, the temporal and spatial features are aggregated and fused through an encoder; finally, depression is identified through segment-level video representation. This embodiment can objectively and accurately automatically identify depressive disorders, thereby promoting early detection, early treatment, and early intervention of depressive disorders.

[0010] Optionally, in one embodiment of the present application, given a key point sequence {C L , N, T} represent the coordinate dimension, the number of key points and the number of frames respectively, and set the relative position set Each relative position The expression is:

[0011]

[0012] in, represents the coordinates of the nose tip position in the Tth frame, h t Indicates the distance between the inner corners of the two eyes;

[0013] The key point graph is represented as:

[0014] G L =(V L ,E L )

[0015] Among them, V L = R is the geometry of the node, the edge set Describes the connections between nodes within each frame.

[0016] Optionally, in one embodiment of the present application, convolution is performed on the key point graph and the action unit graph respectively, including:

[0017] The key point graph and action unit graph are respectively input into independent spatiotemporal graph convolution modules for graph convolution. Each spatiotemporal convolution module contains a graph convolution GCN for modeling the spatial structure of the graph and a temporal convolution TCN for modeling temporal information. The temporal convolution TCN is implemented as a normal 2D convolution operation along the time dimension. The implementation process of the graph convolution GCN includes:

[0018] The graph convolution of a graph G = (V, E) consisting of a node set V and an edge set E is defined as:

[0019]

[0020] Where A represents the learnable adjacency matrix, A is initialized with an all-one matrix and updated during training, I is the identity matrix, D is the degree matrix, and the diagonal elements D of D are ii is the normalized adjacency matrix The sum of the i-th row, X in is the feature matrix of the input, W is the learnable weight matrix for linear transformation, ReLU represents the activation function, X out is the output feature matrix.

[0021] Optionally, in one embodiment of the present application, the graph structure features obtained by convolution are enhanced based on the attention mechanism, including:

[0022] The feature map is enhanced by spatial attention and temporal attention.

[0023] Among them, spatial attention takes graph structure features as input and is expressed as:

[0024]

[0025] Among them, X out is the graph structure feature, C′ represents the spatial dimension, N represents the number of key points, and T represents the number of frames;

[0026] The feature map is enhanced through spatial attention, including:

[0027] Calculate the average value of each node in the time and channel dimensions, and calculate the attention score of each node through the fully connected layer and the Sigmoid function, and compare it with s in multiply;

[0028] Spatial attention takes the output of temporal convolution as input and computes the average over the node and channel dimensions for each frame;

[0029] The above method further includes:

[0030] The enhanced feature maps are processed through the residual layer.

[0031] Optionally, in one embodiment of the present application, the enhanced key point feature map and facial unit feature map of each video segment are fused to obtain a fused feature map of each video segment, including:

[0032] The projection of the action unit feature map on the key point feature map is obtained through cross attention, and it is concatenated with the key point feature to obtain the fused feature;

[0033] Construct a fused feature map based on the fused features.

[0034] Optionally, in one embodiment of the present application, obtaining a segment-level representation of each video segment based on the fused feature map of each video segment includes:

[0035] The fused feature map of each video is passed through at least one stacked spatiotemporal convolution module and global average pooling to obtain a segment-level representation of each video;

[0036] Build video-level representations based on all segment-level representations, including:

[0037] All segment-level representations are concatenated and self-attention is used to capture the relationship between segments to form a video-level representation.

[0038] Depression disorder recognition is performed on video-level representations, and recognition results are obtained, including:

[0039] The video-level representation is input into the fully connected layer and the recognition result is output, where the recognition result includes whether there is a tendency to depression and its severity.

[0040] To achieve the above-mentioned purpose, a second aspect of the present invention provides a depressive disorder recognition system based on facial video sequences, comprising:

[0041] The video processing module is used to segment the video sequence and perform face image detection to obtain the key points and action units of the face image contained in each video segment;

[0042] A graph construction module is used to construct graph structures for the key points and action units of the face image contained in each video, so as to obtain a key point graph and an action unit graph for each video;

[0043] The graph processing module is used to convolve the key point graph and action unit graph of each video respectively, enhance the graph structure features obtained by convolution based on the attention mechanism, and fuse the enhanced key point feature graph and facial unit feature graph of each video to obtain a fused feature graph of each video;

[0044] The recognition module is used to obtain the corresponding segment-level representation based on the fused feature map of each video segment, establish the video-level representation based on all the segment-level representations, and perform depression disorder recognition on the video-level representation to obtain the recognition result.

[0045] Optionally, in one embodiment of the present application, given a key point sequence {C L , N, T} represent the coordinate dimension, the number of key points and the number of frames respectively, and set the relative position set Each relative position The expression is:

[0046]

[0047] in, represents the coordinates of the nose tip position in the tth frame, h t Indicates the distance between the inner corners of the two eyes;

[0048] The key point graph is represented as:

[0049] G L =(V L , E L )

[0050] Among them, V L = R is the geometry of the node, the edge set Describes the connections between nodes within each frame.

[0051] Optionally, in one embodiment of the present application, convolution is performed on the key point graph and the action unit graph respectively, including:

[0052] The key point graph and action unit graph are respectively input into independent spatiotemporal graph convolution modules for graph convolution. Each spatiotemporal convolution module contains a graph convolution GCN for modeling the spatial structure of the graph and a temporal convolution TCN for modeling temporal information. The temporal convolution TCN is implemented as a normal 2D convolution operation along the time dimension. The implementation process of the graph convolution GCN includes:

[0053] The graph convolution of a graph G = (V, E) consisting of a node set V and an edge set E is defined as:

[0054]

[0055] Where A represents the learnable adjacency matrix, A is initialized with an all-one matrix and updated during training, I is the identity matrix, D is the degree matrix, and the diagonal elements D of D are ii is the normalized adjacency matrix The sum of the i-th row, X inis the feature matrix of the input, W is the learnable weight matrix for linear transformation, ReLU represents the activation function, X out is the output feature matrix.

[0056] Optionally, in one embodiment of the present application, the graph structure features obtained by convolution are enhanced based on the attention mechanism, including:

[0057] The feature map is enhanced by spatial attention and temporal attention.

[0058] Among them, spatial attention takes graph structure features as input and is expressed as:

[0059]

[0060] Among them, X out is the graph structure feature, C′ represents the spatial dimension, N represents the number of key points, and T represents the number of frames;

[0061] The feature map is enhanced through spatial attention, including:

[0062] Calculate the average value of each node in the time and channel dimensions, and calculate the attention score of each node through the fully connected layer and the Sigmoid function, and compare it with s in multiply;

[0063] Spatial attention takes the output of temporal convolution as input and computes the average over the node and channel dimensions for each frame;

[0064] The above system also includes a residual processing module, which is specifically used to:

[0065] The enhanced feature maps are processed through the residual layer.

[0066] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0068] Figure 1 A flowchart of a method for identifying depression disorders based on facial video sequences provided in Example 1 of the present application;

[0069] Figure 2 A schematic diagram of the structure of a depressive disorder recognition system based on facial video sequences provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0071] The following describes a method and system for identifying depression disorders based on facial video sequences according to an embodiment of the present application with reference to the accompanying drawings.

[0072] Figure 1 A flowchart of a method for identifying depressive disorders based on facial video sequences provided in Example 1 of the present application.

[0073] like Figure 1 As shown, the method for identifying depression disorders based on facial video sequences includes the following steps:

[0074] Step 101, segmenting the video sequence and performing face image detection to obtain key points and action units of the face image contained in each video segment;

[0075] Specifically, the video sequence is divided into several segments of the same length, the facial image is detected, and the coordinates of 68 key points and 20 action units of each frame of the facial image are calculated respectively.

[0076] Step 102, constructing graph structures for the key points and action units of the face image contained in each video segment, respectively, to obtain a key point graph and an action unit graph for each video segment;

[0077] Specifically, given a key point sequence Where {C L , N, T} represent the coordinate dimension, the number of key points and the number of frames respectively, and use the relative position set To construct the key point map. Each relative position The calculation method is as follows:

[0078]

[0079] in, represents the coordinates of the nose tip position in the tth frame, h t represents the distance between the inner corners of the eyes. L =(V L ,E L ) represents the key point graph, where V L =R is the set of nodes, the set of edges The connection between the internal nodes of each frame is described. This embodiment proposes an adjacency matrix that can be learned through a back-propagation algorithm to automatically explore the internal relationship between different key points, and can effectively learn the spatial dependency relationship between nodes.

[0080] Similarly, given a sequence of facial action units Among them C A represents the dimension of the action unit. Let G A =(V A ,E A ) represents the action unit graph, where V L =Y is the set of nodes, the set of edges Describes the connections between nodes within each frame.

[0081] Step 103, convolving the key point graph and the action unit graph of each video segment respectively, and enhancing the graph structure features obtained by the convolution based on the attention mechanism, and fusing the enhanced key point feature graph and facial unit feature graph of each video segment to obtain a fused feature graph of each video segment;

[0082] Specifically, the key point graph and action unit graph constructed in step 102 are input into independent spatiotemporal graph convolution modules respectively to preliminarily mine the potential semantic information in the graph structure from the perspective of time and space. Each spatiotemporal graph convolution module contains a graph convolution (GCN) for modeling the spatial structure of the graph and a temporal convolution (TCN) for modeling temporal information. The convolution operation of the graph G = (V, E) consisting of a node set V and an edge set E is defined as:

[0083]

[0084] Where A represents a learnable adjacency matrix, which is initialized with an all-one matrix and updated during training, and can better describe the spatial structure of the graph; I is the identity matrix; D is the degree matrix, whose diagonal elements D ii is the normalized adjacency matrix The sum of the i-th row; X in is the feature matrix of the input; W is the learnable weight matrix for linear transformation; ReLU represents the activation function for nonlinear transformation; X out is the feature matrix of the input. Temporal convolution is implemented as a normal 2D convolution operation along the time dimension.

[0085] Specifically, in order to enable the model to pay more attention to the spatial structure and temporal information that are highly related to depression, the present invention introduces spatial attention (SAttention) and temporal attention (TAttention) after each graph convolution and temporal convolution. Spatial attention takes the graph structure features output by the graph convolution as input, that is, Where C' represents the channel dimension. It first calculates the average value of each node in the time and channel dimensions Then the attention score of each node is calculated through two fully connected layers similar to the squeeze-excitation network and a sigmoid function, and then compared with s in Finally, a residual connection is established outside the attention module to stabilize the training process. The difference between temporal attention and spatial attention is that temporal attention takes the output of temporal convolution as input and calculates the average value of each frame in the node and channel dimensions.

[0086] Specifically, the action unit information related to each key point is integrated to form a new graph structure. The process includes:

[0087] Through the early fusion strategy, the action unit information is integrated into each key point to form a graph representation with richer emotional information. Specifically, the projection of the action unit feature graph (AGraph) on the key point feature graph (CGraph) is obtained through cross-attention, and it is spliced ​​with the key point features to obtain the fused features. Then, according to the construction method of the key point graph in 102, a new fused feature graph (FGraph) is constructed.

[0088] Step 104, obtaining a segment-level representation of each video segment based on the fused feature map of each video segment, establishing a video-level representation based on all segment-level representations, and performing depression disorder recognition on the video-level representation to obtain a recognition result.

[0089] Specifically, the fused feature map continues to pass through several stacked spatiotemporal graph convolution modules, comprehensively considering the changes in facial key points and action units to further explore potential depression information. Finally, a global average pooling is applied to form a segment-level representation.

[0090] Specifically, after all the segment-level sentiment representations are concatenated, self-attention is used to capture the relationship between segments, alleviate the density differences of depressive information contained in different segments, form corresponding video-level representations, and input them into the fully connected layer to determine whether there is a depressive tendency and its severity.

[0091] The method for identifying depressive disorders based on facial video sequences in an embodiment of the present application first segments the entire video sequence, constructs a key point coordinate map and an action unit map around the facial image; then, the temporal and spatial features are aggregated and fused through an encoder; finally, depression is identified through segment-level video representation. This embodiment can objectively and accurately automatically identify depressive disorders, thereby promoting early detection, early treatment, and early intervention of depressive disorders.

[0092] In order to implement the above embodiment, the present application also proposes a depressive disorder recognition system based on facial video sequences.

[0093] Figure 2A schematic diagram of the structure of a depressive disorder recognition system based on facial video sequences provided in an embodiment of the present application.

[0094] like Figure 2 As shown, the depression disorder recognition system based on facial video sequences includes:

[0095] The video processing module is used to segment the video sequence and perform face image detection to obtain the key points and action units of the face image contained in each video segment;

[0096] A graph construction module is used to construct graph structures for the key points and action units of the face image contained in each video, so as to obtain a key point graph and an action unit graph for each video;

[0097] The graph processing module is used to convolve the key point graph and action unit graph of each video respectively, enhance the graph structure features obtained by convolution based on the attention mechanism, and fuse the enhanced key point feature graph and facial unit feature graph of each video to obtain a fused feature graph of each video;

[0098] The recognition module is used to obtain the corresponding segment-level representation based on the fused feature map of each video segment, establish the video-level representation based on all the segment-level representations, and perform depression disorder recognition on the video-level representation to obtain the recognition result.

[0099] Optionally, in one embodiment of the present application, given a key point sequence {C L , N, T} represent the coordinate dimension, the number of key points and the number of frames respectively, and set the relative position set Each relative position The expression is:

[0100]

[0101] in, represents the coordinates of the nose tip position in the tth frame, h t Indicates the distance between the inner corners of the two eyes;

[0102] The key point graph is represented as:

[0103] G L =(V L , E L )

[0104] Among them, V L = R is the geometry of the node, the edge set Describes the connections between nodes within each frame.

[0105] Optionally, in one embodiment of the present application, convolution is performed on the key point graph and the action unit graph respectively, including:

[0106] The key point graph and action unit graph are respectively input into independent spatiotemporal graph convolution modules for graph convolution. Each spatiotemporal convolution module contains a graph convolution GCN for modeling the spatial structure of the graph and a temporal convolution TCN for modeling temporal information. The temporal convolution TCN is implemented as a normal 2D convolution operation along the time dimension. The implementation process of the graph convolution GCN includes:

[0107] The graph convolution of a graph G = (V, E) consisting of a node set V and an edge set E is defined as:

[0108]

[0109] Where A represents the learnable adjacency matrix, A is initialized with an all-one matrix and updated during training, I is the identity matrix, D is the degree matrix, and the diagonal elements D of D are ii is the normalized adjacency matrix The sum of the i-th row, X in is the feature matrix of the input, W is the learnable weight matrix for linear transformation, ReLU represents the activation function, X out is the output feature matrix.

[0110] Optionally, in one embodiment of the present application, the graph structure features obtained by convolution are enhanced based on the attention mechanism, including:

[0111] The feature map is enhanced by spatial attention and temporal attention.

[0112] Among them, spatial attention takes graph structure features as input and is expressed as:

[0113]

[0114] Among them, X out is the graph structure feature, C′ represents the spatial dimension, N represents the number of key points, and T represents the number of frames;

[0115] The feature map is enhanced through spatial attention, including:

[0116] Calculate the average value of each node in the time and channel dimensions, and calculate the attention score of each node through the fully connected layer and the Sigmoid function, and compare it with s in multiply;

[0117] Spatial attention takes the output of temporal convolution as input and computes the average over the node and channel dimensions for each frame;

[0118] The above system also includes a residual processing module, which is specifically used to:

[0119] The enhanced feature maps are processed through the residual layer.

[0120] It should be noted that the aforementioned explanation of the embodiment of the method for identifying depression disorders based on facial video sequences is also applicable to the system for identifying depression disorders based on facial video sequences in this embodiment, and will not be repeated here.

[0121] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0123] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0125] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0126] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0128] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for identifying depression disorders based on facial video sequences, characterized in that: include: Segment the video sequence and perform face image detection to obtain the key points and action units of the face image contained in each video segment; Construct graph structures for the key points and action units of the face images contained in each video, and obtain the key point graph and action unit graph of each video; Convolution is performed on the key point graph and action unit graph of each video respectively, and the graph structure features obtained by convolution are enhanced based on the attention mechanism. The enhanced key point feature graph and facial unit feature graph of each video are fused to obtain the fused feature graph of each video. Obtaining a segment-level representation of each video based on the fused feature graph of each video segment, establishing a video-level representation based on all segment-level representations, and performing depression disorder recognition on the video-level representation to obtain a recognition result; Among them, convolution is performed on the key point graph and the action unit graph respectively, including: The key point graph and action unit graph are respectively input into independent spatiotemporal graph convolution modules for graph convolution. Each spatiotemporal convolution module contains a graph convolution GCN for modeling the spatial structure of the graph and a temporal convolution TCN for modeling temporal information. The temporal convolution TCN is implemented as a normal 2D convolution operation along the time dimension. The implementation process of the graph convolution GCN includes: The graph convolution of a graph G = (V, E) consisting of a node set V and an edge set E is defined as: Where A represents the learnable adjacency matrix, A is initialized with an all-one matrix and updated during training, I is the identity matrix, D is the degree matrix, and the diagonal elements D of D are ii is the normalized adjacency matrix The sum of the i-th row, X in is the feature matrix of the input, W is the learnable weight matrix for linear transformation, ReLU represents the activation function, X out is the output feature matrix; The graph structure features obtained by convolution are enhanced based on the attention mechanism, including: The feature map is enhanced by spatial attention and temporal attention. Among them, spatial attention takes graph structure features as input and is expressed as: Among them, X out is the graph structure feature, C′ represents the spatial dimension, N represents the number of key points, and T represents the number of frames; The feature map is enhanced through spatial attention, including: Calculate the average value of each node in the time and channel dimensions, and calculate the attention score of each node through the fully connected layer and the Sigmoid function, and compare it with s in multiply; The spatial attention takes the output of the temporal convolution as input and calculates the average value of each frame in the node and channel dimensions; The method further comprises: The enhanced feature maps are processed through the residual layer.

2. The method according to claim 1, characterized in that Given a sequence of key points CL, N, T represent the coordinate dimension, number of key points and number of frames respectively, setting the relative position set Each relative position The expression is: in, represents the coordinates of the nose tip position in the tth frame, h t Indicates the distance between the inner corners of the two eyes; The key point graph is represented as: G L =(V L ,E L ) Among them, V L = R is the geometry of the node, the edge set Describes the connections between nodes within each frame.

3. The method according to claim 1, characterized in that The enhanced key point feature map and facial unit feature map of each video are fused to obtain a fused feature map of each video, including: The projection of the action unit feature map on the key point feature map is obtained through cross attention, and it is concatenated with the key point feature to obtain the fused feature; Construct a fused feature map based on the fused features.

4. The method according to claim 1, characterized in that Based on the fusion feature map of each video, the segment-level representation of each video is obtained, including: The fused feature map of each video is passed through at least one stacked spatiotemporal convolution module and global average pooling to obtain a segment-level representation of each video; The video-level representation is established based on all the segment-level representations, including: All segment-level representations are concatenated and self-attention is used to capture the relationship between segments to form a video-level representation. The performing depressive disorder recognition on the video-level representation to obtain a recognition result includes: The video-level representation is input into a fully connected layer, and a recognition result is output, wherein the recognition result includes whether there is a tendency to depression and the severity of the depression.

5. A depressive disorder recognition system based on facial video sequences, characterized in that: include: The video processing module is used to segment the video sequence and perform face image detection to obtain the key points and action units of the face image contained in each video segment; A graph construction module is used to construct graph structures for the key points and action units of the face image contained in each video, so as to obtain a key point graph and an action unit graph for each video; The graph processing module is used to convolve the key point graph and action unit graph of each video respectively, enhance the graph structure features obtained by convolution based on the attention mechanism, and fuse the enhanced key point feature graph and facial unit feature graph of each video to obtain a fused feature graph of each video; A recognition module, used to obtain a corresponding segment-level representation based on the fused feature map of each video segment, establish a video-level representation based on all segment-level representations, and perform depression disorder recognition on the video-level representation to obtain a recognition result; Among them, convolution is performed on the key point graph and the action unit graph respectively, including: The key point graph and action unit graph are respectively input into independent spatiotemporal graph convolution modules for graph convolution. Each spatiotemporal convolution module contains a graph convolution GCN for modeling the spatial structure of the graph and a temporal convolution TCN for modeling temporal information. The temporal convolution TCN is implemented as a normal 2D convolution operation along the time dimension. The implementation process of the graph convolution GCN includes: The graph convolution of a graph G = (V, E) consisting of a node set V and an edge set E is defined as: Where A represents the learnable adjacency matrix, A is initialized with an all-one matrix and updated during training, I is the identity matrix, D is the degree matrix, and the diagonal elements D of D are ii is the normalized adjacency matrix The sum of the i-th row, X in is the feature matrix of the input, W is the learnable weight matrix for linear transformation, ReLU represents the activation function, X out is the output feature matrix; The graph structure features obtained by convolution are enhanced based on the attention mechanism, including: The feature map is enhanced by spatial attention and temporal attention. Among them, spatial attention takes graph structure features as input and is expressed as: Among them, X utt is the graph structure feature, C′ represents the spatial dimension, N represents the number of key points, and T represents the number of frames; The feature map is enhanced through spatial attention, including: Calculate the average value of each node in the time and channel dimensions, and calculate the attention score of each node through the fully connected layer and the Sigmoid function, and compare it with s in multiply; The spatial attention takes the output of the temporal convolution as input and calculates the average value of each frame in the node and channel dimensions; The system also includes a residual processing module, which is specifically used for: The enhanced feature maps are processed through the residual layer.

6. The system according to claim 5, characterized in that Given a sequence of key points {C L ,N,T} represent the coordinate dimension, number of key points and number of frames respectively, and set the relative position set Each relative position The expression is: in, represents the coordinates of the nose tip position in the tth frame, h t Indicates the distance between the inner corners of the two eyes; The key point graph is represented as: G L =(V L ,E L ) Among them, V L = R is the geometry of the node, the edge set Describes the connections between nodes within each frame.

Images