Attention mechanism-based depression detection method, system and device and medium

Abstract

The invention discloses a depression detection method, system and device based on an attention mechanism and a medium. The depression detection method comprises the following steps: acquiring an interview video of a user; extracting audio data according to the collected interview video, and obtaining audio features of the audio data; extracting visual features according to the collected interview video; transcriptional annotation is performed on the audio data to obtain text data; and according to the obtained audio features and visual features, predicting the depression degree of the user based on a pre-constructed Transform model of a multi-level cross attention mechanism. According to the invention, by acquiring the features and predicting the depression degree of the user through the model, the potential relation between different modes is supplemented, the method is used for more comprehensive evaluation of the depression symptom of the user, and the detection result is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of depression detection, and in particular to a depression detection method, system, device and medium based on an attention mechanism. Background Art

[0002] Depression is one of the most common mental illnesses today. It manifests as a long-term low mood or loss of interest in activities and loss of pleasure, which can seriously affect work and life.

[0003] Currently, there are two main types of depression detection technologies: the first is based on a single modality (using one of text, voice, and expression), such as analyzing the text content of an individual on social media, collecting key data and extracting corresponding modal features, and then using a self-built model to identify depression. The second is based on multimodality (using multiple of text, voice, and vision), such as collecting the user's voice and expression information, extracting features through a multimodal model, and completing depression identification.

[0004] The shortcomings of existing technologies are that, although the multimodal approach reflects the patient's depressive state more comprehensively than the single modality, the current multimodal fusion technology has the problem of difficulty in taking into account both the internal features of the modalities and the effective interaction between modal features. In addition, current research is mainly based on foreign data sets, so the accuracy of the trained model for Chinese people is not very high. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a depression detection method, system, device and medium based on an attention mechanism are adopted to solve the problems raised in the above background technology.

[0006] A multimodal depression detection method based on a multi-level cross-attention mechanism, characterized by comprising the following steps:

[0007] Step S1, obtaining the interview video of the collecting user;

[0008] Step S2: extracting audio data based on the collected interview video, and obtaining audio features of the audio data;

[0009] Step S3: extracting visual features based on the collected interview video;

[0010] Step S4, transcribing and annotating the audio data to obtain text data;

[0011] Step S5: predict the user's depression level based on the obtained audio features and visual features and a pre-built Transformer model with a multi-level cross-attention mechanism.

[0012] As a further solution of the present invention: the specific steps in step S2 include:

[0013] First, transcribe the audio data and annotate the speaking role, start and end time, duration, and text content of each sentence in the audio;

[0014] Then, the user's voice data in the recording is trimmed according to the marked start and end times;

[0015] Finally, the Librosa library is used to extract audio features.

[0016] As a further solution of the present invention: the visual features include one or any combination of video frame sequences, key feature points of human faces, AUs features, and eye movement data;

[0017] The process of acquiring visual features includes:

[0018] The OpenFace library is used to extract frames from the interview video to obtain the video frame sequence;

[0019] Perform face detection and face alignment on the user in each frame, extract 2D and 3D facial feature points, and the aligned face image;

[0020] Extract the AU feature information of each frame user through the OpenFace library;

[0021] GazeCapture is used to extract user eye detection, eye tracking data, and gaze point estimation.

[0022] As a further solution of the present invention: the specific steps in step S5 include:

[0023] Step S51: cutting the visual features and audio features according to the start and end time of the text data to complete the alignment of the features in the time dimension;

[0024] Step S52: Send the audio features to a pre-built audio high-level feature extraction module. The process formula is:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] Among them, Z f , Z p , Z w , Z s They are the features extracted at the frame stage, phoneme stage, word stage, and sentence stage, respectively. is the final audio feature;

[0034] Step S53: Send the visual features to the pre-built expression extraction module, the formula is:

[0035]

[0036]

[0037]

[0038] Z v =T v (Attention(C v (Z v-1 )));

[0039] in, The features extracted by the CNN branch, is the feature extracted by the Transformer module, Z v It is a comprehensive feature of local features and global features;

[0040] Step S54: respectively send the speech features and the visual features into a pre-built hierarchical cross attention module, which has a layer of cross attention and a layer of self attention module, including:

[0041]

[0042]

[0043]

[0044]

[0045] Step S55: After the processed audio and video features are concatenated, they are sent to the last layer of self-attention module, and then classified through Softmax. The formula is:

[0046]

[0047]

[0048]

[0049] The technical solution of the second aspect: A system including a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any one of the above items, comprising:

[0050] A data collection module is used to collect the original audio and video data of the user in the depression analysis interview environment and transmit the original audio and video data to the server;

[0051] The data processing module is used to process the collected raw audio and video data, obtain audio features, expression features, and perform modal alignment through processing;

[0052] The depression detection module is used to predict the user's depression level based on the acquired voice features and facial expression features using a pre-built Transformer model with a multi-level cross-attention mechanism.

[0053] The technical solution of the third aspect: a device, further comprising:

[0054] at least one processor;

[0055] at least one memory for storing at least one program;

[0056] When the at least one program is executed by the at least one processor, the at least one processor implements a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any of the above items.

[0057] The technical solution of the fourth aspect: a storage medium storing processor-executable instructions, wherein the processor-executable instructions are used to implement a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any of the above items when executed by the processor.

[0058] Compared with the prior art, the present invention has the following technical effects:

[0059] The above technical solution is used to obtain features and then use the pre-built Transformer model with a multi-level cross-attention mechanism to predict the user's depression level. Data collection combines the PHQ-8 depression scale with good application reliability, comprehensively collects data from video and audio modalities, and considers cross-modal feature information, supplementing the potential connections between different modalities, for a more comprehensive assessment of the user's depressive symptoms, and more accurate detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0061] Figure 1 A schematic diagram of the steps of a method for detecting depression provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of a model flow based on phased extraction of speech features provided by an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of a joint expression feature extraction process based on a CNN branch and a Transformer branch provided in an embodiment of the present invention;

[0064] Figure 4 A schematic diagram of a multimodal depression recognition network model based on a multi-level cross-attention mechanism provided in an embodiment of the present invention;

[0065] Figure 5 This is a structural diagram of a depression detection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a multimodal depression detection method based on a multi-level cross attention mechanism comprises the following steps:

[0068] Step S1, obtaining the interview video of the collecting user;

[0069] Step S2: extracting audio data based on the collected interview video, and obtaining audio features of the audio data;

[0070] In this embodiment, Figure 2 As shown, the figure is a schematic diagram of the model process based on extracting speech features in stages;

[0071] Step S3: extracting visual features based on the collected interview video;

[0072] Step S4, transcribing and annotating the audio data to obtain text data;

[0073] Step S5: predict the user's depression level based on the obtained audio features and visual features and a pre-built Transformer model with a multi-level cross-attention mechanism.

[0074] In this embodiment, the specific steps in step S2 include:

[0075] First, transcribe the audio data and annotate the speaking role, start and end time, duration, and text content of each sentence in the audio;

[0076] Then, the user's voice data in the recording is trimmed according to the marked start and end times;

[0077] Finally, the Librosa library is used to extract audio features.

[0078] In this embodiment, the visual features in step S3 include one or any combination of video frame sequence, key feature points of face, AUs features, and eye movement data;

[0079] The process of acquiring visual features includes:

[0080] The OpenFace library is used to extract frames from the interview video to obtain the video frame sequence;

[0081] Perform face detection and face alignment on the user in each frame, extract 2D and 3D facial feature points, and the aligned face image;

[0082] Extract the AU feature information of each frame user through the OpenFace library;

[0083] GazeCapture is used to extract user eye detection, eye tracking data, and gaze point estimation.

[0084] In this embodiment, the specific steps in step S5 include:

[0085] In this embodiment, Figure 3 As shown, the figure is a schematic diagram of the joint expression feature extraction process based on the CNN branch and the Transformer branch;

[0086] In this embodiment, Figure 4 As shown, the diagram is a schematic diagram of a multimodal depression recognition network model based on a multi-level cross-attention mechanism;

[0087] Step S51: cutting the visual features and audio features according to the start and end time of the text data to complete the alignment of the features in the time dimension;

[0088] Step S52: Send the audio features to a pre-built audio high-level feature extraction module. The process formula is:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] Among them, Z f , Z p , Z w , Z s They are the features extracted at the frame stage, phoneme stage, word stage, and sentence stage, respectively. is the final audio feature;

[0098] Step S53: Send the visual features to the pre-built expression extraction module, the formula is:

[0099]

[0100]

[0101]

[0102] Z v =T v (Attention(C v (Z v-1 )));

[0103] in, The features extracted by the CNN branch, is the feature extracted by the Transformer module, Z v It is a comprehensive feature of local features and global features;

[0104] Step S54: respectively send the speech features and the visual features into a pre-built hierarchical cross attention module, which has a layer of cross attention and a layer of self attention module, including:

[0105]

[0106]

[0107]

[0108]

[0109] Step S55: After the processed audio and video features are concatenated, they are sent to the last layer of self-attention module, and then classified through Softmax. The formula is:

[0110]

[0111]

[0112]

[0113] The technical solution of the second aspect: A system including a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any one of the above items, comprising:

[0114] A data collection module is used to collect the original audio and video data of the user in the depression analysis interview environment and transmit the original audio and video data to the server;

[0115] The data processing module is used to process the collected raw audio and video data, obtain audio features, expression features, and perform modal alignment through processing;

[0116] The depression detection module is used to predict the user's depression level based on the acquired voice features and facial expression features using a pre-built Transformer model with a multi-level cross-attention mechanism.

[0117] like Figure 5 As shown, the diagram is a structural diagram of a depression detection system;

[0118] In this embodiment, the system is also provided with a basic support module for providing core public services required by the system, including the realization of functions such as user management, data access, and real-time interaction;

[0119] The technical solution of the third aspect: a device, further comprising:

[0120] at least one processor;

[0121] at least one memory for storing at least one program;

[0122] When the at least one program is executed by the at least one processor, the at least one processor implements a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any of the above items.

[0123] The technical solution of the fourth aspect: a storage medium storing processor-executable instructions, wherein the processor-executable instructions are used to implement a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any of the above items when executed by the processor.

[0124] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] A storage medium storing processor-executable instructions, characterized in that: the processor-executable instructions are used to implement a multimodal depression detection method based on a multi-level cross-attention mechanism as described above when executed by the processor.

[0126] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. A multimodal depression detection method based on a multi-level cross-attention mechanism, characterized in that: The following steps are involved: Step S1, obtaining the interview video of the collecting user; Step S2: extracting audio data based on the collected interview video, and obtaining audio features of the audio data; Step S3: extracting visual features based on the collected interview video; Step S4, transcribing and annotating the audio data to obtain text data; Step S5: predict the user's depression level based on the obtained audio features and visual features and a pre-built Transformer model with a multi-level cross-attention mechanism.

2. According to claim 1, a multimodal depression detection method based on a multi-level cross-attention mechanism is characterized in that: The specific steps in step S2 include: First, transcribe the audio data and annotate the speaking role, start and end time, duration, and text content of each sentence in the audio; Then, the user's voice data in the recording is trimmed according to the marked start and end times; Finally, the Librosa library is used to extract audio features.

3. According to claim 1, a multimodal depression detection method based on a multi-level cross attention mechanism is characterized in that: The visual features include one or any combination of video frame sequences, key facial feature points, AUs features, and eye movement data; The process of acquiring visual features includes: The OpenFace library is used to extract frames from the interview video to obtain the video frame sequence; Perform face detection and face alignment on the user in each frame, extract 2D and 3D facial feature points, and the aligned face image; Extract the AU feature information of each frame user through the OpenFace library; GazeCapture is used to extract user eye detection, eye tracking data, and gaze point estimation.

4. According to claim 1, a multimodal depression detection method based on a multi-level cross-attention mechanism is characterized in that: The specific steps in step S5 include: Step S51: cutting the visual features and audio features according to the start and end time of the text data to complete the alignment of the features in the time dimension; Step S52: Send the audio features to a pre-built audio high-level feature extraction module. The process formula is: Among them, Z f , Z p , Z w , Z s They are the features extracted at the frame stage, phoneme stage, word stage, and sentence stage, respectively. is the final audio feature; Step S53: Send the visual features to the pre-built expression extraction module, the formula is: Z v =T v (Attention(C v (Z v-1 ))); in, The features extracted by the CNN branch, is the feature extracted by the Transformer module, Z v It is a comprehensive feature of local features and global features; Step S54: respectively send the speech features and the visual features into a pre-built hierarchical cross attention module, which has a layer of cross attention and a layer of self attention module, including: Step S55: After the processed audio and video features are concatenated, they are sent to the last layer of self-attention module, and then classified through Softmax. The formula is:

5. A system comprising a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any one of claims 1 to 4, characterized in that: include: A data collection module is used to collect the original audio and video data of the user in the depression analysis interview environment and transmit the original audio and video data to the server; The data processing module is used to process the collected raw audio and video data, obtain audio features, expression features, and perform modal alignment through processing; The depression detection module is used to predict the user's depression level based on the acquired voice features and facial expression features using a pre-built Transformer model with a multi-level cross-attention mechanism.

6. A device, characterized in that: Also includes: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any one of claims 1-4.

7. A storage medium storing instructions executable by a processor, characterized in that: The processor-executable instructions are used to implement a multimodal depression detection method based on a multi-level cross-attention mechanism as described in any one of claims 1-4 when executed by the processor.

Images