Multi-modal data-based depression screening method, device and equipment

Abstract

The invention discloses a depression screening method, device and equipment based on multi-modal data, and the method comprises the steps: extracting video features corresponding to video data through a video branch, extracting audio features corresponding to audio data through an audio branch, and extracting text features corresponding to the text data through the text branches, then fusing video features, audio features and the text features through a cross attention mechanism in the fusion module to obtain fused features, and finally determining depression scale scores of the to-be-screened data according to the fused features through the prediction module. In this way, the video features, the audio features and the text features are fused through a cross attention mechanism, knowledge information carried by the fused features can be increased, redundant features in the video features, the audio features and the text features can be paid attention to, and therefore the screening accuracy of the depression screening model can be improved, and the screening accuracy of the depression screening model can be improved. Therefore, the efficiency and accuracy of multi-mode automatic depression screening can be improved.

Description

Technical Field

[0001] The present application relates to the field of medical information processing technology, and in particular to a depression screening method, device and equipment based on multimodal data. Background Art

[0002] Depression is a common mental disorder. Its symptoms are usually negative facial expressions, low voice, more negative words and first-person words in speech, etc. Depression has seriously affected human mental health, but its screening still generally relies on depression self-rating scales such as PHQ-9 (Patient Health Questionnaire-9) and doctors' experience. The screening efficiency is low and the screening results are subjective.

[0003] Recently, the use of deep learning technology to build a neural network model for depression screening has attracted widespread attention. The neural network model can extract effective information from different modalities in a targeted manner, and fuse multiple modal features through relevant mechanisms. Through continuous training, it can realize automatic screening of patients with depression. However, the current multimodal-based depression screening method does not pay attention to the feature redundancy problem when fusing multimodal features, resulting in poor model performance after feature fusion, which in turn affects the accuracy of depression screening.

[0004] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present application is to provide a method, device and equipment for depression screening based on multimodal data in view of the deficiencies of the existing technology.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a depression screening method based on multimodal data, wherein the depression screening method based on multimodal data specifically includes:

[0007] Acquiring data to be screened, wherein the data to be screened includes video data, audio data, and text data;

[0008] Inputting the video data into a video branch in a trained depression screening model, and extracting video features corresponding to the video data through the video branch;

[0009] Inputting the audio data into an audio branch in a trained depression screening model, and extracting audio features corresponding to the audio data through the audio branch;

[0010] Inputting the text data into a text branch in a trained depression screening model, and extracting text features corresponding to the text data through the text branch;

[0011] Inputting the video features, audio features and text features into a fusion module in the depression screening model, and determining fusion features through a cross-attention mechanism in the fusion module;

[0012] The fusion feature is input into the prediction module in the depression screening model, and the depression scale score of the data to be screened is determined by the prediction module.

[0013] The depression screening method based on multimodal data, wherein the step of inputting the video data into a video branch in a trained depression screening model and extracting video features corresponding to the video data through the video branch specifically includes:

[0014] Determining a video facial action unit frame sequence based on the video data;

[0015] The video facial action unit frame sequence is input into the video feature extraction module in the video branch, and the video feature extraction module performs spatiotemporal feature extraction to obtain video features corresponding to the video data.

[0016] The depression screening method based on multimodal data, wherein the video feature extraction module inputting the video facial action unit frame sequence into the video branch specifically includes:

[0017] For each video facial action unit frame in the video facial action unit frame sequence, extracting spatial features of the video facial action unit frame through a key frame selection module, and generating a one-hot encoding vector based on the spatial features;

[0018] Performing a dot multiplication of the one-hot encoding vector and the video facial action unit frame to obtain a video facial action unit key frame corresponding to the video facial action unit frame;

[0019] A sequence of video facial action unit key frames corresponding to all video facial action unit frames is used as a video facial action unit key frame sequence;

[0020] The video facial action unit key frame sequence is input into the video feature extraction module in the video branch.

[0021] The depression screening method based on multimodal data, wherein the step of inputting the audio data into an audio branch in a trained depression screening model and extracting audio features corresponding to the audio data through the audio branch specifically includes:

[0022] Extracting low-level descriptors, mel-language spectrograms, and wav2vec features corresponding to the audio data;

[0023] Input the low-level descriptor, the Mel-language spectrogram and the wav2vec feature into the audio feature extraction module in the audio branch, and extract the low-level descriptor vector corresponding to the low-level descriptor, the Mel-language spectrogram vector corresponding to the Mel-language spectrogram, and the wav2vec vector corresponding to the wav2vec feature through the audio feature extraction module;

[0024] The low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector are input into a multi-level audio feature interaction module in the audio branch, and the audio features corresponding to the audio data are output through the multi-level audio feature interaction module.

[0025] The depression screening method based on multimodal data, wherein the step of inputting the low-level descriptor vector, the Mel-language spectrogram vector, and the wav2vec vector into the multi-level audio feature interaction module in the audio branch, and outputting the audio features corresponding to the audio data through the multi-level audio feature interaction module specifically includes:

[0026] constructing a first query vector based on the low-level descriptor vector, and constructing a first key vector and a first value vector based on the Mel-language spectrogram vector;

[0027] Determine, based on the first query vector, the first key vector, and the first value vector, a fused audio feature through a first cross attention mechanism unit in a multi-level audio feature interaction module;

[0028] Constructing a second query vector based on the fused audio features, and constructing a second key vector and a second value vector based on the wav2vec vector;

[0029] Based on the second query vector, the second key vector and the second value vector, the audio features corresponding to the audio data are determined through a second cross-attention mechanism unit in a multi-level audio feature interaction module.

[0030] The depression screening method based on multimodal data, wherein the step of inputting the text data into a text branch in a trained depression screening model and extracting text features corresponding to the text data through the text branch specifically includes:

[0031] Inputting the text data into the BERT module in the text branch, and extracting intermediate text features through the BERT model;

[0032] The intermediate text features are input into the FC module in the text branch, and the text features corresponding to the text data are determined by the FC module.

[0033] The depression screening method based on multimodal data, wherein the step of inputting the video features, audio features and text features into a fusion module in the depression screening model and determining the fusion features through a cross-attention mechanism in the fusion module specifically includes:

[0034] Constructing a text query vector based on the text features;

[0035] Constructing a video key vector and a video value vector based on the video features, and constructing an audio key vector and an audio value vector based on the audio features;

[0036] Determining a first fused feature through a cross attention mechanism based on the text query vector, the video key vector, and the video value vector;

[0037] Determining a second fused feature through a cross attention mechanism based on the text query vector, the audio key vector, and the audio value vector;

[0038] A fusion feature is determined based on the first fusion feature and the second fusion feature.

[0039] A second aspect of the present application provides a depression screening device based on multimodal data, wherein the depression screening device based on multimodal data specifically includes:

[0040] An acquisition module, used for acquiring data to be screened, wherein the data to be screened includes video data, audio data and text data;

[0041] A video feature extraction module, used for inputting the video data into a video branch in a trained depression screening model, and extracting video features corresponding to the video data through the video branch;

[0042] An audio feature extraction module, used for inputting the audio data into an audio branch in a trained depression screening model, and extracting audio features corresponding to the audio data through the audio branch;

[0043] A text feature extraction module, used for inputting the text data into a text branch in a trained depression screening model, and extracting text features corresponding to the text data through the text branch;

[0044] A fusion module, used for inputting the video features, audio features and text features into the fusion module in the depression screening model, and determining fusion features through a cross-attention mechanism in the fusion module;

[0045] A prediction module is used to input the fusion feature into the prediction module in the depression screening model, and determine the depression scale score of the data to be screened through the prediction module.

[0046] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the above-described methods for depression screening based on multimodal data.

[0047] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0048] The memory stores a computer-readable program executable by the processor;

[0049] When the processor executes the computer-readable program, the processor implements the steps in any of the above-mentioned methods for screening depression based on multimodal data.

[0050] Beneficial effects: Compared with the prior art, the present application provides a method, device and equipment for screening depression based on multimodal data, the method comprising extracting video features corresponding to the video data through the video branch, extracting audio features corresponding to the audio data through the audio branch, extracting text features corresponding to the text data through the text branch, and then fusing the video features, audio features and text features through the cross-attention mechanism in the fusion module to obtain fusion features, and finally determining the depression scale score of the data to be screened according to the fusion features through the prediction module. In this way, by fusing the video features, audio features and text features through the cross-attention mechanism, the knowledge information carried by the fusion features can be increased, and redundant features in the video features, audio features and text features can also be paid attention to, thereby improving the screening accuracy of the depression screening model, and further improving the efficiency and accuracy of multimodal automatic depression screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A flowchart of a depression screening method based on multimodal data provided in an embodiment of the present application.

[0053] Figure 2 Schematic flow chart for a specific example of a depression screening model.

[0054] Figure 3 The figure is a flow chart showing a specific example of video branching.

[0055] Figure 4 The figure is a schematic flow chart of a specific example of the audio branch.

[0056] Figure 5 A schematic flowchart for a specific example of text branching.

[0057] Figure 6 The figure is a principle flow chart of a specific example of the fusion module.

[0058] Figure 7 This is a schematic diagram of the structure of a depression screening device based on multimodal data provided in an embodiment of the present application.

[0059] Figure 8 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application provide a method, device and apparatus for screening depression based on multimodal data. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0062] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0063] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0064] The inventors have found through research that depression is a common mental disorder, and the symptoms of depression are usually negative facial expressions, low voice, more negative words and first-person words in speech, etc. Depression has seriously affected human mental health, but its screening still generally relies on depression self-rating scales such as PHQ-9 (Patient Health Questionnaire-9) and doctors' experience, with low screening efficiency and subjective screening results.

[0065] Recently, the use of deep learning technology to build a neural network model for depression screening has attracted widespread attention. The neural network model can extract effective information from different modalities in a targeted manner, and fuse multiple modal features through relevant mechanisms. Through continuous training, it can realize automatic screening of patients with depression. However, the current multimodal-based depression screening method does not pay attention to the feature redundancy problem when fusing multimodal features, resulting in poor model performance after feature fusion, which in turn affects the accuracy of depression screening.

[0066] In order to solve the above problems, in an embodiment of the present application, the video data is input into the video branch in the depression screening model, and the video features corresponding to the video data are determined through the video branch; the audio data is input into the audio branch in the depression screening model, and the audio features corresponding to the audio data are determined through the audio branch; the text data is input into the text branch in the depression screening model, and the text features corresponding to the text data are determined through the text branch; the video features, the audio features and the text features are input into the fusion module in the depression screening model, and the fusion features of the video features, the audio features and the text features are determined through the fusion module; the fusion features are input into the prediction module in the depression screening model, and the depression scale score of the data to be screened is determined through the prediction module. The present application processes the features provided by the video data, the features provided by the audio data and the features provided by the text data through the video branch, the audio branch and the text branch respectively to perform depression screening, and can fully extract the features of a single modality by using different branches, and then fully fuse the features of the three modalities by using the fusion module, which can reduce the feature loss on the one hand, and can be fused through the cross-attention mechanism on the other hand, and can avoid the influence of redundant features, which can not only improve the prediction accuracy, but also realize the end-to-end mode.

[0067] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0068] This embodiment provides a depression screening method based on multimodal data, such as Figure 1 As shown, the method includes:

[0069] S10, obtaining data to be screened.

[0070] Specifically, the data to be screened is obtained by collecting data from the subject, wherein the data to be screened includes video data, audio data, and text data. The process of obtaining the data to be screened can be to obtain audio and video data by watching and describing emotional pictures, reading emotional tendency articles, or semi-structured interviews, and finally splitting the audio and video data into video data, audio data, and text data to obtain the data to be screened. In addition, the text data may include basic personal information and mental health status, wherein the basic personal information and mental health status may be a scale questionnaire compiled by gender, age, and patient health questionnaire (PHQ-9) and generalized anxiety scale (GAD-7). Of course, in practical applications, the data to be screened can also be obtained in other ways, for example, a stimulation experimental paradigm for depression detection is pre-set, and then the video data, audio data, and text data of the subject are collected according to the stimulation experimental paradigm for depression detection pre-set to form the data to be screened.

[0071] S20, inputting the video data into a video branch in a trained depression screening model, and extracting video features corresponding to the video data through the video branch.

[0072] Specifically, the depression screening model is a trained deep learning model, and the depression scale score is predicted through the depression screening model. Figure 2 As shown, the depression screening model includes a video branch, an audio branch, a text branch, a fusion module and a prediction model. The video branch, the audio branch and the text branch are parallel and connected to the fusion module, and the fusion module is connected to the prediction model. The input item of the video branch is video data, the input item of the audio branch is audio data, the input item of the text branch is text data, the input item of the fusion module includes the video features output by the video branch, the audio features output by the audio branch and the text features output by the text module, the input item of the prediction module is the fusion features output by the fusion module, and the output item of the prediction module is the depression scale score.

[0073] Further, the video branch is used to extract spatiotemporal features of the video data to obtain video features corresponding to the video data. Exemplarily, the step of inputting the video data into a video branch of a trained depression screening model and extracting the video features corresponding to the video data through the video branch specifically includes:

[0074] S21, determining a video facial action unit frame sequence based on the video data;

[0075] S22, inputting the video facial action unit frame sequence into the video feature extraction module in the video branch, and performing spatiotemporal feature extraction by the video feature extraction module to obtain video features corresponding to the video data.

[0076] Specifically, the video facial action unit frame sequence includes a facial action unit frame corresponding to each video frame in the video data, and each video frame corresponds to a facial action unit frame including a facial action unit in the video frame. That is to say, the video facial action unit frame sequence is composed of facial action unit frames of video frames in the video data each containing the face of the person being detected, wherein the facial action unit frame includes a number of facial action units (for example, each frame is a 68×3 three-dimensional facial action unit, etc.), and the facial action unit is used to reflect the texture features of facial expressions, and the local features of the video frame can be extracted through the facial action unit. In addition, it should be noted that the facial action unit can be obtained using existing methods, for example, using a general computer vision library to crop the face area and extract the facial action unit corresponding to the face area.

[0077] The video feature extraction module is used to extract spatiotemporal features, wherein the video feature extraction module uses the STA module to extract spatiotemporal feature vectors to obtain the video features corresponding to the video data. Specifically, Figure 3 As shown, the STA module may include a spatial attention unit, a temporal attention unit and a fusion unit, wherein the spatial attention unit is used to extract a spatial attention vector and a spatial attention weight, and the temporal attention unit is used to extract a temporal attention vector and a temporal attention weight, wherein the spatial attention weight and the temporal attention weight are fused to obtain a spatial attention weight (for example, dot multiplication of the spatial attention weight and the temporal attention weight, etc.), the spatiotemporal attention weight is dot multiplied with the spatial attention vector to obtain a spatial feature, and the spatiotemporal attention weight is dot multiplied with the temporal attention vector to obtain a temporal feature; finally, the temporal feature and the spatial feature are fused to obtain a video feature (for example, the temporal feature and the spatial feature are added to obtain a video feature, etc.). The embodiment of the present application decouples the temporal attention calculation operation and the spatial attention calculation operation, thereby retaining the temporal structure and spatial structure of the input video frame, avoiding structural damage during the calculation of the coupling operation between the two, and is conducive to improving the screening performance of the model. In addition, the structures of the spatial attention unit and the temporal attention unit can be arranged according to actual conditions. In the embodiment of the present application, both the spatial attention unit and the temporal attention unit include multi-layer CNNs.

[0078] Furthermore, since not every frame of the input image contributes greatly to the screening results during the depression screening process, and the video data itself has the characteristic that continuous frames do not change significantly, before obtaining the video features, the video frames can be screened to remove continuous frames that do not contribute much to the depression screening structure or have no obvious changes. This can reduce the impact of useless features on the screening on the one hand, and reduce the computational complexity of video feature extraction on the other hand, thereby improving the screening speed.

[0079] Based on this, the video feature extraction module that inputs the video facial action unit frame sequence into the video branch specifically includes:

[0080] S211, for each video facial action unit frame in the video facial action unit frame sequence, extracting spatial features of the video facial action unit frame through a key frame selection module, and generating a one-hot encoding vector based on the spatial features;

[0081] S212, performing a dot product of the one-hot encoding vector and the video facial action unit frame to obtain a video facial action unit key frame corresponding to the video facial action unit frame;

[0082] S213, taking a sequence consisting of video facial action unit key frames corresponding to all video facial action unit frames as a video facial action unit key frame sequence;

[0083] S214: Input the video facial action unit key frame sequence into the video feature extraction module in the video branch.

[0084] Specifically, the key frame selection module is used to select the video facial action unit key frames in the video facial action unit frame sequence, wherein the key frame selection module may include a key frame feature extraction unit and a decoding unit, the key frame feature extraction unit is used to extract the spatial features of the video facial action unit frame, and the decoding unit is used to generate a one-hot encoding vector based on the spatial features.

[0085] Exemplarily, the key frame selection module can adopt a DSN module of the structure of CNN+BiLSTM, that is, the DSN module includes a CNN model and a BiLSTM model, the CNN model is used as a key frame feature extraction unit, and the BiLSTM model is used as a decoding unit, that is, the local features of the video facial action unit are extracted by the CNN model to obtain the spatial features of the video facial action unit frame, and the local features of the video facial action unit in the extracted spatial features are decoded by the BiLSTM model to obtain a unique hot encoding vector. Specifically, the DSN module uses CNN to extract the spatial features of the input video facial action unit frame based on the idea of ​​reinforcement learning, uses BiLSTM to decode the spatial features and generates a unique hot encoding vector through Bernoulli distribution mapping, multiplies the unique hot encoding vector with the spatial feature to obtain video key frame information, and optimizes by calculating the representative reward and diversity reward of the key frame, so that the key frame meets the definition of representativeness and diversity. The embodiment of the present application realizes the extraction of key frame information from the input video facial action unit frame through the DSN module, eliminating the influence of redundant frames, so that the model can better process the input sequence.

[0086] In a typical implementation, Figure 3 As shown, the video branch includes a DSN module and a STA module. The DSN module is used as a key frame selection module to determine the key frame sequence of the video facial action unit. The STA module is used as a video feature extraction module to determine the video features corresponding to the video data based on the key frame sequence of the video facial action unit. The DSN module includes a CNN model and a BiLSTM model. The STA module includes a temporal attention unit and a spatial attention unit. The input items of the temporal attention unit and the spatial attention unit are both output items of the DSN module. The embodiment of the present application combines the DSN module and the STA module. On the one hand, key frame information can be extracted from the input video facial action unit frame, eliminating the influence of redundant frames. On the other hand, the temporal attention calculation operation and the spatial attention calculation operation can be decoupled, and the temporal structure and spatial structure of the input video frame can be retained. The structural damage during the calculation of the coupled operation of the two is avoided, which is conducive to improving the screening performance of the model.

[0087] S30: input the audio data into an audio branch in a trained depression screening model, and extract audio features corresponding to the audio data through the audio branch.

[0088] Specifically, the audio data is in the form of a speech signal, and the audio branch determines the audio features corresponding to the audio data by extracting features from the speech information, wherein the audio branch can directly extract features from the speech information to obtain the audio features, or can fuse low-level descriptor features, Mel-language spectrogram features, and wav2vec features to determine the audio features. In an embodiment of the present application, the audio features are determined by fusing low-level descriptor features, Mel-language spectrogram features, and wav2vec features, so that the feature information carried by the audio data can be better captured, and more knowledge information can be provided for the subsequent prediction process.

[0089] Exemplarily, the step of inputting the audio data into an audio branch in a trained depression screening model and extracting audio features corresponding to the audio data through the audio branch specifically includes:

[0090] Extracting low-level descriptors, mel-language spectrograms, and wav2vec features corresponding to the audio data;

[0091] Input the low-level descriptor, the Mel-language spectrogram and the wav2vec feature into the audio feature extraction module in the audio branch, and extract the low-level descriptor vector corresponding to the low-level descriptor, the Mel-language spectrogram vector corresponding to the Mel-language spectrogram, and the wav2vec vector corresponding to the wav2vec feature through the audio feature extraction module;

[0092] The low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector are input into a multi-level audio feature interaction module in the audio branch, and the audio features corresponding to the audio data are output through the multi-level audio feature interaction module.

[0093] Specifically, the low-level descriptor is a set of low-level features extracted from the original waveform of the audio data (ie, speech signal) to reflect the basic properties of the audio data, wherein the low-level features may include various metrics in the time domain and the frequency domain.

[0094] Mel-gram spectrogram is a spectrum diagram of audio data, which is used to represent the energy distribution of audio data in two dimensions of time and frequency. The extraction process of Mel-gram spectrogram can be: first, preprocessing operations (such as denoising and standardization) are performed on the audio data, and the preprocessed audio data is divided into several audio frames, each of which can include part of the audio data (for example, each audio frame includes audio data of tens of milliseconds, etc.); secondly, each audio frame is multiplied by a window function (such as a Hamming window, etc.) to reduce spectrum leakage, and each windowed audio frame is fast Fourier transformed to obtain the spectrum of each windowed audio frame. That is, the time domain signal is converted into a frequency domain signal; again, the frequency spectrum of each windowed audio frame is squared to obtain a power spectrum, and the power spectrum is mapped from a linear frequency scale to a Mel frequency scale to obtain a mapped power spectrum, wherein the Mel scale reflects the human ear's perception characteristics of frequency, that is, the higher the frequency, the lower the human ear's perception resolution; finally, a group of triangular filters are set on the Mel scale, and the mapped power spectrum of each audio frame is passed through the Mel filter group to obtain the output energy of each filter to obtain a Mel spectrogram, wherein the center frequency of the triangular filter is distributed according to the Mel scale, and the triangular filter is used to simulate the human ear's perception characteristics of different frequencies.

[0095] The wav2vec feature is a feature vector extracted from the audio data by the wav2vec model. The wav2vec model is a speech recognition model that uses the structure of an autoencoder to learn to extract features from the audio waveform to obtain a high-dimensional feature vector. The high-dimensional feature vector carries the speech information in the audio data. The wav2vec model uses a high-dimensional feature vector as input and trains the autoencoder in an unsupervised manner to reconstruct the original audio waveform, so that the autoencoder can learn a more informative feature representation. Finally, the wav2vec feature representation obtained from the encoder part of the autoencoder can be used for subsequent speech recognition related tasks. The wav2vec feature contains the structural information of the original audio signal and can effectively represent the semantic information of the audio signal.

[0096] Furthermore, after obtaining the low-level descriptor, mel-language spectrogram and wav2vec features, the audio feature extraction module performs feature extraction on the low-level descriptor, mel-language spectrogram and wav2vec features to obtain a low-level descriptor vector, a mel-language spectrogram vector and a wav2vec vector. Specifically, Figure 4 As shown, the audio feature extraction module includes a low-level descriptor unit (denoted as OpenSmile), a Mel-language spectrogram unit (denoted as NetVLAD) and a wav2vec unit (denoted as wav2vec 2.0). The low-level descriptor unit is used to extract a low-level descriptor vector x LLD, the Mel spectrogram unit is used to extract the Mel spectrogram vector x mel , the wav2vec unit is used to extract the wav2vec vector x wav2vec .

[0097] After obtaining the low-level descriptor vector, the Mel-spectrogram vector and the wav2vec vector, the low-level descriptor vector, the Mel-spectrogram vector and the wav2vec vector can be directly input into the multi-level audio feature interaction module, and the low-level descriptor vector, the Mel-spectrogram vector and the wav2vec vector are fused through the multi-level audio feature interaction module. Alternatively, the low-level descriptor vector, the Mel-spectrogram vector and the wav2vec vector can be respectively passed through the BiLSTM module, and the timing information of the low-level descriptor vector, the Mel-spectrogram vector and the wav2vec vector can be captured through the BiLSTM module, so as to effectively extract the timing features of the low-level descriptor vector, the Mel-spectrogram vector and the wav2vec vector.

[0098] In a typical implementation, Figure 4 As shown, after obtaining the low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector, the low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector are first respectively passed through the BiLSTM module, and then the low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector that have passed through the BiLSTM module are input into the multi-level audio feature interaction module, and feature fusion is performed through the multi-level audio feature interaction module. In other words, the audio branch includes an audio feature extraction module of a low-level descriptor unit, a Mel-language spectrogram unit and a wav2vec unit, three BiLSTM modules and a multi-level audio feature interaction module, the low-level descriptor unit, the Mel-language spectrogram unit and the wav2vec unit are respectively connected to a BiLSTM module, and the three BiLSTM modules are all connected to the multi-level audio feature interaction module.

[0099] Furthermore, the multi-level audio feature interaction module is used to fuse the low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector input therein to obtain the audio features corresponding to the audio data. Specifically, the step of inputting the low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector into the multi-level audio feature interaction module in the audio branch and outputting the audio features corresponding to the audio data through the multi-level audio feature interaction module specifically includes:

[0100] constructing a first query vector based on the low-level descriptor vector, and constructing a first key vector and a first value vector based on the Mel-language spectrogram vector;

[0101] Determine, based on the first query vector, the first key vector, and the first value vector, a fused audio feature through a first cross attention mechanism unit in a multi-level audio feature interaction module;

[0102] Constructing a second query vector based on the fused audio features, and constructing a second key vector and a second value vector based on the wav2vec vector;

[0103] Based on the second query vector, the second key vector and the second value vector, the audio features corresponding to the audio data are determined through a second cross-attention mechanism unit in a multi-level audio feature interaction module.

[0104] Specifically, Figure 4 As shown, the multi-level audio feature interaction module includes a first cross-attention mechanism unit and a second cross-attention mechanism unit, and the first cross-attention mechanism unit and the second cross-attention mechanism unit are connected, wherein the first cross-attention mechanism unit is used to cross-learn the low-level descriptor vector and the Mel-spectrogram vector to obtain a fused audio feature of the low-level descriptor vector and the Mel-spectrogram vector, and the second cross-attention mechanism unit is used to cross-learn the fused audio feature with the wav2vec vector to fuse the low-level descriptor vector, the Mel-spectrogram vector and the wav2vec vector to obtain the audio feature corresponding to the audio data.

[0105] Exemplarily, the multi-level audio feature interaction module MAFIM may include two two-level feature fusion modules DFFM, each DFFM is a cross attention mechanism module FBP based on bilinear pooling gating, wherein the FBP gating unit (FBP Gate) accepts Q and K from two different levels of audio features as input, first linearly transforms the input Q and K, then performs matrix multiplication and accumulation of the linearly transformed feature vectors and normalizes the accumulated results, and finally generates a gating signal using a signal function. The cross attention module accepts Q from the first input and K and V from the second input for attention mechanism calculation. Through FBP gating, the redundancy between audio features at different levels can be effectively filtered to achieve effective fusion of multi-level audio features.

[0106] S40: inputting the text data into a text branch in a trained depression screening model, and extracting text features corresponding to the text data through the text branch.

[0107] Specifically, the text branch is used to extract features from text data. For example, Figure 5As shown, the text branch includes a BERT module and a FC module. The BERT module is a pre-trained model. The BERT module is used to extract features from text data. The FC module is used to perform a full connection operation on the intermediate text features extracted by the BERT model to obtain text features corresponding to the text data.

[0108] Exemplarily, the step of inputting the text data into a text branch in a trained depression screening model and extracting text features corresponding to the text data through the text branch specifically includes:

[0109] Inputting the text data into the BERT module in the text branch, and extracting intermediate text features through the BERT model;

[0110] The intermediate text features are input into the FC module in the text branch, and the text features corresponding to the text data are determined by the FC module.

[0111] Specifically, BERT uses the Encoder architecture in Transformer, which is a bidirectional language model that can consider the previous and next information of the input sequence at the same time, so as to better capture contextual information. For example, when processing a certain word, BERT can use the previous word and the following word information at the same time. The input of the BERT model is the representation corresponding to each token, which consists of three parts, namely word tokens, segmentation tokens, and position embeddings. Word tokens refer to dividing words into a limited set of common sub-word units, which can achieve a compromise between the effectiveness of words and the flexibility of characters. Segmentation tokens are used to distinguish different sentence tokens. Position embedding refers to assigning a position vector to each word or sub-word unit in the input sequence to represent their relative or absolute position in the sequence. This position information is very important for the model to capture the order and dependency between words.

[0112] Regarding the processing flow of the BERT model on text data, specifically, BERT will first perform a word segmentation operation on the input text data, using a word segmenter to cut the input text data into words or subwords; then, in order to convert the text data into a numerical representation, BERT will build a vocabulary (for example, by performing statistics on the training data, etc.), which contains all possible words or subwords; next, BERT will perform an encoding operation, and for each word or subword in the vocabulary, BERT will encode it into a unique integer by using the index in the vocabulary; then, BERT pads or truncates the text so that the model obtains a fixed-length input. Padding means adding special padding marks at the end of the text when the text length is insufficient to make the text reach the specified length. Truncation means deleting part of the text when the text length exceeds the specified length to make it meet the length requirement. After the processing is completed, the BERT model will perform position embedding based on the input text data, and finally determine the intermediate text features through the Encoder in the Transformer based on the position embedding and word embedding.

[0113] The FC module may include one or more fully connected layers, through which the intermediate text features are fully connected to obtain text features corresponding to the text data.

[0114] Of course, it is worth mentioning that the text branch can also adopt other model structures, such as convolution modules, etc., which are not explained here one by one. As long as the model structure can extract features from text data, it can be used.

[0115] S50, inputting the video features, audio features and text features into a fusion module in the depression screening model, and determining fusion features through a cross-attention mechanism in the fusion module.

[0116] Specifically, the fusion module cross-learns the video features, audio features, and text features by adopting a cross-attention mechanism to learn the feature information carried by the video features, audio features, and text features, so that the fusion features carry rich feature information. Among them, when cross-learning the video features, audio features, and text features by using a cross-attention mechanism, a query vector can be constructed based on the text features, a value vector and a key vector can be constructed based on the video features, a query vector can be constructed based on the text features, and a value vector and a key vector can be constructed based on the text features and audio features. A query vector can also be constructed based on the video features, and a value vector and a key vector can be constructed based on the text features and audio features. A query vector can also be constructed based on the audio features, and a value vector and a key vector can be constructed based on the text features and video features.

[0117] In an embodiment of the present application, a query vector is constructed based on text features, and a value vector and a key vector are constructed based on video features and audio features to achieve a text-feature-centric approach. The video features and audio features are fused with the text features through a cross-attention mechanism to obtain fused features. In this way, multi-modal feature fusion can be achieved with the text modality that can effectively express depressive emotions as the main factor, and the audio and video modalities as auxiliary interactions, which can effectively improve the performance of the model.

[0118] Based on this, Figure 6 As shown, the inputting of the video features, audio features and text features into the fusion module in the depression screening model and determining the fusion features through the cross attention mechanism in the fusion module specifically include:

[0119] Constructing a text query vector based on the text features;

[0120] Constructing a video key vector and a video value vector based on the video features, and constructing an audio key vector and an audio value vector based on the audio features;

[0121] Determining a first fused feature through a cross attention mechanism based on the text query vector, the video key vector, and the video value vector;

[0122] Determining a second fused feature through a cross attention mechanism based on the text query vector, the audio key vector, and the audio value vector;

[0123] A fusion feature is determined based on the first fusion feature and the second fusion feature.

[0124] Specifically, the text features are mapped into a query vector Q t , mapping the audio features into the audio key vector K a and the audio value vector V a , map the video features to the video key vector K v and the video value vector V v , where v, a, and t represent video, audio, and text, respectively. This can extract important information from text features, audio features, and video features, making it easier for the neural network to learn more advanced feature representations. Next, calculate Q t The transpose and K a The product of Q t The transpose and K v The product of , respectively, obtains the vector similarity matrix between the text mode and the audio mode and the vector similarity matrix between the text mode and the video mode, and then normalizes the similarity matrix through the Softmax function to obtain the weight matrix W ta and W tv , W ta and Wtv They represent the weight matrix of the interaction between text features and audio features and the weight matrix of the interaction between text features and video features. Finally, the weight matrix W ta and W tv With V a and V v Multiply to get X ta and X tv , and obtain the text modality features after audio modality interaction and the text modality features after video modality interaction respectively. In this way, the multi-modal feature fusion can be realized with the text modality that can effectively express depressive emotions as the leading factor, and the audio and video modalities as auxiliary interactions. This can not only make the video features, audio features and text features fully integrated, so that the model can learn more information, but also make the obtained fusion features better reflect depressive emotions, thereby improving the prediction accuracy of depression scale scores.

[0125] S60: Input the fusion feature into a prediction module in the depression screening model, and determine the depression scale score of the data to be screened through the prediction module.

[0126] Specifically, the prediction module includes two Dropout layers, two Linear layers, a ReLU layer and a Softmax layer. After the fusion features are input, they first pass through a Dropout layer with p=0.3. The function of the Dropout layer is to randomly ignore a part of neurons with a certain probability p during the model training process to prevent the model from overfitting. Then, through a Linear layer and a ReLU layer, ReLU can introduce nonlinearity in the neural network, so that the model can better fit the nonlinear function. And because ReLU has sparsity, it can suppress some neurons to zero, so that the model can better mine relevant features and fit the training data. Then, through a Dropout layer with p=0.3, a Linear layer is used to map the feature dimension to two dimensions, because the model performs a binary classification problem. Finally, the predicted value is normalized by the Softmax function to obtain the final predicted probability, and the predicted probability can be used to determine the depression scale score of the predicted sample.

[0127] To further illustrate the effect of this embodiment, this embodiment adopts the following Figure 2The multi-branch fusion model shown in the figure was used to verify the performance of depression screening on a public depression dataset containing 189 data (130 healthy subjects and 59 depressed subjects). When screening based on video data, the MAE value was 5.33 and the RMSE was 5.80; when screening based on audio data, the MAE value was 5.56 and the RMSE was 5.91; when screening based on text data, the MAE value was 4.96 and the RMSE was 5.54; when screening by integrating video, audio and text trimodal data, the MAE value was 4.32 and the RMSE was 4.55, proving that multimodal fusion can effectively improve the performance of depression screening.

[0128] In summary, this embodiment provides a depression screening method based on multimodal data, the method comprising inputting the video data into the video branch in the depression screening model, and determining the video features corresponding to the video data through the video branch; inputting the audio data into the audio branch in the depression screening model, and determining the audio features corresponding to the audio data through the audio branch; inputting the text data into the text branch in the depression screening model, and determining the text features corresponding to the text data through the text branch; inputting the video features, the audio features and the text features into a fusion module in the depression screening model, and determining the fusion features of the video features, the audio features and the text features through the fusion module; inputting the fusion features into the prediction module in the depression screening model, and determining the depression scale score of the data to be screened through the prediction module. The present application processes the features provided by video data, the features provided by audio data and the features provided by text data through video branches, audio branches and text branches respectively to perform depression screening. It can use different branches to fully extract the features of a single modality, and then use the fusion module to fully fuse the features of the three modalities. On the one hand, it can reduce feature missing, and on the other hand, it can be fused through the cross-attention mechanism to avoid the influence of redundant features. This not only improves the prediction accuracy, but also realizes an end-to-end model.

[0129] Based on the above-mentioned depression screening method based on multimodal data, this embodiment provides a depression screening device based on multimodal data, such as Figure 8 As shown, the depression screening device based on multimodal data specifically includes:

[0130] The acquisition module 100 is used to acquire the data to be screened, wherein the data to be screened includes video data, audio data and text data;

[0131] A video feature extraction module 200, configured to input the video data into a video branch in a trained depression screening model, and extract video features corresponding to the video data through the video branch;

[0132] An audio feature extraction module 300, configured to input the audio data into an audio branch in a trained depression screening model, and extract audio features corresponding to the audio data through the audio branch;

[0133] A text feature extraction module 400, for inputting the text data into a text branch in a trained depression screening model, and extracting text features corresponding to the text data through the text branch;

[0134] A fusion module 500, which is used to input the video features, audio features and text features into the fusion module in the depression screening model, and determine the fusion features through the cross attention mechanism in the fusion module;

[0135] The prediction module 600 is used to input the fusion feature into the prediction module in the depression screening model, and determine the depression scale score of the data to be screened through the prediction module.

[0136] Based on the above-mentioned depression screening method based on multimodal data, this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the depression screening method based on multimodal data as described in the above embodiment.

[0137] Based on the above depression screening method based on multimodal data, the present application also provides a terminal device, such as Figure 8 As shown, it includes at least one processor 20, a display screen 21, and a memory 22, and may also include a communication interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communication interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0138] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0139] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0140] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0141] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A depression screening method based on multimodal data, characterized in that: The depression screening method based on multimodal data specifically includes: Acquiring data to be screened, wherein the data to be screened includes video data, audio data, and text data; Inputting the video data into a video branch in a trained depression screening model, and extracting video features corresponding to the video data through the video branch; Inputting the audio data into an audio branch in a trained depression screening model, and extracting audio features corresponding to the audio data through the audio branch; Inputting the text data into a text branch in a trained depression screening model, and extracting text features corresponding to the text data through the text branch; Inputting the video features, audio features and text features into a fusion module in the depression screening model, and determining fusion features through a cross-attention mechanism in the fusion module; The fusion feature is input into the prediction module in the depression screening model, and the depression scale score of the data to be screened is determined by the prediction module.

2. The depression screening method based on multimodal data according to claim 1, characterized in that: The step of inputting the video data into a video branch of a trained depression screening model and extracting video features corresponding to the video data through the video branch specifically includes: Determining a video facial action unit frame sequence based on the video data; The video facial action unit frame sequence is input into the video feature extraction module in the video branch, and the video feature extraction module performs spatiotemporal feature extraction to obtain video features corresponding to the video data.

3. The depression screening method based on multimodal data according to claim 2, characterized in that: The video feature extraction module for inputting the video facial action unit frame sequence into the video branch specifically includes: For each video facial action unit frame in the video facial action unit frame sequence, extracting spatial features of the video facial action unit frame through a key frame selection module, and generating a one-hot encoding vector based on the spatial features; Performing a dot multiplication of the one-hot encoding vector and the video facial action unit frame to obtain a video facial action unit key frame corresponding to the video facial action unit frame; A sequence of video facial action unit key frames corresponding to all video facial action unit frames is used as a video facial action unit key frame sequence; The video facial action unit key frame sequence is input into the video feature extraction module in the video branch.

4. The depression screening method based on multimodal data according to claim 1, characterized in that: The step of inputting the audio data into an audio branch of a trained depression screening model and extracting audio features corresponding to the audio data through the audio branch specifically includes: Extracting low-level descriptors, mel-language spectrograms, and wav2vec features corresponding to the audio data; Input the low-level descriptor, the Mel-language spectrogram and the wav2vec feature into the audio feature extraction module in the audio branch, and extract the low-level descriptor vector corresponding to the low-level descriptor, the Mel-language spectrogram vector corresponding to the Mel-language spectrogram, and the wav2vec vector corresponding to the wav2vec feature through the audio feature extraction module; The low-level descriptor vector, the Mel-language spectrogram vector and the wav2vec vector are input into a multi-level audio feature interaction module in the audio branch, and the audio features corresponding to the audio data are output through the multi-level audio feature interaction module.

5. The method for screening depression based on multimodal data according to claim 4, characterized in that: The step of inputting the low-level descriptor vector, the Mel-language spectrogram vector, and the wav2vec vector into the multi-level audio feature interaction module in the audio branch, and outputting the audio features corresponding to the audio data through the multi-level audio feature interaction module specifically includes: constructing a first query vector based on the low-level descriptor vector, and constructing a first key vector and a first value vector based on the Mel-language spectrogram vector; Determine, based on the first query vector, the first key vector, and the first value vector, a fused audio feature through a first cross attention mechanism unit in a multi-level audio feature interaction module; Constructing a second query vector based on the fused audio features, and constructing a second key vector and a second value vector based on the wav2vec vector; Based on the second query vector, the second key vector and the second value vector, the audio features corresponding to the audio data are determined through a second cross-attention mechanism unit in a multi-level audio feature interaction module.

6. The method for screening depression based on multimodal data according to claim 1, characterized in that: The step of inputting the text data into a text branch in a trained depression screening model and extracting text features corresponding to the text data through the text branch specifically includes: Input the text data into the BERT module in the text branch, and extract intermediate text features through the BERT model; The intermediate text features are input into the FC module in the text branch, and the text features corresponding to the text data are determined by the FC module.

7. The depression screening method based on multimodal data according to claim 1, characterized in that: Inputting the video features, audio features and text features into a fusion module in the depression screening model, and determining the fusion features through the cross attention mechanism in the fusion module specifically includes: Constructing a text query vector based on the text features; Constructing a video key vector and a video value vector based on the video features, and constructing an audio key vector and an audio value vector based on the audio features; Determining a first fused feature through a cross attention mechanism based on the text query vector, the video key vector, and the video value vector; Determining a second fused feature through a cross attention mechanism based on the text query vector, the audio key vector, and the audio value vector; A fusion feature is determined based on the first fusion feature and the second fusion feature.

8. A depression screening device based on multimodal data, characterized in that: The depression screening device based on multimodal data specifically includes: An acquisition module, used for acquiring data to be screened, wherein the data to be screened includes video data, audio data and text data; A video feature extraction module, used for inputting the video data into a video branch in a trained depression screening model, and extracting video features corresponding to the video data through the video branch; An audio feature extraction module, used for inputting the audio data into an audio branch in a trained depression screening model, and extracting audio features corresponding to the audio data through the audio branch; A text feature extraction module, used for inputting the text data into a text branch in a trained depression screening model, and extracting text features corresponding to the text data through the text branch; A fusion module, used for inputting the video features, audio features and text features into the fusion module in the depression screening model, and determining fusion features through a cross-attention mechanism in the fusion module; A prediction module is used to input the fusion feature into the prediction module in the depression screening model, and determine the depression scale score of the data to be screened through the prediction module.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the depression screening method based on multimodal data as described in any one of claims 1-7.

10. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps of the depression screening method based on multimodal data as described in any one of claims 1 to 7 are implemented.

Images