End-to-end multi-modal fusion depression screening method, device and equipment

Abstract

The application discloses an end-to-end multi-modal fusion depression screening method, device and equipment, the method comprises the following steps: acquiring user data of a testee, wherein the user data comprises video data, audio data and text data; inputting the user data into a feature extraction module of a trained multi-branch fusion model, extracting video features, audio features and text features through the feature extraction module; inputting the video features, the audio features and the text features into a fusion module of the multi-branch fusion model, and acquiring fusion features through the fusion module; inputting the fusion features into a prediction module of the multi-branch fusion model, and outputting depression categories through the prediction module. By adopting the multi-branch fusion model, the video features, the audio features and the text features are fused to obtain the depression categories, so that the prediction efficiency and the accuracy of the depression screening are improved.

Description

Technical Field

[0001] This application relates to the field of medical technology, and in particular to an end-to-end multimodal fusion-based method, device, and equipment for screening depression. Background Technology

[0002] Depression is a common mental illness, clinically characterized by persistent and prolonged low mood and slowed thinking. Currently, depression screening generally relies on self-rating depression scales such as the PHQ-9 (Patient Health Questionnaire-9) and BDI-II (Beck Depression Inventory-II) and physicians' clinical experience. This places a high emphasis on the physician's medical competence and experience, which can affect the objectivity and accuracy of depression screening predictions.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide an end-to-end multimodal fusion-based method, device and equipment for screening depression, which addresses the shortcomings of the existing technology.

[0005] To address the aforementioned technical problems, the first aspect of this application provides an end-to-end multimodal fusion-based depression screening method, wherein the end-to-end multimodal fusion-based depression screening method specifically includes:

[0006] Obtain user data of the test subjects, wherein the user data includes video data, audio data, and text data;

[0007] The user data is input into the feature extraction module of the trained multi-branch fusion model, and video features, audio features, and text features are extracted through the feature extraction module.

[0008] The video features, audio features, and text features are input into the fusion module of the multi-branch fusion model, and the fusion module obtains the fused features.

[0009] The fusion features are input into the prediction module of the multi-branch fusion model, and the prediction module outputs the depression category.

[0010] The feature extraction module comprises a video branch, an audio branch, and a text branch, which are implemented in parallel. The video branch includes a video feature extraction unit, a FA unit, a multiplier, a BiLSTM unit, and a multi-head attention unit. The video feature extraction unit is connected to the FA unit and the multiplier, the FA unit is connected to the multiplier, and the multiplier, the BiLSTM unit, and the multi-head attention unit are connected in sequence.

[0011] In one implementation, the video feature extraction process specifically includes:

[0012] For each video frame in the video data, determine the local binary pattern map of the video frame;

[0013] The local binary pattern map and the video frame are input into the video feature extraction unit, and the video feature extraction unit determines the video spatial feature vector.

[0014] The video space feature vector is input into the FA unit, and the FA unit performs importance weighting on the video space feature vector to obtain a weighted feature vector;

[0015] The weighted feature vector and the video spatial feature vector are multiplied and then input into the BiLSTM unit, which captures the temporal features of the video.

[0016] The video temporal features are input into the multi-head attention unit, and the video features are determined by the multi-head attention unit.

[0017] In one implementation, the video feature extraction unit includes a residual subunit, a Transformer subunit, and a feature fusion subunit; the step of inputting the local binary pattern map and the video frame into the video feature extraction unit, and determining the video spatial feature vector through the video feature extraction unit specifically includes:

[0018] The local binary pattern image is input into the residual sub-unit, and local facial features of the video are extracted through the residual sub-unit;

[0019] The video frame is input into the Transformer subunit, and the global facial features of the video are extracted through the Transformer subunit.

[0020] The local facial features and global facial features in the video are input into the fusion subunit, and the video spatial feature vector is determined by the fusion subunit.

[0021] In one implementation, the audio branch includes a NetVLAD unit, a GRU unit, and a multi-head attention unit; the audio feature extraction process specifically includes:

[0022] Convert the audio data into a Mel language spectrogram;

[0023] The Mel speech spectrogram is input into a NetVLAD unit, and the Mel speech spectrogram vector is determined by the NetVLAD unit;

[0024] The Mel spectrogram vector is input into the GRU unit, and the audio temporal feature vector is extracted through the GRU unit;

[0025] The audio temporal feature vector is input into the multi-head attention unit, and the audio features are determined by the multi-head attention unit.

[0026] In one implementation, the text branch includes a BERT unit and a multi-head attention unit; the text feature extraction process specifically includes:

[0027] The text data is input into the BERT unit, and the initial text features are extracted through the BERT unit.

[0028] The initial text features are input into a multi-head attention unit, and the text features are determined by the multi-head attention unit.

[0029] In one implementation, the process of determining the fusion feature specifically includes:

[0030] The first query vector, the first key vector, and the first value vector are determined based on video features; the second query vector, the second key vector, and the second value vector are determined based on audio features; and the third query vector, the third key vector, and the third value vector are determined based on text features.

[0031] The first query vector, the second query vector, and the third query vector are concatenated into a target query vector; the first key vector, the second key vector, and the third key vector are concatenated into a target key vector; and the first value vector, the second value vector, and the third value vector are concatenated into a target value vector.

[0032] Based on the target query vector and the target key vector, a vector correlation matrix is ​​determined, and the vector correlation matrix is ​​multiplied with the target value vector to obtain candidate fusion features;

[0033] The candidate fusion feature is added to the target value vector to obtain the fusion feature.

[0034] In one implementation, obtaining the test subject's user data specifically includes:

[0035] A data acquisition paradigm is determined, and the data is collected from the test subjects using the acquired paradigm to obtain initial video data.

[0036] The initial video data is cropped to obtain the final video data;

[0037] The initial video data is subjected to audio separation to obtain audio data, and the audio data is then converted into text data.

[0038] The video data, audio data, and text data are used as the user data of the test subject.

[0039] A second aspect of this application provides an end-to-end multimodal fusion-based depression screening device, wherein the device specifically includes:

[0040] The acquisition module is used to acquire the user data of the test subject, wherein the user data includes video data, audio data and text data;

[0041] The control module is used to input the user data into the feature extraction module of the trained multi-branch fusion model, and extract video features, audio features, and text features through the feature extraction module; input the video features, audio features, and text features into the fusion module of the multi-branch fusion model, and obtain fused features through the fusion module; input the fused features into the prediction module of the multi-branch fusion model, and output the depression category through the prediction module.

[0042] The feature extraction module comprises a video branch, an audio branch, and a text branch, which are implemented in parallel. The video branch includes a video feature extraction unit, a FA unit, a multiplier, a BiLSTM unit, and a multi-head attention unit. The video feature extraction unit is connected to the FA unit and the multiplier, the FA unit is connected to the multiplier, and the multiplier, the BiLSTM unit, and the multi-head attention unit are connected in sequence.

[0043] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the end-to-end multimodal fusion depression screening method as described above.

[0044] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory;

[0045] The memory stores a computer-readable program that can be executed by the processor;

[0046] When the processor executes the computer-readable program, it implements the steps in any of the above-described end-to-end multimodal fusion-based depression screening methods.

[0047] Beneficial effects: This application adopts a multi-branch fusion model to fuse video features, audio features and text features to obtain depression categories, thereby improving the predictive efficiency and accuracy of depression screening. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart of an end-to-end multimodal fusion-based depression screening method provided in this application embodiment.

[0050] Figure 2 This is an architecture diagram of the multi-branch fusion model in the end-to-end multimodal fusion depression screening method provided in the embodiments of this application.

[0051] Figure 3 This is a video branch architecture diagram of the multi-branch fusion model in the end-to-end multimodal fusion depression screening method provided in the embodiments of this application.

[0052] Figure 4 This is a diagram illustrating the video spatial feature vector extraction architecture of the video branch in the multi-branch fusion model of the end-to-end multimodal fusion depression screening method provided in this application embodiment.

[0053] Figure 5 The audio branch architecture diagram of the multi-branch fusion model in the end-to-end multimodal fusion depression screening method provided in the embodiments of this application.

[0054] Figure 6 The text branch architecture diagram of the multi-branch fusion model in the end-to-end multimodal fusion depression screening method provided in the embodiments of this application.

[0055] Figure 7 This is an architecture diagram of the fusion module in the multi-branch fusion model of the end-to-end multimodal fusion depression screening method provided in the embodiments of this application.

[0056] Figure 8 This is a schematic diagram of the end-to-end multimodal fusion depression screening device provided in an embodiment of this application.

[0057] Figure 9A schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0058] This application provides an end-to-end multimodal fusion-based method, apparatus, and device for screening depression. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0059] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0061] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0062] Research has shown that depression is a common mental illness, clinically characterized by persistent and prolonged low mood and slowed thinking. Currently, depression screening largely relies on self-rating depression scales such as the PHQ-9 (Patient Health Questionnaire-9) and BDI-II (Beck Depression Inventory-II), as well as physicians' clinical experience. This high dependence on physicians' medical skills and experience can affect the objectivity and accuracy of depression screening predictions. Therefore, the efficiency and accuracy of predicting depression categories need to be improved.

[0063] With the development of artificial intelligence technology, deep learning technology has been widely used in the field of depression screening. Neural network models, by fusing multimodal information, can screen for potential depression patients from the population after proper training, demonstrating good performance. However, most current multimodal depression screening methods do not sufficiently extract single-modal information and fuse multimodal information. For example, in video modalities, they do not simultaneously focus on local and global facial features, and in modal fusion, they only perform simple stitching. In addition, they do not achieve end-to-end input and output, which makes it difficult for the model to learn more effective information from the raw data on its own.

[0064] To address the aforementioned issues, this application embodiment acquires user data from test subjects, including video data, audio data, and text data. The user data is then input into a feature extraction module within a trained multi-branch fusion model to extract video, audio, and text features. These features are then input into a fusion module within the multi-branch fusion model to obtain fused features. Finally, the fused features are input into a prediction module within the multi-branch fusion model, which outputs a depression category. This application, by employing a multi-branch fusion model to fuse video, audio, and text features to obtain a depression category, improves the predictive efficiency and accuracy of depression screening.

[0065] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0066] This embodiment provides an end-to-end multimodal fusion-based method for screening depression, such as... Figure 1 As shown, the method includes:

[0067] S10. Obtain the user data of the test subject.

[0068] Specifically, user data refers to multimodal data reflecting the user's state, obtained through video capture of test subjects. This user data can include video data, audio data, and file data. Video data consists of image information captured during the test, with each video frame containing the test subject's facial region, reflecting facial information such as changes in facial expressions. Audio data consists of speech information captured during the test, and text data consists of the corresponding text information. In other words, video, audio, and text data are collected during a single test.

[0069] In one implementation of this embodiment, obtaining the test subject's user data specifically includes:

[0070] A data acquisition paradigm is determined, and the data is collected from the test subjects using the acquired paradigm to obtain initial video data.

[0071] The initial video data is cropped to obtain the final video data;

[0072] The initial video data is subjected to audio separation to obtain audio data, and the audio data is then converted into text data.

[0073] The video data, audio data, and text data are used as the user data of the test subject.

[0074] Specifically, the data acquisition paradigm can be pre-set, either directly adopting an existing data acquisition paradigm or improving upon an existing one based on actual needs. The paradigm includes one or more data acquisition tasks, setting the execution order and requirements for each task. Testers are evaluated according to the paradigm, and the testing process is video-captured to obtain initial video data. In other words, the initial video data consists of the raw video recorded during the execution of each data acquisition task according to the paradigm. For example, the data acquisition task could be a reading aloud task, an interview task, or both.

[0075] Each video frame in the video data corresponds to an initial video frame in the initial video data, and this video frame is obtained by cropping the facial region of its corresponding initial video frame. In other words, each video frame in the video data is a facial region image. When cropping the facial region of the initial video data, facial region recognition can be performed using traditional edge recognition methods, facial landmark recognition methods, or a trained facial region recognition model.

[0076] In this embodiment, the facial region is determined through facial landmark recognition. Accordingly, the facial region cropping of the initial video data specifically includes:

[0077] Extract the coordinates of facial key points and crop out the facial area based on these coordinates.

[0078] Specifically, the facial key point coordinates can be extracted using a trained facial key point recognition model, for example, using the MediaPipe Face Mesh toolkit. After obtaining the facial key point coordinates, the facial region can be cropped using OpenCV technology, and the size of the facial region can be adjusted and saved. The size of the facial region can be adjusted according to actual needs, and the facial region can be saved as an image file in JPG, PNG, or other formats. In this embodiment, the facial key point coordinates can be the coordinates of 468 3D key points on the face, the size of the facial region can be set to 224×224, and the facial region can be saved as a JPG image file.

[0079] In some implementations, the audio separation of the initial video data to obtain audio data specifically includes:

[0080] The initial video data is timestamped to obtain the timestamped data.

[0081] The audio of the test subject is extracted from the initial video data based on the labeled timestamps to obtain audio data.

[0082] Specifically, the timestamp annotations reflect the audio data corresponding to the test subject. That is, audio data segments of the test subject in the initial video data are selected by annotating the timestamps, and the order of the selected audio data segments can be determined based on the timestamp annotations. Then, the selected audio data segments are combined according to the order of the timestamp annotations to obtain the audio data. The timestamp annotations include several timestamp pairs, each pair including a start timestamp and an end timestamp. The video segments between each timestamp pair contain the test subject's speech, and based on the timestamp pairs, all video segments include the speech of all test subjects in the original video data. Furthermore, the timestamp annotations can be done manually or through semantic recognition using a speech recognition model. The audio data can be stored as a WAV file or other audio format files.

[0083] In some implementations, the text conversion of the audio data to obtain text data specifically includes:

[0084] The audio data was converted into initial text data using a speech-to-text conversion tool.

[0085] The initial text data is proofread to obtain the text data.

[0086] Specifically, the initial text data is text directly obtained using a phonetic-to-text conversion tool; therefore, the obtained text may contain many errors. Based on this, this embodiment employs manual proofreading to revise the initial text data. For example, typos are corrected while listening to the audio, and missing modal particles are added to preserve complete text information and ensure the accuracy and integrity of the text data. The proofread text is then used as the final text data. The text data can be stored in formats such as CSV.

[0087] S20. Input the user data into the feature extraction module of the trained multi-branch fusion model, and extract video features, audio features and text features through the feature extraction module.

[0088] Specifically, such as Figure 2 As shown, the multi-branch fusion model is a trained network model used to predict the depression category of a test subject. The multi-branch fusion model includes a feature extraction module, a fusion module, and a prediction module. The feature extraction module is connected to the fusion module, and the fusion module is connected to the prediction module. Specifically, the feature extraction module extracts video features, audio features, and text features; the fusion module determines fused features based on the video features, audio features, and text features; and the prediction module determines the depression category based on the fused features.

[0089] like Figure 2 As shown, the feature extraction module operates in parallel with video, audio, and text branches, all of which are connected to the fusion module. The input to the video branch is video data, the input to the audio branch is audio data, and the input to the text branch is text data. Figure 3 As shown, the video branch includes a video feature extraction unit, an FA unit, a multiplier, a BiLSTM unit, and a multi-head attention unit. The video feature extraction unit is connected to both the FA unit and the multiplier, and the FA unit is connected to the multiplier. The multiplier, the BiLSTM unit, and the multi-head attention unit are connected sequentially. Wherein, as... Figure 3 and 4 As shown, the video feature extraction unit includes a residual subunit, a Transformer subunit, and a feature fusion subunit, both of which are connected to the feature fusion subunit. The FA unit includes a Tanh unit and a Softmax unit, wherein the feature extraction unit is connected to the Tanh unit, the Tanh unit is connected to the Softmax unit, and both the feature extraction unit and the Softmax unit are connected to the multiplier.

[0090] like Figure 5 As shown, the audio branch includes a NetVLAD unit, a GRU unit, and a multi-head attention unit, wherein the NetVLAD unit is connected to the GRU unit, and the GRU unit is connected to the multi-head attention unit.

[0091] like Figure 6 As shown, the text branch includes a BERT unit and a multi-head attention unit, wherein the BERT unit is connected to the multi-head attention unit.

[0092] In one implementation of this application, the video feature extraction process specifically includes:

[0093] S210. For each video frame in the video data, determine the local binary pattern map of the video frame;

[0094] S211. Input the local binary pattern map and the video frame into the video feature extraction unit, and determine the video spatial feature vector through the video feature extraction unit;

[0095] S212. Input the video space feature vector into the FA unit, and perform importance weighting on the video space feature vector through the FA unit to obtain a weighted feature vector;

[0096] S213. The weighted feature vector and the video spatial feature vector are multiplied and then input into the BiLSTM unit to capture the video temporal features through the BiLSTM unit.

[0097] S214. Input the video temporal features into the multi-head attention unit, and determine the video features through the multi-head attention unit.

[0098] Specifically, in step S210, the local binary pattern image is a single-channel image, and the video frames in the video data are RGB images. The video frames in the video data are used to reflect the global features of the face, and the local binary pattern image is used to reflect the local features of the face in the video frames. Furthermore, the image size of the local binary pattern image is the same as the image size of the video frames in the video data. For example, if the video frame is a 224×224 RGB image of the face, then the local binary pattern image is a 224×224 single-channel image. Global facial features can be extracted from the RGB facial image, and local facial features can be extracted from the local binary pattern image.

[0099] Furthermore, the process of obtaining the local binary pattern map can be as follows: convert the pixels of the facial RGB image into a grayscale image, and then extract the local texture features of each pixel based on the grayscale value of each pixel in the grayscale image and the grayscale values ​​of its surrounding neighboring pixels to obtain the local binary pattern map. This embodiment of the application utilizes the rotation invariance, grayscale invariance, and texture features that can reflect facial expressions of the local binary pattern map, which can effectively obtain facial texture features, providing rich facial information for subsequent depression category prediction, thereby improving the accuracy of depression category prediction.

[0100] In step S211, the video spatial feature vector is a feature vector obtained by fusing local and global features of the test subject's face. In this embodiment, inputting the local binary pattern map and the video frame into the video feature extraction unit, and determining the video spatial feature vector through the video feature extraction unit specifically includes:

[0101] The local binary pattern image is input into the residual sub-unit, and local facial features of the video are extracted through the residual sub-unit;

[0102] The video frame is input into the Transformer subunit, and the global facial features of the video are extracted through the Transformer subunit.

[0103] The local facial features and global facial features in the video are input into the fusion subunit, and the video spatial feature vector is determined by the fusion subunit.

[0104] Specifically, the residual subunit includes a ResNet34 model for acquiring local facial features from the video. In this embodiment, 16 frames from the local binary pattern image are input into the ResNet34 model. The ResNet34 model fully extracts facial texture features from the local binary pattern image, and these facial texture features are used as local facial features in the video. The ResNet34 model is a network model with controllable depth and moderate complexity, which can effectively extract image texture features. Furthermore, the ResNet34 model avoids gradient vanishing, gradient exploding, and overfitting problems by using residual connections. In other words, the residual subunit can effectively extract the local facial features from the video.

[0105] The Transformer subunit includes a Swin-Transformer model for acquiring global facial features from a video. The Swin-Transformer model is a Transformer-based network model employing a hierarchical design similar to CNNs. Each layer of the Swin-Transformer model consists of a Patch Merging layer and multiple Transformer Block layers. The Patch Merging layer reduces the image resolution. In other words, the facial RGB image is input into the Swin-Transformer model of the Transformer subunit. The Swin-Transformer model expands the receptive field layer by layer while simultaneously reducing the resolution of the input feature map to extract global facial features. Furthermore, when calculating attention, the Swin-Transformer model also encodes the relative positions of the facial RGB image, meaning it considers the local pixel positional relationships within the facial RGB image. Therefore, the Swin-Transformer model can effectively capture the global facial features of the facial RGB image.

[0106] The fusion subunit includes a convolutional network module. This convolutional network module employs a residual connection module and an attention mechanism. The local facial features and global facial features from the video are input into the convolutional network module. The residual connection module in the convolutional network module adds the residuals of the local facial features and the global facial features to obtain a feature vector. Furthermore, it calculates the importance weights of elements at different positions in the local and global facial features using a Softmax function, and multiplies the importance weights by the feature vector to obtain a video space feature vector. This embodiment of the application, by adding the residuals of the local and global facial features, can further extract effective information while retaining the original feature information, thus reducing feature loss. Multiplying the importance weights by the feature vector highlights important information in the feature vector, which is beneficial for improving the model's screening performance.

[0107] In step S212, since not every input video frame contributes significantly to the screening results during the depression screening process—that is, there are differences in importance among the video frames—it is necessary to distinguish the importance of each video frame. In this embodiment, the weighted feature vector reflects the importance of each video frame. Based on the importance of each video frame, the video space feature vector of each video frame is weighted to obtain a weighted feature vector. Specifically, the video space feature vector is input into the FA unit, and the FA unit performs importance weighting on the video space feature vector to obtain the weighted feature vector, specifically including:

[0108] The video space feature vector is sequentially activated by the Tanh activation function and the Softmax activation function to obtain the importance weight score of the video space feature vector.

[0109] The importance weight score and the video spatial feature vector are input into a multiplier to obtain a weighted feature vector.

[0110] This application embodiment, by weighting the video space feature vectors according to their importance, can better highlight the features of key video frames and weaken the influence of non-key video frames on the output of the multi-branch fusion model, so that the multi-branch fusion model can mainly extract effective features from key video frames.

[0111] In step S213, the video temporal features are the temporal features of the test subject's facial expression changes in each video frame. These video temporal features are determined by multiplying the weighted feature vectors and video space feature vectors of each video frame element-wise using a multiplier, and then inputting the result into a BiLSTM unit. In other words, the BiLSTM unit further extracts the temporal features of the test subject's facial expression changes in each video frame based on the weighted feature vectors and video space feature vectors of each video frame, thus obtaining the video temporal features. Since the BiLSTM unit can capture the bidirectional dependencies between feature vectors and effectively solve the long-term dependency problem, this embodiment of the application, by employing the BiLSTM unit, can effectively extract the temporal information of the test subject's facial expression changes and ensure that the extracted information is not lost due to the continuous training of the multi-branch fusion model.

[0112] In step S214, the multi-head attention unit is a combination of multiple single-head attention units. Each single-head attention unit focuses only on different parts of the input video temporal features, and then their outputs are merged to form a final attention score. The video features are determined based on this attention score. This embodiment of the application, by employing the multi-head attention unit, can improve the ability of the multi-branch fusion model to capture long sequence information, enabling the multi-branch fusion model to better handle complex input feature sequences.

[0113] Furthermore, in this embodiment of the application, for ease of understanding, the video feature extraction process can be represented as follows:

[0114] MHA(BiLSTM(FA(RSTDBN(Img_face,Img_lbp)))),

[0115] Wherein, Img_face represents the video frame, Img_lbp represents the local binary pattern graph, RSTDBN represents the video feature extraction unit, FA represents the FA unit, BiLSTM represents the BiLSTM unit, and MHA represents the multi-head attention unit.

[0116] In one implementation of this application, the audio feature extraction process specifically includes:

[0117] S220. Convert the audio data into a Mel language spectrogram;

[0118] S221. Input the Mel speech spectrogram into the NetVLAD unit, and determine the Mel speech spectrogram vector through the NetVLAD unit;

[0119] S222. Input the Mel spectrogram vector into the GRU unit, and extract the audio temporal feature vector through the GRU unit;

[0120] S223. Input the audio temporal feature vector into the multi-head attention unit, and determine the audio features through the multi-head attention unit.

[0121] Specifically, in step S220, the Mel spectrogram is used to extract the audio features of the test subject, and can be obtained by data conversion of the audio data. Before data conversion of the audio data, data preprocessing is required, such as denoising and standardization. After data preprocessing, converting the audio data into a Mel spectrogram specifically includes:

[0122] The audio data is sequentially subjected to frame segmentation, windowing, and fast Fourier transform to obtain the audio spectrum;

[0123] Squaring the audio spectrum yields the audio power spectrum.

[0124] The audio power spectrum is mapped to a Mel scale to obtain a Mel filter bank;

[0125] The Mel spectrogram is obtained by passing the audio power spectrum through a Mel filter bank.

[0126] Specifically, the framing operation involves dividing the preprocessed audio data into many small frames, each typically containing tens of milliseconds of audio data, to obtain framed audio data. The framed audio data is a sequence of audio frames after the framing operation. The windowing operation multiplies the framed audio data of each audio frame by a window function, typically a Hamming window. The purpose of the windowing operation is to reduce spectral leakage. The purpose of the Fast Fourier Transform is to convert the time-domain signal into a frequency-domain signal.

[0127] Furthermore, mapping the audio power spectrum to a Mel scale involves mapping a linear audio power scale to a Mel frequency scale. The Mel scale reflects the human ear's frequency perception characteristics; that is, the higher the frequency, the lower the perceptual resolution of the human ear. By setting a set of triangular filters on the Mel scale, with the center frequency distributed according to the Mel scale, a Mel filter bank is obtained. This Mel filter bank, through the triangular filters, can simulate the human ear's perceptual characteristics of different audio frequencies. Passing the audio power spectrum of each audio frame through the Mel filter bank allows the acquisition of the output energy of each filter, and these output energies can constitute a Mel spectrogram.

[0128] In step S221, the NetVLAD unit is a convolutional neural network architecture for weakly supervised learning scene recognition, improved from VLAD, used to further extract the Mel spectrogram vector. Specifically, the NetVLAD unit includes a K-means clustering algorithm to cluster the Mel spectrogram to achieve dimensionality reduction, thereby obtaining audio feature vectors. Each audio feature vector is normalized individually, and all audio feature vectors are then normalized using the overall L2 norm to obtain the Mel spectrogram vector. This embodiment of the application processes the Mel spectrogram using the NetVLAD unit, which can unify the dimensionality of the Mel spectrogram vector and effectively represent the global information of the Mel spectrogram.

[0129] In step S222, the GRU unit is a type of RNN network model that can effectively suppress gradient vanishing or gradient exploding while capturing semantic associations in long sequences. The GRU unit works by introducing an update gate and a reset gate to selectively pass past state information to the current state. The update gate determines whether to pass past state information to the current state, with a value range of [0,1]. If the update gate is close to 0, the contribution of the past state to the current state will be small; if the update gate is close to 1, the contribution of the past state to the current state will be large. The reset gate determines whether to ignore past states, also with a value range of [0,1]. If the reset gate is close to 0, the past state will be completely ignored; otherwise, it will be completely retained. Therefore, using the GRU unit can reduce model complexity while also extracting effective audio temporal feature vectors from the audio.

[0130] In step S223, the network structure of the multi-head attention unit is the same as that of the multi-head attention unit in step S214. The multi-head attention unit determines audio features through an attention mechanism and audio temporal feature vectors. Furthermore, in this embodiment, for ease of understanding, the audio feature extraction process can be represented as follows:

[0131] MHAi(GRU(NetVLAD(MS)));

[0132] Wherein, MS represents the Mel-language spectrogram, NetVLAD represents the NetVLAD unit, GRU represents the GRU unit, and MHA represents the multi-head attention unit.

[0133] In one implementation of this application, the text feature extraction process specifically includes:

[0134] S230. Input the text data into the BERT unit and extract initial text features through the BERT unit;

[0135] S231. Input the initial text features into the multi-head attention unit, and determine the text features through the multi-head attention unit.

[0136] Specifically, in step S230, the BERT unit includes a BERT model, which is used to extract features from the text data to obtain initial text features. In this embodiment, the BERT unit uses an encoder from the Transformer model to encode the text data. The BERT unit is a bidirectional language model that can simultaneously consider the preceding and following information of the input sequence, thereby better capturing contextual information. For example, when processing a word in the text data, BERT can utilize information from both the preceding and following words. The input to the BERT model is the representation corresponding to each word, which consists of three parts: word token, segmentation token, and position embedding. A word token refers to dividing a word into a finite set of common sub-word units, achieving a trade-off between word effectiveness and character flexibility. Segmentation tokens are used to distinguish different sentence tokens. Position embedding refers to assigning a position vector to each word or sub-word unit in the input sequence to represent their relative or absolute position in the sequence. This positional information is crucial for the model to capture the order and dependencies between words.

[0137] In this embodiment of the application, inputting the text data into the BERT unit and extracting initial text features through the BERT unit specifically includes:

[0138] The text data is input into the BERT unit for word segmentation to obtain several words or sub-words;

[0139] A vocabulary is constructed based on a number of words or sub-words, wherein the vocabulary contains all possible words or sub-words;

[0140] Each word or sub-word in the vocabulary is input into the encoder in the BERT unit for encoding to obtain the word vector of each word or sub-word, and the word vector of each word or sub-word is used as the initial text feature.

[0141] Specifically, constructing a vocabulary based on a number of words or sub-words can be achieved by statistically analyzing these words or sub-words, converting them into numerical representations, and then using the BERT unit to construct a vocabulary containing all possible words or sub-words. Each word or sub-word in the vocabulary is input into the encoder of the BERT unit for encoding. For each word or sub-word in the vocabulary, the encoder encodes it as a unique integer using an index from the vocabulary. To ensure that the input to the multi-head attention unit is a fixed-length feature vector, i.e., to obtain initial text features of a fixed length, the BERT unit performs padding or truncation operations on the text. Padding involves adding special padding markers to the end of the text when the text length is insufficient to reach the specified length, while truncation involves deleting a portion of the text when the text length exceeds the specified length to meet the length requirement.

[0142] In step S231, the multi-head attention unit has the same network structure as the multi-head attention unit in step S214. The multi-head attention unit determines text features by introducing an attention mechanism based on the initial text features. Furthermore, in this embodiment, for ease of understanding, the text feature extraction process can be represented as follows:

[0143] MHA(BERT(Text));

[0144] Wherein, Text represents the text data, BERT represents the BERT unit, and MHA represents the multi-head attention unit.

[0145] This application embodiment extracts features from video data, audio data, and text data by using video branches, audio branches, and text branches respectively. This allows for the full extraction of feature information from multiple modalities, thereby fully acquiring local and global facial features, audio features, and text features, and thus improving the predictive efficiency and accuracy of depression screening.

[0146] S30. Input the video features, the audio features, and the text features into the fusion module of the multi-branch fusion model, and obtain the fusion features through the fusion module.

[0147] Specifically, the fused features are features obtained by fusing features from video, audio, and text modalities. The fusion module employs attention and residual mechanisms to fuse the feature vectors from these three modalities, i.e., fusing the video features, the audio features, and the text features.

[0148] In one implementation of this embodiment, such as Figure 7 As shown, the process of determining the fusion features specifically includes:

[0149] The first query vector, the first key vector, and the first value vector are determined based on video features; the second query vector, the second key vector, and the second value vector are determined based on audio features; and the third query vector, the third key vector, and the third value vector are determined based on text features.

[0150] The first query vector, the second query vector, and the third query vector are concatenated into a target query vector; the first key vector, the second key vector, and the third key vector are concatenated into a target key vector; and the first value vector, the second value vector, and the third value vector are concatenated into a target value vector.

[0151] Based on the target query vector and the target key vector, a vector correlation matrix is ​​determined, and the vector correlation matrix is ​​multiplied with the target value vector to obtain candidate fusion features;

[0152] The candidate fusion feature is added to the target value vector to obtain the fusion feature.

[0153] Specifically, the first query vector, first key vector, and first value vector are obtained by mapping the video features to a first vector space. The second query vector, second key vector, and second value vector are obtained by mapping the audio features to a second vector space. The third query vector, third key vector, and third value vector are obtained by mapping the text features to a third vector space. In other words, the video features, audio features, and text features are mapped to three different vector spaces to obtain the first query vector Q. v First key vector K v First value vector V v Second query vector Q a Second bond vector K a Second value vector V a Third query vector Q t Third bond vector K t Third value vector V t This application embodiment maps the video features, audio features, and text features to three different vector spaces, which enables the extraction of important information from the video features, audio features, and text features, facilitating the multi-branch fusion model to learn higher-level feature representations.

[0154] The vector concatenation method uses the `torch.stack()` function to stack feature vectors. Specifically, the `torch.stack()` function stacks the query vector Q, key vector K, and value vector V respectively to obtain the target query vector, target key vector, and target value vector. It can be understood that the target query vector, target key vector, and target value vector are feature vectors obtained through a simple fusion of video features, audio features, and text features. The target query vector, target key vector, and target value vector can be represented as follows:

[0155] Q s = stack(Q v Q a Q t );

[0156] K s =stack(K) v ,K a ,K t );

[0157] V s = stack(V v V a V t );

[0158] Among them, Q s K represents the target query vector. s V represents the target key vector. s Q represents the target value vector. v Let Q represent the first query vector. a Let Q represent the second query vector. t K represents the third query vector. v Let K represent the first key vector. a K represents the second bond vector. t V represents the third bond vector. v V represents the first value vector. a V represents the second value vector. t This represents the third value vector, and stack represents the torch.stack() function.

[0159] The inter-vector correlation matrix is ​​determined by multiplying the transpose of the target query vector and the target key vector using the `torch.matmul()` function. The inter-vector correlation matrix is ​​then normalized using the `Softmax` function to obtain a weight matrix, which represents the importance of elements at the same position among video, audio, and text features. The candidate fusion feature is determined by multiplying the weight matrix and the target value vector. The fusion feature is determined by adding the candidate fusion feature to the target value vector, where the fusion feature can be expressed as:

[0160] FF = torch.marmul(W, V s )+V s ;

[0161] Where FF represents the candidate fusion feature, W represents the weight matrix, and V s Represents the target value vector, torch.matmul(W, V) s ) represents candidate fusion features, and torch.matmul represents the torch.matmul() function.

[0162] For ease of understanding, in this embodiment, the process of feature fusion of video features, audio features, and text features can be represented as follows:

[0163] ((σ(Q s ×K s ))×V s )+V s ;

[0164] Where σ represents the Softmax function, Q s K represents the target query vector. s V represents the target key vector. s represents the target value vector, × represents matrix multiplication, and + represents the addition of corresponding elements of the eigenvectors.

[0165] This application embodiment obtains candidate fusion features by multiplying the inter-vector correlation matrix and the target value vector. It fuses important information from different modalities according to their importance, while also retaining the complete information before fusion. This enables the model to fully learn the effective information from different modalities, thereby improving the model's screening performance.

[0166] S40. Input the fusion features into the prediction module of the multi-branch fusion model, and output the depression category through the prediction module.

[0167] Specifically, the depression category is a depression classification. The prediction module includes two Dropout layers, two Linear layers, one ReLU layer, and one Softmax layer. One Dropout layer is connected to one Linear layer, one Linear layer is connected to the ReLU layer, the ReLU layer is connected to the other Dropout layer, the other Dropout layer is connected to the other Linear layer, and the other Linear layer is connected to the Softmax layer. The Dropout layer's function is to randomly ignore a portion of neurons with a certain probability during model training, preventing overfitting. Specifically, the ReLU layer introduces non-linearity into the neural network, allowing the model to better fit non-linear functions. Furthermore, due to the sparsity of the ReLU layer, some neurons can be suppressed to zero, enabling the model to better mine relevant features and fit the training data. In this embodiment, the fused features are input into a Dropout layer with p=0.3, then through a Linear layer and a ReLU layer, then through another Dropout layer with p=0.3, and then through a Linear layer to map the feature dimension to two dimensions. Finally, the predicted value is normalized by the Softmax function to obtain the final predicted probability. The depression category of the test subject can be determined by the predicted probability.

[0168] For ease of understanding, in this embodiment, the specific process of outputting the depression category through the prediction module can be represented as follows:

[0169] Softmax(Linear(Dropout(ReLU(Linear(Dropout(FF))))));

[0170] Wherein, FF represents the fusion feature, Dropout represents one of the Dropout layers, Linear represents one of the Linear layers, ReLU represents the ReLU layer, Dropout represents another Dropout layer, Linear represents another Linear layer, and Softmax represents the Softmax function.

[0171] To further illustrate the effectiveness of this embodiment, 240 validation data points (including 120 healthy subjects and 120 depressed subjects) were used to validate the depression screening performance of the multi-branch fusion model. When screening based on video data, the F1 score was 0.65 and the sensitivity was 0.64; when screening based on audio data, the F1 score was 0.62 and the sensitivity was 0.61; when screening based on text data, the F1 score was 0.64 and the sensitivity was 0.64; when screening using a fusion of video, audio, and text data, the F1 score was 0.76 and the sensitivity was 0.78, demonstrating that multimodal fusion can effectively improve the performance of depression screening.

[0172] In summary, this embodiment provides an end-to-end multimodal fusion-based depression screening method. The method specifically includes: acquiring user data of the test subject, wherein the user data includes video data, audio data, and text data; inputting the user data into a feature extraction module in a trained multi-branch fusion model, extracting video features, audio features, and text features through the feature extraction module; inputting the video features, audio features, and text features into a fusion module in the multi-branch fusion model, obtaining fused features through the fusion module; and inputting the fused features into a prediction module in the multi-branch fusion model, outputting a depression category through the prediction module. This application improves the prediction efficiency and accuracy of depression screening by employing a multi-branch fusion model to fuse video features, audio features, and text features to obtain a depression category.

[0173] Based on the aforementioned end-to-end multimodal fusion depression screening method, this embodiment provides an end-to-end multimodal fusion depression screening device, such as... Figure 8 As shown, the device specifically includes:

[0174] The acquisition module 100 is used to acquire the user data of the test subject, wherein the user data includes video data, audio data and text data;

[0175] The control module 200 is used to input the user data into the feature extraction module of the trained multi-branch fusion model, and extract video features, audio features, and text features through the feature extraction module; input the video features, audio features, and text features into the fusion module of the multi-branch fusion model, and obtain fused features through the fusion module; input the fused features into the prediction module of the multi-branch fusion model, and output the depression category through the prediction module.

[0176] The feature extraction module comprises a video branch, an audio branch, and a text branch, which are implemented in parallel. The video branch includes a video feature extraction unit, a FA unit, a multiplier, a BiLSTM unit, and a multi-head attention unit. The video feature extraction unit is connected to the FA unit and the multiplier, the FA unit is connected to the multiplier, and the multiplier, the BiLSTM unit, and the multi-head attention unit are connected in sequence.

[0177] Based on the above-described end-to-end multimodal fusion depression screening method, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the end-to-end multimodal fusion depression screening method as described in the above embodiment.

[0178] Based on the aforementioned end-to-end multimodal fusion depression screening method, this application also provides a terminal device, such as... Figure 9 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0179] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0180] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0181] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0182] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal fusion-based depression screening method based on end-to-end, characterized in that, The end-to-end multimodal fusion-based depression screening method includes: Obtain user data of the test subjects, wherein the user data includes video data, audio data, and text data; The user data is input into the feature extraction module of the trained multi-branch fusion model, and video features, audio features, and text features are extracted through the feature extraction module. The video features, audio features, and text features are input into the fusion module of the multi-branch fusion model, and the fusion module obtains the fused features. The fusion features are input into the prediction module of the multi-branch fusion model, and the prediction module outputs the depression category. The feature extraction module comprises a video branch, an audio branch, and a text branch, which are implemented in parallel. The video branch includes a video feature extraction unit, an FA unit, a multiplier, a BiLSTM unit, and a multi-head attention unit. The video feature extraction unit is connected to the FA unit and the multiplier, the FA unit is connected to the multiplier, and the multiplier, the BiLSTM unit, and the multi-head attention unit are connected in sequence. The video feature extraction process specifically includes: For each video frame in the video data, determine the local binary pattern map of the video frame; The local binary pattern map and the video frame are input into the video feature extraction unit, and the video feature extraction unit determines the video spatial feature vector. The video space feature vector is input into the FA unit, and the FA unit performs importance weighting on the video space feature vector to obtain a weighted feature vector; The weighted feature vector and the video spatial feature vector are multiplied and then input into the BiLSTM unit, which captures the temporal features of the video. The video temporal features are input into the multi-head attention unit, and the video features are determined by the multi-head attention unit.

2. The end-to-end multimodal fusion-based depression screening method according to claim 1, characterized in that, The video feature extraction unit includes a residual subunit, a Transformer subunit, and a feature fusion subunit; the step of inputting the local binary pattern map and the video frame into the video feature extraction unit, and determining the video spatial feature vector through the video feature extraction unit, specifically includes: The local binary pattern image is input into the residual sub-unit, and local facial features of the video are extracted through the residual sub-unit; The video frame is input into the Transformer subunit, and the global facial features of the video are extracted through the Transformer subunit. The local facial features and global facial features in the video are input into the fusion subunit, and the video spatial feature vector is determined by the fusion subunit.

3. The end-to-end multimodal fusion-based depression screening method according to claim 1, characterized in that, The audio branch includes a NetVLAD unit, a GRU unit, and a multi-head attention unit; the audio feature extraction process specifically includes: Convert the audio data into a Mel language spectrogram; The Mel speech spectrogram is input into a NetVLAD unit, and the Mel speech spectrogram vector is determined by the NetVLAD unit; The Mel spectrogram vector is input into the GRU unit, and the audio temporal feature vector is extracted through the GRU unit; The audio temporal feature vector is input into the multi-head attention unit, and the audio features are determined by the multi-head attention unit.

4. The end-to-end multimodal fusion-based depression screening method according to claim 1, characterized in that, The text branch includes BERT units and multi-head attention units; the text feature extraction process specifically includes: The text data is input into the BERT unit, and the initial text features are extracted through the BERT unit. The initial text features are input into a multi-head attention unit, and the text features are determined by the multi-head attention unit.

5. The end-to-end multimodal fusion-based depression screening method according to claim 1, characterized in that, The process of determining the fusion features specifically includes: The first query vector, the first key vector, and the first value vector are determined based on video features; the second query vector, the second key vector, and the second value vector are determined based on audio features; and the third query vector, the third key vector, and the third value vector are determined based on text features. The first query vector, the second query vector, and the third query vector are concatenated into a target query vector; the first key vector, the second key vector, and the third key vector are concatenated into a target key vector; and the first value vector, the second value vector, and the third value vector are concatenated into a target value vector. Based on the target query vector and the target key vector, a vector correlation matrix is ​​determined, and the vector correlation matrix is ​​multiplied with the target value vector to obtain candidate fusion features; The candidate fusion feature is added to the target value vector to obtain the fusion feature.

6. The end-to-end multimodal fusion-based depression screening method according to claim 1, characterized in that, The acquisition of user data from test subjects specifically includes: A data acquisition paradigm is determined, and the data is collected from the test subjects using the acquired paradigm to obtain initial video data. The initial video data is cropped to obtain the final video data; The initial video data is subjected to audio separation to obtain audio data, and the audio data is then converted into text data. The video data, audio data, and text data are used as the user data of the test subject.

7. A multimodal fusion-based depression screening device based on end-to-end, characterized in that, The end-to-end multimodal fusion depression screening device specifically includes: The acquisition module acquires the user data of the test subject, wherein the user data includes video data, audio data, and text data; The control module is used to input the user data into the feature extraction module of the trained multi-branch fusion model, and extract video features, audio features, and text features through the feature extraction module; input the video features, audio features, and text features into the fusion module of the multi-branch fusion model, and obtain fused features through the fusion module; input the fused features into the prediction module of the multi-branch fusion model, and output the depression category through the prediction module. The feature extraction module comprises a video branch, an audio branch, and a text branch, which are implemented in parallel. The video branch includes a video feature extraction unit, an FA unit, a multiplier, a BiLSTM unit, and a multi-head attention unit. The video feature extraction unit is connected to the FA unit and the multiplier, the FA unit is connected to the multiplier, and the multiplier, the BiLSTM unit, and the multi-head attention unit are connected in sequence. The video feature extraction process specifically includes: For each video frame in the video data, determine the local binary pattern map of the video frame; The local binary pattern map and the video frame are input into the video feature extraction unit, and the video feature extraction unit determines the video spatial feature vector. The video space feature vector is input into the FA unit, and the FA unit performs importance weighting on the video space feature vector to obtain a weighted feature vector; The weighted feature vector and the video spatial feature vector are multiplied and then input into the BiLSTM unit, which captures the temporal features of the video. The video temporal features are input into the multi-head attention unit, and the video features are determined by the multi-head attention unit.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the end-to-end multimodal fusion-based depression screening method as described in any one of claims 1-6.

9. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps of the end-to-end multimodal fusion-based depression screening method as described in any one of claims 1-6.

Images