A depression assessment method and device based on audio-visual multi-modal data fusion

Abstract

The application discloses a depression evaluation method based on audio-visual multi-modal data fusion, comprising the following steps: step 1, obtaining the facial video of a subject and the audio of the subject; step 2, obtaining the low-level visual features and low-level audio features of the subject; step 3, inputting the low-level visual features and low-level audio features obtained in step 2 into a parallel multi-scale bridge fusion depression evaluation PMBFN network for processing and obtaining a depression rating; by constructing a spatial coding module of a visual and audio branch, a parallel multi-scale dynamic convolution module and a space-time attention pooling module, multi-scale deep features are quickly and efficiently extracted from audio-visual multi-modal data, dynamic performance of depression behavior is comprehensively captured, and under the adjustment of a multi-modal bridge fusion module, data between modes is fully interacted, the utilization rate of multi-modal data is improved, and therefore the accuracy and efficiency of automatic depression evaluation are improved.

Description

Technical Field

[0001] This invention relates to the field of depression assessment methods, specifically a depression assessment method and device based on audiovisual multimodal data fusion. Background Technology

[0002] Depression is a common mental disorder characterized by persistent low mood, loss of interest, impaired cognitive function, sleep disturbances, and appetite disorders. These symptoms severely impact quality of life and, in extreme cases, can lead to suicidal behavior. Therefore, early assessment of the severity of depression is crucial. In clinical depression assessment, physicians typically observe patients' symptoms and assess their severity through structured or semi-structured interviews. Standardized self-rating scales for depression severity are often used to aid in quantifying the severity. However, this approach, combining interviews and scale assessments, has a degree of subjectivity and requires significant human, resource, and time investment. With continuous breakthroughs in artificial intelligence, particularly the rapid development of deep learning technology, automated depression assessment has opened up new possibilities.

[0003] Current automated depression assessment technologies primarily focus on using deep learning to learn feature representations reflecting depressive information from multimodal data. By extracting deep features, they assess the severity of depression, aiming to provide a convenient, objective, and efficient way to quickly screen individuals with depression. However, existing automated depression assessment methods typically observe multimodal data only at a single scale, failing to fully explore multimodal representations. The extracted deep features are susceptible to noise interference, thus failing to adequately capture the subtle short-term manifestations and dynamic expressions over long periods of depression. Multimodal fusion technology is a crucial step in automated depression assessment utilizing multimodal data. However, existing fusion methods lack interactivity in the fusion stage and rely excessively on single-stage fusion, failing to fully leverage the complementary information between multimodal data. These shortcomings prevent achieving accurate and efficient depression assessment when screening large and complex populations. Summary of the Invention

[0004] The purpose of this invention is to provide a depression assessment method based on audiovisual multimodal data fusion to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for assessing depression based on audiovisual multimodal data fusion includes the following steps:

[0007] Step 1: Obtain facial video and audio of the subject;

[0008] Step 2: Process the acquired facial video and audio of the subject to obtain the subject's low-level visual and audio features.

[0009] Step 3: Input the low-level visual features and low-level audio features obtained in Step 2 into the parallel multi-scale bridge fusion depression assessment PMBFN network for processing and obtain a depression rating.

[0010] The multi-scale bridging network for depression assessment, PMBFN, includes a visual spatial coding module (SE), an audio spatial coding module (SE), a parallel multi-scale dynamic convolutional network (PMDCLN) combined with LSTM, a multi-modal bridging network (MBFN), and a regression network (RN).

[0011] As a further aspect of the present invention: the acquisition of low-level visual features in step 2 includes the following steps:

[0012] S1. Visual features are extracted from the collected facial videos of the subjects using facial behavior analysis tools to obtain low-level visual descriptors such as eye gaze direction vector, head posture coordinates, and facial action units. Frame data with a confidence level of 1 are retained to obtain the original dataset of various low-level visual descriptors.

[0013] S2. Sample multiple frames of raw low-level visual descriptor data at the same time according to a fixed time interval t to obtain the eye gaze vector, head pose coordinates, and T, which are uniformly set by the facial motion unit. V Frame visual data;

[0014] S3. Perform min-max normalization on the time dimension of the data in each channel of eye gaze vector, head posture coordinates, and facial motion unit to obtain normalized eye gaze vector, head posture coordinates, and facial motion unit data.

[0015] S4. The standardized eye gaze vectors, head pose coordinates, and facial motion unit data are concatenated frame by frame to obtain the merged multi-visual low-level descriptor temporal sequence features, the length of which is denoted as T. V ;

[0016] S5. Divide the merged multiple visual low-level descriptor temporal sequences along the time dimension into segments of a set length S. V Frame time series segments, to obtain T V / S V Time series fragments of low-level visual features

[0017] Where N V =T V / S V , D represents a time series segment of low-level visual features. V This represents the initial dimensional size of low-level visual features.

[0018] As a further aspect of the present invention: the acquisition of low-level audio features in step 2 includes the following steps;

[0019] L1. Extract the speech portion of the subject's speech from the acquired subject audio data according to the timestamp to obtain several audio segment data;

[0020] L2. Concatenate several audio segments in chronological order to obtain merged audio data;

[0021] L3. The merged audio data is processed through frame division, windowing, short-time Fourier transform, Mel filtering, and logarithmic operations to obtain the log-Mel spectrum audio time sequence characteristics.

[0022] L4. Extracting pre-defined T values ​​from the log-Mel spectrum audio time-series features. G From frame data, a fixed-length log-Mel spectrum audio time-series sequence feature is obtained;

[0023] L5. Divide the log-Mel spectrum audio time-series features along the time dimension into segments of a set length S. A Frame time series segments, to obtain T A / S A Time series segments with log-Mel spectrum features

[0024] Where N A =T A / S A , D represents a time series segment of the log-Mel spectrum characteristics. A This represents the initial dimension size of the log-Mel spectrum feature.

[0025] As a further aspect of the present invention: the spatial coding module SE re-encodes visual and audio features through a 1D convolutional layer, and performs residual-like processing on the original features through a single-layer convolution.

[0026]

[0027] In the formula: The data is re-encoded; The data is before re-encoding; Convs(·) represents a 1D convolutional layer, and L(·) represents a single-layer convolutional operation with adjusted channels.

[0028] As a further aspect of the present invention: the parallel multi-scale dynamic convolutional network PMDCLN combined with LSTM consists of M parallel multi-scale dynamic convolutional PMDCL modules combined with LSTM and N max pooling layers.

[0029] As a further aspect of the present invention: the parallel multi-scale dynamic convolution module PMDCL module is composed of three parallel convolution branches and LSTM;

[0030] Three parallel convolutional branches can be used to process single-modal segment features to obtain spatiotemporal features at three scales. 1,M,i ,Scale 2,M,i ,Scale 3,M,i M∈{A,V} represents the visual or audio modality;

[0031]

[0032]

[0033]

[0034] Where Conv represents a 1D convolution operation, DConv represents a 1D dilated convolution operation, and σ represents the ReLU activation function;

[0035] The multi-scale hierarchical feature Z is obtained by using spatiotemporal features at three scales. M,i

[0036] Z M,i =Concat(Scale) 1,M,i ,Scale 1,M,i +Scale 2,M,i ,Scale 1,M,i +Scale 2,M,i +Scale 3,M,i )

[0037] Wherein, Concat(·) represents the operation of splicing features along the channel direction;

[0038] Z M,i Then, by accumulating the data through LSTM layers and residual-like connections, the aggregated multi-scale features are obtained. Finally, through batch normalization, the multi-scale aggregated features L of a single modality and single segment are obtained. M,i :

[0039] L M,i =BN(LSTMs(Z) M,i )+Conv(Z M,i ))

[0040] Where LSTMs represents LSTM layer operations, Conv represents 1D convolutional layer, and BN represents batch normalization operation.

[0041] As a further aspect of the present invention: the multimodal bridge fusion network MBFN is composed of U spatiotemporal attention pooling STAP modules for each of the two modes and V multimodal bridge fusion MBF modules.

[0042] As a further aspect of the present invention: the regression network RN consists of two linear layers and a ReLU activation function, and the output of the multimodal bridge fusion module in the last layer is processed to obtain the evaluated PHQ-8 score.

[0043] As a further aspect of the present invention: the establishment of the parallel multi-scale bridge fusion depression assessment network PMBFN includes the following steps:

[0044] Step 3.1: Input the low-level visual features and low-level audio features into the visual spatial coding module SE and the audio spatial coding module SE respectively for re-encoding to obtain visual spatiotemporal segment features and audio spatiotemporal segment features;

[0045] Step 3.2: Input the visual spatiotemporal segment features obtained in Step 3.1 into the parallel multi-scale dynamic convolutional network PMDCLN combined with LSTM for processing visual data, and input the audio spatiotemporal segment features into the parallel multi-scale dynamic convolutional network PMDCLN combined with LSTM for processing audio data, to obtain multi-level and multi-scale spatiotemporal depth visual features and multi-level and multi-scale spatiotemporal depth audio features.

[0046] Step 3.3: Input the obtained multi-level, multi-scale spatiotemporal depth visual features and multi-level, multi-scale spatiotemporal depth audio features into the Multimodal Bridge Fusion Network (MBN) to obtain the global fused features.

[0047] Step 3.4: Convert the output MF of the Multimodal Bridge Fusion Module (MBF) last The input is fed into the regression network RN to obtain the PHQ-8 score predicted by the network.

[0048]

[0049] Wherein, RN(·) represents the operation performed by the regression network;

[0050] Step 3.5: Calculate the PHQ-8 score predicted by the network using the mean squared error loss function MSELoss. The loss between the actual PHQ-8 score y and the network parameters are updated via backpropagation.

[0051] Step 3.6: Iteratively train the parallel multi-scale bridge fusion depression assessment network PMBFN; during the training process, the SGD optimizer is used to optimize all parameters of the depression assessment network PMBFN, and the training is iterated for C epochs.

[0052] If the loss on the validation set does not decrease for I consecutive cycles, the iterative training is terminated early; the trained parallel multi-scale bridge fusion depression assessment network PMBFN is obtained, which is used to automatically assess the PHQ-8 score of depression.

[0053] An electronic device is characterized by comprising a memory and a processor, wherein the memory stores a computer program that is executed by the processor when running, which is an automatic depression assessment method based on audiovisual multimodal data fusion, and the processor is used to execute the program stored in the memory.

[0054] Compared with the prior art, the beneficial effects of the present invention are:

[0055] 1. This invention constructs a spatial coding module for visual and audio branches, a parallel multi-scale dynamic convolution module, and a spatiotemporal attention pooling module to quickly and efficiently extract multi-scale deep features from audiovisual multimodal data, comprehensively capture the dynamic manifestations of depressive behavior, and, under the adjustment of the multimodal bridge fusion module, enables full interaction of data between modalities, improves the utilization rate of multimodal data, and thus improves the accuracy and efficiency of automatic depression assessment.

[0056] 2. This invention uses 1D convolution technology to re-encode various low-level visual features and log-Mel spectrum features to enhance the connection between spatial features and reduce noise interference in the original data, thereby making the network more robust in extracting temporal depression information and more in line with real-world applications.

[0057] 3. This invention extracts short-term, multi-scale depressive features by combining a parallel multi-scale dynamic convolution module of LSTM. Specifically, it employs parallel 1D convolutions of different scales and types with LSTM technology to fully extract and aggregate temporal contextual information at different levels and expand the model's receptive field, thereby more comprehensively capturing the potential visual or auditory depressive expressions of the subjects. Even when facing a large and complex population, it can still discriminate the performance of patients with different degrees of depression. Simultaneously, residual-like connections further enhance the model's generalization and robustness, and batch normalization technology accelerates network training, ultimately enabling the rapid and efficient extraction of key information reflecting the severity of depression from visual and audio data.

[0058] 4. This invention uses spatiotemporal attention pooling to aggregate deep spatiotemporal features from multiple short time segments into global spatiotemporal features over a long period of time. This allows for long-term observation of subjects and adaptive assessment of the importance of depressive information reflected by subjects in different time ranges. This reduces the burden on the network to find effective information and is more conducive to assisting the network in judging the depression score of subjects.

[0059] 5. This invention integrates multi-level visual depth features and audio depth features through a bridge fusion strategy, which can make full use of multiple synchronous data sources to improve the accuracy of depression assessment, avoid information bias caused by single-modal data assessment of depression, and effectively improve the utilization efficiency of multimodal data in automatic depression assessment. Attached Figure Description

[0060] Figure 1 This is a diagram showing the overall architecture of the PMBFN network of this invention;

[0061] Figure 2 This is a schematic diagram of the spatial coding module SE in the PMBFN network of the present invention;

[0062] Figure 3 This is a schematic diagram of the parallel multi-scale dynamic convolution module PMDCL combined with LSTM in the PMBFN network of this invention.

[0063] Figure 4 This is a schematic diagram of the spatiotemporal attention pooling module STAP in the PMBFN network of this invention;

[0064] Figure 5 This is a schematic diagram of the Multimodal Bridge Fusion Module (MBF) in the PMBFN network of this invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Please see Figure 1-5 In this embodiment of the invention, a method for assessing depression based on audiovisual multimodal data fusion is characterized by comprising the following steps:

[0067] Step 1: Obtain the subject's facial video and audio. In this embodiment, the subject's facial video and audio include the subject's eye gaze vector, head pose coordinates, facial action units (FAUs), WAV format audio, and PHQ-8 score labels. The DAIC-WOZ and E-DAIC datasets are used for preprocessing.

[0068] Step 2: Process the acquired facial video and audio of the subject to obtain the subject's low-level visual and audio features.

[0069] The acquisition of low-level visual features in step 2 includes the following steps:

[0070] S1. Visual features are extracted from the collected facial videos of the subjects using facial behavior analysis tools to obtain low-level visual descriptors such as eye gaze direction vector, head posture coordinates, and facial action units. Frame data with a confidence level of 1 are retained to obtain the original dataset of various low-level visual descriptors.

[0071] S2. Sample multiple frames of raw low-level visual descriptor data at the same time according to a fixed time interval t to obtain the eye gaze vector, head pose coordinates, and T, which are uniformly set by the facial motion unit. V Frame visual data;

[0072] S3. Perform min-max normalization on the time dimension of the data in each channel of eye gaze vector, head posture coordinates, and facial motion unit to obtain normalized eye gaze vector, head posture coordinates, and facial motion unit data.

[0073] S4. The standardized eye gaze vectors, head pose coordinates, and facial motion unit data are concatenated frame by frame to obtain the merged multi-visual low-level descriptor temporal sequence features, the length of which is denoted as T. V ;

[0074] S5. Divide the merged multiple visual low-level descriptor temporal sequences along the time dimension into segments of a set length S. V Frame time series segments, to obtain T V / S V Time series fragments of low-level visual features

[0075] Where N V =T V / S V , D represents a time series segment of low-level visual features. V This represents the initial dimensional size of low-level visual features.

[0076] The acquisition of low-level audio features in step 2 includes the following steps;

[0077] L1. Extract the speech portion of the subject's speech from the acquired subject audio data according to the timestamp to obtain several audio segment data;

[0078] L2. Concatenate several audio segments in chronological order to obtain merged audio data;

[0079] L3. The merged audio data is processed through frame division, windowing, short-time Fourier transform, Mel filtering, and logarithmic operations to obtain the log-Mel spectrum audio time sequence characteristics.

[0080] L4. Extracting pre-defined T values ​​from the log-Mel spectrum audio time-series features. A From frame data, a fixed-length log-Mel spectrum audio time-series sequence feature is obtained;

[0081] L5. Divide the log-Mel spectrum audio time-series features along the time dimension into segments of a set length S. A Frame time series segments, to obtain T A / S A Time series segments with log-Mel spectrum features

[0082] Where N A =T A / S A , D represents a time series segment of the log-Mel spectrum characteristics. A This represents the initial dimensionality of the log-Mel spectrum feature.

[0083] Step 3: Input the low-level visual features and low-level audio features obtained in Step 2 into the parallel multi-scale bridge fusion depression assessment PMBFN network for processing and obtain a depression rating.

[0084] The multi-scale bridge fusion depression assessment PMBFN network, its overall framework and assessment process are as follows: Figure 1 As shown, it consists of spatial coding modules SE (visual and audio branches), a parallel multi-scale dynamic convolutional network PMDCLN (combined with LSTM), a multimodal bridge fusion network MBFN, and a regression network RN.

[0085] The parallel multi-scale dynamic convolutional network PMDCLN combined with LSTM consists of M=4 parallel multi-scale dynamic convolutional PMDCL modules combined with LSTM and N=3 max pooling layers.

[0086] The Multimodal Bridge Fusion Network (MBN) consists of two modalities, each with U=4 spatiotemporal attention pooling STAP modules and N=4 multimodal bridge fusion MBF modules.

[0087] The regression network RN consists of two linear layers and a ReLU activation function. The output of the multimodal bridge fusion module in the last layer is processed to obtain the evaluated PHQ-8 score.

[0088] For visual and audio modal temporal sequence data, firstly, spatial features exhibit strong correlations, and low-dimensional features cannot fully represent this complex relationship. Furthermore, the original data suffers from varying noise interference due to different subjects' responses. Therefore, PMBFN utilizes a spatial coding module to obtain more robust high-dimensional spatial feature representations. Secondly, audiovisual multimodal temporal sequence data contains dynamic emotional changes in patients with depression over time. Therefore, extracting key depressive information from temporal data is crucial for accurately assessing depression. However, different patients with depression do not react synchronously in terms of response rate and performance. Relying solely on single-scale feature extraction methods results in low generalization. PMBFN employs a parallel multi-scale feature extraction architecture to comprehensively capture temporal depressive expressions at different levels of visual and auditory dimensions in patients with depression. Furthermore, since multimodal data contains richer information than unimodal data, adopting an effective multimodal fusion method is key to fully utilizing multimodal data. However, using a single-stage fusion method can lead to information bias due to insufficient interaction between modalities, failing to leverage the advantages of multimodal depression assessment compared to unimodal depression assessment. Therefore, PMBFN constructed a multimodal bridge fusion depression assessment network, which fully integrates multi-level information across modalities through a recursive, multi-stage, interactive fusion approach to obtain a more discriminative global multimodal depression feature representation, thereby improving the accuracy of multimodal depression assessment.

[0089] The establishment of the parallel multi-scale bridging depression assessment network PMBFN includes the following steps:

[0090] Step 3.1: Input the low-level visual features and low-level audio features into the visual spatial coding module SE and the audio spatial coding module SE respectively for re-encoding to obtain visual spatiotemporal segment features and audio spatiotemporal segment features;

[0091] Features of each low-level visual segment The spatially recoded N is obtained through the spatial coding module SE of the visual branch. V =25 visual spatiotemporal segment features Enhance the connections between features in the visual modal space and reduce noise interference in visual information;

[0092]

[0093] In equation (1), SE(·) represents the operation performed by the visual branch space coding module, and D′ V =64 indicates the dimensionality of the recoded low-level visual features;

[0094] Features of each log-Mel spectrum segment The spatially recoded N is obtained through the spatial coding module SE of the audio branch. A =9 audio spatiotemporal segment features Enhance the connections between features in the audio modal space and reduce noise interference in audio information;

[0095]

[0096] In equation (2), SE(·) represents the operation performed by the audio branch space coding module, and D′ A =128 indicates the dimensionality of the recoded log-Mel spectrum features;

[0097] Step 3.2: Input the visual spatiotemporal segment features obtained in Step 3.1 into the parallel multi-scale dynamic convolutional network PMDCLN combined with LSTM for processing visual data, and input the audio spatiotemporal segment features into the parallel multi-scale dynamic convolutional network PMDCLN combined with LSTM for processing audio data, to obtain multi-level and multi-scale spatiotemporal depth visual features and multi-level and multi-scale spatiotemporal depth audio features.

[0098] Each spatiotemporal segment feature of the vision is input into the parallel multi-scale dynamic convolutional network PMDCLN combined with LSTM in the vision branch to obtain multi-level and multi-scale spatiotemporal depth visual features. The acquisition of multi-level and multi-scale spatiotemporal depth visual features includes the following steps:

[0099] A1. Features of each spatiotemporal segment of the visual modality By combining the first layer of the visual branch with the parallel multi-scale dynamic convolution module PMDCL of LSTM, the output first-layer multi-scale visual spatiotemporal depth features are obtained.

[0100]

[0101] In equation (3), PMDCL V,1 (·) indicates the operation performed by the parallel multi-scale dynamic convolution module PMDCL combined with LSTM in the first layer of the visual branch;

[0102] A2. The first-layer multi-scale visual spatiotemporal depth features output from the visual modality branch. After max pooling, the dimensionality-reduced visual modal spatiotemporal features are obtained.

[0103]

[0104] In equation (4), MaxPooling V,1 (·) represents the max pooling operation in the first layer of the visual modality branch. Max pooling reduces redundant features in the temporal domain, extracts more refined temporal information, and effectively reduces the training and testing time of the network. τ = 2 represents the compression ratio of the max pooling layer.

[0105] A3. Max-pool the output of the previous layer in the visual modality branch. The input is fed into the next layer's parallel multi-scale dynamic convolutional module PMDCL. Following steps 3.3.1 and 3.3.2, the output of the visual modality branch's l-th layer parallel multi-scale dynamic convolutional module PMDCI, which combines LSTM, is obtained sequentially. Output of max pooling layer

[0106] Features of each spatiotemporal segment of the audio modality The input is fed into the parallel multi-scale dynamic convolutional network PMDCI, which combines LSTM with the audio branch, to obtain multi-level, multi-scale spatiotemporal depth visual features. The acquisition of multi-level, multi-scale spatiotemporal depth visual features includes the following steps:

[0107] B1. Features of each spatiotemporal segment of the audio modality The first layer of the audio branch uses the parallel multi-scale dynamic convolution module PMDCI, which combines LSTM with the first layer of the audio branch, to obtain the first layer of multi-scale visual spatiotemporal depth features.

[0108]

[0109] In equation (5), PMDCI A,1 (·) indicates the operation performed by the parallel multi-scale dynamic convolution PMDCL module combined with LSTM in the first layer of the audio modality branch;

[0110] B2. The first-layer multi-scale visual spatiotemporal depth features output from the audio modality branch. After max pooling, the dimensionality-reduced audio modal spatiotemporal features are obtained.

[0111]

[0112] In equation (6), MaxPooling A,1 (·) represents the max pooling operation in the first layer of the audio modality branch;

[0113] B3. The output of the previous max-pooling layer in the audio modality branch. The input is fed into the next layer's parallel multi-scale dynamic convolutional module PMDCL. Following steps 3.4.1 and 3.4.2, the output of the audio modality branch's l-th layer parallel multi-scale dynamic convolutional module PMDCL, which combines LSTM, is obtained sequentially. Output of max pooling layer

[0114] Step 3.3: Input the obtained multi-level, multi-scale spatiotemporal depth visual features and multi-level, multi-scale spatiotemporal depth audio features into the Multimodal Bridge Fusion Network (MBN) to obtain the global fused features.

[0115] Step 3.3.1: Obtain the visual features from each layer. With audio features The global feature representations of the visual and audio data in this layer are obtained respectively through the spatiotemporal attention pooling module of this layer. and

[0116]

[0117]

[0118] In equation (7), STAP V,l (·) represents the operation performed by the spatial attention pooling module of the l-th visual modality. In equation (8), STAP A,l (·) represents the operation performed by the spatial attention pooling module of the l-th audio modality;

[0119] Step 3.3.2: Determine if the current layer is the first layer. If it is the first layer, then represent the global spatiotemporal features of the visual and audio data for this layer. and The data is fused using the Multimodal Bridge Fusion (MBF) module of this layer; if it is not the first layer, the visual and audio global spatiotemporal features of this layer are represented. and The output MF of the multimodal bridge fusion module MBF in the previous layer l-1 The features are fused together using the Multimodal Bridge Fusion (MBF) module of this layer to obtain the multimodal fusion feature (MF) of this layer. l :

[0120]

[0121] In equation (9), MBF(·) represents the operation performed by the multimodal bridge fusion module MBF;

[0122] Step 3.4: The output MF of the last layer's multimodal bridge fusion module MBF is... last The input is fed into the regression network RN to obtain the PHQ-8 score predicted by the network.

[0123]

[0124] In equation (10), RN(·) represents the operation performed by the regression network;

[0125] Step 3.5: Calculate the PHQ-8 score predicted by the network using the mean squared error loss function MSELoss. The loss between the actual PHQ-8 score y and the network parameters are updated via backpropagation.

[0126]

[0127] In equation (11), y i This represents the true PHQ-8 score of the sample. This represents the PHQ-8 score output by the network, and N represents the number of samples in a batch. In this embodiment, N = 8.

[0128] Step 3.6: Iteratively train the parallel multi-scale bridge fusion depression assessment network PMBFN; during the training process, the SGD optimizer is used to optimize all parameters of the depression assessment network PMBFN, and the training is iterated for C epochs.

[0129] If the loss on the validation set does not decrease for I consecutive cycles, the iterative training is terminated early; the trained parallel multi-scale bridge fusion depression assessment network PMBFN is obtained, which is used to automatically assess the PHQ-8 score of depression.

[0130] In this embodiment, the parallel multi-scale bridge fusion depression assessment network PMBFN is iteratively trained based on the preprocessed dataset. During the training process, the SGD optimizer is used to optimize all parameters of the depression assessment network PMBFN. The iterative training continues for C = 150 epochs. If the loss on the validation set does not decrease for I = 5 consecutive epochs, the iterative training is terminated early. The trained parallel multi-scale bridge fusion depression assessment network PMBFN is obtained and used to automatically assess the PHQ-8 score of depression.

[0131] Spatial coding modules (SE) in the visual and audio branches, such as Figure 2 As shown, visual features are processed through 1D convolutional layers. With audio features Re-encode, and perform residual processing on the original features through a single-layer convolution using Equation (12). Residual processing prevents the network from degrading during training.

[0132]

[0133] In equation (12), Convs(·) represents a 1D convolutional layer, and I(·) represents a single-layer convolutional operation with adjusted channels;

[0134] The parallel multi-scale dynamic convolutional module PMDCL for both visual and audio branches consists of three parallel convolutional branches and an LSTM, as follows: Figure 3 As shown;

[0135] Three parallel convolutional branches process the single-modal segment features according to formulas (12), (13), and (14) to obtain spatiotemporal features at three scales. 1,M,i ,Scale 2,M,i ,Scale 3,M,i M∈{A,V} represents the visual or audio modality;

[0136]

[0137]

[0138]

[0139] In equations (13), (14), and (15), Conv represents a 1D convolution operation. Specifically, Conv1 represents a 1D convolution operation with a kernel size of 1, Conv2 represents a 1D convolution operation with a kernel size of 3, and Conv3 represents a 1D convolution operation with a kernel size of 5, thus capturing deep depressive features at different levels. DConv represents a 1D dilated convolution operation. Specifically, DConv1, DConv2, and DConv3 represent 1D dilated convolution operations with a kernel size of 3 and dilation rates of 1, 3, and 5, respectively. Dilated convolution further expands the receptive field of the network model, enabling the model to observe a wider range of visual and speech dynamic changes. Furthermore, parallel operations can minimize the time required for the network to extract features related to depressive information. σ represents the ReLU activation function.

[0140] The spatiotemporal features at three scales are used to obtain the complete multi-scale hierarchical feature Z according to formula (16). M,i :

[0141] Z M,i =Concat(Scale) 1,M,i ,Scale 1,M,i +Scale 2,M,i ,Scale 1,M,i +Scale 2,M,i +Scale 3,M,i ) (16)

[0143] In equation (16), Concat(·) represents the operation of splicing features along the channel direction;

[0144] Z M,i Then, by accumulating the data through LSTM layers and residual-like connections, the aggregated multi-scale features are obtained. Finally, through batch normalization, the multi-scale aggregated features L of a single modality and single segment are obtained. M,i :

[0145] L M,i =BN(LSTMs(Z) M,i )+Conv(Z M,i (17)

[0146] In Equation (17), LSTMs represents LSTM layer operations, Conv represents 1D convolutional layer, and BN represents batch normalization operation. Batch normalization can accelerate network training.

[0147] The spatiotemporal attention pooling module STAP in the visual and audio branches, such as Figure 4 As shown, the single-modal spatiotemporal attention weight Att is obtained through formula (18). M :

[0148]

[0149] In Equation (18), GAP represents the global average pooling operation, GMP represents the global max pooling operation, MLP represents the multilayer perceptron operation, and σ2 represents the Sigmoid activation function.

[0150] The single-modal spatiotemporal attention weight Att M Multiplying by each spatiotemporal segment and then summing them up yields the single-modal global spatiotemporal feature Z. M This enhances the network's ability to adapt to large amounts of multimodal data;

[0151]

[0152] In equation (19), ∑ denotes the Kronecker inner product, and ∑ denotes the summation operation over multiple fragment features;

[0153] Multimodal Bridge Fusion Module (MBF), such as Figure 5 As shown, cross-modal temporal cross-attention weights are generated using formulas (19) and (20), respectively. and

[0154]

[0155]

[0156] In equations (20) and (21), and represent the temporal attention weights for generating the visual modality from the audio modality features, respectively. The temporal attention weights for generating audio modalities from visual modal features are represented, Avg represents the average pooling operation along the channel direction, and W represents the average attention weights for generating audio modalities from visual modal features. A With W V This represents the weight matrix for generating the cross-attention linear layer, b. A With b V These are the bias vectors for the corresponding linear layers;

[0157] Cross-modal temporal cross-attention weights and The formula is used to further aggregate the global spatiotemporal features of a single modality into a global tensor representation. This interactive aggregation enhances the sharing and communication of information across different modalities.

[0158]

[0159]

[0160] In equations (22) and (23), V A With V V These represent the global tensor representations of audio and video, respectively. ∑ denotes the Kronecker inner product, and ∑ denotes the summation operation along the time dimension;

[0161] Following the strategy in step 3.3.2, formula (9) can be further refined into formula (23):

[0162]

[0163] In formula (24), Dense(·) represents the fusion layer composed of the fully connected layer and the ReLU activation function. This recursive interactive fusion strategy compensates for the information bias of the single mode in a way that supplements information. Through multi-stage and multi-level fusion, it makes full use of multi-modal data, obtains more discriminative multi-modal fusion features, and thus improves the accuracy of automatic depression assessment.

[0164] An electronic device includes a processor and a memory for storing a computer program for the above-described method, and the processor is used to execute the program stored in the memory.

[0165] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0166] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for assessing depression based on audiovisual multimodal data fusion, characterized in that, Includes the following steps: Step 1: Obtain facial video and audio of the subject; Step 2: Process the acquired facial video and audio of the subject to obtain the subject's low-level visual and audio features. Step 3: Input the low-level visual and low-level audio features obtained in Step 2 into the parallel multi-scale bridging fusion depression assessment. The network processes the data and obtains a depression rating; Parallel multiscale bridging depression assessment The network includes a visual spatial coding module. Audio spatial encoding module Combination Parallel multi-scale dynamic convolutional networks and multimodal bridge fusion network and regression network ; The Parallel Multiscale Bridging Fusion Depression Assessment The establishment of a network includes the following steps: Step 3.1: Input the low-level visual features and low-level audio features into the visual spatial coding module SE1 and the audio spatial coding module SE2 respectively for re-encoding to obtain visual spatiotemporal segment features and audio spatiotemporal segment features; Step 3.2: Input the visual spatiotemporal segment features obtained in Step 3.1 into the combination used for processing visual data. Parallel multi-scale dynamic convolutional networks The audio spatiotemporal segment features are input into a combination used to process audio data. Parallel multi-scale dynamic convolutional networks This yields multi-level, multi-scale spatiotemporal depth visual features and multi-level, multi-scale spatiotemporal depth audio features. Step 3.3: Input the obtained multi-level, multi-scale spatiotemporal depth visual features and multi-level, multi-scale spatiotemporal depth audio features into the multimodal bridge fusion network. In this process, global fusion features are obtained; Step 3.4: Integrate the multimodal bridge fusion module Output Input into the regression network In the middle, the network prediction is obtained. Fraction : in, This indicates the operations performed by the regression network; Step 3.5: Use the mean squared error loss function Computational network prediction Fraction With the real Fraction Losses between And update the network parameters through backpropagation; Step 3.6: Develop a parallel multi-scale bridging network for depression assessment. Iterative training is performed; during the training process, the following methods are used: Optimizer for Depression Assessment Network All parameters were optimized, and iterative training was performed. indivual until; If the loss on the validation set is continuous If the performance does not decrease within a certain number of cycles, the iterative training is terminated early; the trained parallel multi-scale bridging fusion depression assessment network is obtained. Used for automated assessment of depression Fraction.

2. The method for assessing depression based on audiovisual multimodal data fusion according to claim 1, characterized in that, The acquisition of low-level visual features in step 2 includes the following steps: S1. Visual features are extracted from the collected facial videos of the subjects using facial behavior analysis tools to obtain low-level visual descriptors such as eye gaze direction vector, head posture coordinates, and facial action units. Frame data with a confidence level of 1 are retained to obtain the original dataset of various low-level visual descriptors. S2. The original multiple low-level visual descriptor multi-frame data are processed simultaneously at fixed time intervals. Sampling was performed to obtain eye gaze vectors, head pose coordinates, and facial motion units with uniform settings. Frame visual data; S3. Perform time-dimension processing on the data within each channel of the eye gaze vector, head pose coordinates, and facial motion units. Standardization yields standardized eye gaze vectors, head posture coordinates, and facial motion unit data. S4. The standardized eye gaze vectors, head pose coordinates, and facial motion unit data are concatenated frame by frame to obtain the merged multi-visual low-level descriptor temporal sequence features, the length of which is denoted as [missing information]. ; S5. Divide the merged multiple visual low-level descriptor time sequences along the time dimension into segments of a set length. Frame time series segments, obtained Time series fragments of low-level visual features ; in , , A time series segment representing low-level visual features. This represents the initial dimensional size of low-level visual features.

3. The method for assessing depression based on audiovisual multimodal data fusion according to claim 1, characterized in that, The acquisition of low-level audio features in step 2 includes the following steps: L1. Extract the speech portion of the subject's speech from the acquired subject audio data according to the timestamp to obtain several audio segment data; L2. Concatenate several audio segments in chronological order to obtain merged audio data; L3. The merged audio data is processed through frame division, windowing, short-time Fourier transform, Mel filtering, and logarithmic operations to obtain the log-Mel spectrum audio time sequence characteristics. L4. Extracting pre-defined features from the log-Mel spectrum audio time-series sequence. From frame data, a fixed-length log-Mel spectrum audio time-series sequence feature is obtained; L5. Divide the log-Mel spectrum audio time-series features along the time dimension into segments of a set length. Frame time series segments, obtained Time series segments with log-Mel spectrum features ; in , , This represents a time series segment representing the log-Mel spectrum characteristics. This represents the initial dimension size of the log-Mel spectrum feature.

4. The method for assessing depression based on audiovisual multimodal data fusion according to claim 1, characterized in that, The spatial coding module Visual and audio features are re-encoded using 1D convolutional layers, and the original features are processed using a single-layer convolutional layer to perform residual-like processing. In the formula: The data is re-encoded; This is the data before re-encoding; Indicates a 1D convolutional layer. This indicates a single-layer convolution operation that adjusts the channels.

5. The method for assessing depression based on audiovisual multimodal data fusion according to claim 1, characterized in that, The combination Parallel multi-scale dynamic convolutional networks Depend on A combination Parallel multi-scale dynamic convolution Modules and It consists of a maximum pooling layer.

6. The method for assessing depression based on audiovisual multimodal data fusion according to claim 5, characterized in that, The parallel multi-scale dynamic convolution module The module consists of three parallel convolutional branches and constitute; Three parallel convolutional branches process single-modal segment features to obtain spatiotemporal features at three scales. , , , Indicates visual or audio modality; in, This represents a 1D convolution operation. This represents a 1D dilated convolution operation. express Activation function; Multi-scale hierarchical features are obtained by using spatiotemporal features at three scales. in, This indicates an operation that splices features along the channel direction; Then through Layer and class residual connections are accumulated to obtain aggregated multi-scale features. Then, batch normalization is performed to obtain multi-scale aggregated features of a single modality and a single fragment. : in, express Layer operations, Indicates a 1D convolutional layer. This indicates batch normalization operation.

7. The method for assessing depression based on audiovisual multimodal data fusion according to claim 1, characterized in that, The multimodal bridge fusion network Each of the two modes Spatiotemporal attention pooling Modules and Multimodal bridge fusion Module composition.

8. The method for assessing depression based on audiovisual multimodal data fusion according to claim 1, characterized in that, The regression network Composed of two linear layers and The activation function is used to process the output of the last layer's multimodal bridge fusion module to obtain the evaluation. Fraction.

9. An electronic device, characterized in that, The device includes a memory and a processor. The memory stores a computer program that is executed by the processor when it runs, which is a method for assessing depression based on audiovisual multimodal data fusion according to any one of claims 1-8. The processor is used to execute the program stored in the memory.

Images