Multi-modal depression assessment system and method based on gradient embedding and modal complementation

Abstract

The invention relates to a multi-modal depression assessment system and method based on gradient embedding and modal complementation. The system comprises a long-time video face extraction and coding module, a long-time voice Mel spectrum extraction module, a sequence normalization and short-time sequence generation module, a first-order gradient extraction module, a second-order gradient extraction module, a mixed-order gradient embedding module, a multi-head attention module with multi-stage vectorization and a dimension regularity module. A modal complementation expression generation module, a modal complementation module and a depression degree prediction module; by introducing a first-order and second-order gradient embedding mechanism, the capability of capturing dynamic features in an audio and video sequence is effectively enhanced; utilizing a multi-stage attention mechanism to fully extract multi-level representation in a time sequence evolution process; and accurate complementation between audio and video information is realized by constructing a cross-modal complementary fusion mechanism. According to the system, the recognition capability and the evaluation accuracy of depression-related behavior clues are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction, particularly to the fields of affective computing and multimodal depression level prediction, specifically to a multimodal depression assessment system and method based on gradient embedding and modal complementarity. Background Technology

[0002] Early identification and assessment of depression are crucial for intervention and treatment. However, traditional diagnostic methods rely heavily on professional psychiatrists, and due to the uneven distribution of medical resources, patients often struggle to receive timely and effective screening and diagnosis.

[0003] In recent years, AI-based assisted diagnostic technologies have offered new solutions to these problems. Studies have shown that patients with depression exhibit certain regular changes in behavioral modalities such as facial expression dynamics and speech prosody. Therefore, using computer vision and speech analysis technologies to automatically extract behavioral features from long-term video and speech signals of individuals and assess their level of depression has become a research direction with significant application value.

[0004] Currently, existing automated depression assessment schemes typically follow a multimodal processing framework: first, high-dimensional feature sequences are extracted from video and audio modalities separately; then, these features are fused and input into a prediction model. However, such methods still have significant limitations in performance and reliability, specifically in the following aspects: First, at the feature extraction level, existing models have limited ability to capture the subtle dynamic changes in audio and video sequences that are closely related to the depressive state. These local dynamic cues are crucial for accurately distinguishing between healthy individuals and depressed patients, but existing architectures struggle to model them adequately and effectively.

[0005] Secondly, in terms of utilizing sequence information, existing methods do not fully integrate the multi-level and multi-stage evolutionary information of temporal features. This results in the model's vectorized representation of the sequence failing to comprehensively and profoundly reflect its complete state changes throughout the evolution process, thus losing some important information that can be used for discrimination.

[0006] Finally, at the multimodal fusion level, existing technologies still have a relatively superficial understanding of the deep complementary relationships between features of different modalities (such as video and audio). Simple feature splicing or mixing strategies fail to explicitly model and utilize the inherent, discriminative complementary information between modalities, which limits the upper limit of the model's final representation ability and makes model optimization lack clear guidance.

[0007] Therefore, there is an urgent need in this field for a novel depression assessment model that can more effectively capture the dynamic details of sequences, make fuller use of multi-stage temporal information, and achieve more accurate modal complementarity, so as to improve the accuracy and reliability of automatic assessment. Summary of the Invention

[0008] This invention provides a multimodal depression assessment system and method based on gradient embedding and modal complementarity to address the technical problems of existing automatic depression assessment schemes, such as insufficient capture of local dynamic details of audio and video sequences, inadequate utilization of multi-stage information in the temporal evolution process, and unclear mining and fusion mechanisms of deep complementary information among multiple modalities.

[0009] In a first aspect, the present invention provides a multimodal depression assessment system based on gradient embedding and modal complementarity, comprising: a long-term video face extraction and encoding module for extracting a facial image sequence from an input long-term video and encoding it into a video temporal sequence; a long-term speech Mel spectrum extraction module for extracting Mel spectra from the input long-term speech to obtain an audio temporal sequence; and a sequence normalization and short-term sequence generation module, connecting the long-term video face extraction and encoding module and the long-term speech Mel spectrum extraction module, for cutting and generating fixed-length short-term video sequences and audio temporal sequences from the video temporal sequence and the audio temporal sequence. A short-time audio sequence is generated, and the short-time video sequence and short-time audio sequence are normalized respectively; a first-order gradient extraction module, connected to the sequence normalization and short-time sequence generation module, is used to extract the first-order gradient of the short-time video sequence and short-time audio sequence respectively; a second-order gradient extraction module, connected to the sequence normalization and short-time sequence generation module, is used to extract the second-order gradient of the short-time video sequence and short-time audio sequence respectively; a mixed-order gradient embedding module, connected to the sequence normalization and short-time sequence generation module, the first-order gradient extraction module and the second-order gradient extraction module, is used to embed the first-order gradient and the second-order gradient into a single gradient. The algorithm performs gradient fusion to generate mixed gradient information, which is then embedded into corresponding short-time video or audio sequences to obtain video gradient enhancement sequences and audio gradient enhancement sequences. A multi-head attention module with multi-stage vectorization, connected to the mixed gradient embedding module, performs multi-stage processing and aggregation on the video gradient enhancement sequences and audio gradient enhancement sequences to obtain video vectorized representations and audio vectorized representations. A dimension normalization module, connected to the multi-head attention module with multi-stage vectorization, maps the video vectorized representations and audio vectorized representations to the same dimension space to obtain video normalized representations and audio normalized representations, respectively. A modal complementarity representation generation module, connected to the dimension normalization module, calculates the difference between the video normalized representation and the audio normalized representation to generate a modal complementarity representation. A modal complementarity module, connected to the modal complementarity representation generation module, uses the modal complementarity representation to perform weighted fusion of the corresponding video normalized representation or audio normalized representation to obtain a complementary fused representation. A depression level prediction module, connected to the modal complementarity module, outputs a depression level prediction score based on the fused representation.

[0010] In some embodiments, the mixed-order gradient embedding module includes: a gradient joint module connected to the first-order gradient extraction module and the second-order gradient extraction module, configured to concatenate the first-order gradient and the second-order gradient, and generate mixed-order gradient weights through a convolutional layer and a Softmax function; and a gradient adjustment module connected to the gradient joint module and the sequence normalization and short-time sequence generation module, configured to embed the mixed-order gradient weights into the corresponding short-time video sequence or short-time audio sequence through a Hadamard product operation.

[0011] In some embodiments, the multi-head attention module with multi-stage vectorization includes: a multi-head sequence vectorization module, connected to the mixed-order gradient embedding module, configured to process the video gradient enhancement sequence and the audio gradient enhancement sequence respectively through an N-layer multi-head self-attention mechanism, and to perform one-dimensional convolution on the output of each layer to obtain a staged sequence vector; and a sequence aggregation module, connected to the multi-head sequence vectorization module, configured to concatenate the N staged sequence vectors and aggregate them into the video vectorization representation or the audio vectorization representation through an attention mechanism.

[0012] In some embodiments, the modal complement representation generation module is configured to: subtract the audio regularization representation from the video regularization representation to obtain a video-to-audio complementary representation; and / or, subtract the video regularization representation from the audio regularization representation to obtain an audio-to-video complementary representation.

[0013] In some embodiments, the modal complementation module is configured to: concatenate the video regularization representation or audio regularization representation with its corresponding modal complementation representation; input the concatenation result into a fully connected layer and a Softmax function to generate corresponding complementation weights; and use the complementation weights to perform weighted fusion of the video regularization representation or audio regularization representation and the corresponding modal complementation representation to obtain a complementary fused representation.

[0014] In some embodiments, the first-order gradient extraction module is configured to extract the first-order gradient of the short-time video sequence or short-time audio sequence through a two-dimensional convolution operation using a predefined first-order differential convolution kernel.

[0015] In some embodiments, the second-order gradient extraction module is configured to extract the second-order gradient of the short-time video sequence or short-time audio sequence through a two-dimensional convolution operation using a predefined second-order difference convolution kernel.

[0016] In some embodiments, the sequence normalization and short-time sequence generation module is configured to: generate the short-time video sequence and short-time audio sequence by cutting from the long-time video and long-time speech with a fixed duration window and overlap rate; and perform mean-variance normalization on the short-time video sequence and short-time audio sequence respectively based on the mean and standard deviation of the video time sequence and the audio time sequence.

[0017] Secondly, the present invention provides a multimodal depression assessment method based on gradient embedding and modal complementarity, applied to the multimodal depression assessment system based on gradient embedding and modal complementarity as described in any one of the first aspects. The method includes: acquiring input long-duration video and corresponding long-duration speech; extracting facial image sequences from the input long-duration video and encoding them into a video temporal sequence; extracting Mel spectrum from the input long-duration speech to obtain an audio temporal sequence; segmenting the video temporal sequence and audio temporal sequence to generate fixed-length short-duration video sequences and short-duration audio sequences, and normalizing the short-duration video sequences and short-duration audio sequences respectively; extracting the first-order gradients of the short-duration video sequences and short-duration audio sequences respectively; extracting the second-order gradients of the short-duration video sequences and short-duration audio sequences respectively; and converting the first-order gradients into the second-order gradients of the second-order gradients of the second-order gradients of the third-order gradients of the fourth-order gradients of the fifth-order gradients of the sixth-order gradients of the seventh-order gradients of the eighth-order gradients of the ninth-order gradients of the tenth ... The first and second-order gradients are fused to generate mixed-order gradient information, which is then embedded into the corresponding short-time video or audio sequences to obtain video gradient enhancement sequences and audio gradient enhancement sequences. These sequences undergo multi-stage processing and aggregation to obtain video vectorized representations and audio vectorized representations. The video and audio vectorized representations are mapped to the same dimensional space to obtain video regularization representations and audio regularization representations, respectively. The difference between the video regularization representation and the audio regularization representation is calculated to generate a modal complementarity representation. The modal complementarity representation is then used to perform weighted fusion of the corresponding video regularization representation or audio regularization representation to obtain a complementary fused representation. Finally, a depression level prediction score is output based on the fused representation.

[0018] Thirdly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor, when executing the programs stored in the memory, implements the functions of each module in the multimodal depression assessment system based on gradient embedding and modal complementarity as described in any of the first aspects.

[0019] The multimodal depression assessment system and method based on gradient embedding and modal complementarity provided by this invention firstly extracts the first and second gradients of each element in the sequence along the time axis through a first-order gradient extraction module and a second-order gradient extraction module, respectively. Then, the gradient information of different orders is fused and embedded into the original sequence by a mixed-order gradient embedding module, thereby enhancing the model's ability to capture subtle differences between audio and video sequences of healthy individuals and patients with depression.

[0020] Secondly, by using a multi-head attention module with multi-stage vectorization, the phase sequence vectors output by each layer of multi-head attention are integrated to form a vectorized representation that fully reflects the sequence evolution process, thereby improving the model's ability to characterize temporal features in multiple stages and its classification performance.

[0021] Finally, the difference between different modal representations is calculated by the modal complementarity representation generation module, the complementary representation between audio and video modalities is explicitly constructed, and the adaptive fusion of cross-modal information is achieved through the modal complementarity module, thereby effectively enhancing the model's overall modeling ability for multimodal depressive behavior cues. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of a multimodal depression assessment system based on gradient embedding and modal complementarity provided in an embodiment of the present invention; Figure 2 This invention provides an application flowchart of a multimodal depression assessment system based on gradient embedding and modal complementarity, as provided in an embodiment of the invention. Figure 3 An application flow diagram of a hybrid gradient embedding module provided in an embodiment of the present invention; Figure 4 An application flow diagram of a multi-head attention module with multi-stage vectorization is provided for an embodiment of the present invention; Figure 5 A flowchart illustrating a multimodal depression assessment method based on gradient embedding and modal complementarity provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention; 1-Long-term video face extraction and encoding module; 2-Long-term speech Mel spectrum extraction module; 3-Sequence normalization and short-term sequence generation module; 4-First-order gradient extraction module; 5-Second-order gradient extraction module; 6-Mixed-order gradient embedding module; 61-Gradient joint module; 62-Gradient adjustment module; 7-Multi-head attention module with multi-stage vectorization; 71-Multi-head sequence vectorization module; 72-Sequence aggregation module; 8-Dimensional normalization module; 9-Modal complementarity representation generation module; 10-Modal complementarity module; 11-Depression level prediction module. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Figure 1 This is a schematic diagram of the structure of a multimodal depression assessment system based on gradient embedding and modal complementarity, provided as an embodiment of the present invention. Figure 1 As shown, the system includes: Long-time video face extraction and encoding module 1 is used to extract facial image sequences from the input long-time video and encode them into a video temporal sequence; Long-time speech Mel spectrum extraction module 2 is used to extract the Mel spectrum from the input long-time speech to obtain the audio time sequence; The sequence normalization and short-time sequence generation module 3 is connected to the long-time video face extraction and encoding module 1 and the long-time speech Mel spectrum extraction module 2. It is used to cut and generate fixed-length short-time video sequences and short-time audio sequences from the video time sequence and the audio time sequence, and normalize the short-time video sequences and short-time audio sequences respectively. The first-order gradient extraction module 4 is connected to the sequence normalization and short-time sequence generation module 3, and is used to extract the first-order gradients of the short-time video sequence and the short-time audio sequence, respectively. The second-order gradient extraction module 5 is connected to the sequence normalization and short-time sequence generation module 3, and is used to extract the second-order gradients of the short-time video sequence and the short-time audio sequence respectively. The mixed-order gradient embedding module 6 is connected to the sequence normalization and short-time sequence generation module 3, the first-order gradient extraction module 4, and the second-order gradient extraction module 5. It is used to fuse the first-order gradient and the second-order gradient to generate mixed-order gradient information, and embed the mixed-order gradient information into the corresponding short-time video sequence or short-time audio sequence to obtain the video gradient enhancement sequence and the audio gradient enhancement sequence. A multi-head attention module 7 with multi-stage vectorization is connected to the hybrid gradient embedding module 6, and is used to perform multi-stage processing and aggregation on the video gradient enhancement sequence and the audio gradient enhancement sequence respectively to obtain the video vectorization representation and the audio vectorization representation. The dimension normalization module 8 is connected to the multi-head attention module 7 with multi-stage vectorization, and is used to map the video vectorization representation and the audio vectorization representation to the same dimension space to obtain the video normalization representation and the audio normalization representation respectively. Modal complement representation generation module 9, connected to the dimension normalization module 8, is used to calculate the difference between the video normalization representation and the audio normalization representation to generate a modal complement representation; Modal complementarity module 10, connected to the modal complementarity representation generation module 9, is used to perform weighted fusion of the corresponding video normalization representation or audio normalization representation using the modal complementarity representation to obtain the complementary fused representation; The depression level prediction module 11 is connected to the modality complementarity module 10 and is used to output a depression level prediction score based on the fusion representation.

[0027] Specifically, the long-term video face extraction and encoding module 1 is used to extract facial image sequences from the input long-term video and encode them into a video temporal sequence to achieve a serialized representation of the long-term facial video. To address the need for a regression task predicting depression levels, this embodiment uses a deep convolutional neural network (e.g., ResNet50) pre-trained on the ImageNet dataset as the base model, and fine-tunes its loss function to a mean squared error loss function suitable for regression tasks. In the specific implementation, a face detection tool (e.g., OpenFace) is first used to extract facial image sequences from the long-term video. Then, each facial image is input into the fine-tuned pre-trained model, and its deep feature representation is extracted as the encoding result. Finally, all facial images in the long-term video are encoded into a video temporal sequence, denoted as... ,in It is the dimension of the feature. It is the length of the sequence.

[0028] Long-time speech Mel spectrum extraction module 2 is used to extract Mel spectrum features from the input long-time speech signal, converting the speech signal into a time-frequency representation. The Mel spectrum feature sequence output by this module is denoted as... ,in It is the dimension of the feature. It is the length of the sequence.

[0029] The sequence normalization and short-time sequence generation module 3 is connected to the long-time video face extraction and encoding module 1 and the long-time speech Mel spectrum extraction module 2, and is used to obtain the normalized sequence and generate a short-time audio and video sequence with a fixed length.

[0030] In some embodiments, the sequence normalization and short-time sequence generation module 3 is configured to: generate the short-time video sequence and the short-time audio sequence from the long-time video and long-time speech with a fixed duration window and overlap rate; and perform mean-variance normalization on the short-time video sequence and the short-time audio sequence based on the mean and standard deviation of the video time sequence and the audio time sequence, respectively.

[0031] Specifically, firstly, based on the complete video temporal sequence output by the long-time video face extraction and encoding module 1... Calculate the mean of each row. and standard deviation Simultaneously, based on the complete audio time sequence output by the long-time speech Mel spectrum extraction module 2... Calculate the mean of each row. and standard deviation Next, to ensure temporal alignment of the audio and video modalities, a fixed duration window (e.g., 3 seconds) and a fixed overlap rate (e.g., 50%) are used to synchronously segment the original long-duration video and corresponding long-duration audio, resulting in a series of aligned short-duration video and audio segments. Subsequently, for each short-duration video segment, the face image sequence is extracted and its features are encoded using a pre-trained deep convolutional network to obtain the short-duration video sequence. ,in The dimension of the sequence. The sequence length is represented by the length of the short-time audio segment; for each short-time audio segment, its Mel-spectral features are extracted to obtain the short-time audio sequence. ,in Indicates the dimension of the Mel spectrum. This indicates the frame number of the Mel spectrum. Finally, using... and right Each row is normalized using mean-variance normalization to obtain a normalized short-time video sequence. Similarly, using and being opposed Each row is normalized using mean-variance normalization to obtain the normalized short-time audio sequence. .

[0032] The first-order gradient extraction module 4 is connected to the sequence normalization and short-time sequence generation module 3, and is used to extract the first-order gradient of the short-time audio and video sequence. In some embodiments, the first-order gradient extraction module 4 is configured to extract the first-order gradient of the short-time video sequence or short-time audio sequence through a two-dimensional convolution operation using a predefined first-order differential convolution kernel.

[0033] Specifically, this embodiment constructs a specific form (e.g.) The first-order difference template of ) is used as the parameter of the two-dimensional convolutional layer, respectively for ) and Perform a two-dimensional convolution operation to extract the first-order gradient of each element in the sequence along the time dimension. The final first-order gradient result of the short video sequence is denoted as... The first-order gradient result of a short audio sequence is denoted as... .

[0034] The second-order gradient extraction module 5, connected to the sequence normalization and short-time sequence generation module 3, is used to extract the second-order gradient of the short-time audio and video sequence. In some embodiments, the second-order gradient extraction module 5 is configured to extract the second-order gradient of the short-time video sequence or short-time audio sequence through a two-dimensional convolution operation using a predefined second-order difference convolution kernel.

[0035] Specifically, this embodiment constructs a specific size (e.g.) The second-order difference template of ) is used as the parameter of the two-dimensional convolutional layer, respectively for ) and A two-dimensional convolution operation is performed to extract the second-order gradient of each element in the sequence along the time dimension. The final second-order gradient result of the short video sequence is denoted as... The second gradient result of a short audio sequence is denoted as... .

[0036] The mixed-order gradient embedding module 6, connected to the sequence normalization and short-time sequence generation module 3, the first-order gradient extraction module 4, and the second-order gradient extraction module 5, is used to embed the mixed information of the first-order gradient and the second-order gradient into the short-time audio and video sequence to obtain the audio and video gradient enhancement sequence.

[0037] A multi-head attention module 7 with multi-stage vectorization, connected to a hybrid-order gradient embedding module 6, is used to capture the inter-frame dependencies in the audio-video gradient enhancement sequence and realize the vectorized representation of the audio-video gradient enhancement sequence, wherein the video vectorized representation is denoted as... Audio vectorization is denoted as .

[0038] The dimension normalization module 8, connected to the multi-head attention module 7 with multi-stage vectorization, is used to map the vectorized representation of the audio and video gradient enhancement sequence to the same dimensional space. Specifically, this embodiment uses two fully connected layers to vectorize the different modalities... and Mapping to a representation space of the same dimension yields a video regularization representation. and audio regularization representation To ensure consistency in semantic space among regular representations of different modalities, the following loss function is introduced as a constraint during the model training phase: (1) in, It is to handle and Shared linear layer, These are the actual depression level labels corresponding to the samples. N It represents the total number of training samples.

[0039] The modal complementarity representation generation module 9, connected to the dimension normalization module 8, is used to generate complementary representations between audio and video modalities. In some embodiments, the modal complementarity representation generation module 9 is configured to: subtract the audio normalization representation from the video normalization representation to obtain a complementary representation of video to audio; and / or, subtract the video normalization representation from the audio normalization representation to obtain a complementary representation of audio to video.

[0040] Specifically, this embodiment defines To represent the complementary relationship between the video modality and the audio modality, define... This represents the complementary representation of the audio modality to the video modality.

[0041] The modal complementarity module 10, connected to the modal complementarity representation generation module 9, is used to achieve information complementarity between audio and video. In some embodiments, the modal complementarity module 10 is configured to: concatenate the video normalization representation or audio normalization representation with its corresponding modal complementarity representation; input the concatenation result into a fully connected layer and a Softmax function to generate corresponding complementarity weights; and use the complementarity weights to perform weighted fusion of the video normalization representation or audio normalization representation and the corresponding modal complementarity representation to obtain a complementary fused representation.

[0042] Specifically, in the process of supplementing the audio modality with the video modality, the audio normalization representation is first performed. Complementary representation of audio to video The concatenation is performed to form a column vector; this column vector is then input into a fully connected layer to obtain intermediate features. By using the Reshape layer Shape transformation The weight matrix is ​​obtained by using the Softmax function along the column dimensions. The final result of supplementing the audio with the video is obtained through the following weighted calculation. : (2) in, and They are The first and second lines.

[0043] Similarly, in the process of supplementing the video mode with the audio mode, the video frequency normalization representation is first performed. Complementary representation of audio to video The concatenation is performed to form a column vector; this column vector is then input into a fully connected layer to obtain intermediate features. By using the Reshape layer Shape transformation The weight matrix is ​​obtained by using the Softmax function along the column dimensions. Finally, the audio-to-video supplementary result is obtained through weighted calculation. For reference, see formula (2).

[0044] The depression severity prediction module 11, connected to the modality complementarity module 10, is used to output a depression severity prediction result, such as a depression score, based on the fused representation. This module will... and The inputs are fed into two fully connected layers, resulting in two prediction scores, denoted as follows: and Then, and The elements are concatenated into a two-element column vector and then fed into a final fully connected layer for integration, yielding the final depression level prediction score. .

[0045] During model training, the system is optimized using the following loss function: (3) in, and These are two weighting coefficients.

[0046] Figure 2This is an application flowchart of a multimodal depression assessment system based on gradient embedding and modal complementarity provided in an embodiment of the present invention. It should be noted that the modules repeatedly presented in the diagram (such as sequence normalization and short-time sequence generation module 3, first-order gradient extraction module 4, second-order gradient extraction module 5, etc.) are actually the same functional module. Since the system needs to process both "video" and "audio" input modalities simultaneously, the same module process needs to be executed independently for each modality. Therefore, the repeated drawing reflects the parallel processing logic.

[0047] like Figure 2 As shown, the system's input is divided into two parallel processing paths: a video path and an audio path. Video path flow: After inputting the video, it goes through long-term video face extraction and encoding, sequence normalization and short-term sequence generation, first-order gradient extraction and second-order gradient extraction, mixed-order gradient embedding, multi-stage vectorization multi-head attention, and finally completes dimension normalization. Audio path flow: After inputting audio, it sequentially goes through long-time speech Mel spectrum extraction, sequence normalization and short-time sequence generation, first-order gradient extraction and second-order gradient extraction, mixed-order gradient embedding, multi-stage vectorization multi-head attention, and finally completes dimension normalization. After both processes have completed dimensional normalization, the system will summarize the results from both processes and pass them sequentially through the modal complement representation generation module 9, the modal complement module 10, and the depression level prediction module 11, finally outputting the prediction result of the depression level.

[0048] Figure 3 An application flow architecture diagram of a hybrid-order gradient embedding module provided in an embodiment of the present invention is shown below. Figure 3 As shown, the mixed-order gradient embedding module 6 includes: a gradient joint module 61, connected to the first-order gradient extraction module 4 and the second-order gradient extraction module 5, configured to concatenate the first-order gradient and the second-order gradient, and generate mixed-order gradient weights through a convolutional layer and a Softmax function; and a gradient adjustment module 62, connected to the gradient joint module 61 and the sequence normalization and short-time sequence generation module 3, configured to embed the mixed-order gradient weights into the corresponding short-time video sequence or short-time audio sequence through a Hadamard product operation.

[0049] Specifically, the gradient joint module 61 is connected to the first-order gradient extraction module 4 and the second-order gradient extraction module 5. This model will extract the gradients from the video sequence. and Concatenate along the channel axis to form a tensor with two channels. ; to the audio sequence and Concatenate along the channel axis to form a tensor with two channels. .

[0050] Taking audio modal processing as an example, according to formula (4), the number of channels is increased and compressed through two convolutional layers, and the result is denoted as... The corresponding processing results for the video modality are denoted as .

[0051] (4) in It is the ReLU activation function. It is a two-dimensional convolutional layer.

[0052] Subsequently, according to Equation (5), the Softmax function is used along the channel axis to obtain the weight tensor. The weight tensor of the video modality is denoted as .

[0053] (5) Finally, according to formula (6), the gradient joint result is obtained through weighted fusion. The joint gradient result of the video modalities is denoted as .

[0054] (6) in and They are The first and second channels.

[0055] Gradient adjustment module 62 is connected to gradient joint module 61 and sequence normalization and short-time sequence generation module 3, respectively, and adjusts the mixed-order joint gradient through Hadamard product. and Embedded into sequence and middle.

[0056] Figure 4 An application flow diagram of a multi-head attention module with multi-stage vectorization is provided for an embodiment of the present invention, such as... Figure 4 As shown, the multi-head attention module 7 with multi-stage vectorization includes: a multi-head sequence vectorization module 71, connected to the mixed-order gradient embedding module 6, configured to process the video gradient enhancement sequence and the audio gradient enhancement sequence respectively through an N-layer multi-head self-attention mechanism, and perform one-dimensional convolution on the output of each layer to obtain a staged sequence vector; and a sequence aggregation module 72, connected to the multi-head sequence vectorization module 71, configured to concatenate the N staged sequence vectors and aggregate them into the video vectorized representation or the audio vectorized representation through an attention mechanism.

[0057] Specifically, the multi-head sequence vectorization module 71 is connected to the mixed-order gradient embedding module 6. The result of this module is input into the multi-head attention mechanism, as shown in formula (7), and the corresponding result is denoted as and .

[0058] (7) MHSA is an abbreviation for Multi-Head Self-Attention. This indicates that MHSA will be executed N times. It is the Hadamard product operation.

[0059] Taking audio modality processing as an example, to obtain the vectorized result of the sequence, as shown in formula (8), a one-dimensional convolution layer is used to summarize the output of each MHSA, and the corresponding result is denoted as Correspondingly, the processing result of the video modality is denoted as: .

[0060] (8) in Represents one-dimensional convolution. This indicates that MHSA is executing the first The output of the next number of times.

[0061] Sequence aggregation module 72 is connected to multi-head sequence vectorization module 71. This module aggregates N stage sequence vectors ( , The feature matrix is ​​constructed by concatenating elements along the column dimensions, i.e. , .

[0062] Then, according to formula (9), the attention mechanism is used to obtain the vectorized result, denoted as... , .

[0063] (9) in It is a parameter matrix. This refers to matrix multiplication.

[0064] The software implementation of the system described in this invention is based on the Python programming language, possessing good cross-platform compatibility and stable operation in mainstream operating system environments such as Windows and Linux. It should be noted that the implementation of this invention is not limited to a specific programming language or operating platform; it can also be implemented using other programming languages ​​and deployed and executed on corresponding hardware platforms.

[0065] At the application level, this invention can be deployed on a variety of computing devices, including but not limited to computer terminals such as desktop computers and laptops, as well as portable mobile devices such as smartphones and tablets, providing flexible technical support for the assessment of depression levels in different scenarios.

[0066] The multimodal depression assessment system based on gradient embedding and modal complementarity in this embodiment first extracts the first and second gradients of each element in the sequence along the time axis through the first-order gradient extraction module and the second-order gradient extraction module, respectively. Then, the gradient information of different orders is fused and embedded into the original sequence by the mixed-order gradient embedding module, thereby enhancing the model's ability to capture subtle differences between audio and video sequences of healthy individuals and patients with depression.

[0067] Secondly, by using a multi-head attention module with multi-stage vectorization, the phase sequence vectors output by each layer of multi-head attention are integrated to form a vectorized representation that fully reflects the sequence evolution process, thereby improving the model's ability to characterize temporal features in multiple stages and its classification performance.

[0068] Finally, the difference between different modal representations is calculated by the modal complementarity representation generation module, the complementary representation between audio and video modalities is explicitly constructed, and the adaptive fusion of cross-modal information is achieved through the modal complementarity module, thereby effectively enhancing the model's overall modeling ability for multimodal depressive behavior cues.

[0069] Figure 5 This is a flowchart illustrating a multimodal depression assessment method based on gradient embedding and modal complementarity provided in an embodiment of the present invention. It is applied to the aforementioned multimodal depression assessment system based on gradient embedding and modal complementarity, as shown below. Figure 5 As shown, the method includes: Step S501: Obtain the input long-duration video and the corresponding long-duration audio.

[0070] Step S502: Extract facial image sequences from the input long-term video and encode them into a video temporal sequence.

[0071] Step S503: Extract the Mel spectrum from the input long-term speech to obtain the audio time sequence.

[0072] Step S504: Cut short-time video sequences and short-time audio sequences of fixed length from the video time sequence and audio time sequence, and normalize the short-time video sequences and short-time audio sequences respectively.

[0073] Step S505: Extract the first-order gradients of the short-time video sequence and the short-time audio sequence respectively.

[0074] Step S506: Extract the second-order gradients of the short-time video sequence and the short-time audio sequence respectively.

[0075] Step S507: Fuse the first-order gradient and the second-order gradient to generate mixed-order gradient information, and embed the mixed-order gradient information into the corresponding short-time video sequence or short-time audio sequence to obtain video gradient enhancement sequence and audio gradient enhancement sequence.

[0076] Step S508: Perform multi-stage processing and aggregation on the video gradient enhancement sequence and the audio gradient enhancement sequence respectively to obtain the video vectorization representation and the audio vectorization representation.

[0077] Step S509: Map the video vectorized representation and the audio vectorized representation to the same dimensional space to obtain the video regularization representation and the audio regularization representation, respectively.

[0078] Step S510: Calculate the difference between the video regularization representation and the audio regularization representation to generate a modal complementary representation.

[0079] Step S511: Use the modal complementarity representation to perform weighted fusion on the corresponding video regularization representation or audio regularization representation to obtain the complementary fused representation.

[0080] Step S512: Output a depression level prediction score based on the fusion representation.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and corresponding beneficial effects of the multimodal depression assessment method based on gradient embedding and modal complementarity described above can be found in the corresponding process in the previous system example, and will not be repeated here.

[0082] like Figure 6 As shown, this embodiment of the invention provides an electronic device, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other via the communication bus 604. Memory 603 is used to store computer programs; In one embodiment of the present invention, when the processor 601 executes the program stored in the memory 603, it implements the functions of each module in any of the aforementioned multimodal depression assessment systems based on gradient embedding and modal complementarity.

[0083] The electronic device provided in this embodiment of the invention has a similar implementation principle and technical effect to the above embodiments, and will not be described again here.

[0084] The aforementioned memory 603 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Memory 603 has storage space for program code used to perform any of the method steps described above. For example, the storage space for program code may include individual program codes for implementing the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, optical discs (CDs), memory cards, or floppy disks. Such computer program products are typically portable or fixed storage units. The storage unit may have storage segments or storage spaces arranged similarly to memory 603 in the aforementioned electronic device. The program code may be compressed, for example, in a suitable form. Typically, the storage unit includes programs for performing the method steps according to embodiments of the invention, i.e., code that can be read by a processor such as 601, which, when run by the electronic device, causes the electronic device to perform the various steps in the methods described above.

[0085] Embodiments of the present invention also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the functions of each module in the multimodal depression assessment system based on gradient embedding and modal complementarity as described above.

[0086] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of the present invention.

[0087] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0088] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A multimodal depression assessment system based on gradient embedding and modal complementarity, characterized in that, include: The long-time video face extraction and encoding module is used to extract facial image sequences from the input long-time video and encode them into a video temporal sequence; The long-time speech Mel spectrum extraction module is used to extract the Mel spectrum from the input long-time speech to obtain the audio time sequence; The sequence normalization and short-time sequence generation module is connected to the long-time video face extraction and encoding module and the long-time speech Mel spectrum extraction module. It is used to cut and generate fixed-length short-time video sequences and short-time audio sequences from the video time sequence and the audio time sequence, and normalize the short-time video sequences and short-time audio sequences respectively. A first-order gradient extraction module, connected to the sequence normalization and short-time sequence generation modules, is used to extract the first-order gradients of the short-time video sequence and the short-time audio sequence, respectively. The second-order gradient extraction module, connected to the sequence normalization and short-time sequence generation modules, is used to extract the second-order gradients of the short-time video sequence and the short-time audio sequence, respectively. The mixed-order gradient embedding module connects the sequence normalization and short-time sequence generation module, the first-order gradient extraction module, and the second-order gradient extraction module. It is used to fuse the first-order gradient and the second-order gradient to generate mixed-order gradient information, and embed the mixed-order gradient information into the corresponding short-time video sequence or short-time audio sequence to obtain the video gradient enhancement sequence and the audio gradient enhancement sequence. A multi-head attention module with multi-stage vectorization is connected to the hybrid gradient embedding module, which is used to perform multi-stage processing and aggregation on the video gradient enhancement sequence and the audio gradient enhancement sequence respectively to obtain the video vectorization representation and the audio vectorization representation. The dimension normalization module, connected to the multi-head attention module with multi-stage vectorization, is used to map the video vectorization representation and the audio vectorization representation to the same dimension space, so as to obtain the video normalization representation and the audio normalization representation respectively. A modal complement representation generation module, connected to the dimension normalization module, is used to calculate the difference between the video normalization representation and the audio normalization representation to generate a modal complement representation; The modal complementation module, connected to the modal complementation representation generation module, is used to perform weighted fusion of the corresponding video normalization representation or audio normalization representation using the modal complementation representation to obtain the complementary fused representation; The depression severity prediction module, connected to the modality complementarity module, is used to output a depression severity prediction score based on the fused representation.

2. The system according to claim 1, characterized in that, The hybrid-order gradient embedding module includes: The gradient joint module, connected to the first-order gradient extraction module and the second-order gradient extraction module, is configured to concatenate the first-order gradient and the second-order gradient, and generate mixed-order gradient weights through a convolutional layer and a Softmax function. The gradient adjustment module, connected to the gradient joint module and the sequence normalization and short-time sequence generation module, is configured to embed the mixed-order gradient weights into the corresponding short-time video sequence or short-time audio sequence through Hadamard product operation.

3. The system according to claim 1, characterized in that, The multi-head attention module with multi-stage vectorization includes: The multi-head sequence vectorization module, connected to the mixed-order gradient embedding module, is configured to process the video gradient enhancement sequence and the audio gradient enhancement sequence respectively through an N-layer multi-head self-attention mechanism, and perform one-dimensional convolution on the output of each layer to obtain a staged sequence vector. The sequence aggregation module, connected to the multi-head sequence vectorization module, is configured to concatenate N staged sequence vectors and aggregate them into the video vectorization representation or audio vectorization representation through an attention mechanism.

4. The system according to claim 1, characterized in that, The modal complement representation generation module is configured as follows: Subtracting the audio normalization representation from the video normalization representation yields the complementary representation of the audio from the video; and / or, Subtracting the video regularization representation from the audio regularization representation yields the complementary representation of audio to video.

5. The system according to claim 1 or 4, characterized in that, The modal complementarity module is configured as follows: The video or audio normalization representation is spliced ​​together with its corresponding modal complementary representation. The concatenated result is input into a fully connected layer and a Softmax function to generate corresponding complementary weights; The complementary weights are used to perform weighted fusion of the video regularization representation or audio regularization representation and the corresponding modal complementary representation to obtain the complementary fused representation.

6. The system according to claim 1, characterized in that, The first-order gradient extraction module is configured to extract the first-order gradient of the short-time video sequence or short-time audio sequence through a two-dimensional convolution operation using a predefined first-order differential convolution kernel.

7. The system according to claim 1 or 6, characterized in that, The second-order gradient extraction module is configured to extract the second-order gradient of the short-time video sequence or short-time audio sequence through a two-dimensional convolution operation using a predefined second-order difference convolution kernel.

8. The system according to claim 1, characterized in that, The sequence normalization and short-time sequence generation module is configured as follows: The short-time video sequence and short-time audio sequence are generated by segmenting the long-time video and long-time speech using a fixed duration window and overlap rate; Based on the mean and standard deviation of the video time series and the audio time series, mean-variance normalization is performed on the short-time video series and the short-time audio series respectively.

9. A multimodal depression assessment method based on gradient embedding and modal complementarity, characterized in that, The method applied to the multimodal depression assessment system based on gradient embedding and modal complementarity as described in any one of claims 1-8, the method comprising: Obtain the input long-duration video and its corresponding long-duration audio; Extract facial image sequences from the input long-term video and encode them into a video temporal sequence; Extract the Mel spectrum from the input long-term speech to obtain the audio time sequence; Fixed-length short-time video sequences and short-time audio sequences are generated by cutting from the video time-series sequence and the audio time-series sequence, and the short-time video sequences and short-time audio sequences are normalized respectively; Extract the first-order gradients of the short-time video sequence and the short-time audio sequence respectively; Extract the second-order gradients of the short-time video sequence and the short-time audio sequence respectively; The first-order gradient and the second-order gradient are fused to generate mixed-order gradient information, and the mixed-order gradient information is embedded into the corresponding short-time video sequence or short-time audio sequence to obtain video gradient enhancement sequence and audio gradient enhancement sequence. The video gradient enhancement sequence and the audio gradient enhancement sequence are processed and aggregated in multiple stages to obtain the video vectorization representation and the audio vectorization representation. The video vectorization representation and the audio vectorization representation are mapped to the same dimensional space to obtain the video regularization representation and the audio regularization representation, respectively. The difference between the video warping representation and the audio warping representation is calculated to generate a modal complementary representation; The modal complementarity representation is used to perform weighted fusion of the corresponding video regularization representation or audio regularization representation to obtain the complementary fused representation. The depression level prediction score is output based on the fusion representation.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the functions of each module in the multimodal depression assessment system based on gradient embedding and modal complementarity as described in any one of claims 1-8.

Images