A depression recognition method based on voice and video modalities

Abstract

The application relates to the technical field of depression recognition and discloses a depression recognition method based on voice and video modalities, which comprises the following steps: preprocessing and initial feature extraction are performed on obtained original voice data and original video data; initial voice features are input into a voice branch, features are extracted by combining local window attention and global attention based on local sensitive hashing, a memory propagation mechanism is used to enhance the context continuity of long voice sequences across sections, and voice feature representation is obtained; initial video features are input into a video branch, fine-grained expression dynamics are reserved through hierarchical feature learning, and video feature representation is obtained; the features of the two modalities are aligned, and dynamic weighted fusion is performed through an adaptive post-fusion mechanism to obtain joint feature representation; and finally, the joint feature representation is input into a classification module to output a depression state classification result. The application solves the problems of long voice semantic fragmentation and high overhead, reserves micro-expression, and improves the robustness and accuracy of the system.

Description

Technical Field

[0001] This invention relates to the field of depression recognition technology, and more specifically, to a depression recognition method based on voice and video modalities. Background Technology

[0002] Depression is a common and prevalent mental disorder that can lead to self-harm or suicide in severe cases, significantly impacting both individual and public health. Therefore, early screening and timely intervention for depression are crucial. Currently, clinical diagnosis relies primarily on patient self-reporting, scale assessments, and physician judgment. However, these methods have limited objectivity and consistency and are constrained by medical resources, making large-scale screening difficult. In recent years, with the development of artificial intelligence technology, utilizing multimodal information such as voice and video for depression-assisted identification has gradually become an important research direction.

[0003] Among related technologies, existing multimodal depression recognition methods based on voice and video still have significant limitations.

[0004] On the one hand, in the process of speech feature modeling, existing methods based on Transformer or self-attention mechanisms usually face the problem of large computational complexity and storage overhead when processing long sequence speech data. On the other hand, some methods that adopt local attention mechanisms can improve processing efficiency to a certain extent, but they are difficult to fully model long-range contextual information, thus affecting the effective capture of depression-related speech features.

[0005] On the other hand, in the process of video feature modeling, video data has both temporal and spatial features. Directly performing overall modeling usually introduces a high computational and storage burden, while overly coarse-grained modeling methods are prone to causing the loss of fine-grained facial expression dynamic information and local regional features.

[0006] Furthermore, in the multimodal fusion stage, most existing methods adopt fixed weights or relatively coarse-grained decision combination methods, which make it difficult to dynamically adjust the modal contribution according to the changes in the information quality of speech and video modalities in different samples, resulting in the inability to fully utilize the complementary information between modalities.

[0007] To address this, we propose a depression identification method based on both voice and video modalities. Summary of the Invention

[0008] This invention provides a depression recognition method based on speech and video modalities. It addresses the technical problems of existing methods based on Transformer or self-attention mechanisms, which suffer from high computational complexity and storage overhead when processing long-sequence speech data; high computational and storage burden during video feature modeling; and the loss of fine-grained facial expression dynamics and local region features due to overly coarse-grained modeling. Furthermore, in the multimodal fusion stage, most existing methods use fixed weights or relatively coarse-grained decision combinations, making it difficult to dynamically adjust modal contributions based on changes in the information quality of speech and video modalities in different samples, thus hindering the full utilization of complementary information between modalities.

[0009] This invention provides a method for depression recognition based on speech and video modalities, comprising the following steps: S1, multimodal data preprocessing and initial feature extraction: acquiring the original speech data and original video data of the object to be identified, and preprocessing and extracting initial features from the original speech data and original video data respectively to obtain initial speech features and initial video features; S2, speech feature encoding and context enhancement: inputting the initial speech features into the speech branch for encoding to obtain speech feature representation; S3, video hierarchical feature encoding: inputting the initial video features into the video branch for hierarchical encoding to obtain video feature representation; S4, multimodal feature adaptive fusion: uniformly aligning the speech feature representation and video feature representation, and dynamically weighting and fusing them through an adaptive post-fusion mechanism to obtain a joint feature representation; S5, depression state classification output: inputting the joint feature representation into the classification module to output the depression state classification result.

[0010] As a further improvement of the present invention, the steps of preprocessing and initial feature extraction of the original speech data include S11, segmenting the original speech data into sentences, and generating enhanced speech samples based on sentence-level substitution; S12, extracting deep acoustic features from the enhanced speech samples to obtain initial speech features.

[0011] As a further improvement of the present invention, the steps of preprocessing and initial feature extraction of the original video data include S13, performing temporal filtering, face localization and alignment, image cropping and normalization processing on the original video data to obtain preprocessed original video data; S14, extracting initial video features for subsequent hierarchical coding from the preprocessed original video data to obtain initial video features.

[0012] As a further improvement of the present invention, step S13 includes the following steps: S131, resampling the original video data and obtaining the timestamp of the speaking interval based on the audio channel; S132, performing face localization, cross-frame association and key point alignment on each frame in the speaking interval, and discarding frames that do not meet the preset quality conditions; S133, performing outward cropping and uniform scaling on the retained face region, and extracting a fixed-length continuous video segment as the initial video feature to obtain the preprocessed original video data.

[0013] As a further improvement of the present invention, in step S2, the speech branch includes a hybrid attention mechanism and a memory propagation mechanism, which is used to segment the long speech sequence and introduce historical memory information of the previous segment during the current segment encoding process to enhance the contextual continuity of the long speech sequence across segments; the initial speech features that take into account both local dependencies and global contextual information are extracted through the hybrid attention mechanism, and the contextual continuity of the long speech sequence across segments is enhanced through the memory propagation mechanism to obtain a speech feature representation.

[0014] As a further improvement of the present invention, the hybrid attention mechanism includes local window attention and global attention based on local sensitive hashing, which combines local window attention with global attention based on local sensitive hashing to take into account both local acoustic pattern modeling and long-range context dependency modeling.

[0015] As a further improvement of the present invention, the hybrid attention mechanism also includes a context gating block, which is used to adaptively weight the outputs of local window attention and global attention based on local sensitive hashing to obtain speech feature representation.

[0016] As a further improvement of the present invention, in step S3, the video branch includes a block segmentation module, which is used to segment the spatial dimension of the video frame and map it to a serialized marker representation, so as to form a video feature input suitable for subsequent encoding while preserving local details; a video coding block, which is used to perform feature modeling on each local region based on local window attention, and to realize cross-window information interaction through an alternating window offset mechanism, so as to enhance the correlation modeling capability between different spatial regions; a spatial dimensionality reduction and merging block, which is used to aggregate and reduce the dimensionality of the markers of adjacent spatial locations, so as to realize the gradual reduction of spatial scale and the gradual enhancement of feature semantics, and obtain the video feature representation; through spatial segmentation, multi-scale hierarchical modeling, local window attention, cross-window interaction, and spatial dimensionality reduction and merging, hierarchical feature learning is performed on the initial video data to obtain the video feature representation.

[0017] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a depression recognition method based on voice and video modalities.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a depression recognition method based on voice and video modalities.

[0019] The beneficial effects of this invention are as follows: This invention introduces a hybrid attention mechanism and a memory propagation mechanism during the speech feature modeling process to balance the efficiency of long speech sequence processing with the utilization of global contextual information; during the video feature modeling process, a hierarchical video coding method is adopted to enhance the representation ability of visual local regions and fine-grained dynamic features; during the audio-video fusion process, an adaptive post-fusion mechanism is adopted to improve the model's adaptability to quality fluctuations in different modalities, thereby achieving more stable and accurate depression recognition. Attached Figure Description

[0020] Figure 1 This is a flowchart of a depression recognition method based on voice and video modalities according to the present invention; Figure 2 This is a structural diagram of the hybrid attention mechanism in a depression recognition method based on voice and video modalities according to the present invention; Figure 3 This is a structural diagram of the memory propagation mechanism in a depression recognition method based on voice and video modalities according to the present invention; Figure 4 This is a stacked structure diagram of two video coding blocks in a depression recognition method based on speech and video modalities according to the present invention; Figure 5 This is a structural diagram of the spatial dimension reduction and merging block in a depression recognition method based on voice and video modalities according to the present invention. Detailed Implementation

[0021] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0022] like Figures 1-5As shown in the figure, a depression recognition method based on speech and video modalities includes the following steps: S1. Multi-modal data preprocessing and initial feature extraction: Obtain the original speech data and original video data of the object to be recognized, and perform preprocessing and initial feature extraction on the original speech data and original video data respectively to obtain initial speech features and initial video features.

[0023] Specifically, for speech data, first segment the original interview recording, i.e., the original speech data, into N independent sentences as basic semantic units; then use a sliding window with a fixed length of M (M < N) to traverse the sentence sequence, and randomly reorganize the M consecutive sentences within each window to generate enhanced samples with a new context structure. Through the iterative advancement of the sliding window (step size: M sentences), cover all the corpus, and use zero-padding to handle the possible insufficient sentences in the end window to ensure that all samples have the same dimension. After generating augmented samples using the aforementioned data augmentation method, use the pre-trained Wav2Vec2.0 model to extract deep acoustic features from the original speech data.

[0024] For video data, first resample the original video data and screen the speaking intervals in the time dimension.

[0025] Among them, obtain the timestamps of continuous speaking segments based on the audio channel, and combine the mouth movement intensity to constrain the transition frames to obtain an effective video frame sequence.

[0026] Then, perform face localization, cross-frame association, and key point alignment on each frame in the effective video frame sequence to normalize the facial pose and scale differences. On this basis,剔除 frames that do not meet the preset quality conditions, and perform outer expansion cropping and unified scaling on the remaining face regions.

[0027] Subsequently, perform normalization processing on the cropped face images, intercept continuous video segments with a fixed length from the screened speaking intervals, and sample them in a sliding window manner. In the training stage, further apply lightweight semantic-preserving augmentation to the video segments, and finally obtain standardized initial video features.

[0028] S2. Speech feature encoding and context enhancement: Input the initial speech features into the speech branch for encoding to obtain speech feature representations.

[0029] Specifically, input the initial speech features obtained in S1 into the speech branch for encoding. The speech branch includes a hybrid attention mechanism and a memory propagation mechanism. Among them, first use the hybrid attention mechanism to perform feature encoding on the initial speech features to take into account both local dependency modeling and global context information extraction.

[0030] Then, for the segmentation process of long speech sequences, a memory propagation mechanism is introduced to maintain and transmit historical information between adjacent speech segments, so as to enhance the contextual continuity of long speech sequences across segments.

[0031] The final speech feature representation is obtained.

[0032] Furthermore, S2 includes the following sub-steps: S2.1, Hybrid Attention Mechanism Encoding: The initial input speech features are subjected to hybrid attention encoding. This hybrid attention mechanism combines local window attention with global attention based on locality-sensitive hashing, and introduces a context-gated block to adaptively weight the outputs of different attention branches. This improves the modeling ability for long-range contextual information while controlling computational complexity. Its structure diagram is shown below. Figure 2 As shown.

[0033] Let the input of the hybrid attention module be X∈R B×T×D ,in B Indicates batch size, T Indicates the sequence length. D Let D represent the embedding dimension. Let H be the total number of attention heads. Assuming D is divisible by H, then the dimension of each attention head is... d k =D / H For the first h A person's attention h Query vector Q h Key vector K h AND value vector V h We obtain this through linear projection: (1) in, W s,h ∈R D×dk For the first h The projection matrix shared by the key and the query in each attention head. W v,h ∈R D×dk Let be the head-specific projection matrix of the value vector, and Q h ,K h ,V h ∈R B×T×dk .

[0034] In the hybrid attention mechanism, H Each attention head is divided into two subsets: the first... Hlocal Attention points (numbered 1 to...) H local ) is used for local window attention, the rest H local Attention point (number) H local +1 to H This is used for global attention based on locality-sensitive hashing. This partitioning method is a predefined hyperparameter that remains fixed during the training and inference phases.

[0035] For the global attention component, this paper adopts the attention mechanism based on locality-sensitive hashing proposed in Reformer. Specifically, the... h The global attention output of each attention head is: (2) For the local window attention part, the first h Attention is focused on each time step t ∈ 1,…,T Local attention is computed independently on the first level. Specifically, in the first level... t This mechanism operates only at a fixed window width. W Attention calculation is performed internally, that is, from K h With V h Selected from t−[W / 2] to t+[W / 2] Each of the tokens constitutes a subtensor. K h,t ∈R B×W×dk and V h,t ∈R B ×W×dk .

[0036] For the local window attention part, the first h Attention is focused on each time step t ∈ 1,…,T Local attention is computed independently on the first level. Specifically, in the first level... t This mechanism operates only at a fixed window width. W Attention calculation is performed internally, that is, from K h With V h Selected from t−[W / 2] to t+[W / 2] Each of the tokens constitutes a subtensor. K h,t ∈R B×W×dk and V h,t ∈R B ×W×dk .

[0037] At the same time, using Q h,t ∈R B×dk Q represents h The tth Each token. Local attention at time step. t The output is defined as: (3) According to formula (3), by concatenating the outputs of all time steps, the complete local attention result of the i-th attention head can be obtained: (4) After completing the calculations for local window attention and global attention based on locality-sensitive hashing, the output representations of each attention head can be obtained. Att h local and Att h global The multi-head attention results described above will serve as input to the subsequent context gating module. This module adaptively scales the outputs of different attention branches at the head level to generate the final hybrid attention representation. The specific operation is as follows: By performing average pooling on the input sequence along the time dimension, the global context vector is obtained. c ∈R B×D . Let X t ∈R B×D Represents the input sequence X The Middle t Given several tokens, the vector c is calculated as follows: (5) Obtaining the global context vector c Then, a gated vector is generated through a gated function containing a two-layer feedforward network. g =[ g 1 ,…,g H ]∈R B×H The network employs the GELU activation function and uses a Sigmoid nonlinear mapping in the output layer, with the following form: (6) in, W 1 ∈R D×D / 2 and W 2 ∈R D / 2×H For learnable weight matrix, b1 and b2σ(⋅) is the bias term, representing the Sigmoid function.

[0038] In the gate vector g In, each scalar g h Used to control the corresponding number h The contribution strength of each attention head is determined. Then, the outputs of all attention heads are scaled using these gating coefficients. Specifically, the contribution strength of each attention head is determined. h Weighted input of individual attention points O h ∈R B×T×dk Defined as: (7) Finally, the weighted outputs of all attention heads are concatenated and linearly projected to obtain the final output of the hybrid attention mechanism: (8) in, W O ∈R Hdk×D This indicates the output projection matrix.

[0039] S2.2, Contextual Enhancement of Memory Propagation Mechanism: The long speech sequence encoded using a hybrid attention mechanism is segmented, and the memory state corresponding to the previous segment is introduced during the feature encoding process of the current segment. The current subsequence and historical memory information are jointly input to achieve the transfer of contextual information across segments. After the current segment encoding is completed, the current output and historical memory state are updated to generate a new memory state for subsequent segmentation, thereby obtaining an enhanced speech feature representation. Its structure diagram is shown below. Figure 3 As shown.

[0040] In the proposed memory propagation mechanism, for ultra-long input sequences X ∈R B×T×D First, divide it into τ max A length of L The non-overlapping subsequences are denoted as follows: Regarding the first τ Subsequences X τ (1≤ τ ≤ τ max ), introduce the corresponding memory state m τ−1 ∈R B×M×D ,in M This indicates a fixed memory length and an initial memory state. m 0 Set to empty.

[0041] Given the current subsequence X τ and their corresponding memory states m τ−1 First, the two are concatenated along the time dimension and used as input to the hybrid attention module to obtain the hybrid attention output of the current segment. This method allows the use of contextual information from historical segments when processing the current segment.

[0042] In obtaining Then, a memory update operation is performed to generate the memory state used by the next segment. m τ Specifically, first, the current output... Memory state of the previous moment m τ−1 The data is then stitched together and transformed using linear projection. Subsequently, a truncation operation is performed to retain only the nearest features. M Each time step forms a new memory state. m τ 。 This process is repeated on all subsequences to obtain the output of each segment sequentially. Finally, the attention output of each segment is mixed. L The effective time steps are concatenated to obtain the overall output. Att mixed ∈R B×T×D .

[0043] S3, Hierarchical Video Feature Encoding: The initial video features are input into the video branch for hierarchical encoding to obtain the video feature representation.

[0044] Specifically, the initial video features obtained in S1 are input into the video branch for hierarchical feature encoding. The video branch includes a segmentation module, a video coding block, and a spatial dimensionality reduction and merging block. First, the segmentation module locally partitions and embeds representations of the video frames to obtain a labeled sequence suitable for subsequent encoding. Then, the video coding block performs local in-window feature modeling and cross-window information interaction on the labeled sequence to enhance the association representation capability between different spatial regions. Finally, the spatial dimensionality reduction and merging block aggregates and reduces the dimensionality of the features layer by layer, reducing spatial resolution while enhancing high-level semantic expression capabilities to obtain the video feature representation.

[0045] Furthermore, S3 includes the following sub-steps: S3.1, Block module processing: The initial video features are spatially segmented, dividing each video frame into multiple local blocks. Each local block is then embedded and mapped to obtain a labeled representation that preserves the spatial grid structure. To compensate for the weakening of absolute position information caused by the segmentation operation, position encoding can be introduced into the labeled representation to form the input features for subsequent video coding blocks.

[0046] Let the spatial resolution of the preprocessed video clip be... H×W If the value is 288×288, then we have V ∈R T×288×288×3 This module only divides the video into blocks in the spatial dimension, while keeping the temporal dimension unchanged, thus representing the video as a sequence of elements. T The frame sequence is composed of frames, each frame is independently spatially divided and embedded, and finally stacked in the time dimension to obtain a temporal marker sequence.

[0047] Specifically, for any time t frame image V t ∈R 288×288×3 Using a side length of p Non-overlapping spatial blocks are obtained in both the height and width directions. Each block forms a unit of... × A two-dimensional mesh composed of local blocks. To ensure strict alignment with the subsequent layer-by-layer merging of the four-layer structure, the modules are fixed. p =4, therefore each frame is divided into 72×72 local blocks. Each local block covers a 4×4 pixel area of ​​the original image and contains 3 color channels, which can be regarded as a local tensor. x t,i,j ∈R p×p×3 ,in i ∈1,…,72, j ∈1,…,72 are the row and column indices of the spatial grid, respectively.

[0048] To map local blocks to a uniform-dimensional labeled representation, this module flattens each local block and applies a linear mapping. Let the flattening operator be denoted as... vec (⋅), then the vectorized representation of the local block is Introducing a learnable projection matrix With bias b e ∈R C This is mapped to a C-dimensional embedding vector, et,i,j=vec(xt,i,j)We+be, where C is the channel dimension of the first layer. After flattening and mapping each local block, each frame can be represented as an embedding tensor on a regular grid. E t ∈R 72×72×CThen stack the time dimension to get E ∈R T×72×72×C To facilitate subsequent coding blocks performing calculations in local windows, this module retains its grid structure in its implementation.

[0049] Since block segmentation itself weakens pixel-level absolute positional information, to supplement spatial structure priors, this module introduces two-dimensional positional encoding for each frame, which is added to the block embedding to form the input label. Let the learnable positional table be... P ∈R 72 ×72×C The final input is Z t = E t + P And obtained by stacking in the time dimension Z ∈R T×72×72×C .

[0050] S3.2, Feature modeling of video coding blocks: Video coding blocks are represented by markers output by the chunking module. Z As input, while maintaining the scalability of the hierarchical structure, fine-grained facial expression region modeling is achieved using self-attention within local windows, and cross-window information interaction is established through an alternating window offset mechanism. For ease of description, a unified tensor notation is first given. Let the _i_ be a _i_, ... ℓ The layer receives input from the block module. ,in H ℓ ×W ℓ For the first ℓ The spatial resolution of the layer, C ℓ For the first ℓ The feature dimensions of each layer. Video coding blocks are repeatedly stacked several times in each layer. Its core consists of a window attention mechanism sublayer and a feedforward sublayer, and layer normalization and residual connections are used to stabilize training. Let the th layer... ℓ If the video coding blocks of a layer are stacked twice, its structure is as follows: Figure 4 As shown.

[0051] First, the window division and label organization method are given. For any time step... t Take the marker grid of this frame. This module first performs layer normalization on it, and then performs layer normalization on non-overlapping layers. M × M The window is divided into sections, where M Let be the window side length. Serialize the markers within each window to obtain the window sequence. Performing self-attention within a window can reduce computational complexity from the global (H0) to a minimum. ℓ Wℓ ) 2 Reduced to This order of magnitude allows for computationally controllable local detail modeling at higher resolutions. This operation applies to all time steps. t They are performed independently, but share parameters in the time dimension to emphasize the temporal transferability of expression patterns.

[0052] In the window self-attention mechanism sublayer, given a window sequence X Three sets of learnable linear mappings are introduced. W Q ,W K , W V Map it to query, key, and value representation: (9) in Further, the channel dimension is divided into several subspaces to form a multi-head mechanism, denoted by the dimension of each head. and will Q,K,V Break it down into The attention output of the i-th head is defined as: (10) in B This is the relative position offset matrix within the window, used to compensate for the lack of absolute coordinate information after block segmentation and to enhance the prior knowledge of local geometric structure. The outputs of each header are concatenated according to the channel dimension and then linearly mapped. W O Get the window output: (11) in Then, collapse it back into the grid according to the original window layout to restore the spatial structure.

[0053] Let WindowAttn represent the combined operator of window partitioning, multi-head self-attention within the window, and collapse / restoration described above. To align with the residual structure, the window self-attention sub-layer is calculated using layer normalization. For any time step... t It can be written as: (12) Here, LN refers to the layer normalization operation.

[0054] A window is connected to a position-wise feedforward sublayer after the attention sublayer to enhance nonlinear representation capabilities and improve channel mixing efficiency. For each position... C ℓ dimensional label vector z The feedforward sublayer is defined as a combination of two linear mappings and nonlinear activations: (13) in , , r To expand the multiplier, σ It is an element-wise nonlinear function. Consistent with the attention sublayer, the feedforward sublayer also employs layer normalization and residual connections. (14) Applying the above two steps in parallel over time yields the output of this coded block for the entire video. .

[0055] To prevent relying solely on a fixed window from limiting cross-window information interaction, this module alternately introduces a window offset mechanism in adjacent encoding regions, such as... Figure 4 The video coding block 2 is shown. Specifically, before the next window attention, the marker grid is cyclically translated in the spatial dimension, and the translation amount is [ M / 2], to obtain the offset grid representation, then repeat the same process. M × M The system divides windows according to rules and calculates attention within each window. Offsets allow tags that were originally on opposite sides of different window boundaries to enter the same window under the new window division, thus achieving cross-window information fusion. Since cyclic translation introduces connections from non-adjacent regions at window boundaries, the module introduces a mask matrix for the offset attention calculation to shield tag pairs that shouldn't interact, ensuring that attention only propagates within geometrically reasonable local neighborhoods. After attention calculation, a reverse translation is performed to restore the mesh to its original coordinate system. By alternately stacking non-offset window attention and offset window attention, the model can gradually expand its effective receptive field while maintaining local computational advantages.

[0056] S3.3, Spatial Dimensionality Reduction and Merging: The features processed by the video coding block are input into the spatial dimensionality reduction and merging block. Adjacent spatial location labels are aggregated and dimensionality reduced, resulting in a progressively decreasing spatial scale. During downsampling, local facial expression details and cross-regional correlation information are preserved as much as possible to obtain a video feature representation for subsequent multimodal fusion. Its structure diagram is shown below. Figure 5 As shown.

[0057] Specifically, this module is connected after several video coding blocks, and its input is the first... ℓ Feature tensors of layers The output is the feature of the next layer. ,in , Based on the aforementioned settings, the four-layer structure will sequentially yield spatial grid scales of 72×72, 36×36, 18×18, and 9×9, while the time length...T It remains unchanged throughout the entire process.

[0058] This merging operation uses a 2×2 local neighborhood as the basic unit, aggregating the labels of four adjacent spatial locations into a new label. For any time step... t Let the input grid be . For each output position ( i,j The corresponding input neighborhood consists of four labels, namely: (15) in i ∈{0,…, },j∈{0,…, }. Concatenating these four markers along the channel dimension yields a 4C. ℓ Dimensional vector: (16) This forms the spliced ​​tensor. Simply stitching together the data leads to a sharp expansion of the channel dimensions and introduces redundancy. Therefore, linear projection is further introduced to achieve dimensionality reduction and channel reshaping. Let the projection matrix be... , bias is The merged output is then defined as: (17) Multiplication and addition are performed positionally by position in the last dimension. To stabilize training and suppress distribution drift caused by splicing, the module is pre-projected. Apply layer normalization, i.e., use layer normalization Alternative Enter linear projection. The above operation is equivalent to learning an adaptive information convergence map in each 2×2 neighborhood. Its ability is significantly stronger than fixed-weight average pooling or max pooling because it allows the model to learn different contribution weights and cross-channel interactions among the four sub-blocks, thereby better preserving the local texture differences and asymmetric cues corresponding to micro-expressions.

[0059] Regarding channel dimension settings, the module adopts a capacity allocation strategy consistent with the hierarchical structure: every time the spatial scale is halved, the channel dimension is moderately increased to compensate for the representation capacity loss caused by the reduction in the number of labels. Specific values ​​are... C ℓ+1 =2 C ℓ Therefore, spatial dimensionality reduction merging blocks not only achieve spatial scale reduction, but also provide stronger semantic abstraction capabilities for higher layers, enabling the model to gradually transition from fine-grained muscle texture to more stable facial expression patterns and cross-regional relational representations.

[0060] S4, Multimodal Feature Adaptive Fusion: The speech feature representation and video feature representation are uniformly aligned and dynamically weighted and fused through an adaptive post-fusion mechanism to obtain a joint feature representation.

[0061] Specifically, we first define the output representations for two modalities. Let the global representation vector of the speech processing branch output in S2 be a∈R. Da In S3, the video processing branch obtains the feature tensor after the fourth layer of encoding. Where C4 is the dimension of the fourth layer channel. Since post-fusion is better suited for handling fixed-dimensional global vector representations, this method is suitable for Z... (4) Perform spatiotemporal convergence to obtain a global representation of the video. v ∈R Dv Specifically, the time series features are first obtained by globally averaging and converging the spatial dimensions: (18) in , t =1,…, T Then, average convergence is performed on the time dimension to obtain the global vector of the video. (19) In obtaining a and v Subsequently, a learnable gating post-fusion mechanism is employed to generate joint representations. z ∈R D First, the two modes are linearly aligned so that they are projected onto the same dimensional space. Let... D To integrate the latent space dimension, a projection matrix is ​​introduced. W a ∈R Da ×D , W v ∈R Dv×D and corresponding bias b a ,b v ∈R D ,definition: (20) in a,v ∈R D Then, the gating vector is constructed. g ∈R D The contribution ratio of the two modes is adaptively adjusted. Gating is generated jointly by the aligned representations of the two modes, defined as: (twenty one) in , ∈RD×D , b g ∈R D Let σ(⋅) be a learnable parameter, and let σ(⋅) be an element-wise compression function, defined as follows: ,make g Each dimension takes a value between 0 and 1. The final joint representation is defined as: (twenty two) Where ⊙ represents element-wise multiplication.

[0062] To further enhance the nonlinear expressive power of joint representation, the method... z A lightweight feedforward transform is superimposed on top, preserving the residual form. For scaling factor... r f Then there is the final multimodal fusion representation. z' : (twenty three) in, , , , .

[0063] S5, Depression state classification output: Input the joint feature representation into the classification module and output the depression state classification result.

[0064] Specifically, the joint feature representation obtained in S4 is input into the classification module, which is used to classify the joint feature representation and output the classification result of the depression state of the object to be identified.

[0065] Furthermore, the classification module receives the final multimodal fusion representation after adaptive fusion of multimodal features and maps it to the corresponding category discrimination space to obtain the prediction results for each category; then, based on the prediction results, the classification result of the depressive state of the object to be identified is determined, thereby completing the identification of depression based on speech and video modalities.

[0066] As an optional embodiment, the present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a depression recognition method based on voice and video modalities.

[0067] As an optional embodiment, the present invention further includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a depression recognition method based on voice and video modalities.

[0068] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A method for identifying depression based on voice and video modalities, characterized in that, Includes the following steps: S1, Multimodal data preprocessing and initial feature extraction: Obtain the original speech data and original video data of the object to be identified, and perform preprocessing and initial feature extraction on the original speech data and original video data respectively to obtain the initial speech features and initial video features; S2, Speech feature encoding and context enhancement: The initial speech features are input into the speech branch for encoding to obtain the speech feature representation; S3, Hierarchical Video Feature Encoding: The initial video features are input into the video branch for hierarchical encoding to obtain the video feature representation; S4, Multimodal Feature Adaptive Fusion: The speech feature representation and video feature representation are uniformly aligned and dynamically weighted and fused through an adaptive post-fusion mechanism to obtain a joint feature representation; S5, Depression state classification output: Input the joint feature representation into the classification module and output the depression state classification result.

2. The method for depression recognition based on voice and video modalities according to claim 1, characterized in that, The steps for preprocessing and initial feature extraction of raw speech data include: S11, perform sentence segmentation on the original speech data, and generate enhanced speech samples based on sentence-level substitution; S12 extracts deep acoustic features from the enhanced speech samples to obtain initial speech features.

3. The method for depression recognition based on voice and video modalities according to claim 1, characterized in that, The steps for preprocessing and initial feature extraction of raw video data include: S13, perform time-series filtering, face localization and alignment, image cropping and normalization on the original video data to obtain the preprocessed original video data; S14: Extract initial video features from the preprocessed raw video data for subsequent hierarchical coding to obtain initial video features.

4. The method for depression recognition based on voice and video modalities according to claim 3, characterized in that, Step S13 includes the following steps S131, resample the original video data and obtain the timestamp of the speaking interval based on the audio channel; S132, perform face localization, cross-frame association and key point alignment on each frame in the speaking interval, and remove frames that do not meet the preset quality conditions; S133, the retained face region is cropped and uniformly scaled, and a fixed-length continuous video segment is extracted as the initial video feature to obtain the preprocessed original video data.

5. The method for depression recognition based on voice and video modalities according to claim 1, characterized in that, In step S2, the speech branch includes Hybrid attention mechanisms; The memory propagation mechanism is used to segment long speech sequences and introduce historical memory information of previous segments into the current segment encoding process to enhance the contextual continuity of long speech sequences across segments. The initial speech features are extracted by the hybrid attention mechanism, which takes into account both local dependencies and global contextual information. The memory propagation mechanism is used to enhance the contextual continuity of long speech sequences across segments, thus obtaining speech feature representations.

6. The method for depression recognition based on voice and video modalities according to claim 5, characterized in that, The hybrid attention mechanism includes Local window attention; Locality-Sensitive Hashing-based global attention combines local window attention with locality-sensitive hashing-based global attention to balance local acoustic pattern modeling and long-range context dependency modeling.

7. The method for depression recognition based on voice and video modalities according to claim 6, characterized in that, The hybrid attention mechanism also includes A context gating block is used to adaptively weight the outputs of local window attention and global attention based on local sensitive hashing to obtain a speech feature representation.

8. The method for depression recognition based on voice and video modalities according to claim 6, characterized in that, In step S3, the video branch includes The segmentation module is used to segment the spatial dimension of video frames and map it to a serialized token representation, so as to form video feature input suitable for subsequent encoding while preserving local details; The video coding block is used to model the features of each local region based on local window attention, and to realize cross-window information interaction through an alternating window offset mechanism, so as to enhance the correlation modeling ability between different spatial regions. Spatial dimensionality reduction merging blocks are used to aggregate and reduce the dimensionality of labels of adjacent spatial locations to achieve a gradual decrease in spatial scale and a gradual enhancement of feature semantics, thereby obtaining video feature representations. By spatial segmentation, multi-scale hierarchical modeling, local window attention, cross-window interaction, and spatial dimensionality reduction merging, hierarchical feature learning is performed on the initial video data to obtain video feature representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a depression recognition method based on voice and video modalities as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements a computer-readable storage medium as described in any one of claims 1 to 8, having a computer program stored thereon, characterized in that, when executed by a processor, the program implements a depression recognition method based on voice and video modalities as described in any one of claims 1 to 8.

Images