Anxiety disorder detection system based on hierarchical cross-modal attention fusion

Abstract

The application discloses an anxiety disorder detection system based on hierarchical cross-modal attention fusion, applied to the technical field of anxiety disorder detection, comprising: a modal feature coding module: based on a pre-trained BERT model, global semantic features of patient text are extracted; through a stacked double-layer bidirectional long short-term memory network, time sequence dynamic characteristics and context-dependent information of patient audio features are captured; based on a deep convolutional neural network, deep video features of patient speech video are extracted; a hierarchical cross-modal attention fusion module: based on a hierarchical bidirectional attention fusion strategy, bidirectional attention interaction fusion of text and audio modal features and bidirectional attention interaction fusion of fusion results and video modal features are carried out respectively; a residual enhancement multilayer perceptron classification decision module: multichannel fusion features are input to a residual enhancement structure-based multilayer perceptron classification decision model, and an anxiety disorder detection result is obtained. The application effectively improves the anxiety disorder detection precision.

Description

Technical Field

[0001] This invention relates to the field of anxiety disorder detection technology, and in particular to an anxiety disorder detection system based on hierarchical cross-modal attention fusion. Background Technology

[0002] Anxiety disorders are a common mental illness resulting from the combined effects of genetics, neurotransmitter abnormalities, psychosocial stress, and environmental factors. Anxiety disorders not only affect patients' daily lives and social abilities but are also closely related to comorbidities such as depression, hypertension, and cardiovascular disease. Therefore, timely identification of anxiety disorders is of great importance.

[0003] In current clinical diagnosis, the identification of anxiety disorders mainly relies on the subjective judgment of doctors and traditional psychological scales, which suffers from low efficiency and susceptibility to human error. Due to the often subtle early symptoms of anxiety disorders, the scarcity of medical resources, and limitations in testing methods, there is an urgent need for an automated anxiety disorder detection technology to aid in timely and accurate diagnosis.

[0004] With the development of artificial intelligence technology, affective computing and psychological state recognition methods based on multimodal data have gradually emerged, becoming an important research direction for intelligent auxiliary assessment of mental health. Especially in the highly interactive and semantically dense application scenario of clinical interviews, patients often simultaneously convey psychological signals through multiple unstructured modalities, including language content (text), tone of voice (audio), and facial expressions and body movements (video), when expressing their emotions and psychological feelings. Compared to single-modality methods, multimodal information can more comprehensively and meticulously reflect an individual's psychological state from multiple levels, including semantics, emotion, and physiology, helping to improve the accuracy and objectivity of anxiety detection. Therefore, fusion modeling based on multimodal information has become an important trend in the research of anxiety and other mental disorders. However, although existing research has made some progress in multimodal anxiety detection, there are still many shortcomings in modeling methods and fusion strategies. Most methods only use shallow fusion strategies such as simple feature splicing, average weighting, or early fusion, ignoring the importance of semantic co-modeling between modalities and failing to fully capture the complex and subtle interactive relationships between different modalities. Meanwhile, due to the inconsistent time steps and semantic differences in multimodal data, existing methods often lack effective temporal alignment mechanisms, leading to asynchronous issues in feature fusion between modalities and affecting the accuracy of the final semantic mapping. Furthermore, current models mostly rely on flat or single-layer representations, lacking hierarchical structural modeling from local to global and from low-level features to high-level semantics, thus failing to comprehensively depict the dynamic process of psychological state changes.

[0005] Therefore, how to provide an anxiety disorder detection system based on hierarchical cross-modal attention fusion that can effectively improve the abstraction ability of internal representations of different modalities, model the deep semantic interaction relationship between modalities, fully explore the correlation information between multimodalities, and achieve accurate and efficient identification of anxiety disorders is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention proposes an anxiety disorder detection system based on hierarchical cross-modal attention fusion.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] An anxiety disorder detection system based on hierarchical cross-modal attention fusion includes:

[0009] Modality feature encoding module: used to extract global semantic features of patient text in clinical interviews based on pre-trained BERT model to obtain text modality features; used to capture the temporal dynamic characteristics and context-dependent information of patient audio features in clinical interviews through stacked two-layer bidirectional long short-term memory network to obtain audio modality features; used to extract deep video features of patient speaking videos in clinical interviews based on deep convolutional neural network to obtain video modality features.

[0010] The hierarchical cross-modal attention fusion module is used to perform bidirectional attention interaction fusion of text modal features and audio modal features based on a hierarchical bidirectional attention fusion strategy, as well as bidirectional attention interaction fusion of the fusion results of text modal features and audio modal features with video modal features, to obtain multimodal fusion features.

[0011] The residual-enhanced multilayer perceptron classification decision module is used to input multimodal fusion features into a multilayer perceptron classification decision model based on residual enhancement structure to obtain anxiety disorder detection results.

[0012] Optionally, global semantic features of patient texts from clinical interviews can be extracted based on a pre-trained BERT model, specifically:

[0013] The semantic context is modeled based on the bidirectional Transformer architecture. The WordPiece tokenizer is used to map the text into a token sequence and further embed it into a vector of a preset dimension. After processing by a 24-layer Transformer structure, the output vector [0] at the [CLS] position is extracted as the global semantic feature of the text.

[0014] Optionally, the temporal dynamics and context-dependent information of patient audio features in clinical interviews can be captured by stacking a two-layer bidirectional long short-term memory network, specifically:

[0015] The audio features are extended in the time dimension by using a first-layer bidirectional long short-term memory network with a first preset dimension, and the output of the first-layer bidirectional long short-term memory network is input into a second-layer bidirectional long short-term memory network with a second preset dimension for deep modeling to capture the temporal dynamic characteristics and contextual dependency information of the audio features; wherein, the second dimension is smaller than the first dimension.

[0016] Optionally, after capturing the temporal dynamics and contextual dependency information of the audio sequence, the method further includes: extracting the hidden state vector of the last layer of the output sequence of the bidirectional long short-term memory network, and processing it using layer normalization and random dropout operations to obtain audio modal features.

[0017] Optionally, after extracting the global semantic features of the patient's text and the deep video features of the patient's speech video, the method further includes: inputting the extracted features into an improved residual fully connected block and performing batch normalization and conditional Dropout processing; wherein, the improved residual fully connected block includes a fully connected layer with ReLU activation and a dimension-aligned shortcut connection, and when the input and output dimensions are inconsistent, the shortcut path introduces a linear transformation to achieve dimension matching.

[0018] Optionally, before fusing multimodal features based on a hierarchical bidirectional attention fusion strategy, the method further includes: mapping the input text, audio, and video modal features to a unified low-dimensional feature space based on a learnable modality-specific linear projection network, as follows:

[0019]

[0020] in, The result obtained by mapping the i-th multimodal feature to a unified low-dimensional feature space; γ i W i b i These represent the scaling factor, projection weight matrix, and bias term corresponding to the i-th multimodal feature, respectively; f i is the original feature of the i-th multimodal feature; text, audio, and video are text, audio, and video, respectively.

[0021] Optionally, based on a hierarchical bidirectional attention fusion strategy, bidirectional attention interaction fusion of text modality features and audio modality features is performed, specifically as follows:

[0022] A multi-head attention mechanism is used to construct a bidirectional interaction relationship between text modal features and audio modal features, as follows:

[0023]

[0024] Among them, A text→audio A audio→textThis represents the bidirectional interaction between text modal features and audio modal features; MultiHead is a multi-head attention mechanism; Q, K, and V are the query, key, and value vectors, respectively. These are text modal features and audio modal features, respectively.

[0025] The bidirectional interaction relationship between text modality features and audio modality features is added to the original modality features via a residual connection, and then layer normalization is performed to obtain the bidirectional attention interaction fusion result of text modality features and audio modality features, as follows:

[0026]

[0027] Among them, F TA This represents the result of bidirectional attention interaction fusion of text modality features and audio modality features; LayerNorm is the layer normalization operation; || is the feature concatenation operation.

[0028] Optionally, based on a hierarchical bidirectional attention fusion strategy, the fusion results of text modal features and audio modal features are fused with video modal features through bidirectional attention interaction, specifically as follows:

[0029] A bidirectional interaction relationship is constructed using a multi-head attention mechanism to fuse the fusion results of text modal features and audio modal features with video modal features, as follows:

[0030]

[0031] in, This represents the bidirectional interaction between the fusion results of text modal features and audio modal features and the bidirectional attention interaction fusion of video modal features; MultiHead represents the multi-head attention mechanism; Q, K, and V are the query, key, and value vectors, respectively; F TA , These are the fusion results of text modal features and audio modal features, and video modal features, respectively.

[0032] The fusion results of text and audio modal features are combined with the bidirectional attention interaction of video modal features. The bidirectional interaction relationship is then added to the original modal features via residual connections, followed by layer normalization, to obtain the multimodal fused features, as follows:

[0033]

[0034] Among them, F fused is for multimodal fusion features; LayerNorm is for layer normalization; || is for feature concatenation.

[0035] Optionally, a multilayer perceptron classification decision model based on residual enhancement structure extracts and nonlinearly transforms multimodal fusion features layer by layer through three cascaded residual fully connected blocks, and finally obtains the prediction probability of the binary classification task through a sigmoid activation function.

[0036] Optionally, it also includes: using cross-entropy as the loss function, as follows:

[0037]

[0038] Among them, y i ∈{0,1} represents the true label of the i-th sample; The predicted probability N output by the model is the total number of samples.

[0039] As can be seen from the above technical solution, compared with the prior art, this invention proposes an anxiety disorder detection system based on hierarchical cross-modal attention fusion. This invention fully utilizes multimodal information such as textual language expression, voice tone changes, and facial expression dynamics during doctor-patient clinical interviews, combined with the patient's actual communication style, to construct a complete psychological state representation system. Based on this, by designing an independent feature extraction structure, the abstraction ability of representations within different modalities is improved, and a hierarchical cross-modal attention mechanism is used to model the deep semantic interaction relationship between modalities, achieving accurate identification of potential anxiety states. By introducing hierarchical feature modeling and cross-modal attention mechanisms, the correlation information between text, audio, and video modalities is fully explored, achieving more accurate and efficient anxiety state identification. This invention not only improves the anxiety detection model's perception and understanding of complex multimodal inputs, significantly enhancing the system's robustness and generalization ability, but also enables low-cost, non-invasive, and highly efficient intelligent psychological screening in actual clinical scenarios, possessing high application and promotion value. Especially in remote mental health assessment, online consultation-assisted diagnosis, and mental illness monitoring systems, this invention can effectively improve diagnostic efficiency and quality, providing technical support for mental health intervention and public health management. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the system structure of the present invention.

[0042] Figure 2 This is a schematic diagram of the hierarchical cross-modal attention fusion module structure of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1:

[0045] Embodiment 1 of this invention discloses an anxiety disorder detection system based on hierarchical cross-modal attention fusion, such as... Figure 1 As shown, it includes:

[0046] Modality feature encoding module: used to extract global semantic features of patient text in clinical interviews based on pre-trained BERT model to obtain text modality features; used to capture temporal dynamic characteristics and contextual dependency information of patient audio features in clinical interviews through stacked two-layer bidirectional long short-term memory network to obtain audio modality features; used to extract deep video features of patient speaking videos in clinical interviews based on deep convolutional neural network to obtain video modality features.

[0047] The global semantic features of patient text in clinical interviews are extracted based on a pre-trained BERT model (based on bert-large-uncased). Specifically:

[0048] The semantic context is modeled based on the bidirectional Transformer architecture. The WordPiece tokenizer is used to map the text into a token sequence and further embed it into a vector of a preset dimension (d = 1024 dimensions). After processing by a 24-layer Transformer structure, the output vector [0] at the [CLS] position is extracted as the global semantic feature of the text.

[0049] A semantic encoder based on the BERT structure can fully model contextual information, thereby capturing the psychological characteristics expressed by patients in conversations.

[0050] To extract potential emotional and psychological features from patient speech, this invention proposes a multidimensional acoustic feature extraction method, combined with recursive neural structures for feature modeling. Using timestamp information from the patient-doctor dialogue transcript, only patient speech segments are extracted to ensure temporal synchronization with other modalities.

[0051] In terms of acoustic feature extraction, the librosa tool was used to extract multiple types of descriptors, constructing a total of 185-dimensional audio feature vectors (audio features), including but not limited to the following feature types: Mel frequency cepstral coefficients (MFCC): 20-dimensional, used to characterize the spectral envelope; Chroma features: 24-dimensional, based on a 24-bin discrete pitch space; Mel spectrogram features: 128-dimensional, reflecting human auditory characteristics; Low-order time-frequency features: 13-dimensional, including zero-crossing rate, energy, spectral centroid, spectral flatness, etc.

[0052] To adapt to multimodal learning frameworks, a stacked two-layer bidirectional long short-term memory network is used to capture the temporal dynamics and context-dependent information of patient audio features in clinical interviews. Specifically:

[0053] The audio features are extended in the time dimension by using a first-layer bidirectional long short-term memory network with a first preset dimension (256) to adapt it to the input requirements of sequence models, as follows:

[0054] h1 = BiLSTM 256 (ExpandDims(A,axis=1));

[0055] Where h1 is the output of the first layer of bidirectional long short-term memory network; ExpandDims is the insertion of a dimension at a specified position; A is the audio feature matrix composed of the above audio feature vectors; axis=1 is the insertion of a new dimension at the dimension position with index 1.

[0056] The output of the first-layer bidirectional long short-term memory network is input into a second-layer bidirectional long short-term memory network with a second preset dimension (the second preset dimension is smaller than the first preset dimension, and the second preset dimension is 128) for deep modeling, capturing the temporal dynamic characteristics and contextual dependency information of audio features, as follows:

[0057] h2 = BiLSTM 128 (h1);

[0058] Where h2 is the output of the second-layer bidirectional long short-term memory network.

[0059] After capturing the temporal dynamics and contextual dependencies of the audio sequence, the process also includes: extracting the hidden state vector of the last layer of the output sequence of the bidirectional long short-term memory network, and processing it with layer normalization and random dropout operations (dropout rate p is set to 0.3) to obtain audio modal features.

[0060] This method enhances the feature discrimination capability of audio modalities in complex speech scenarios, and is particularly suitable for individual recognition and emotional state recognition tasks in multi-speaker scenarios in clinical contexts.

[0061] To effectively acquire patients' visual emotional expression information, this invention employs a facial video feature extraction method based on deep convolutional neural networks. Patient speech segments are extracted from the dialogue video according to their timestamps, and frames are uniformly extracted from each video segment, sampling a fixed number of keyframes. All image frames are scaled to 224×224 pixels and uniformly standardized. The frame images are input into a DenseNet201 model pre-trained on ImageNet to extract 1920-dimensional deep visual features for each frame. To generate a global representation of the entire video, the feature vectors of all frames are average-pooled to obtain the global visual vector of the video segment.

[0062] To enhance the nonlinear expressive power of features, this invention constructs an improved residual fully connected block, comprising a fully connected layer with ReLU activation and a dimension-aligned shortcut connection. When the input and output dimensions are inconsistent, the shortcut path introduces a linear transformation to achieve dimension matching. Therefore, after extracting the global semantic features of the patient's text and the deep video features of the patient's speech video, the invention further includes: inputting the extracted features into the improved residual fully connected block, and performing batch normalization and conditional dropout processing (with a dropout rate p set to 0.2) to improve the robustness and generalization ability of the model.

[0063] The hierarchical cross-modal attention fusion module is used to perform bidirectional attention interaction fusion of text modal features and audio modal features based on a hierarchical bidirectional attention fusion strategy, as well as bidirectional attention interaction fusion of the fusion results of text modal features and audio modal features with video modal features, to obtain multimodal fusion features.

[0064] To address the shortcomings of existing multimodal fusion methods in handling semantic inconsistencies and dynamic modal dependencies, this invention proposes a hierarchical cross-modal attention fusion module, aiming to achieve effective alignment and deep interaction of multimodal features. The core technical idea of ​​this module is to introduce a local-global two-stage attention mechanism, achieving fine-grained alignment between two modalities and overall integration across all modalities in stages. Through the multi-head attention mechanism, the model can autonomously learn the interaction strength and context sensitivity between modalities, while employing a cross-layer connection mechanism to preserve the original information of each modality and mitigate information loss during feature fusion. Figure 2 As shown, the hierarchical cross-modal attention fusion module mainly consists of the following two core components:

[0065] Modality Alignment Projection Layer: To address the issues of inconsistent dimensionality and heterogeneous distribution in multimodal feature spaces, this invention designs a learnable modality-specific linear projection network. Before fusing multimodal features based on a hierarchical bidirectional attention fusion strategy, the learnable modality-specific linear projection network maps the input text, audio, and video modal features to a unified low-dimensional feature space, as follows:

[0066]

[0067] in, The result obtained by mapping the i-th multimodal feature to a unified low-dimensional feature space; γ i W i b i Let W be the scaling factor, projection weight matrix, and bias term corresponding to the i-th multimodal feature, respectively, and let d = 256 be the output dimension. To ensure independence and balance among different modalities, the weight matrix W is... i Orthogonal initialization method, scaling factor γ i These are trainable parameters used to dynamically adjust the contribution of each modality to the final fused representation; f i is the original feature of the i-th multimodal feature; text, audio, and video are text, audio, and video, respectively.

[0068] Bidirectional cross-modal attention layer: Building upon the feature alignment described above, to achieve more refined interactions between different modalities, this invention further proposes a hierarchical bidirectional attention fusion strategy. Based on this hierarchical bidirectional attention fusion strategy, bidirectional attention interaction fusion of text modal features and audio modal features is performed, specifically as follows:

[0069] A multi-head attention mechanism is used to construct a bidirectional interaction relationship between text modal features and audio modal features, as follows:

[0070]

[0071] Among them, A text→audio A audio→text It represents the bidirectional interaction between text modal features and audio modal features; MultiHead is a multi-head attention mechanism; Q, K, and V are the query, key, and value vectors, respectively, and the weight parameters are learnable variables; These are text modal features and audio modal features, respectively.

[0072] The bidirectional interaction relationship between text modality features and audio modality features is added to the original modality features via a residual connection, and then layer normalization is performed to obtain the bidirectional attention interaction fusion result of text modality features and audio modality features, as follows:

[0073]

[0074] Among them, F TA This represents the result of bidirectional attention interaction fusion of text modality features and audio modality features; LayerNorm is the layer normalization operation; || is the feature concatenation operation. Then, F... TA Perform global average pooling to compress its temporal dimension.

[0075] Based on a hierarchical bidirectional attention fusion strategy, the fusion results of text modal features and audio modal features are fused with video modal features through bidirectional attention interaction, specifically as follows:

[0076] A bidirectional interaction relationship is constructed using a multi-head attention mechanism to fuse the fusion results of text modal features and audio modal features with video modal features, as follows:

[0077]

[0078] in, This represents the bidirectional interaction between the fusion results of text modal features and audio modal features and the bidirectional attention interaction fusion of video modal features; MultiHead represents the multi-head attention mechanism; Q, K, and V are the query, key, and value vectors, respectively; F TA , These are the fusion results of text modal features and audio modal features, and video modal features, respectively.

[0079] The fusion results of text and audio modal features are combined with the bidirectional attention interaction of video modal features. The bidirectional interaction relationship is then added to the original modal features via residual connections, followed by layer normalization, to obtain the multimodal fused features, as follows:

[0080]

[0081] Among them, F fused The multimodal fusion features can be directly input into the subsequent classifier module for decision-making; LayerNorm is the layer normalization operation; || is the feature concatenation operation.

[0082] The residual-enhanced multilayer perceptron classification decision module is used to input multimodal fusion features into a multilayer perceptron classification decision model based on residual enhancement structure to obtain anxiety disorder detection results.

[0083] To improve the discriminative power of classification models and enhance the nonlinear expressive power of fused features, this invention proposes a multilayer perceptron (MLP) classification decision module based on residual enhancement structure.

[0084] The multilayer perceptron classification decision model based on residual enhancement structure adopts a multimodal feature fusion vector F. fused As input, the multimodal fusion features are extracted and nonlinearly transformed layer by layer through three cascaded residual fully connected blocks, as follows:

[0085] h1 = ResBlock(F fused ,1024),h1=BN(h1),Dorpout(p=0.5);

[0086] h2=ResBlock(h1,512),h2=BN(h2),Dorpout(p=0.4);

[0087] h3=ResBlock(h2,256), h3=BN(h3);

[0088] Where h1 is the multimodal feature fusion vector F fused The output after the first residual block; BN is the batch normalization process; h2 is the output of h1 after the second residual block; h3 is the output of h2 after the third residual block; finally, h3 is activated by a Sigmoid activation function to obtain the predicted probability for the binary classification task, as follows:

[0089]

[0090] Where σ is the Sigmoid activation function, used to convert the model output value Compress the data to the [0,1] interval to achieve binary classification probability output; b is the bias term.

[0091] It also includes: for the anxiety state recognition binary classification task handled by this invention, cross-entropy is used as the loss function to measure the difference between the model's predicted value and the actual label, as follows:

[0092]

[0093] Among them, y i ∈{0,1} represents the true label of the i-th sample; The predicted probability N output by the model is the total number of samples.

[0094] This invention discloses an anxiety disorder detection system based on hierarchical cross-modal attention fusion. First, in the module feature encoding module, a pre-trained BERT model is used to encode the dialogue text between the patient and doctor to obtain a context-aware semantic representation. The Librosa tool is used to extract low-level acoustic features of the audio, and stacked two-layer BiLSTM is used to capture the temporal characteristics of the speech. Simultaneously, a DenseNet201 deep convolutional neural network is used to extract visual emotional features from the dialogue video frames, and residual connections and normalization processing are used to enhance feature representation capabilities. Next, in the hierarchical cross-modal attention fusion module, a two-level bidirectional cross-modal attention mechanism is designed. The first stage realizes fine-grained interaction between the text and speech modalities to obtain an intermediate fused representation; the second stage further fuses this intermediate representation with the video modal, thereby achieving hierarchical information integration across modalities and enhancing the deep association modeling of semantics and emotion. Finally, the fused feature representations are input into the residual-enhanced multilayer perceptron classification module. Through a multilayer fully connected network and residual connection structure, combined with batch normalization and Dropout techniques, overfitting is effectively prevented, and the model's generalization ability is improved, ultimately achieving binary classification to determine whether a patient is experiencing anxiety. This invention balances intramodal temporal information modeling and intermodal correlation modeling, possessing good interpretability and scalability. Experimental results show that this method has higher accuracy and robustness in anxiety recognition tasks compared to traditional single-modal or simple fusion methods, and is applicable to various scenarios such as intelligent healthcare, mental health assessment, and remote diagnosis and treatment.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0096] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An anxiety disorder detection system based on hierarchical cross-modal attention fusion, characterized in that, include: Modal feature encoding module: used to extract global semantic features of patient text in clinical interviews based on a pre-trained BERT model, and obtain text modal features; This is used to capture the temporal dynamics and context-dependent information of patient audio features in clinical interviews through stacked two-layer bidirectional long short-term memory networks, thus obtaining audio modal features; and to extract deep video features from patient speaking videos in clinical interviews based on deep convolutional neural networks, thus obtaining video modal features. By using a stacked two-layer bidirectional long short-term memory network to capture the temporal dynamics and context-dependent information of patient audio features in clinical interviews, specifically: The audio features are extended in the time dimension by a first-layer bidirectional long short-term memory network with a first preset dimension, and the output of the first-layer bidirectional long short-term memory network is input into a second-layer bidirectional long short-term memory network with a second preset dimension for deep modeling to capture the temporal dynamic characteristics and contextual dependency information of the audio features; wherein, the second dimension is smaller than the first dimension. The hierarchical cross-modal attention fusion module is used to perform bidirectional attention interaction fusion of the text modal features and audio modal features based on a hierarchical bidirectional attention fusion strategy, as well as bidirectional attention interaction fusion of the fusion results of the text modal features and audio modal features with the video modal features, to obtain multimodal fusion features. Based on a hierarchical bidirectional attention fusion strategy, bidirectional attention interaction fusion of the text modal features and audio modal features is performed, specifically as follows: A multi-head attention mechanism is used to construct the bidirectional interaction relationship between the text modal features and the audio modal features; The bidirectional interaction relationship between the text modal features and the audio modal features is added to the original modal features through residual connections, and then layer normalization is performed to obtain the bidirectional attention interaction fusion result of the text modal features and the audio modal features. Residual-enhanced multilayer perceptron classification decision module: used to input the multimodal fusion features into the multilayer perceptron classification decision model based on residual enhancement structure to obtain anxiety disorder detection results; The multilayer perceptron classification decision model based on residual enhancement structure extracts and nonlinearly transforms the multimodal fusion features layer by layer through three cascaded residual fully connected blocks, and finally obtains the prediction probability of the binary classification task through a sigmoid activation function. After extracting the global semantic features of the patient's text and the deep video features of the patient's speech video, the system further includes: inputting the extracted features into an improved residual fully connected block and performing batch normalization and conditional Dropout processing; wherein, the improved residual fully connected block includes a fully connected layer with ReLU activation and a dimension-aligned shortcut connection, and when the input and output dimensions are inconsistent, the shortcut path introduces a linear transformation to achieve dimension matching.

2. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that, Global semantic features of patient text from clinical interviews are extracted based on a pre-trained BERT model, specifically: Semantic context is modeled based on a bidirectional Transformer architecture. The WordPiece tokenizer is used to map the text into a token sequence, which is then further embedded into a vector of a preset dimension. After processing through a 24-layer Transformer structure, the output vector at the [CLS] position is extracted. This serves as a global semantic feature of the text.

3. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that, After capturing the temporal dynamics and contextual dependency information of the audio sequence, the method further includes: extracting the hidden state vector of the last layer of the output sequence of the bidirectional long short-term memory network, and processing it using layer normalization and random dropout operations to obtain the audio modal features.

4. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that, Before fusing multimodal features based on a hierarchical bidirectional attention fusion strategy, the process also includes: mapping the input text, audio, and video modal features to a unified low-dimensional feature space based on a learnable modality-specific linear projection network, as follows: in, For the first The result obtained by mapping various multimodal features to a unified low-dimensional feature space; , , The first Scaling factors, projection weight matrices, and bias terms corresponding to various multimodal features; For the first The original features of a multimodal feature; , , These are text, audio, and video, respectively.

5. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that, The bidirectional interaction relationship between the text modal features and the audio modal features is constructed using a multi-head attention mechanism, as follows: in, , This refers to the bidirectional interaction relationship between the text modal features and the audio modal features; This is a multi-head attention mechanism; , , These are the query, key, and value vector, respectively. , These are the text modal features and the audio modal features, respectively; The bidirectional interaction relationship between the text modal features and audio modal features is added to the original modal features via a residual connection, and then layer normalization is performed to obtain the bidirectional attention interaction fusion result of the text modal features and audio modal features, as follows: in, This is the result of bidirectional attention interaction fusion of the text modal features and audio modal features; For layer normalization operation; This is a feature splicing operation.

6. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that, Based on a hierarchical bidirectional attention fusion strategy, the fusion results of the text modal features and audio modal features are fused with the bidirectional attention interaction of the video modal features, specifically as follows: A bidirectional interaction relationship is constructed using a multi-head attention mechanism to fuse the fusion results of the text modal features and audio modal features with the bidirectional attention interaction of the video modal features, as follows: in, , This refers to the bidirectional interactive relationship between the fusion result of the text modal features and audio modal features and the bidirectional attention interaction fusion of the video modal features; This is a multi-head attention mechanism; , , These are the query, key, and value vector, respectively. , These are the fusion results of the text modal features and audio modal features, and the video modal features, respectively. The fusion result of the text modal features and audio modal features is combined with the bidirectional interaction relationship of the video modal features through bidirectional attention interaction. This relationship is then added to the original modal features via residual connections, and layer normalization is performed to obtain the multimodal fusion features, as follows: in, This is a multimodal fusion feature; For layer normalization operation; This is a feature splicing operation.

7. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that, Also includes: Cross-entropy is used as the loss function, as follows: in, For the first The true label of each sample; The predicted probability output by the model The total number of samples.

Images