Anxiety disorder detection system based on hierarchical cross-modal attention fusion

Abstract

The invention discloses an anxiety disorder detection system based on hierarchical cross-modal attention fusion, which is applied to the technical field of anxiety disorder detection, and comprises a modal feature coding module for extracting global semantic features of a patient text based on a pre-trained BERT model; capturing time sequence dynamic characteristics and context dependence information of audio features of the patient by stacking a double-layer bidirectional long-short-term memory network; extracting deep video features of the video speaking by the patient based on a deep convolutional neural network; the hierarchical cross-modal attention fusion module is used for respectively carrying out bidirectional attention interaction fusion of text and audio modal features and bidirectional attention interaction fusion of a fusion result and video modal features based on a hierarchical bidirectional attention fusion strategy; and the residual enhancement multilayer perceptron classification decision module is used for inputting the multi-modal fusion features to a multilayer perceptron classification decision model based on a residual enhancement structure to obtain an anxiety disorder detection result. The anxiety disorder detection precision is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of anxiety disorder detection, and in particular to an anxiety disorder detection system based on hierarchical cross-modal attention fusion. Background Art

[0002] Anxiety disorders are common psychiatric conditions, resulting from a combination of genetics, neurotransmitter abnormalities, psychosocial stress, and environmental factors. They not only impact patients' daily lives and social abilities but are also closely associated with comorbidities such as depression, hypertension, and cardiovascular disease. Therefore, timely identification of anxiety disorders is crucial.

[0003] Current clinical diagnosis relies primarily on subjective physician judgment and traditional psychological scales, which are inefficient and susceptible to human interference. Given the hidden early symptoms of anxiety disorders, limited medical resources, and limited testing methods, there is an urgent need for automated anxiety disorder detection technology to facilitate timely and accurate diagnosis.

[0004] With the development of artificial intelligence (AI) technology, affective computing and mental state recognition methods based on multimodal data have gradually emerged as important research directions for intelligently assisted mental health assessment. In particular, in highly interactive and semantically dense clinical interviews, patients often convey their emotions and psychological feelings simultaneously through multiple unstructured modalities, including language content (text), voice intonation (audio), and facial expressions and body movements (video). Compared to single modal information, multimodal information can more comprehensively and meticulously reflect an individual's mental state from multiple perspectives, including semantics, emotion, and physiology, helping to improve the accuracy and objectivity of anxiety detection. Therefore, fusion modeling based on multimodal information has become a key trend in the identification of psychological disorders such as anxiety. However, while existing research has made some progress in multimodal anxiety detection, many shortcomings remain in modeling approaches and fusion strategies. Most methods employ shallow fusion strategies such as simple feature concatenation, average weighting, or early fusion, neglecting the importance of semantic co-modeling between modalities and failing to fully capture the complex and detailed interactions between different modalities. At the same time, due to the inconsistent time steps and differences in semantic expression of multimodal data, existing methods often lack effective temporal alignment mechanisms, resulting in asynchronous feature fusion between modalities and affecting the accuracy of the final semantic mapping. Furthermore, current models often rely on flat or single-layer representations, lacking hierarchical structural modeling from the local to the global, from low-level features to high-level semantics, and are unable to fully capture the dynamic process of psychological state changes.

[0005] To this end, how to provide an anxiety disorder detection system based on hierarchical cross-modal attention fusion that can effectively improve the abstract ability of internal representations of different modalities, model deep semantic interaction relationships between modalities, fully mine the correlation information between multiple modalities, and achieve accurate and efficient identification of anxiety disorders is a problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0006] In view of this, the present invention proposes an anxiety disorder detection system based on hierarchical cross-modal attention fusion.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] An anxiety disorder detection system based on hierarchical cross-modal attention fusion, comprising:

[0009] Modal Feature Encoding Module: This module is used to extract global semantic features from patient texts in clinical interviews based on a pre-trained BERT model, thereby obtaining text modal features. It is used to capture the temporal dynamics and contextual dependency information of patient audio features in clinical interviews through a stacked two-layer bidirectional long short-term memory network, thereby obtaining audio modal features. It is used to extract deep video features from patient speech videos in clinical interviews based on a deep convolutional neural network, thereby obtaining video modal features.

[0010] Hierarchical cross-modal attention fusion module: Based on the hierarchical bidirectional attention fusion strategy, it performs bidirectional attention interaction fusion of text modality features and audio modality features, and bidirectional attention interaction fusion of the fusion results of text modality features and audio modality features with video modality features to obtain multimodal fusion features;

[0011] Residual enhanced multilayer perceptron classification decision module: used to input multimodal fusion features into the multilayer perceptron classification decision model based on the residual enhanced structure to obtain anxiety disorder detection results.

[0012] Optionally, extract global semantic features of patient text in clinical interviews based on a pre-trained BERT model, specifically:

[0013] The semantic context is modeled based on the bidirectional Transformer architecture, and the WordPiece word segmenter is used to map the text into a token sequence, which is further embedded into a vector of a preset dimension. After processing through a 24-layer Transformer structure, the output vector [0] at the [CLS] position is extracted as the global semantic feature of the text.

[0014] Optionally, a stacked two-layer bidirectional long short-term memory network is used to capture the temporal dynamics and contextual dependency of the patient audio features in the clinical interview. Specifically:

[0015] The audio features are expanded in the time dimension through a first-layer bidirectional long short-term memory network of a first preset dimension, and the output of the first-layer bidirectional long short-term memory network is input into a second-layer bidirectional long short-term memory network of a second preset dimension for deep modeling to capture the temporal dynamic characteristics and context-dependent information of the audio features; wherein the second dimension is smaller than the first dimension.

[0016] Optionally, after capturing the temporal dynamic characteristics and context dependency information of the audio sequence, the method further includes: extracting the last hidden state vector of the output sequence of the bidirectional long short-term memory network, and processing it using layer normalization and random Dropout operations to obtain audio modal features.

[0017] Optionally, after extracting the global semantic features of the patient's text and the deep video features of the patient's speech video, the method further includes: inputting the extracted features into an improved residual fully connected block, and performing batch normalization and conditional Dropout processing; wherein, the improved residual fully connected block includes a fully connected layer with ReLU activation and a dimensionally aligned shortcut connection. When the input and output dimensions are inconsistent, the shortcut path introduces a linear transformation to achieve dimensional matching.

[0018] Optionally, before fusing multimodal features based on the hierarchical bidirectional attention fusion strategy, the method further includes: mapping the input text, audio, and video modality features into a unified low-dimensional feature space based on a learnable modality-specific linear projection network, as follows:

[0019]

[0020] in, is the result of mapping the i-th multimodal feature to a unified low-dimensional feature space; γ i 、W i 、b i are the scaling factor, projection weight matrix, and bias term corresponding to the i-th multimodal feature; f i is the original feature of the i-th multimodal feature; text, audio, and video are text, audio, and video respectively.

[0021] Optionally, based on a hierarchical bidirectional attention fusion strategy, bidirectional attention interaction fusion of text modality features and audio modality features is performed, specifically:

[0022] The multi-head attention mechanism is used to construct a bidirectional interactive relationship between text modality features and audio modality features as follows:

[0023]

[0024] Among them, A text→audio 、A audio→textis the bidirectional interaction between text modality features and audio modality features; MultiHead is a multi-head attention mechanism; Q, K, and V are query, key, and value vectors, respectively; They are text modality features and audio modality features respectively;

[0025] The bidirectional interaction relationship between text modality features and audio modality features is added to the original modality features through residual connection, and layer normalization is performed to obtain the bidirectional attention interaction fusion result of text modality features and audio modality features, as follows:

[0026]

[0027] Among them, F TA is the result of the bidirectional attention interaction fusion of text modality features and audio modality features; LayerNorm is the layer normalization operation; || is the feature splicing operation.

[0028] Optionally, based on a hierarchical bidirectional attention fusion strategy, the fusion results of text modality features and audio modality features are fused with bidirectional attention interaction of video modality features, specifically:

[0029] The multi-head attention mechanism is used to construct a bidirectional interactive relationship between the fusion results of text modality features and audio modality features and the bidirectional attention interaction fusion of video modality features, as follows:

[0030]

[0031] in, is the bidirectional interaction between the fusion result of text modality features and audio modality features and the bidirectional attention interaction fusion of video modality features; MultiHead is the multi-head attention mechanism; Q, K, V are query, key and value vectors respectively; F TA 、 They are the fusion results of text modality features and audio modality features, and video modality features respectively;

[0032] The bidirectional interaction relationship of the fusion result of the text modality feature and the audio modality feature and the bidirectional attention interaction fusion of the video modality feature is added to the original modality feature through residual connection, and the layer normalization operation is performed to obtain the multimodal fusion feature, as follows:

[0033]

[0034] Among them, F fused is the multimodal fusion feature; LayerNorm is the layer normalization operation; || is the feature splicing operation.

[0035] Optionally, a multi-layer perceptron classification decision model based on a residual enhancement structure extracts and nonlinearly transforms multimodal fusion features layer by layer through three layers of residual fully connected blocks connected in series, and finally passes through a Sigmoid activation function to obtain the prediction probability of the binary classification task.

[0036] Optionally, it also includes: using cross entropy as the loss function, as follows:

[0037]

[0038] Among them, y i ∈{0, 1} is the true label of the i-th sample; The predicted probability output by the model is N, which is the total number of samples.

[0039] As can be seen from the above technical solutions, compared with the existing technology, the present invention proposes an anxiety disorder detection system based on hierarchical cross-modal attention fusion. This invention fully utilizes multimodal information such as textual language expression, voice intonation changes, and facial expression dynamics during the doctor-patient clinical interview, and combines it with the patient's actual communication methods to construct a complete psychological state representation system. On this basis, by designing independent feature extraction structures, the abstraction capability of the internal representation of different modalities is enhanced, and the hierarchical cross-modal attention mechanism is used to model the deep semantic interactions between modalities, achieving accurate identification of potential anxiety states. By introducing hierarchical feature modeling and cross-modal attention mechanisms, the correlation information between text, audio, and video modalities is fully exploited, achieving more accurate and efficient anxiety state identification. This invention not only improves the anxiety detection model's ability to perceive and understand complex multimodal inputs, significantly enhancing the system's robustness and generalization capabilities, but also enables low-cost, non-invasive, and highly efficient intelligent psychological screening in real clinical scenarios, with high application and promotion value. In particular, in remote mental health assessment, online consultation-assisted diagnosis, and mental illness monitoring systems, this invention can effectively improve the efficiency and quality of diagnosis and treatment, providing technical support for mental health intervention and public health management. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 Schematic diagram of the system structure of the present invention.

[0042] Figure 2 Schematic diagram of the structure of the hierarchical cross-modal attention fusion module of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] Example 1:

[0045] Example 1 of the present invention discloses an anxiety disorder detection system based on hierarchical cross-modal attention fusion, such as Figure 1 As shown, including:

[0046] Modal feature encoding module: used to extract the global semantic features of patient texts in clinical interviews based on the pre-trained BERT model to obtain text modal features; used to capture the temporal dynamic characteristics and context-dependent information of patient audio features in clinical interviews by stacking a double-layer bidirectional long short-term memory network to obtain audio modal features; used to extract deep video features of patient speech videos in clinical interviews based on a deep convolutional neural network to obtain video modal features.

[0047] The pre-trained BERT (based on bert-large-uncased) model extracts global semantic features of patient text in clinical interviews, specifically:

[0048] The semantic context is modeled based on the bidirectional Transformer architecture, and the text is mapped into a token sequence using the WordPiece word segmenter, which is further embedded into a vector of a preset dimension (d = 1024 dimensions). After processing through a 24-layer Transformer structure, the output vector [0] at the [CLS] position is extracted as the global semantic feature of the text.

[0049] The semantic encoder based on the BERT structure can fully model contextual information, thereby capturing the psychological characteristics expressed by patients in conversations.

[0050] To extract the underlying emotional and psychological characteristics of patients' speech, this paper proposes a multidimensional acoustic feature extraction method, combined with recursive neural architecture for feature modeling. Using timestamp information from the patient-doctor conversation transcript, only the patient's speech fragments are captured to ensure temporal synchronization with other modalities.

[0051] In terms of acoustic feature extraction, the librosa tool is used to extract multiple descriptors and construct an audio feature vector (audio feature) with a total dimension of 185, including but not limited to the following feature types: Mel-frequency cepstral coefficients (MFCC): 20 dimensions, used to characterize the spectral envelope; Chroma features: 24 dimensions, based on a 24-bin discrete pitch space; Mel spectrogram features: 128 dimensions, reflecting human auditory characteristics; low-order time-frequency features: 13 dimensions, including zero-crossing rate, energy, spectral centroid, spectral flatness, etc.

[0052] To adapt to the multimodal learning framework, a stacked two-layer bidirectional long short-term memory network is used to capture the temporal dynamics and contextual dependency information of patient audio features in clinical interviews. Specifically:

[0053] The audio features are expanded in the time dimension through the first-layer bidirectional long short-term memory network of the first preset dimension (the first preset dimension is 256) to adapt them to the sequence model input requirements as follows:

[0054] h1=BiLSTM 256 (ExpandDims(A,axis=1));

[0055] Where h1 is the output of the first-layer bidirectional long short-term memory network; ExpandDims is to insert a dimension at the specified position; A is the audio feature matrix composed of the above audio feature vectors; axis=1 is to insert a new dimension at the dimension position with index 1.

[0056] The output of the first-layer bidirectional long short-term memory network is input into the second-layer bidirectional long short-term memory network of the second preset dimension (the second preset dimension is smaller than the first preset dimension, and the second preset dimension is 128) for deep modeling to capture the temporal dynamic characteristics and context-dependent information of the audio features, as follows:

[0057] h2=BiLSTM 128 (h1);

[0058] Among them, h2 is the output of the second-layer bidirectional long short-term memory network.

[0059] After capturing the temporal dynamic characteristics and context-dependent information of the audio sequence, it also includes: extracting the last layer hidden state vector of the output sequence of the bidirectional long short-term memory network, and using layer normalization and random Dropout operations (the dropout rate p is set to 0.3) to obtain audio modal features.

[0060] This method enhances the feature discrimination ability of audio modalities in complex speech scenarios, and is particularly suitable for individual recognition and emotional state recognition tasks in multi-speaker scenarios in clinical contexts.

[0061] To effectively obtain the patient's visual emotional expression information, the present invention adopts a facial video feature extraction method based on a deep convolutional neural network. The patient's speech segments are intercepted from the conversation video according to the timestamp, and frames are evenly extracted from each video, sampling a fixed number of key frames. All image frames are scaled to 224×224 size and uniformly standardized. The frame image is input into the DenseNet201 model pre-trained on ImageNet to extract 1920-dimensional deep visual features for each frame. To generate a global representation of the entire video, the feature vectors of all frames are average pooled to obtain the global visual vector of the video segment.

[0062] To enhance the nonlinear expressiveness of features, the present invention constructs an improved residual fully connected block, which includes a fully connected layer with ReLU activation and a dimensionally aligned shortcut connection. When the input and output dimensions are inconsistent, the shortcut path introduces a linear transformation to achieve dimensionality matching. Therefore, after extracting the global semantic features of the patient's text and the deep video features of the patient's speech video, the extracted features are also input into the improved residual fully connected block and subjected to batch normalization and conditional dropout processing (the dropout rate p is set to 0.2) to improve the robustness and generalization ability of the model.

[0063] Hierarchical cross-modal attention fusion module: It is used to perform bidirectional attention interaction fusion of text modality features and audio modality features based on a hierarchical bidirectional attention fusion strategy, as well as bidirectional attention interaction fusion of the fusion results of text modality features and audio modality features with video modality features to obtain multimodal fusion features.

[0064] In view of the shortcomings of existing multimodal fusion methods in dealing with semantic inconsistency and dynamic modal dependency, the present invention proposes a hierarchical cross-modal attention fusion module, which aims to achieve effective alignment and deep interaction of multimodal features. The core technical idea of ​​this module is to introduce a local-global two-stage attention mechanism to achieve fine-grained alignment between the two modalities and overall integration between all modalities in stages. Through the multi-head attention mechanism, the model can autonomously learn the interaction strength and context sensitivity between modalities, while adopting a cross-layer connection mechanism to retain the original information of each modality and alleviate the problem of information loss in the feature fusion process. Figure 2 As shown in Figure 2, the hierarchical cross-modal attention fusion module mainly consists of the following two core components:

[0065] Modality-aligned projection layer: To address the dimensionality inconsistency and distribution heterogeneity of multimodal feature spaces, this paper designs a learnable modality-specific linear projection network. Before fusing multimodal features based on a hierarchical bidirectional attention fusion strategy, the input text, audio, and video modality features are mapped to a unified low-dimensional feature space based on a learnable modality-specific linear projection network, as follows:

[0066]

[0067] in, is the result of mapping the i-th multimodal feature to a unified low-dimensional feature space; γ i 、W i 、b i are the scaling factor, projection weight matrix, and bias term corresponding to the i-th multimodal feature, and the output dimension is uniformly set to d = 256; to ensure the independence and balance between different modalities, the weight matrix W i Using orthogonal initialization, the scaling factor γ i is a trainable parameter used to dynamically adjust the contribution of each modality to the final fusion representation; f i is the original feature of the i-th multimodal feature; text, audio, and video are text, audio, and video respectively.

[0068] Bidirectional cross-modal attention layer: Based on the above feature alignment, in order to achieve more refined interaction between different modalities, this paper further proposes a hierarchical bidirectional attention fusion strategy. Based on the hierarchical bidirectional attention fusion strategy, bidirectional attention interaction fusion of text modality features and audio modality features is performed, specifically:

[0069] The multi-head attention mechanism is used to construct a bidirectional interactive relationship between text modality features and audio modality features as follows:

[0070]

[0071] Among them, A text→audio 、A audio→text is the bidirectional interaction between text modality features and audio modality features; MultiHead is a multi-head attention mechanism; Q, K, and V are query, key, and value vectors, respectively, and the weight parameter is a learnable variable; They are text modality features and audio modality features respectively;

[0072] The bidirectional interaction relationship between text modality features and audio modality features is added to the original modality features through residual connection, and layer normalization is performed to obtain the bidirectional attention interaction fusion result of text modality features and audio modality features, as follows:

[0073]

[0074] Among them, F TA is the result of the bidirectional attention interaction fusion of text modality features and audio modality features; LayerNorm is the layer normalization operation; || is the feature concatenation operation. TA A global average pooling operation is performed to compress its temporal dimension.

[0075] Based on the hierarchical bidirectional attention fusion strategy, the fusion results of text modality features and audio modality features are fused with bidirectional attention interaction of video modality features. Specifically:

[0076] The multi-head attention mechanism is used to construct a bidirectional interactive relationship between the fusion results of text modality features and audio modality features and the bidirectional attention interaction fusion of video modality features, as follows:

[0077]

[0078] in, is the bidirectional interaction between the fusion result of text modality features and audio modality features and the bidirectional attention interaction fusion of video modality features; MultiHead is the multi-head attention mechanism; Q, K, V are query, key and value vectors respectively; F TA 、 They are the fusion results of text modality features and audio modality features, and video modality features respectively;

[0079] The bidirectional interaction relationship of the fusion result of the text modality feature and the audio modality feature and the bidirectional attention interaction fusion of the video modality feature is added to the original modality feature through residual connection, and the layer normalization operation is performed to obtain the multimodal fusion feature, as follows:

[0080]

[0081] Among them, F fused It is a multimodal fusion feature that can be directly input into the subsequent classifier module for decision-making; LayerNorm is a layer normalization operation; || is a feature splicing operation.

[0082] Residual enhanced multilayer perceptron classification decision module: used to input multimodal fusion features into the multilayer perceptron classification decision model based on the residual enhanced structure to obtain anxiety disorder detection results.

[0083] In order to improve the discrimination ability of the classification model and enhance the nonlinear expression ability of the fusion features, the present invention proposes a multi-layer perceptron (MLP) classification decision module based on the residual enhancement structure.

[0084] Multi-layer perceptron classification decision model based on residual enhancement structure adopts multimodal feature fusion vector F fused As input, the multimodal fusion features are extracted and nonlinearly transformed layer by layer through three layers of residual fully connected blocks in series, as follows:

[0085] h1=ResBlock(F fused ,1024),h1=BN(h1),Dorpout(p=0.5);

[0086] h2=ResBlock(h1,512),h2=BN(h2),Dorpout(p=0.4);

[0087] h3=ResBlock(h2,256), h3=BN(h3);

[0088] Among them, h1 is the multimodal feature fusion vector F fused The output after the first layer of residual blocks; BN is batch normalization; h2 is the output of h1 after the second layer of residual blocks; h3 is the output of h2 after the third layer of residual blocks; finally, h3 passes through a Sigmoid activation function to obtain the predicted probability of the binary classification task, as follows:

[0089]

[0090] Among them, σ is the Sigmoid activation function, which is used to convert the model output value Compress to the interval [0,1] to achieve binary classification probability output; b is the bias term.

[0091] It also includes: for the anxiety state recognition binary classification task handled by the present invention, cross entropy is used as the loss function to measure the difference between the model prediction value and the actual label, as follows:

[0092]

[0093] Among them, y i ∈{0, 1} is the true label of the i-th sample; The predicted probability output by the model is N, which is the total number of samples.

[0094] The embodiment of the present invention discloses an anxiety disorder detection system based on hierarchical cross-modal attention fusion. First, in the module feature encoding module, the pre-trained BERT model is used to encode the conversation text between the patient and the doctor to obtain a context-aware semantic representation; the Librosa tool is used to extract the low-level acoustic features of the audio, and the speech timing characteristics are captured by stacking a double-layer BiLSTM; at the same time, the DenseNet201 deep convolutional neural network is used to extract the visual emotion features of the conversation video frames, and the feature expression ability is enhanced through residual connection and normalization processing. Then, in the hierarchical cross-modal attention fusion module, a two-level bidirectional cross-modal attention mechanism is designed. The first stage realizes the fine-grained interaction between the text and speech modalities to obtain an intermediate fusion representation; the second stage further fuses the intermediate representation with the video modality, thereby realizing hierarchical information integration between modalities and enhancing the deep correlation modeling of semantics and emotions. Finally, the fused feature representation is input into the residual enhanced multilayer perceptron classification module. Through the multi-layer fully connected network and residual connection structure, combined with batch normalization and Dropout technology, overfitting is effectively prevented, the generalization ability of the model is improved, and finally a binary classification is achieved to determine whether the patient is in an anxiety state. The present invention takes into account both the modeling of internal temporal information of the modality and the modeling of the correlation between modalities, and has good interpretability and scalability. Experimental results show that this method has higher accuracy and robustness in anxiety recognition tasks than traditional single modality or simple fusion methods, and is suitable for various scenarios such as smart medical care, mental health assessment, and remote diagnosis and treatment.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0096] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An anxiety disorder detection system based on hierarchical cross-modal attention fusion, characterized by: include: Modal feature encoding module: used to extract global semantic features of patient text in clinical interviews based on the pre-trained BERT model to obtain text modal features; It is used to capture the temporal dynamic characteristics and contextual dependency information of patient audio features in clinical interviews by stacking a double-layer bidirectional long short-term memory network to obtain audio modality features; it is used to extract deep video features of patient speech videos in clinical interviews based on a deep convolutional neural network to obtain video modality features; Hierarchical cross-modal attention fusion module: used for performing bidirectional attention interaction fusion of the text modality features and the audio modality features, and bidirectional attention interaction fusion of the fusion results of the text modality features and the audio modality features with the video modality features based on a hierarchical bidirectional attention fusion strategy, to obtain multimodal fusion features; Residual enhanced multilayer perceptron classification decision module: used to input the multimodal fusion features into the multilayer perceptron classification decision model based on the residual enhanced structure to obtain the anxiety disorder detection results.

2. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: The pre-trained BERT model is used to extract global semantic features of patient text in clinical interviews, specifically: The semantic context is modeled based on the bidirectional Transformer architecture, and the text is mapped into a token sequence using the WordPiece word segmenter, which is further embedded into a vector of a preset dimension. After processing through a 24-layer Transformer structure, the output vector [0] at the [CLS] position is extracted as the global semantic feature of the text.

3. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: The temporal dynamics and context-dependent information of patient audio features in clinical interviews are captured by stacking a two-layer bidirectional long short-term memory network. Specifically: The audio features are expanded in the time dimension through a first-layer bidirectional long short-term memory network of a first preset dimension, and the output of the first-layer bidirectional long short-term memory network is input into a second-layer bidirectional long short-term memory network of a second preset dimension for deep modeling to capture the temporal dynamic characteristics and context-dependent information of the audio features; wherein the second dimension is smaller than the first dimension.

4. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: After capturing the temporal dynamic characteristics and context-dependent information of the audio sequence, it also includes: extracting the last layer hidden state vector of the output sequence of the bidirectional long short-term memory network, and using layer normalization and random Dropout operations to obtain the audio modal features.

5. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: After extracting the global semantic features of the patient text and the deep video features of the patient speech video, it also includes: inputting the extracted features into an improved residual fully connected block, and performing batch normalization and conditional Dropout processing; wherein, the improved residual fully connected block includes a fully connected layer with ReLU activation and a dimensionally aligned shortcut connection. When the input and output dimensions are inconsistent, the shortcut path introduces a linear transformation to achieve dimensional matching.

6. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: Before fusing multimodal features based on the hierarchical bidirectional attention fusion strategy, the following steps are also included: Based on a learnable modality-specific linear projection network, the input text, audio, and video modality features are mapped to a unified low-dimensional feature space, as follows: in, is the result of mapping the i-th multimodal feature to a unified low-dimensional feature space; γ i 、W i 、b i are the scaling factor, projection weight matrix, and bias term corresponding to the i-th multimodal feature; f i is the original feature of the i-th multimodal feature; test, audio, and video are text, audio, and video respectively.

7. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: Based on the hierarchical bidirectional attention fusion strategy, the bidirectional attention interactive fusion of the text modality features and the audio modality features is performed, specifically: The multi-head attention mechanism is used to construct a bidirectional interactive relationship between the text modality features and the audio modality features as follows: Among them, A text→audio 、A audio→text is the bidirectional interactive relationship between the text modality features and the audio modality features; MultiHead is a multi-head attention mechanism; Q, K, and V are query, key, and value vectors, respectively; are the text modal features and audio modal features respectively; The bidirectional interaction relationship between the text modality features and the audio modality features is added to the original modality features through residual connection, and layer normalization is performed to obtain the bidirectional attention interaction fusion result of the text modality features and the audio modality features, as follows: Among them, F TA is the result of the bidirectional attention interaction fusion of the text modality features and the audio modality features; LayerNorm is the layer normalization operation; || is the feature splicing operation.

8. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: Based on the hierarchical bidirectional attention fusion strategy, the bidirectional attention interactive fusion of the fusion results of the text modality features and the audio modality features and the video modality features is performed, specifically: The multi-head attention mechanism is used to construct a bidirectional interactive relationship between the fusion result of the text modality feature and the audio modality feature and the bidirectional attention interaction fusion of the video modality feature, as follows: in, is the bidirectional interaction relationship between the fusion result of the text modality feature and the audio modality feature and the bidirectional attention interaction fusion of the video modality feature; MultiHead is a multi-head attention mechanism; Q, K, and V are query, key, and value vectors respectively; F TA 、 are respectively the fusion result of the text modality feature and the audio modality feature, and the video modality feature; The bidirectional interaction relationship of the fusion result of the text modality feature and the audio modality feature and the bidirectional attention interaction fusion of the video modality feature is added to the original modality feature through residual connection, and layer normalization operation is performed to obtain the multimodal fusion feature, as follows: Among them, F fused is the multimodal fusion feature; LayerNorm is the layer normalization operation; || is the feature splicing operation.

9. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: The multi-layer perceptron classification decision model based on the residual enhancement structure extracts and nonlinearly transforms the multimodal fusion features layer by layer through three layers of residual fully connected blocks connected in series, and finally obtains the prediction probability of the binary classification task through a Sigmoid activation function.

10. The anxiety disorder detection system based on hierarchical cross-modal attention fusion according to claim 1, characterized in that: Also includes: The cross entropy is used as the loss function as follows: Among them, y i ∈{0, 1} is the true label of the i-th sample; The predicted probability output by the model is N, which is the total number of samples.

Images