Multi-modal psychological state assessment system based on adaptive multi-branch fusion
By constructing an adaptive multi-branch fusion multimodal mental state assessment system, the problems of modal heterogeneity and insufficient cross-modal interaction modeling in multimodal mental state identification are solved, achieving fine differentiation and efficient identification of mental states, and is suitable for auxiliary screening of mental health and remote medical diagnosis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2026-02-13
- Publication Date
- 2026-05-15
AI Technical Summary
Existing multimodal mental state recognition methods suffer from modal heterogeneity, insufficient cross-modal interaction modeling, and a lack of adaptability in fusion strategies, making it difficult to meet the needs for fine differentiation and efficient deployment of multiple categories of mental states.
A multimodal mental state assessment system based on adaptive multi-branch fusion is constructed. It employs a feature encoding module, an adaptive multi-branch fusion module, and a classification module to extract features from text, speech, and video modalities, respectively. The optimal fusion weight is adaptively calculated through a dynamic adaptive cross-modal attention fusion module and a multi-branch fusion mechanism, and finally, a multilayer perceptron is used for classification decision.
It enhances the ability to utilize multimodal information, improves the accuracy, stability and generalization of psychological state recognition, and is applicable to scenarios such as psychological health auxiliary screening and remote medical diagnosis.
Smart Images
Figure CN122050718A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of psychological state assessment technology, and more specifically to a multimodal psychological state assessment system based on adaptive multi-branch fusion. Background Technology
[0002] With the accelerating pace of society and increasing life pressures, the incidence of mental health problems such as anxiety disorders and depressive disorders is rising year by year, becoming a significant factor affecting public mental and physical health and social stability. In clinical practice, the assessment of mental state mainly relies on psychiatrists or psychological counselors to make comprehensive judgments through clinical interviews, scale assessments, and long-term follow-ups. This process is highly dependent on professional experience and suffers from problems such as strong subjectivity, low efficiency, and difficulty in large-scale promotion, especially in primary healthcare institutions and telemedicine scenarios where the scarcity of professional mental health resources is particularly prominent. Therefore, exploring an objective, automated, and scalable mental state identification technology has significant practical significance and application value.
[0003] With the development of artificial intelligence technology, emotion computing and mental state recognition methods based on multimodal data have gradually emerged, becoming an important research direction for intelligent auxiliary assessment of mental health. However, existing multimodal mental state recognition methods still have many shortcomings. On the one hand, different modalities exhibit significant heterogeneity in data distribution, feature scale, and semantic expression. Simple feature splicing or fixed-weight fusion cannot fully explore the complementary information between modalities and easily introduces redundant or noisy features. On the other hand, some methods only focus on modeling features within a modality, neglecting the deep semantic interaction relationships between text, speech, and video, resulting in insufficient utilization of cross-modal association information. In addition, existing methods mostly adopt a single path or a single fusion strategy, lacking the ability to adaptively model the differences in contributions from different fusion modes. When facing different individual differences and complex clinical scenarios, the robustness and generalization ability of the models are still limited. At the classification decision level, existing technologies are mostly concentrated on binary classification tasks, which are difficult to meet the needs of fine differentiation of multiple mental states such as normal, anxious, and depressed in actual clinical practice. At the same time, some model structures are relatively complex, training is unstable, and it is difficult to deploy and promote them efficiently in real applications.
[0004] Therefore, how to provide a multimodal mental state assessment system based on adaptive multi-branch fusion that can fully utilize multimodal information, has adaptive fusion capabilities, supports multi-category mental state discrimination, and has good stability and generalization performance is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the present invention provides a multimodal psychological state assessment system based on adaptive multi-branch fusion. It aims to improve the automated identification capability of normal, anxious, and depressive psychological states in clinical patients. Addressing the problems of significant modal heterogeneity, insufficient cross-modal interaction modeling, and lack of adaptability in fusion strategies in existing psychological state identification methods during multimodal data fusion, an end-to-end deep learning framework is constructed to jointly model and intelligently discriminate text, voice, and video multimodal information in clinical interview scenarios.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal mental state assessment system based on adaptive multi-branch fusion includes: Feature encoding module, adaptive multi-branch fusion module, and classification module; The feature encoding module includes: a text modality feature extraction module, an audio modality feature extraction module, and a video modality feature extraction module; Text modality feature extraction module: used to encode patient interview texts using the large-scale Chinese semantic model SentenceTransformer to obtain sentence vectors; Audio modal feature extraction module: Used to employ Whisper Large-V3 as a speech encoder and combine it with low-level acoustic features to form a hybrid representation of semantic vectors and acoustic features; Video modality feature extraction module: used to extract visual depth features frame by frame using pre-trained EfficientNet-B2, while fusing inter-frame motion information; Adaptive multi-branch fusion module: This module is used for dynamic adaptive cross-modal attention fusion and introduces attention enhancement and cross-modal interaction, multi-branch fusion and adaptive gating mechanisms. In the three-class classification task, it adaptively calculates the optimal fusion weights based on the signal-to-noise ratio of different patient modal features. Classification module: Used to input the fused features into the multilayer perceptron to complete the final classification decision.
[0007] Optionally, the patient interview text is encoded using the large-scale Chinese semantic model SentenceTransformer to obtain sentence vectors, specifically: The system automatically segments all responses from each patient into sentences based on punctuation, breaking long conversations into shorter sentences. Represent all the speech fragments of a given patient, and assign each sentence to... Encode into sentence vectors using Transformer. .
[0008] Optionally, after obtaining the sentence vector, the process may also include: processing the sentence vector... The operations for average pooling and max pooling are performed as follows:
[0009] After concatenating the double-pooled sentence vectors, a fixed-dimensional text vector is obtained through a linear mapping layer and a normalization mapping layer. Finally, the text feature representation is obtained. .
[0010] Optionally, Whisper Large-V3 is used as the speech encoder, and combined with low-level acoustic features to form a hybrid representation of semantic vectors and acoustic features, specifically: Based on the timestamp information in the patient's text JSON file, audio segments during the patient's speech are extracted and input into a Whisper Processor for encoding to obtain the hidden state of the time dimension. Its average vector is used as a speech semantic embedding. ; Additional traditional acoustic features, including MFCC, F0, RMS, ZCR, spectral centroid, and spectral contrast, are extracted and represented as high-dimensional speech behavior features. The final speech feature representation is obtained. The audio features were then normalized using Z-score.
[0011] Optionally, pre-trained EfficientNet-B2 is used for frame-by-frame visual depth feature extraction, while fusing inter-frame motion information, specifically: The entire video is sampled uniformly over time, and each frame is encoded using EfficientNet-B2. The model then calculates three types of statistics and concatenates them into frame-level aggregated features. ; Farneback optical flow was calculated based on adjacent frames. The mean motion amplitude between frames in the patient's clinical interview video was extracted, and the mean optical flow amplitude, standard deviation, median, and proportion of high-motion frames were extracted to characterize potential psychological and behavioral signals, including emotional arousal, eye movement frequency, and facial change rate. This was denoted as... ; Ultimately, the video features of different patients are composed of depth static features and optical flow dynamic features, represented as follows: .
[0012] Optionally, it may also include: a feature enhancement module: used to enhance the extracted text modality features. Audio modal features and video modal features L2 Normalization is uniformly used for intramodal normalization, specifically as follows: Perform on the extracted modal features Normalized text feature vectors were obtained respectively. Speech feature vectors and video feature vectors ; It automatically fills in zero vectors when any of the text, voice, or video modalities are missing. Each modality operates independently with try-except, and all three modalities strictly output fixed-dimensional embeddings.
[0013] Optionally, before the dynamically adaptive cross-modal attention fusion module, which introduces attention enhancement and cross-modal interaction, multi-branch fusion, and adaptive gating mechanisms, adaptively calculates the optimal fusion weights based on the signal-to-noise ratio of different patient modal features in a three-class classification task, the module further includes: The obtained feature vectors are normalized using LayerNorm, and non-linear transformations are performed using the GELU activation function and Dropout, ultimately mapping them to a unified 512-dimensional feature space. The mapped text, speech, and video features are represented as follows: , , .
[0014] Optionally, a dynamic adaptive cross-modal attention fusion module is used, incorporating attention enhancement and cross-modal interaction, multi-branch fusion, and adaptive gating mechanisms. In the three-class classification task, the optimal fusion weights are adaptively calculated based on the signal-to-noise ratio of different patient modal features. Specifically: It mainly consists of three core components: global self-attention enhancement, three-branch parallel fusion structure, and sample-level branch gating fusion mechanism; Global self-attention enhancement: used to achieve semantic alignment between different modalities and model higher-order associations between modalities, specifically: Features of the three modalities , , Stacked as a sequence The input is fed into the multi-head self-attention module for modal interaction processing, as follows:
[0015] Each attention head is defined as follows: ;in, , , From linear mapping Each The dimension is The enhanced modality is split into unimodal representations, allowing each modality to reference the semantic information of other modalities when calculating its representation, thus generating refined features. , , ; The three-branch parallel fusion structure includes three parallel branches: early splicing, attention-enhanced fusion, and cross-modal fusion, with each branch focusing on different information fusion levels; Early stitching: Directly stitching by channel in the projection space, stitching together the projection features of the three modes to obtain... The features are normalized using LayerNorm and overfitted using Dropout. The final feature representation is as follows:
[0016] Attention-enhanced fusion: Average pooling is performed on the output of the attention-enhanced sequence to obtain the global context semantic vector. and will Repeat three times according to the feature dimension to obtain Subsequently, this residual signal is added to the original modal splicing features to obtain an enhanced representation. Finally, the attention-enhanced fusion features are obtained through normalization and Dropout:
[0017] Cross-modal fusion: used to further enhance the deep semantic interaction capabilities between text, speech, and video modalities, specifically: The refined modal features are input into the cross-modal fusion module, where three directional cross-modal attention mechanisms are calculated sequentially, each employing a standard multi-head attention method. MHA These are extracted semantics from text via audio, text via video, and audio via video, resulting in three cross-modal augmented features. , and Then, the three cross-modal attention features are concatenated and... GMU Linear gating generates cross-modal fusion weights ;in, This is the gated weight matrix; for Sigmoid Activation; then fusion is performed using learnable gating to obtain... Among them, when When it approaches 1, it leans towards video-enhanced semantics; when it approaches 1... 0 At that time, the audio-biased enhancement was used to strengthen the semantics; subsequently, the fused features were... Input to position feedforward network FFN In the middle, we obtained ;in, Use the GELU activation function; , The transformation matrix is linear; the final output is:
[0018] A sample-level branch-gated fusion mechanism is used to further improve the model's adaptability in real-world clinical scenarios, specifically as follows: Using a modal-gated fusion strategy, the model can dynamically adjust the fusion weights between modalities based on the different modal qualities of different patients; firstly, the feature representations of the three branches are concatenated according to channels, and the concatenated feature representation is as follows: The data is then input into a gating network, and the dynamic branch weights for each sample are calculated using a two-layer neural network, as follows:
[0019] To each Do Obtain the branch weights for each sample The final fusion feature is the weighted sum of the branches, i.e. .
[0020] Optionally, the fused features are input into a multilayer perceptron to complete the final classification decision, specifically:
[0021] in, Predictive logits for three states: normal, anxious, and depressed; The entire network is trained end-to-end. For the three-class classification task, the model outputs... A softmax function is applied to obtain the class probability distribution, and the discrimination boundaries for three categories—anxiety, depression, and normality—are learned through a multi-class cross-entropy loss function.
[0022] in, This represents the number of training samples; Number of categories; For the sample The true label in class c; This represents the probability of the corresponding class predicted by the model.
[0023] As can be seen from the above technical solution, compared with the prior art, this invention discloses a multimodal psychological state assessment system based on adaptive multi-branch fusion. First, in the modal feature encoding module, feature extraction strategies are designed for the characteristics of different modal data. For the text modality, the SentenceTransformer, a large-scale Chinese semantic pre-trained model, is used to encode the patient's clinical interview text to obtain a global semantic representation with contextual semantic awareness. For the speech modality, Whisper Large-V3 is used as the speech semantic encoder, and multiple low-level acoustic features are fused to construct a hybrid speech representation that takes into account both high-level semantic information and fine-grained acoustic characteristics. For the video modality, the patient's speaking video is processed frame-by-frame based on a pre-trained EfficientNet-B2 network to extract high-level visual depth features, thereby representing the emotional information in facial expressions and behavioral cues. Second, in the adaptive multi-branch fusion module, the model introduces a multi-branch fusion structure to characterize multimodal relationships at different levels. The multimodal features are initially refined through a global attention mechanism, and on this basis, the original splicing fusion branch, the attention-enhanced fusion branch, and the cross-modal interaction fusion branch are constructed in parallel. The cross-modal interaction fusion branch utilizes a multi-head cross-modal attention and gating fusion mechanism to achieve deep semantic interaction modeling between text, speech, and video modalities. Furthermore, a branch weight gating mechanism is designed to adaptively assign weights to each fusion branch based on sample-level features, and the multi-branch outputs are weighted and aggregated to obtain a more discriminative and robust multimodal fusion representation. Finally, in the classification decision module, the adaptively fused multimodal features are input into a multilayer perceptron classification network. Through multi-level linear mapping, nonlinear activation, normalization, and regularization operations, redundant information is gradually compressed and discriminative ability is enhanced, ultimately achieving a three-class classification of the patient's psychological state, outputting three recognition results: normal, anxious, and depressed. The system proposed in this invention can simultaneously consider intramodal feature modeling and intermodal correlation modeling, effectively improving the accuracy, stability, and generalization ability of psychological state recognition in complex clinical interview scenarios. It has good scalability and practical value, and is suitable for applications such as psychological health auxiliary screening, remote medical diagnosis, and intelligent human-computer interaction. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0025] Picture 1 This is a schematic diagram of the system structure provided by the present invention. Detailed Implementation
[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] Example 1: Embodiment 1 of this invention discloses a multimodal mental state assessment system based on adaptive multi-branch fusion, such as... Picture 1 As shown, it includes: a feature encoding module, an adaptive multi-branch fusion module, and a classification module.
[0028] The feature encoding module includes: a text modality feature extraction module, an audio modality feature extraction module, and a video modality feature extraction module.
[0029] The text modality feature extraction module (for doctor-patient dialogue scenarios, analyzing psychological health signals in patients' language) is used to encode patient interview texts using the large-scale Chinese semantic model SentenceTransformer to obtain high-quality sentence vectors suitable for subsequent dynamic fusion modules. A text representation method of "sentence-level encoding + multi-pooling fusion" is designed to fully capture the semantic information in clinical interview dialogues between patients and doctors, and avoid losing local information such as emotional fluctuations and strong emotional signals in short sentences.
[0030] The patient interview text was encoded using the large-scale Chinese semantic model SentenceTransformer to obtain high-quality sentence vectors suitable for the subsequent dynamic fusion module, specifically: Extract the text content from the patient's speech from the text file, and automatically segment all responses for each patient according to punctuation marks ("。!?;"), breaking long dialogues into several shorter sentences. Represent all the speech fragments of a given patient, and assign each sentence to... Encode into sentence vectors using Transformer. .
[0031] To simultaneously capture global semantic trends and key information, after obtaining the sentence vector, the process also includes: processing the sentence vector... The operations for average pooling and max pooling are performed as follows:
[0032] Furthermore, to improve the model's robustness to semantic distributions and enable long interview texts and short text segments to be encoded in a unified feature space, the double-pooled sentence vectors are concatenated, and then a fixed-dimensional text vector is obtained through a linear mapping layer and a normalization mapping layer. Finally, the text feature representation is obtained. .
[0033] A patient's speech modality not only carries semantic information but also reflects their emotions, physiological state, and psychological symptoms. This invention designs a hybrid audio modality feature extraction module that integrates deep semantic vectors and traditional prosodic features: it uses Whisper Large-V3 as a speech encoder and combines rich low-level acoustic features to form a hybrid representation of semantic vectors and fine acoustic features.
[0034] Whisper Large-V3 is used as the speech encoder, and combined with low-level acoustic features, a hybrid representation of semantic vectors and acoustic features is constructed, specifically as follows: To fully capture the semantic information, timbre, emotion, pauses, and natural spoken features of clinical dialogue, the audio segments during the patient's speech are first extracted based on the timestamp information in the patient's text JSON file and then input into a Whisper Processor for encoding to obtain the time-dimensional hidden state. Its average vector is used as a speech semantic embedding. ; To enhance the model's sensitivity to the emotional speech features of different patients, additional traditional acoustic features, including MFCC, F0, RMS, ZCR, spectral centroid, and spectral contrast, are extracted and represented as high-dimensional speech behavior features. The final speech feature representation is obtained. Furthermore, Z-score normalization is applied to the audio features to improve their compatibility with other modalities.
[0035] In clinical interviews, patients' emotional state and behavioral patterns are often reflected through facial expressions, movement amplitude, and energy changes. Videos of patients with mood disorders contain physiological and behavioral signals such as micro-expressions, head movements, and decreased movement amplitude, which can be better used to distinguish different categories of patients. Based on this, this invention proposes a video modality feature extraction module: used to extract frame-by-frame visual depth features using pre-trained EfficientNet-B2, while fusing inter-frame motion information.
[0036] Frame-by-frame visual depth feature extraction is performed using a pre-trained EfficientNet-B2, while simultaneously fusing inter-frame motion information, specifically as follows: To enhance model robustness and avoid bias towards short segments, the entire video is sampled uniformly over time, and each frame is encoded using EfficientNet-B2. Subsequently, in order to capture the intensity of the patient's movements and changes in facial expressions, the model calculated three types of statistics and concatenated them into frame-level aggregated features. ; To better capture the movement characteristics of different patients, this invention also calculates Farneback optical flow based on adjacent frames, extracts the mean motion amplitude between frames in the patient's clinical interview video, and extracts the mean optical flow amplitude, standard deviation, median, and proportion of high-motion frames to characterize potential psychological and behavioral signals, including emotional arousal, eye movement frequency, and facial change speed, denoted as... ; Ultimately, the video features of different patients are composed of depth static features and optical flow dynamic features, represented as follows: .
[0037] It also includes: a feature enhancement module: used to improve the comparability of trimodal features in the shared space and the stability of cross-modal fusion, by enhancing the extracted text modal features. Audio modal features and video modal features L2 Normalization is uniformly used for intra-modal standardization to ensure that there is no scale bias between different modalities. This significantly improves the model's robustness to multimodal scale differences and reduces gradient instability during training in the fusion phase. Specifically: Perform on the extracted modal features Normalized text feature vectors were obtained respectively. Speech feature vectors and video feature vectors ; Furthermore, throughout the feature construction process, zero vectors are automatically filled when any of the text, speech, or video modalities are missing, ensuring consistent data dimensionality and preventing interruption of the training process. This mechanism enhances the model's stability in handling large-scale real-world clinical data. Each modality operates independently using try-except, and all three modalities strictly output fixed-dimensional embeddings, providing a unified input shape for subsequent fusion stages.
[0038] Adaptive multi-branch fusion module: To fully explore the complementarity between text, speech and video modalities and improve the model's dynamic adaptability to the importance of different modalities, it is based on a dynamic adaptive cross-modal attention fusion module and introduces attention enhancement and cross-modal interaction, multi-branch fusion and adaptive gating mechanism. In the three-class classification task, it adaptively calculates the optimal fusion weights according to the signal-to-noise ratio of different patient modal features.
[0039] The module, based on dynamic adaptive cross-modal attention fusion and incorporating attention enhancement, cross-modal interaction, multi-branch fusion, and adaptive gating mechanisms, adaptively calculates the optimal fusion weights based on the signal-to-noise ratio of different patient modalities in a three-class classification task. Before this, it also includes: To ensure numerical stability and prevent model overfitting, the obtained feature vectors are normalized using LayerNorm, and non-linear transformations are performed using the GELU activation function and Dropout, ultimately mapping them to a unified 512-dimensional feature space to prepare for subsequent modality fusion. The mapped text, speech, and video features are represented as follows: , , .
[0040] Based on a dynamically adaptive cross-modal attention fusion module, and incorporating attention enhancement and cross-modal interaction, multi-branch fusion, and adaptive gating mechanisms, the optimal fusion weights are adaptively calculated in a three-class classification task according to the signal-to-noise ratio of different patient modalities. Specifically: It mainly consists of three core components: global self-attention enhancement, three-branch parallel fusion structure, and sample-level branch gating fusion mechanism; Global self-attention enhancement: used to achieve semantic alignment between different modalities and model higher-order associations between modalities, specifically: Features of the three modalities , , Stacked as a sequence The input is fed into the multi-head self-attention module for modal interaction processing, as follows:
[0041] Each attention head is defined as follows: ;in, , , From linear mapping Each The dimension is The enhanced modality is split into unimodal representations, allowing each modality to reference the semantic information of other modalities when calculating its representation, thus generating refined features. , , This structure can retain the characteristics of a single modality and enhance its ability to interact with other modalities, thereby minimizing feature drift between different modalities.
[0042] After completing the global self-attention enhanced interaction, each modality has contained semantic cues from other modalities. Based on this, a three-branch parallel fusion structure was constructed, including: early splicing, attention-enhanced fusion, and cross-modal fusion. Each branch focuses on different information fusion levels. Early stitching: First, stitch the projection features of the three modes directly in the projection space by channel to obtain... The features are normalized using LayerNorm and overfitted using Dropout. The final feature representation is as follows:
[0043] This process can better capture shallow modal information, preserve the original representation of each modality and its local information, and provide baseline fusion information.
[0044] Attention-enhanced fusion: After completing global cross-modal self-attention, the output of the attention-enhanced sequence is first subjected to average pooling to obtain the global context semantic vector. To maintain dimensional alignment with the trimodal concatenation vector from the previous branch, we will... Repeat three times according to the feature dimension to obtain Subsequently, this residual signal is added to the original modal splicing features to obtain an enhanced representation. Finally, the attention-enhanced fusion features are obtained through normalization and Dropout:
[0045] in, As a cross-modal global semantic compensation term from global sub-attention, it is injected into the first branch structure in a residual manner, so that each modality obtains guidance information from the global context, thereby achieving more refined attention enhancement fusion based on early concatenation.
[0046] Cross-modal fusion: used to further enhance the deep semantic interaction capabilities between text, speech, and video modalities, specifically: The refined modal features are input into the cross-modal fusion module, where three directional cross-modal attention mechanisms are calculated sequentially, each employing a standard multi-head attention method. MHA These are extracted semantics from text via audio, text via video, and audio via video, resulting in three cross-modal augmented features. , and Then, the three cross-modal attention features are concatenated and... GMU Linear gating generates cross-modal fusion weights ;in, This is the gated weight matrix; for Sigmoid Activation; then fusion is performed using learnable gating to obtain... Among them, when When it approaches 1, it leans towards video-enhanced semantics; when it approaches 1... 0 At the same time, the design favors audio-based enhancement of semantics, avoiding multimodal gradient conflicts; subsequently, the fused features are... Input to position feedforward network FFN In the middle, we obtained ;in, Use the GELU activation function; , The transformation matrix is linear; the final output is:
[0047] This structure integrates directional cross-modal attention and gating mechanisms, enabling it to capture fine-grained inter-modal reasoning signals, dynamically adjust information flow in multimodal contention, provide cross-modal global guidance, and ultimately combine local attention reasoning capabilities with global consistency constraints, allowing the model to capture complex cross-modal semantic interactions.
[0048] A sample-level branch-gated fusion mechanism is used to further improve the model's adaptability in real-world clinical scenarios, specifically as follows: Using a modal-gated fusion strategy, the model can dynamically adjust the fusion weights between modalities based on the different modal qualities of different patients; firstly, the feature representations of the three branches are concatenated according to channels, and the concatenated feature representation is as follows: The data is then input into a gating network, and the dynamic branch weights for each sample are calculated using a two-layer neural network, as follows:
[0049] To each Do Obtain the branch weights for each sample The final fusion feature is the weighted sum of the branches, i.e. This gating structure uses a sample set rather than a batch or class level, allowing each sample to adaptively select the most useful fusion path. Furthermore, the weights are derived from the data itself, rather than from fixed learning parameters. This approach can better model the modal contribution differences of real clinical information, thereby significantly improving the stability and accuracy of the anxiety, depression, and normality tri-classification tasks.
[0050] The classification module is used to improve the discriminative ability of the classification model and enhance the nonlinear expressive ability of the fused features. After completing the multi-branch adaptive fusion, the fused features are input into the multilayer perceptron to complete the final classification decision.
[0051] The fused features are then input into a multilayer perceptron to complete the final classification decision, specifically:
[0052] in, The model predicts logits for three states: normal, anxious, and depressed. The MLP's structural design balances representational power and training stability, consisting of multi-level linear transformations, nonlinear activation, normalization, and regularization. This structure progressively compresses redundant information and learns more discriminative decision boundaries in a higher-level space, thereby improving the model's generalization ability in complex multimodal scenarios.
[0053] The entire network is trained end-to-end. For the three-class classification task, the model outputs... A softmax function is applied to obtain the class probability distribution, and the discrimination boundaries for three categories—anxiety, depression, and normality—are learned through a multi-class cross-entropy loss function.
[0054] in, This represents the number of training samples; Number of categories; For the sample The true label in class c; This represents the probability of the corresponding class predicted by the model.
[0055] Therefore, the multimodal mental state assessment system based on adaptive multi-branch fusion proposed in this invention can comprehensively extract behavioral and psychological features from multiple dimensions such as patients' language content, tone of voice, and facial expressions. By achieving deep information interaction and fusion between modalities, it effectively improves the accuracy and robustness of the three-class classification task in mental health testing. By introducing an adaptive multi-branch fusion mechanism, this system overcomes the limitations of traditional single-modal or static fusion methods in collaborative modeling of multi-source heterogeneous information. It can more comprehensively and finely characterize the psychological features of patients in terms of language content, speech expression, and non-verbal behavior during clinical interviews, thereby significantly improving the ability to distinguish between normal, anxious, and depressed mental states. Simultaneously, the end-to-end deep learning framework constructed in this invention possesses good stability and generalization ability, effectively adapting to individual differences among patients and complex real-world scenarios, providing reliable technical support for mental health assisted screening, early risk warning, and telemedicine. This system not only improves the objectivity and automation level of mental state assessment but also provides a new technical path and practical paradigm for the in-depth application of multimodal artificial intelligence technology in the field of clinical mental health.
[0056] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0057] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multimodal mental state assessment system based on adaptive multi-branch fusion, characterized in that, include: Feature encoding module, adaptive multi-branch fusion module, and classification module; The feature encoding module includes: a text modality feature extraction module, an audio modality feature extraction module, and a video modality feature extraction module; The text modality feature extraction module is used to encode patient interview texts using the large-scale Chinese semantic model SentenceTransformer to obtain sentence vectors. The audio modal feature extraction module is used to employ Whisper Large-V3 as a speech encoder and combine it with low-level acoustic features to form a hybrid representation of semantic vectors and acoustic features. The video modality feature extraction module is used to extract visual depth features frame by frame using pre-trained EfficientNet-B2, while fusing inter-frame motion information. The adaptive multi-branch fusion module is used for a dynamic adaptive cross-modal attention fusion module, and introduces attention enhancement and cross-modal interaction, multi-branch fusion and adaptive gating mechanism. In the three-class classification task, it adaptively calculates the optimal fusion weight based on the signal-to-noise ratio of different patient modal features. The classification module is used to input the fused features into the multilayer perceptron to complete the final classification decision.
2. The multimodal psychological state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, The patient interview text was encoded using the large-scale Chinese semantic model SentenceTransformer to obtain sentence vectors, specifically: The system automatically segments all responses from each patient into sentences based on punctuation, breaking long conversations into shorter sentences. Represent all the speech fragments of a given patient, and assign each sentence to... Encode into sentence vectors using Transformer. .
3. The multimodal psychological state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, After obtaining the sentence vector, the process further includes: processing the sentence vector... The operations for average pooling and max pooling are performed as follows: After concatenating the double-pooled sentence vectors, a fixed-dimensional text vector is obtained through a linear mapping layer and a normalization mapping layer. Finally, the text feature representation is obtained. .
4. The multimodal psychological state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, Whisper Large-V3 is used as the speech encoder, and combined with low-level acoustic features, a hybrid representation of semantic vectors and acoustic features is constructed, specifically as follows: Based on the timestamp information in the patient's text JSON file, audio segments during the patient's speech are extracted and input into a Whisper Processor for encoding to obtain the hidden state of the time dimension. Its average vector is used as a speech semantic embedding. ; Additional traditional acoustic features, including MFCC, F0, RMS, ZCR, spectral centroid, and spectral contrast, are extracted and represented as high-dimensional speech behavior features. The final speech feature representation is obtained. The audio features were then normalized using Z-score.
5. The multimodal psychological state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, Frame-by-frame visual depth feature extraction is performed using a pre-trained EfficientNet-B2, while simultaneously fusing inter-frame motion information, specifically as follows: The entire video is sampled uniformly over time, and each frame is encoded using EfficientNet-B2. The model then calculates three types of statistics and concatenates them into frame-level aggregated features. ; Farneback optical flow was calculated based on adjacent frames. The mean motion amplitude between frames in the patient's clinical interview video was extracted, and the mean optical flow amplitude, standard deviation, median, and proportion of high-motion frames were extracted to characterize potential psychological and behavioral signals, including emotional arousal, eye movement frequency, and facial change rate. This was denoted as... ; Ultimately, the video features of different patients are composed of depth static features and optical flow dynamic features, represented as follows: .
6. The multimodal mental state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, Also includes: Feature enhancement module: used to enhance the extracted text modality features. Audio modal features and video modal features L2 Normalization is uniformly used for intramodal normalization, specifically as follows: Perform on the extracted modal features Normalized text feature vectors were obtained respectively. Speech feature vectors and video feature vectors ; It automatically fills in zero vectors when any of the text, voice, or video modalities are missing. Each modality operates independently with try-except, and all three modalities strictly output fixed-dimensional embeddings.
7. The multimodal psychological state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, The module, based on dynamic adaptive cross-modal attention fusion and incorporating attention enhancement, cross-modal interaction, multi-branch fusion, and adaptive gating mechanisms, adaptively calculates the optimal fusion weights based on the signal-to-noise ratio of different patient modalities in a three-class classification task. Before this, it also includes: The obtained feature vectors are normalized using LayerNorm, and non-linear transformations are performed using the GELU activation function and Dropout, ultimately mapping them to a unified 512-dimensional feature space. The mapped text, speech, and video features are represented as follows: , , .
8. The multimodal psychological state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, Based on a dynamically adaptive cross-modal attention fusion module, and incorporating attention enhancement and cross-modal interaction, multi-branch fusion, and adaptive gating mechanisms, the optimal fusion weights are adaptively calculated in a three-class classification task according to the signal-to-noise ratio of different patient modalities. Specifically: It mainly consists of three core components: global self-attention enhancement, three-branch parallel fusion structure, and sample-level branch gating fusion mechanism; The global self-attention enhancement is used to achieve semantic alignment between different modalities and model higher-order associations between modalities, specifically: Features of the three modalities , , Stacked as a sequence The input is fed into the multi-head self-attention module for modal interaction processing, as follows: Each attention head is defined as follows: ;in, , , From linear mapping Each The dimension is The enhanced modality is split into unimodal representations, allowing each modality to reference the semantic information of other modalities when calculating its representation, thus generating refined features. , , ; The three-branch parallel fusion structure includes three parallel branches: early splicing, attention-enhanced fusion, and cross-modal fusion, with each branch focusing on different information fusion levels; The early stitching: directly stitching along channels in the projection space, stitching together the projection features of the three modes to obtain... The features are normalized using LayerNorm and overfitted using Dropout. The final feature representation is as follows: The attention-enhanced fusion involves performing average pooling on the output of the attention-enhanced sequence to obtain a global context semantic vector. and will Repeat three times according to the feature dimension to obtain Subsequently, this residual signal is added to the original modal splicing features to obtain an enhanced representation. Finally, the attention-enhanced fusion features are obtained through normalization and Dropout: The cross-modal fusion is used to further enhance the deep semantic interaction capabilities between text, speech, and video modalities, specifically as follows: The refined modal features are input into the cross-modal fusion module, where three directional cross-modal attention mechanisms are calculated sequentially, each employing a standard multi-head attention method. MHA These are three methods: extracting semantics from text using audio, extracting semantics from text using video, and extracting semantics from audio using video, ultimately resulting in three cross-modal enhanced features. , and Then, the three cross-modal attention features are concatenated and... GMU Linear gating generates cross-modal fusion weights ;in, This is the gate weight matrix; for Sigmoid Activation; then fusion is performed using learnable gating to obtain... Among them, when When it approaches 1, it leans towards video-enhanced semantics; when it approaches 1... 0 At that time, the audio-biased enhancement was used to strengthen the semantics; subsequently, the fused features were... Input to position feedforward network FFN In the middle, we get ;in, Use the GELU activation function; , The transformation matrix is linear; the final output is: The sample-level branch-gated fusion mechanism is used to further improve the model's adaptability in real clinical scenarios, specifically as follows: Using a modal-gated fusion strategy, the model can dynamically adjust the fusion weights between modalities for different modal qualities in different patients; firstly, the feature representations of the three branches are concatenated according to channels, and the concatenated feature representation is as follows: The data is then input into a gating network, and the dynamic branch weights for each sample are calculated using a two-layer neural network, as follows: To each Do Obtain the branch weights for each sample The final fusion feature is the weighted sum of the branches, i.e. .
9. A multimodal psychological state assessment system based on adaptive multi-branch fusion according to claim 1, characterized in that, The fused features are then input into a multilayer perceptron to complete the final classification decision, specifically: in, Predictive logits for three states: normal, anxious, and depressed; The entire network is trained end-to-end. For the three-class classification task, the model outputs... A softmax function is applied to obtain the class probability distribution, and the discrimination boundaries for three categories—anxiety, depression, and normality—are learned through a multi-class cross-entropy loss function. in, This represents the number of training samples; Number of categories; For the sample The true label in class c; This represents the probability of the corresponding class predicted by the model.