Cross-attention multi-modal psychological representation extraction method and system based on semantic prior guidance

Abstract

The application discloses a cross-attention multi-modal psychological representation extraction method and system based on semantic prior guidance. The method collects text, speech, facial expression and physiological signal multi-modal psychological signal data; constructs a psychological field semantic prior knowledge base, generates a prior feature vector through knowledge graph embedding; performs single-modal feature coding on each modal data respectively to obtain independent feature representation; injects the prior feature vector into the cross-modal cross-attention module as attention bias, and completes inter-modal feature alignment and fusion through a multi-head cross-attention mechanism guided by the prior to generate multi-modal fusion features; inputs a hierarchical psychological representation projection head, and outputs a three-level multi-granularity psychological representation spectrum of fine-grained emotion, medium-grained symptom and coarse-grained syndrome. The application effectively fuses professional knowledge in the field of mental health, improves the accuracy and clinical relevance of multi-modal feature alignment, generates a hierarchical representation structure, and meets the progressive analysis needs of clinical decision-making.

Description

Technical Field

[0001] This application relates to the technical field of cross-modal artificial intelligence, specifically to a method and system for extracting cross-attention multimodal mental representations based on semantic prior guidance. Background Technology

[0002] With the development of artificial intelligence technology, multimodal mental representation extraction has significant application value in fields such as mental health assessment, human-computer interaction, and affective computing. Mental representation refers to the way the human cognitive system encodes external stimuli and internal psychological states. Accurate extraction of mental representations is of great significance for understanding human cognitive processes and achieving precise psychological intervention.

[0003] Existing multimodal mental representation extraction technologies suffer from three main shortcomings: First, multimodal feature fusion often employs a purely data-driven approach without prior knowledge, lacking guidance from professional knowledge in the field of mental health. This makes the models prone to learning spurious associations in the data, resulting in representations that do not conform to the logic of clinical psychological diagnosis. Second, intermodal alignment primarily relies on general attention mechanisms, failing to utilize prior knowledge of cross-modal manifestations of psychological symptoms. This leads to insufficient accuracy in cross-modal semantic alignment and weak generalization ability in small-sample clinical data scenarios. Third, mental representation outputs are mostly single-level classification results, unable to simultaneously output multi-granular hierarchical representations ranging from fine-grained emotions to coarse-grained syndromes. This makes it difficult to support the full-process clinical decision-making needs in psychological counseling scenarios, from emotion recognition to symptom assessment to syndrome judgment. In AI-driven mental health service scenarios, multimodal psychological assessment needs to simultaneously process users' textual expressions, speech acoustic features, facial micro-expressions, and physiological signals such as heart rate and electrodermal activity. Furthermore, the assessment results must conform to the diagnostic criteria for mental disorders and the logic of clinical psychological assessment. Existing purely data-driven deep learning solutions cannot meet the professionalism and interpretability requirements of clinical-level psychological assessment. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for extracting cross-attention multimodal mental representations based on semantic prior guidance. This method can effectively integrate professional knowledge in the field of mental health, improve the accuracy and clinical relevance of multimodal feature alignment, and generate a hierarchical representation structure from emotion to symptoms to syndromes, thereby meeting the needs of progressive analysis in clinical decision-making.

[0005] In a first aspect, this application provides a method for extracting cross-attention multimodal mental representations based on semantic prior guidance, the method comprising: Collect multimodal psychological signal data, which includes text modal data, speech modal data, facial expression modal data, and physiological signal modal data; Construct a semantic prior knowledge base for the psychological domain, embed entities and relations in the semantic prior knowledge base into a knowledge graph, and generate prior feature vectors. The text modal data, the speech modal data, the facial expression modal data, and the physiological signal modal data are each encoded using single-modal feature encoding to generate independent feature representations for each modality; The prior feature vector is injected as an attention bias into the cross-modal cross-attention module. The prior-guided multi-head cross-attention mechanism is used to perform inter-modal feature alignment and fusion on the independent feature representations of each modality to generate multi-modal fused features. The multimodal fusion features are input into a hierarchical mental representation projection head, and a multi-granular mental representation spectrum is output through a multi-level fully connected projection network. The multi-granular mental representation spectrum includes fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features.

[0006] Optionally, the step of constructing a semantic prior knowledge base for the psychological domain, and embedding entities and relations in the semantic prior knowledge base into a knowledge graph to generate prior feature vectors, includes: By integrating diagnostic criteria for mental disorders, norms of psychological assessment scales, knowledge system of cognitive behavioral therapy, and clinical counseling annotation data, a knowledge graph in the field of psychology is constructed. The knowledge graph in the psychological domain includes symptom entities, syndrome entities, emotion entities, intervention strategy entities, and the relationships between entities; The TransE knowledge graph embedding model is used to learn vector representations of entities and relations in the psychological domain knowledge graph, generating entity embedding vectors and relation embedding vectors. Based on the evaluation scenario corresponding to the current multimodal psychological signal, relevant entity subgraphs are retrieved from the psychological domain knowledge graph, and the relevant entity embedding vectors are aggregated to generate the prior feature vector.

[0007] Optionally, the step of performing single-modal feature encoding on the text modal data, the speech modal data, the facial expression modal data, and the physiological signal modal data to generate independent feature representations for each modality includes: The text modal data is semantically encoded using a finely tuned Chinese pre-trained language model in the field of mental health, generating a text modal feature representation. The speech modal data is acoustically encoded using a pre-trained wav2vec2 speech model to generate a speech modal feature representation. The facial expression modality data is spatiotemporally encoded using a lightweight residual convolutional network to generate a facial expression modality feature representation. The physiological signal modal data is feature-encoded using a one-dimensional convolutional temporal network to generate physiological signal modal feature representations. The four types of modal feature representations are then mapped to a feature space of a unified dimension through modality-specific linear projection layers to generate independent feature representations for each modality.

[0008] Optionally, the step of injecting the prior feature vector as an attention bias into the cross-modal cross-attention module, and performing inter-modal feature alignment and fusion on the independent feature representations of each modality through a prior-guided multi-head cross-attention mechanism to generate multi-modal fused features includes: A cross-modal cross-attention module is constructed, which includes a text-speech cross-attention sublayer, a text-expression cross-attention sublayer, a text-physiology cross-attention sublayer, and a speech-expression-physiology trimodal cross-attention sublayer; During the attention weight calculation process of each cross-attention sub-layer, the prior feature vector is transformed linearly to generate an attention bias matrix, and the attention bias matrix is ​​summed with the scaled dot product attention weights; The attention distribution with prior bias is generated using the Softmax activation function; The independent feature representations of each modality are weighted and aggregated based on the attention distribution after adding prior bias to generate cross-modal interactive features; The cross-modal interaction features output from all cross-attention sublayers are residually connected and layer normalized to generate the multimodal fusion features.

[0009] Optionally, the cross-modal cross-attention module employs cross-modal contrastive learning loss for inter-modal spatial alignment constraints during the training phase, including: Construct an intermodal contrastive learning loss function, wherein the contrastive learning loss function uses the features of different modalities under the same psychological state as positive sample pairs and the features of different modalities under different psychological states as negative sample pairs; By contrastive learning loss function, different modal features of the same mental state are kept close to each other in a unified feature space, while modal features of different mental states are kept far apart. In the process of calculating the contrastive learning loss function, sample weights guided by prior knowledge are introduced, and the weight coefficients of positive and negative sample pairs are adjusted according to the strength of the association between entities in the knowledge graph of the psychological domain.

[0010] Optionally, the multimodal fusion features are input into a hierarchical mental representation projection head, and a multi-granularity mental representation spectrum is output through a multi-level fully connected projection network. This multi-granularity mental representation spectrum includes fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features, including: A three-layer cascaded projection structure is constructed: the first layer is a fine-grained emotion projection head, the second layer is a medium-grained symptom projection head, and the third layer is a coarse-grained syndrome projection head. The multimodal fusion features are input into a fine-grained emotion projection head, and fine-grained emotion layer features of a preset dimension are output through a two-layer fully connected network. The fine-grained emotion layer features are concatenated with the multimodal fusion features, input into the medium-grained symptom projection head, and output medium-grained symptom layer features. The features of the medium-grained symptom layer are concatenated with the features of the fine-grained emotion layer and the multimodal fusion features. The result is input into the coarse-grained syndrome projection head and output as coarse-grained syndrome layer features.

[0011] Optionally, it also includes a multi-granularity mental representation consistency verification step: constructing a multi-granularity representation consistency verification matrix based on diagnostic rules in the semantic prior knowledge base of the psychological domain; The logical consistency among the fine-grained emotion layer features, the medium-grained symptom layer features, and the coarse-grained syndrome layer features is verified using a consistency verification matrix. Feature correction is performed on representation dimensions that do not conform to the diagnostic rule logic, and the corrected multi-granularity mental representation spectrum is output.

[0012] Secondly, this application also proposes a cross-attention multimodal mental representation extraction system based on semantic prior guidance, the system comprising: The multimodal data acquisition module is configured to acquire multimodal psychological signal data, which includes text modal data, speech modal data, facial expression modal data, and physiological signal modal data. The semantic prior construction module is configured to construct a semantic prior knowledge base for the psychological domain, and to embed entities and relations in the semantic prior knowledge base for the psychological domain into a knowledge graph to generate prior feature vectors. The single-modal coding module is configured to perform single-modal feature encoding on the text modal data, the speech modal data, the facial expression modal data, and the physiological signal modal data respectively, to generate independent feature representations for each modality; The prior-guided fusion module is configured to inject the prior feature vector as an attention bias into the cross-modal cross-attention module, and perform inter-modal feature alignment and fusion on the independent feature representations of each modality through the prior-guided multi-head cross-attention mechanism to generate multi-modal fusion features; The hierarchical representation output module is configured to input the multimodal fusion features into the hierarchical mental representation projection head, and output a multi-granularity mental representation spectrum through a multi-level fully connected projection network. The multi-granularity mental representation spectrum includes fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features.

[0013] Thirdly, this application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0014] Fourthly, this application also proposes a non-transitory computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0015] Compared with the prior art, this application has the following advantages: This application provides a method and system for extracting multimodal mental representations based on semantic prior guidance and cross-attention. It generates prior feature vectors by constructing a semantic prior knowledge base in the psychological domain, and uses them as attention bias to guide a cross-modal cross-attention mechanism to achieve multimodal feature alignment and fusion. Finally, it outputs a multi-granular mental representation spectrum containing emotion layer, symptom layer and syndrome layer. It effectively solves the representation bias problem caused by lack of professional knowledge in the prior art, and meets the needs of clinical decision-making for a progressive analysis process. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a cross-attention multimodal mental representation extraction method based on semantic prior guidance, provided for an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of the structure of a cross-attention multimodal mental representation extraction system based on semantic prior guidance, provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0020] Firstly, such as Figure 1 As shown, this application provides a method for extracting cross-attention multimodal mental representations based on semantic prior guidance. The method includes: S100. Collect multimodal psychological signal data, including text modal data, speech modal data, facial expression modal data, and physiological signal modal data; Specifically, various sensors and recording devices can be used to acquire users' text, voice, facial expressions, and physiological signals. For example, text data can be obtained through speech-to-text technology or direct text input; voice data can be recorded using a microphone; facial expression data can be captured using a camera; and physiological signal data can be measured using wearable devices or specialized medical equipment (such as electrocardiographs and skin conductance meters). This collected data will serve as the basis for subsequent psychological representation extraction.

[0021] S200. Construct a semantic prior knowledge base for the psychological domain, embed entities and relations in the semantic prior knowledge base for the psychological domain into a knowledge graph, and generate prior feature vectors. The knowledge base can be constructed based on expert experience, existing psychology literature, or publicly available datasets through manual organization and annotation. For example, concepts and their associations from psychology dictionaries and diagnostic manuals can be collected. The knowledge graph embedding process can employ various vectorization techniques, such as matrix factorization-based methods or simple neural network models, to map discrete entities and relationships in the knowledge base into a continuous vector space to capture their semantic information. The resulting prior feature vectors will serve as guiding information for subsequent modality fusion.

[0022] S300: Perform single-modal feature encoding on text modal data, speech modal data, facial expression modal data, and physiological signal modal data respectively to generate independent feature representations for each modality; Specifically, text modal data can be feature-extracted using a bag-of-words model or a simple recurrent neural network; speech modal data can be encoded using traditional acoustic feature extraction methods such as Mel-frequency cepstral coefficients (MFCC) or linear predictive coding (LPC); facial expression modal data can have key points or texture features extracted using image processing techniques; and physiological signal modal data can have statistical features extracted using time-domain or frequency-domain analysis methods. Subsequently, these features from different modalities can be linearly transformed into a unified feature space, forming independent feature representations for each modality.

[0023] S400. The prior feature vector is injected into the cross-modal cross-attention module as an attention bias. The multi-head cross-attention mechanism guided by prior is used to perform inter-modal feature alignment and fusion on the independent feature representations of each modality to generate multi-modal fusion features. Specifically, the cross-modal attention module can consist of multiple attention layers, each responsible for handling interactions between different modalities. During the calculation of attention weights, prior feature vectors can be directly added to the query or key vectors to influence the attention score calculation, thus guiding the model to focus more on cross-modal associations related to prior knowledge during fusion. The multi-head mechanism allows the model to learn intermodal dependencies from different perspectives, enhancing the robustness of fusion. In this way, the independent feature representations of each modality are effectively aligned and fused, forming a multimodal fusion feature containing rich semantic information.

[0024] S500 inputs multimodal fusion features into a hierarchical mental representation projection head, and outputs a multi-granular mental representation spectrum through a multi-level fully connected projection network. The multi-granular mental representation spectrum includes fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features.

[0025] Specifically, the hierarchical mental representation projector can be designed as a series of interconnected fully connected network layers. The first layer projects multimodal fusion features onto a fine-grained emotion space, outputting intensity values ​​of basic emotions such as happiness and sadness. The second layer further processes the output of the first layer with the original multimodal fusion features, projecting it onto a medium-grained symptom space, outputting scores for symptoms such as sleep problems and changes in appetite. The third layer, based on the outputs of the first two layers and the multimodal fusion features, projects onto a coarse-grained syndrome space, outputting scores for dimensions such as depressive syndrome and anxiety syndrome. Through this hierarchical projection method, a comprehensive understanding of psychological states from the micro to the macro level can be obtained.

[0026] In some possible embodiments, a semantic prior knowledge base in the psychological domain is constructed, and knowledge graph embedding is performed on the entities and relations in the semantic prior knowledge base to generate prior feature vectors, including: By integrating diagnostic criteria for mental disorders, norms of psychological assessment scales, knowledge system of cognitive behavioral therapy, and clinical counseling annotation data, a knowledge graph in the field of psychology is constructed. The knowledge graph in the field of psychology includes symptom entities, syndrome entities, emotion entities, intervention strategy entities, and the relationships between entities; The TransE knowledge graph embedding model is used to learn vector representations of entities and relations in the knowledge graph of the psychology domain, generating entity embedding vectors and relation embedding vectors. Based on the evaluation scenario corresponding to the current multimodal psychological signals, relevant entity subgraphs are retrieved from the knowledge graph of the psychological domain, and the embedding vectors of relevant entities are aggregated to generate prior feature vectors.

[0027] Specifically, to construct a comprehensive and authoritative knowledge graph in the field of psychology, this embodiment integrates multi-source heterogeneous professional knowledge. Among these, diagnostic criteria for mental disorders typically refer to internationally recognized diagnostic manuals, such as the Diagnostic and Statistical Manual of Mental Disorders (DSM) or the International Classification of Diseases (ICD), which define, diagnose, and classify various mental disorders. These standards provide the knowledge graph with core syndrome entities and their symptom associations. Psychological assessment scale norms include the structure, items, scoring rules, and normal distribution data of standardized psychological measurement tools (such as depression scales and anxiety scales) in different populations. This data helps define the intensity and assessment dimensions of symptoms. The cognitive behavioral therapy knowledge system provides a theoretical framework, specific techniques, and strategies for psychological intervention, enriching the intervention strategy entities in the knowledge graph and their associations with symptoms and emotions. Clinical consultation annotation corpus consists of text data from actual psychological counseling dialogues, medical records, or expert annotations. By extracting information from these corpora, subtle relationships and contextual information between symptoms, emotions, and syndromes in the real world can be captured. By structuring, identifying entities, extracting relationships, and fusing knowledge from these heterogeneous data sources, a psychological domain knowledge graph is ultimately constructed, containing symptom entities, syndrome entities, emotion entities, intervention strategy entities, and the rich relationships between them. For example, a "depression" syndrome entity may be connected to symptom entities such as "low mood," "insomnia," and "loss of appetite" through an "inclusion" relationship, while the "low mood" entity may be connected to the "sadness" emotion entity through a "manifestation" relationship.

[0028] Building upon this foundation, to transform the symbolic knowledge graph into a computable numerical representation, this embodiment employs the TransE knowledge graph embedding model to learn vector representations of entities and relations in the psychological domain knowledge graph. The TransE model is a classic knowledge graph embedding method. Its core idea is to model relations as a translation operation in the entity embedding space. Specifically, for a fact triple (head entity h, relation r, tail entity t), the model expects the vector representation of head entity h plus the vector representation of relation r to approximately equal the vector representation of tail entity t (h + r ≈ t). By minimizing a distance-based loss function, the model can learn low-dimensional, dense vector representations of each entity and relation—namely, entity embedding vectors and relation embedding vectors. These vectors effectively capture semantic similarity and relational patterns between entities, making semantically related entities closer in the vector space, and entity pairs with the same relation exhibiting consistency in the relation vector direction. Besides TransE, other knowledge graph embedding models, such as TransR, DistMult, or ComplEx, can also be used to adapt to different types of relation complexity and knowledge graph structures.

[0029] As an implementation method, to ensure that the generated prior feature vectors are highly context-relevant, this embodiment retrieves relevant entity subgraphs from the psychological domain knowledge graph based on the assessment scenario corresponding to the current multimodal psychological signal, and aggregates the embedding vectors of relevant entities. The assessment scenario can be determined based on initial user input, preset assessment tasks (such as "depression risk screening" or "anxiety emotion recognition"), or the analysis results of preliminary multimodal signals. Once the assessment scenario is identified, the system starts with the core entity related to the scenario (such as the "depression" syndrome entity) and extracts an entity subgraph most relevant to the current assessment task from the vast knowledge graph using graph traversal algorithms (such as breadth-first search or depth-first search) or graph query language. This subgraph contains entities of symptoms, emotions, syndromes, and intervention strategies directly related to the assessment scenario, as well as their interrelationships. Subsequently, the embedding vectors of all relevant entities in this subgraph are aggregated to generate the final prior feature vector. Aggregation methods can include, but are not limited to, average pooling (averaging all entity embedding vectors), max pooling (maximizing all entity embedding vectors in each dimension), or more complex graph neural network (GNN) models, such as graph convolutional networks (GCN) or graph attention networks (GAT), to encode the subgraph structure and generate a context-aware vector that comprehensively reflects the semantic information of the subgraph.

[0030] In some possible embodiments, single-modal feature encoding is performed on text modal data, speech modal data, facial expression modal data, and physiological signal modal data respectively to generate modal-independent feature representations, including: Semantic encoding of text modal data is performed using a finely tuned Chinese pre-trained language model in the field of mental health to generate text modal feature representations; The speech modal data is encoded using a pre-trained speech model (wav2vec2) to generate speech modal feature representations. A lightweight residual convolutional network is used to encode the spatiotemporal features of facial expression modal data to generate facial expression modal feature representations. Physiological signal modal data are feature-encoded using a one-dimensional convolutional temporal network to generate physiological signal modal feature representations. The four types of modal feature representations are then mapped to a feature space of a unified dimension through modality-specific linear projection layers to generate independent feature representations for each modality.

[0031] Specifically, for text modal data, this embodiment employs a finely tuned Chinese pre-trained language model in the field of mental health for semantic encoding. Based on general language understanding, this model undergoes secondary training or fine-tuning on a large corpus of mental health-related texts (e.g., psychological counseling records, diagnostic criteria for mental illness, psychological research literature, etc.) to better understand and capture deep semantic information such as psychological concepts, emotional expressions, and symptom descriptions contained within the text. For example, mainstream Chinese pre-trained language models such as BERT, RoBERTa, or ERNIE can be selected as the foundation. By adding task-specific output layers and training on labeled mental health text datasets, highly semantic text modal feature representations can be generated. This encoding method effectively avoids the semantic information loss or ambiguity problems that may occur when traditional bag-of-words models or simple embedding models process complex psychological texts.

[0032] For speech modal data, this embodiment uses a pre-trained speech model (wav2vec2) for acoustic feature encoding. The Wav2vec2 model is a self-supervised learning-based speech representation model that can directly learn rich, context-dependent acoustic features from the original audio waveform. This model learns the inherent structure of speech by predicting masked speech segments, thereby capturing acoustic cues closely related to psychological states, such as speech rate, intonation, pitch, volume, and pauses. Inputting the original speech signal into this model extracts high-dimensional, highly discriminative speech modal feature representations. These features not only contain linguistic content but, more importantly, reflect the speaker's emotional state, cognitive load, and potential signs of psychological distress.

[0033] In processing facial expression modality data, this embodiment employs a lightweight residual convolutional network for spatiotemporal feature encoding. Facial expressions are important visual signals conveying emotions and psychological states, and their changes exhibit temporal and spatial continuity. The lightweight residual convolutional network, by introducing residual connections, enables the network to construct deeper feature extraction paths while effectively mitigating the gradient vanishing problem in deep network training. Its "lightweight" design ensures that the model maintains high feature extraction capabilities while possessing lower computational complexity and faster processing speed, making it suitable for real-time or near real-time psychological assessment scenarios. This network can capture subtle facial muscle movements, dynamic changes in expressions, and temporal features of facial key points from video frame sequences, thereby generating a refined representation of facial expression modality features.

[0034] For physiological signal modal data, this embodiment employs a one-dimensional convolutional temporal network for feature encoding. Physiological signals (such as electrocardiograms, skin conductance responses, and electroencephalograms) are objective indicators reflecting an individual's physiological arousal and emotional state, typically represented as one-dimensional time-series data. One-dimensional convolutional temporal networks are particularly suitable for processing this type of data. They can slide across the time dimension using convolutional kernels of different sizes, automatically learning and extracting local patterns, frequency components, waveform features, and temporal dependencies from the signal. For example, they can identify specific patterns in heart rate variability, peak changes in skin conductance, or rhythmic characteristics of brain waves—all important biomarkers for assessing psychological states such as stress, anxiety, and depression. This network can effectively extract stable and biologically meaningful physiological signal modal feature representations from raw physiological signals.

[0035] Finally, to ensure effective fusion of features from different modalities, this embodiment maps the four modal feature representations to a feature space of a unified dimension through modality-specific linear projection layers. This means that each modality's features have an independent linear transformation layer responsible for adjusting its original feature dimensions to a preset unified dimension. This modality-specific projection method allows each modality to maintain its uniqueness during the transformation process while ensuring dimensional consistency across all modal features, providing standardized and compatible input for subsequent cross-modal attention mechanisms.

[0036] In some possible embodiments, prior feature vectors are injected as attention biases into the cross-modal cross-attention module. A prior-guided multi-head cross-attention mechanism is used to align and fuse the independent feature representations of each modality, generating multimodal fused features, including: A cross-modal cross-attention module is constructed, which includes a text-speech cross-attention sublayer, a text-expression cross-attention sublayer, a text-physiology cross-attention sublayer, and a speech-expression-physiology trimodal cross-attention sublayer; During the calculation of attention weights in each cross-attention sub-layer, the prior feature vector is transformed linearly to generate an attention bias matrix, and the attention bias matrix is ​​summed with the scaled dot product attention weights. The attention distribution with prior bias is generated using the Softmax activation function; The independent feature representations of each modality are weighted and aggregated based on the attention distribution after adding prior bias to generate cross-modal interactive features; Residual connections and layer normalization are performed on the cross-modal interaction features output from all cross-attention sublayers to generate multimodal fusion features.

[0037] Specifically, the cross-modal attention module aims to achieve information interaction and fusion between features from different modalities. This module can be implemented using a Transformer-based cross-attention mechanism, where features from one modality serve as queries, and features from the other modality serve as keys and values. Attention weights are obtained by calculating the dot product similarity between the query and the key, followed by Softmax normalization. These attention weights are then weighted and summed with the values, thereby enabling the transfer and fusion of information from one modality to another. To more precisely capture the specific associations between different modal combinations in multimodal psychological signal data, this module is designed to include multiple specialized cross-attention sublayers. For example, the text-speech cross-attention sublayer can be used to capture the synergistic relationship between speech content and speech features such as tone and rate of speech; the text-expression cross-attention sublayer can be used to model the matching or mismatch between text semantics and facial expressions; and the text-physiology cross-attention sublayer can explore the potential connections between text expression and physiological responses such as heart rate and skin conductance. Furthermore, the speech-expression-physiological trimodal cross-attention sublayer can further integrate deep information from nonverbal modalities to capture more complex nonverbal psychological states. These sublayers can operate in parallel, each focusing on a specific modal interaction pattern.

[0038] In standard scaled dot product attention mechanisms, attention weights are typically calculated by taking the dot product of the query vector and the key vector and dividing by a scaling factor. To effectively incorporate prior feature vectors generated from a semantic prior knowledge base in the psychological domain into attention computation, this embodiment proposes using them as a learnable bias. Specifically, the prior feature vectors are first processed through a linear transformation layer (e.g., a fully connected layer) to adjust their dimensions to match the attention weight matrix, thereby generating an attention bias matrix. This linear transformation layer can learn how to map high-dimensional semantic prior information to the bias space required by the attention mechanism. Subsequently, this generated attention bias matrix is ​​directly added to the original scaled dot product attention weights. This summation operation allows prior knowledge to directly influence the attention score of each query-key pair, thus semantically guiding and adjusting the model's focus before calculating the attention distribution.

[0039] After summing the attention bias matrix and the scaled dot product attention weights, the result is processed by the Softmax activation function. The Softmax function transforms any real-valued vector into a probability distribution, ensuring that the sum of all attention weights is 1 and that each weight value is between 0 and 1. By applying Softmax after summing the biases, it is ensured that the prior knowledge adjustment of the attention weights is effective, and the final generated attention distribution is normalized and can be directly used for subsequent weighted aggregation.

[0040] After obtaining the attention distribution with prior bias, this distribution is used as weights to perform a weighted summation of the independent feature representations (usually value vectors) of the corresponding modality. Specifically, each value vector is multiplied by its corresponding attention weight, and then all weighted value vectors are summed to generate a cross-modal interaction feature that integrates information from the query modality and the key / value modality. This interaction feature not only contains the semantic information of the original modality but also strengthens the intermodal correlations related to the current psychological assessment scenario through the guidance of prior knowledge, making the interaction feature more targeted and interpretable.

[0041] To effectively integrate the outputs from different cross-attention sub-layers and ensure the stability and performance of model training, this embodiment employs residual connections and layer normalization techniques. Residual connections, also known as skip connections, allow information to flow directly from input to output, helping to alleviate the vanishing gradient problem in deep networks and making it easier for the network to learn identity mappings, thus preserving original feature information. Specifically, the cross-modal interaction features output by each cross-attention sub-layer can be residually connected to the input features of that sub-layer. Subsequently, layer normalization is applied to the residually connected features. Layer normalization normalizes the feature dimensions of each sample, helping to stabilize the training process, accelerate convergence, and reduce sensitivity to initialization parameters. By integrating the residually connected and layer normalized features output from all cross-attention sub-layers (e.g., through concatenation or summation), a multimodal fusion feature with rich semantics and cross-modal interaction information is finally generated.

[0042] In some possible embodiments, the cross-modal cross-attention module employs cross-modal contrastive learning loss for inter-modal spatial alignment constraints during the training phase, including: Construct an intermodal contrastive learning loss function. The contrastive learning loss function uses the features of different modalities under the same mental state as positive sample pairs and the features of different modalities under different mental states as negative sample pairs. This loss function aims to optimize the feature space by explicitly bringing similar samples closer together and pushing dissimilar samples further apart. Here, similar samples are defined as feature representations of different modalities under the same mental state, while dissimilar samples are feature representations of modalities under different mental states. Its core function is to force the model to learn a shared, semantically consistent feature space, causing multimodal inputs related to the same mental state to converge spatially, while inputs related to different mental states remain separate.

[0043] By contrastive learning loss function, different modal features of the same mental state are kept close to each other in a unified feature space, while modal features of different mental states are kept far apart. This loss function forces the aforementioned objective by minimizing the distance between positive sample pairs (e.g., Euclidean distance or cosine similarity) and maximizing the distance between negative sample pairs. This ensures that features from different modalities are semantically effectively aligned before fusion.

[0044] In the process of calculating the contrastive learning loss function, sample weights guided by prior knowledge are introduced, and the weight coefficients of positive and negative sample pairs are adjusted according to the strength of the association between entities in the knowledge graph of the psychological domain.

[0045] These weights are determined based on the strength of associations between entities in the psychological domain knowledge graph. For example, if the knowledge graph shows a strong psychological correlation between "anxiety" and "fear," the weights of sample pairs involving these two states may be appropriately adjusted when constructing negative sample pairs to reflect their inherent similarity and avoid over-penalization. Specifically, this is achieved by quantifying the strength of associations between entities in the psychological domain knowledge graph into weight coefficients and applying them to the loss contribution of positive and negative sample pairs. This means that for psychological states considered more correlated in the knowledge graph, their corresponding positive and negative sample pairs may have different influences in the loss calculation, thus enabling the feature space learned by the model to not only distinguish different psychological states but also reflect the complex hierarchy and associations between psychological states.

[0046] In some possible embodiments, multimodal fusion features are input into a hierarchical mental representation projection head, and a multi-granularity mental representation spectrum is output through a multi-level fully connected projection network. This multi-granularity mental representation spectrum includes fine-grained emotion layer features, medium-granularity symptom layer features, and coarse-grained syndrome layer features, including: A three-layer cascaded projection structure is constructed: the first layer is a fine-grained emotion projection head, the second layer is a medium-grained symptom projection head, and the third layer is a coarse-grained syndrome projection head. The multimodal fusion features are input into the fine-grained emotion projection head, and the fine-grained emotion layer features of a preset dimension are output through a two-layer fully connected network. The fine-grained emotion layer features include the intensity values ​​of eight basic emotions: happiness, sadness, anger, fear, disgust, surprise, anxiety, and calmness. Fine-grained emotion layer features are concatenated with multimodal fusion features, input to a medium-grained symptom projection head, and output medium-grained symptom layer features, which include scores for symptom dimensions such as sleep problems, changes in appetite, decreased attention, and loss of interest. The medium-grained symptom layer features are concatenated with the fine-grained emotion layer features and multimodal fusion features. The input is a coarse-grained syndrome projection head, and the output is a coarse-grained syndrome layer feature. The coarse-grained syndrome layer feature includes scores for dimensions such as depression syndrome, anxiety syndrome, obsessive-compulsive syndrome, and stress-related syndrome.

[0047] This three-tiered cascaded projection structure aims to simulate the hierarchical reasoning process in psychological assessment and diagnosis, progressing from basic emotions to specific symptoms and finally to a syndrome. Through this structure, the system can progressively refine and abstract the representation of psychological states. The first-tier, fine-grained emotion projection head captures the most basic and direct emotional responses; the second-tier, medium-grained symptom projection head, building upon this, incorporates more comprehensive information to identify specific psychological symptoms; and the third-tier, coarse-grained syndrome projection head further integrates all information to form a judgment of the overall psychological syndrome. This cascaded design ensures the logical connection between different granular features and the effectiveness of information transmission.

[0048] The fine-grained emotion projector is the first step in hierarchical mental representation extraction, receiving multimodal fusion features after intermodal alignment and fusion. Internally, this projector employs a two-layer fully connected network to map high-dimensional fusion features to a pre-defined, low-dimensional emotion feature space. These two fully connected layers learn and extract discriminative information related to basic emotions from the multimodal fusion features. The output fine-grained emotion layer features are specifically represented by intensity values ​​for eight basic emotions (happiness, sadness, anger, fear, disgust, surprise, anxiety, and calmness). These intensity values ​​are typically floating-point numbers between 0 and 1, representing the degree to which an individual experiences each emotion at the current moment. This design enables the system to quantify and distinguish subtle emotional fluctuations in individuals.

[0049] The medium-granularity symptom projector is responsible for capturing psychological symptoms at a more macroscopic level. Its input is not merely multimodal fusion features, but rather a concatenation of extracted fine-grained emotion layer features with the original multimodal fusion features. This concatenation operation allows the medium-granularity symptom projector to consider not only the original multimodal information but also fully utilize quantified emotional information when identifying symptoms. In this way, the system can learn the potential association between emotions and symptoms; for example, persistent sadness may be associated with loss of interest. The output medium-granularity symptom layer features are represented by scores across a series of symptom dimensions, such as sleep problems, changes in appetite, decreased attention, and loss of interest. These scores reflect the severity of an individual's symptoms across each dimension.

[0050] The coarse-grained syndrome projector represents the highest level of hierarchical mental representation extraction, designed to identify and quantify psychological syndromes. Its input is a richer set of information, including mid-grained symptom layer features, fine-grained emotion layer features, and a concatenation of original multimodal fusion features. This integration of multi-level information allows the coarse-grained syndrome projector to comprehensively consider emotions, symptoms, and original multimodal signals, resulting in a more accurate diagnosis of syndromes. For example, combining sadness, loss of interest symptoms, and abnormal physiological signals can more reliably determine the presence of depressive syndromes. The output coarse-grained syndrome layer features are represented by scores for dimensions such as depressive syndromes, anxiety syndromes, obsessive-compulsive syndromes, and stress-related syndromes; these scores can be directly used to assist in clinical diagnosis and assessment.

[0051] In some possible embodiments, a multi-granularity mental representation consistency verification step is also included: constructing a multi-granularity representation consistency verification matrix based on diagnostic rules in the semantic prior knowledge base of the mental domain; The logical consistency among the fine-grained emotion layer features, the medium-grained symptom layer features, and the coarse-grained syndrome layer features is verified using a consistency verification matrix. Feature correction is performed on representation dimensions that do not conform to the diagnostic rule logic, and the corrected multi-granularity mental representation spectrum is output.

[0052] This multi-granularity mental representation consistency verification step aims to ensure that the generated multi-granularity mental representation spectrum maintains logical coherence and clinical rationality across different granularities. Its core lies in incorporating domain expert knowledge to calibrate the original features output by the model, thereby improving the reliability of the evaluation results.

[0053] The diagnostic rules in the semantic prior knowledge base of the psychological domain are derived from authoritative diagnostic criteria in psychology and psychiatry (e.g., DSM-5, ICD-11), clinical practice guidelines, manuals for interpreting psychological assessment scales, and expert consensus. They are stored in a structured form within the semantic prior knowledge base of the psychological domain; for example, they can be expressed as "If there is persistent loss of interest and sleep disturbance, the likelihood of depressive syndrome increases" or "If the intensity of sadness is high and the duration is long, it may be accompanied by symptoms such as changes in appetite." These rules provide explicit constraints on the logical relationships between psychological representations of different granularities.

[0054] The construction of a multi-granularity representation consistency verification matrix is ​​based on diagnostic rules from a semantic prior knowledge base in the psychological domain. It is used to quantify or encode the strength of associations and logical dependencies between psychological representations of different granularities (such as emotions, symptoms, and syndromes). For example, an element in the matrix can represent the strength of the positive correlation between "sadness" and "depressive syndrome," or the association weight between "sleep problems" and "anxiety syndrome." This matrix can be a binary matrix indicating the existence of an association; it can be a weighted matrix representing the strength or probability of the association; or it can be a set of rule-based logical expressions. Its construction process may involve encoding expert knowledge, statistical analysis, or learning from large amounts of clinical data using machine learning methods.

[0055] The logical consistency among fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features is verified using a consistency verification matrix. This verification process involves comparing the original multi-granularity psychorepresentation spectrum output by the model with the pre-defined consistency verification matrix. Specifically, based on the rules or weights defined in the verification matrix, it can be assessed whether there are contradictions or situations that do not conform to clinical logic among the current fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features. For example, if the verification matrix indicates that "high anxiety is often accompanied by sleep problems," but the current output shows a high anxiety syndrome score and a low sleep problem score, then an inconsistency is considered to exist. Verification methods can include threshold-based rule matching, calculating consistency scores, or using optimization algorithms to find the feature combination that best conforms to the constraints of the verification matrix.

[0056] Feature correction is performed on representation dimensions that do not conform to the logic of diagnostic rules. When inconsistencies are detected, the corresponding representation dimensions need to be adjusted. Correction methods may include: gaining or attenuating feature values ​​based on weights or rules in the verification matrix; using iterative optimization algorithms to minimize the deviation from the original features while satisfying consistency constraints; or introducing regularization terms to encourage the generation of more consistent features during model training. The goal of correction is to make the adjusted multi-granularity mental representation spectrum more consistent with the logic of clinical diagnosis, while preserving as much predictive information as possible from the original model.

[0057] The system outputs a revised multi-granularity mental representation spectrum. After consistency verification and feature correction, the system generates a final, calibrated multi-granularity mental representation spectrum. This revised spectrum maintains multimodal fusion information while ensuring logical coherence between features of different granularities, thus providing a more reliable and clinically meaningful basis for mental health assessment and intervention.

[0058] It should be noted that the above description describes some embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0059] Secondly, based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also proposes a cross-attention multimodal mental representation extraction system based on semantic prior guidance, such as... Figure 2 As shown: The multimodal data acquisition module 100 is configured to acquire multimodal psychological signal data, which includes text modal data, speech modal data, facial expression modal data, and physiological signal modal data. The semantic prior construction module 200 is configured to construct a semantic prior knowledge base for the psychological domain, embed entities and relations in the semantic prior knowledge base for the psychological domain into a knowledge graph, and generate prior feature vectors. The single-modal coding module 300 is configured to perform single-modal feature encoding on text modal data, speech modal data, facial expression modal data and physiological signal modal data respectively, and generate independent feature representations for each modality; The prior-guided fusion module 400 is configured to inject the prior feature vector as an attention bias into the cross-modal cross-attention module, and perform inter-modal feature alignment and fusion on the independent feature representations of each modality through the prior-guided multi-head cross-attention mechanism to generate multi-modal fusion features; The hierarchical representation output module 500 is configured to input multimodal fusion features into the hierarchical mental representation projection head and output a multi-granular mental representation spectrum through a multi-level fully connected projection network. The multi-granular mental representation spectrum includes fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features.

[0060] The system described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0061] Thirdly, based on the same inventive concept, embodiments of this application also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0062] Fourthly, based on the same inventive concept, embodiments of this application also provide a non-transitory computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0063] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for extracting cross-attention multimodal mental representations based on semantic prior guidance, characterized in that, The method includes: Collect multimodal psychological signal data, which includes text modal data, speech modal data, facial expression modal data, and physiological signal modal data; A semantic prior knowledge base for the psychological domain is constructed. Entities and relationships within this prior knowledge base are embedded into a knowledge graph to generate prior feature vectors. This process includes: integrating diagnostic criteria for mental disorders, norms of psychological assessment scales, a knowledge system of cognitive behavioral therapy, and clinical consultation annotation corpora to construct a psychological domain knowledge graph. This knowledge graph includes symptom entities, syndrome entities, emotion entities, intervention strategy entities, and the relationships between entities. Vector representation learning is performed on the entities and relationships within the psychological domain knowledge graph using the TransE knowledge graph embedding model to generate entity embedding vectors and relationship embedding vectors. Based on the assessment scenario corresponding to the current multimodal psychological signal, relevant entity subgraphs are retrieved from the psychological domain knowledge graph, and the relevant entity embedding vectors are aggregated to generate the prior feature vectors. The text modal data, the speech modal data, the facial expression modal data, and the physiological signal modal data are each encoded using single-modal feature encoding to generate independent feature representations for each modality; The prior feature vector is injected as an attention bias into the cross-modal cross-attention module. A prior-guided multi-head cross-attention mechanism is used to align and fuse the modal-independent feature representations, generating multimodal fusion features. This includes: constructing a cross-modal cross-attention module comprising text-speech cross-attention sublayers, text-expression cross-attention sublayers, text-physiology cross-attention sublayers, and speech-expression-physiology trimodal cross-attention sublayers; during the attention weight calculation of each cross-attention sublayer, the prior feature vector is linearly transformed to generate an attention bias matrix, and the attention bias matrix is ​​summed with the scaled dot product attention weights; an attention distribution with prior bias is generated using the Softmax activation function; the modal-independent feature representations are weighted and aggregated according to the attention distribution with prior bias to generate cross-modal interaction features; and the cross-modal interaction features output from all cross-attention sublayers are residually connected and layer normalized to generate the multimodal fusion features. The multimodal fusion features are input into a hierarchical mental representation projection head, and a multi-granular mental representation spectrum is output through a multi-level fully connected projection network. The multi-granular mental representation spectrum includes fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features. This involves: constructing a three-layer cascaded projection structure: the first layer is a fine-grained emotion projection head, the second layer is a medium-grained symptom projection head, and the third layer is a coarse-grained syndrome projection head; inputting the multimodal fusion features into the fine-grained emotion projection head, and outputting fine-grained emotion layer features of a preset dimension through a two-layer fully connected network; concatenating the fine-grained emotion layer features with the multimodal fusion features, inputting them into the medium-grained symptom projection head, and outputting medium-grained symptom layer features; concatenating the medium-grained symptom layer features with the fine-grained emotion layer features and the multimodal fusion features, inputting them into the coarse-grained syndrome projection head, and outputting coarse-grained syndrome layer features.

2. The method according to claim 1, characterized in that, The step of performing single-modal feature encoding on the text modal data, the speech modal data, the facial expression modal data, and the physiological signal modal data to generate independent feature representations for each modality includes: The text modal data is semantically encoded using a finely tuned Chinese pre-trained language model in the field of mental health, generating a text modal feature representation. The speech modal data is acoustically encoded using a pre-trained wav2vec2 speech model to generate a speech modal feature representation. The facial expression modality data is spatiotemporally encoded using a lightweight residual convolutional network to generate a facial expression modality feature representation. The physiological signal modal data is feature-encoded using a one-dimensional convolutional temporal network to generate physiological signal modal feature representations. The four types of modal feature representations are then mapped to a feature space of a unified dimension through modality-specific linear projection layers to generate independent feature representations for each modality.

3. The method according to claim 1, characterized in that, The cross-modal cross-attention module employs cross-modal contrastive learning loss during the training phase for inter-modal spatial alignment constraints, including: Construct an intermodal contrastive learning loss function, wherein the contrastive learning loss function uses the features of different modalities under the same psychological state as positive sample pairs and the features of different modalities under different psychological states as negative sample pairs; By contrastive learning loss function, different modal features of the same mental state are kept close to each other in a unified feature space, while modal features of different mental states are kept far apart. In the process of calculating the contrastive learning loss function, sample weights guided by prior knowledge are introduced, and the weight coefficients of positive and negative sample pairs are adjusted according to the strength of the association between entities in the knowledge graph of the psychological domain.

4. The method according to claim 1, characterized in that, It also includes a multi-granularity mental representation consistency verification step: constructing a multi-granularity representation consistency verification matrix based on diagnostic rules in the semantic prior knowledge base of the psychological domain; The logical consistency among the fine-grained emotion layer features, the medium-grained symptom layer features, and the coarse-grained syndrome layer features is verified using a consistency verification matrix. Feature correction is performed on representation dimensions that do not conform to the diagnostic rule logic, and the corrected multi-granularity mental representation spectrum is output.

5. A cross-attention multimodal mental representation extraction system based on semantic prior guidance, characterized in that, The system includes: The multimodal data acquisition module is configured to acquire multimodal psychological signal data, which includes text modal data, speech modal data, facial expression modal data, and physiological signal modal data. The semantic prior construction module is configured to build a semantic prior knowledge base for the psychological domain. It embeds entities and relationships in the semantic prior knowledge base into a knowledge graph to generate prior feature vectors. This includes: integrating diagnostic criteria for mental disorders, norms of psychological assessment scales, the knowledge system of cognitive behavioral therapy, and clinical consultation annotation corpora to construct a psychological domain knowledge graph; the psychological domain knowledge graph contains symptom entities, syndrome entities, emotion entities, intervention strategy entities, and the relationships between entities; it learns vector representations of entities and relationships in the psychological domain knowledge graph using the TransE knowledge graph embedding model to generate entity embedding vectors and relationship embedding vectors; based on the assessment scenario corresponding to the current multimodal psychological signal, it retrieves relevant entity subgraphs from the psychological domain knowledge graph, aggregates the relevant entity embedding vectors, and generates the prior feature vectors. The single-modal coding module is configured to perform single-modal feature encoding on the text modal data, the speech modal data, the facial expression modal data, and the physiological signal modal data respectively, to generate independent feature representations for each modality; The prior-guided fusion module is configured to inject the prior feature vector as an attention bias into the cross-modal cross-attention module. It performs inter-modal feature alignment and fusion on the independent feature representations of each modality through a prior-guided multi-head cross-attention mechanism to generate multimodal fusion features. This includes: constructing a cross-modal cross-attention module comprising text-speech cross-attention sub-layers, text-expression cross-attention sub-layers, text-physiology cross-attention sub-layers, and speech-expression-physiology trimodal cross-attention sub-layers; during the attention weight calculation process of each cross-attention sub-layer, the prior feature vector is linearly transformed to generate an attention bias matrix, and the attention bias matrix is ​​summed with the scaled dot product attention weights; an attention distribution with prior bias is generated using a Softmax activation function; the independent feature representations of each modality are weighted and aggregated according to the attention distribution with prior bias to generate cross-modal interaction features; and the cross-modal interaction features output from all cross-attention sub-layers are residually connected and layer normalized to generate the multimodal fusion features. The hierarchical representation output module is configured to input the multimodal fusion features into a hierarchical mental representation projection head, and output a multi-granular mental representation spectrum through a multi-level fully connected projection network. The multi-granular mental representation spectrum includes fine-grained emotion layer features, medium-grained symptom layer features, and coarse-grained syndrome layer features. This includes: constructing a three-layer cascaded projection structure, with the first layer being a fine-grained emotion projection head, the second layer a medium-grained symptom projection head, and the third layer a coarse-grained syndrome projection head; inputting the multimodal fusion features into the fine-grained emotion projection head, and outputting fine-grained emotion layer features of a preset dimension through a two-layer fully connected network; concatenating the fine-grained emotion layer features with the multimodal fusion features, inputting the concatenation into the medium-grained symptom projection head, and outputting medium-grained symptom layer features; and concatenating the medium-grained symptom layer features with the fine-grained emotion layer features and the multimodal fusion features, inputting the concatenation into the coarse-grained syndrome projection head, and outputting coarse-grained syndrome layer features.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images