Multimodal psychological intention understanding method based on zero sample and intention perception alignment

Abstract

This invention discloses a multimodal psychological intent understanding method based on zero-shot and intent-aware alignment, relating to the technical field of mental health monitoring and intelligent dialogue systems. The method includes the following steps: Step 1: Collecting time-synchronized dialogue data containing text, audio, and video to form multimodal data; Step 2: Constructing a multi-objective fusion model by combining time-aligned multimodal data with a late-stage fusion strategy to obtain a multi-objective joint optimization function; Step 3: Using an adaptive inference strategy, dynamically allocating the multimodal data to obtain predicted intent classification results. This invention features the advantages of being able to start and run without relying on large-scale manually labeled data, fundamentally solving the bottlenecks of high data labeling costs and difficulty in obtaining data in the field of psychological dialogue, achieving accurate fusion of cross-modal information through time alignment, and significantly improving the accuracy of analysis.

Description

Technical Field

[0001] This invention relates to the field of mental health monitoring and intelligent dialogue system technology, and more specifically, to a multimodal psychological intent understanding method based on zero-shot and intent perception alignment. Background Technology

[0002] With the deep integration of artificial intelligence (AI) technology into the fields of mental health and clinical psychology, the automated and intelligent analysis of interpersonal dialogues has entered a new stage. Compared to traditional questionnaires or single interviews, integrating natural language processing, speech emotion computing, and computer vision can yield more continuous, objective, and richer multimodal behavioral data. However, this brings unprecedented challenges to data processing and clinical interpretation capabilities. Because high-quality clinically labeled data is extremely scarce and costly to acquire, utilizing advanced AI technologies to improve data utilization efficiency and optimize analytical performance has become a core issue for the industry's development.

[0003] Psychological dialogue intent understanding technology is an intelligent analysis technology. Common analysis methods include text-based sentiment analysis, speech-based emotion recognition, and micro-expression-based behavior coding. Multimodal psychological intent analysis integrates these technologies into dialogue records composed of text, audio, and video streams. Through information fusion, the analysis system can more comprehensively understand the psychological state of the speakers.

[0004] A typical scenario is that clinical research or auxiliary diagnostic systems use automated technology to analyze psychological interview recordings. While this effectively solves the problems of low efficiency and strong subjectivity of manual coding, a major problem is the difficulty in fragmenting and aligning multimodal information. This is unavoidable and urgently needs to be addressed in complex real-world scenarios. Moreover, due to the asynchronous nature of psychological expression and the presence of noise and obstruction in the recording environment, the interference or contradiction between different modal features will increase as the number of analysis dimensions increases, seriously affecting the reliability of the analysis.

[0005] Multimodal fusion technology is a commonly used analytical method. Different fusion strategies (such as early fusion and late fusion) can be combined in different ways to form various performance models. Due to the complexity of psychological intent and the sensitivity of privacy data, a large amount of high-quality labeled data is required to train a high-precision model. Moreover, when the analysis task is expanded to new intent types, the model often needs to be re-labeled and retrained, and the resulting time and money costs are a significant factor.

[0006] Currently, related technical solutions in the field of multimodal psychoanalysis only focus on using more complex neural network architectures or fusion modules to complete classification tasks, and improving model performance by enhancing single-modal feature extraction algorithms to improve the reliability of results. However, the semantic context and emotional expression of psychological dialogues change rapidly, so models are required to have stronger contextual understanding and cross-modal reasoning capabilities. As a result, the number of model parameters and complexity increase. In addition, many advanced models require processing centers with powerful GPU computing power, but many medical institutions currently have limited local deployment conditions and insufficient computing resources, which can lead to analysis delays or impracticality.

[0007] In the context of psychological dialogue analysis and mental health assessment, auxiliary diagnostic systems and clinical experts achieve a deep understanding of patients' psychological states by fusing multimodal information. Common analysis models include unimodal and multimodal fusion-based methods. When using a pure text analysis model, the system cannot simultaneously utilize paralinguistic information in speech and nonlinguistic information in vision. In this model, to infer emotions and intentions from text, the system needs to rely on extremely precise and rich semantic understanding. If the patient's expression is implicit, contradictory, or contains culturally specific metaphors, the system can capture very few effective clues, making it difficult to identify key intentions such as suicide risk. In a simple early multimodal fusion model, the system is allowed to process signals from multiple modalities simultaneously, but it requires that the features of each modality be strictly aligned in time and have stable quality. When faced with common real-world scenarios such as missing signals, noise interference, or cross-modal contradictions, the analysis cannot be performed reliably. Furthermore, when dealing with new clinical intention recognition needs with scarce labeled data, such as identifying a specific post-traumatic response or adapting to cultural differences in different regions where large-scale labeled data is unavailable, traditional models often cannot be accurately and effectively adjusted and deployed.

[0008] There are currently no systematic solutions to these problems using existing technologies. Summary of the Invention

[0009] The purpose of this invention is to design and develop a multimodal psychological intent understanding method based on zero-shot and intent-aware alignment. Through zero-shot knowledge distillation and intent-aware alignment mechanism, it achieves psychological intent understanding without labeling. Combined with late fusion technology, it realizes the collaboration of multimodal data and achieves accurate identification of complex psychological intent.

[0010] The technical solution provided by this invention is as follows:

[0011] A multimodal psychological intent understanding method based on zero-shot and intent-aware alignment includes the following steps:

[0012] Step 1: Collect time-synchronized dialogue data containing text, audio, and video to form multimodal data;

[0013] Step 2: Construct a multi-objective fusion model. By combining time-aligned multimodal data with a late-stage fusion strategy, a multi-objective joint optimization function is obtained.

[0014] Step 3: Using an adaptive reasoning strategy, multimodal data is dynamically allocated to obtain the predicted intent classification results.

[0015] Preferably, step two specifically includes the following steps:

[0016] Step 1: Preprocess the text data to obtain a preliminary intent probability distribution and confidence score;

[0017] Step 2: Extract features from the multimodal data, and perform temporal alignment on the extracted multimodal data based on the text data. Combine this with an intent-aware attention mechanism to obtain the modality alignment loss function.

[0018] Step 3: After encoding the audio feature vector and video feature vector, perform late-stage fusion to obtain the final intent probability distribution;

[0019] Step 4: Calculate the multi-objective joint optimization function using the classification loss function, modality alignment loss function, regularization loss function, and consistency loss function;

[0020] Step 5: Train the multi-objective fusion model until the verification loss no longer decreases for 20 consecutive rounds. Once training is complete, select the intent with the highest probability from the final probability intent distribution as the predicted intent classification result.

[0021] Preferably, the preliminary intent probability distribution is obtained by sequentially passing the text fragment through a pre-trained inference model, a projection layer, and a normalization layer.

[0022] Preferably, the pre-trained inference model is The encoder, wherein the projection layer is a linear transformation function layer.

[0023] Preferably, the intent-aware attention mechanism satisfies:

[0024] ;

[0025] In the formula, To perceive aligned features, For querying the matrix, The key matrix, For value matrices, It is a time sequence mask matrix. Indicates the dimension of the key.

[0026] Preferably, the query matrix, key matrix, and value matrix satisfy:

[0027] ;

[0028] ;

[0029] ;

[0030] In the formula, For the first Text feature vectors extracted from text fragments, This is the matrix resulting from concatenating the audio and video feature vectors. To query the projection matrix, Let be the key projection matrix.

[0031] Preferably, both the audio feature vector and the video feature vector are obtained through a multilayer perceptron.

[0032] Preferably, during the multi-objective fusion model training process in step 5, preliminary intent understanding pseudo-labels are generated by labeling the training set data, and training is performed using the preliminary intent understanding pseudo-labels.

[0033] The preliminary intent understanding pseudo-tag satisfies:

[0034] ;

[0035] In the formula, It is the first Preliminary intent pseudo-labels for a text fragment, For the first Intent-based prompt template, For the first in the text sequence A text fragment, Used to calculate the semantic matching degree between text and template.

[0036] Preferably, the dynamic calculation allocation specifically includes:

[0037] When the confidence score of the complete model is lower than the confidence threshold during the initial inference or when the confidence score of the text data is lower than the confidence threshold, the multi-objective fusion model is enabled to predict the intent classification results.

[0038] When the confidence score of the text data reaches or exceeds the confidence threshold, the zero-sample text inference results are directly adopted.

[0039] Preferably, it also includes:

[0040] Step 4: Output interpretability results for the predicted intent classification results, key evidence fragments, modality contribution analysis graphs, and time series evolution reports.

[0041] The beneficial effects of this invention are as follows:

[0042] (1) The present invention designs and develops a multimodal psychological intent understanding method based on zero-sample and intent perception alignment. It uses zero-sample knowledge distillation technology to bypass the dependence on labeled data and adaptively integrates multimodal evidence through the dynamic fusion mechanism of intent perception, reducing misjudgments caused by signal quality problems or modal conflicts. Moreover, the system has extremely low startup and expansion costs, and its clinical practicality and deployment agility have significant advantages over traditional strategies that rely on fixed data and rules. At the same time, this data-driven and knowledge-driven mode has strong scenario generalization ability and is less affected by the completeness of data for specific populations.

[0043] (2) The multimodal psychological intent understanding method based on zero-sample and intent perception alignment designed and developed in this invention uses multimodal alignment and fusion technology to associate discrete text, speech and visual features with semantic intent through timestamps. By controlling parameters such as the weight of each modality and the fusion method, the confidence of the analysis system in the main signal dimension is maximized, while the impact of noise interference is minimized. This achieves a highly robust and accurate intent understanding, which not only reduces the risk of misjudgment, but also overcomes the analysis bias caused by the one-sidedness of single-modal information.

[0044] (3) The present invention designs and develops a multimodal psychological intent understanding method based on zero-sample and intent perception alignment, which simultaneously satisfies the three goals of high accuracy, strong generalization and low data dependence. It realizes a clinically practical analysis system with low annotation cost, which is a novel technical paradigm that breaks through the data bottleneck. It can also ensure the balance between accuracy and efficiency, and greatly improve the practicality and deployment flexibility of the system through the adaptive reasoning mechanism of dynamic calculation allocation. Attached Figure Description

[0045] Figure 1 This is a schematic diagram illustrating the training process and multimodal data processing of the multi-objective fusion model described in this invention.

[0046] Figure 2 This is a schematic diagram of the late-stage fusion process of the multi-objective fusion model described in this invention;

[0047] Figure 3 This is a schematic diagram of the confusion matrix described in this invention. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0049] The present invention provides a multimodal psychological intent understanding method based on zero-shot and intent-aware alignment, comprising the following steps:

[0050] Step 1: Collect time-synchronized dialogue data containing text, audio, and video, i.e., multimodal data;

[0051] Among them, text data, audio data, and video data are all independent datasets;

[0052] Step 2: Construct a multi-objective fusion model. By combining time-aligned multimodal data with a late-stage fusion strategy, a multi-objective joint optimization function is obtained. This includes the following steps:

[0053] like Figure 1 As shown, step 1 involves preprocessing the text data:

[0054] Construct a pre-trained inference model for the text data to generate a preliminary intent probability distribution and confidence score, specifically including:

[0055] For the input text data, the pre-trained inference model obtains its semantic representation vector through the BART encoder. It is then mapped to the intent category space through a learnable projection layer (linear transformation), and the initial intent category recognition is obtained through a normalization layer:

[0056] ;

[0057] In the formula, For the first in the text sequence A text fragment, For the first Each intent category, and , For the number of intent categories, For the first The text fragment belongs to the first The probability of each intent category For learnable projection matrices, For bias terms, , , The dimension of the hidden layer in the BART encoder;

[0058] Therefore, the confidence score is calculated:

[0059] ;

[0060] In the formula, For the first Confidence score of each text segment Let be the information entropy, used to measure the uncertainty of a probability distribution, and satisfy the following:

[0061] ;

[0062] Step 2: Extract features from the multimodal data, and perform temporal alignment on the extracted multimodal data based on the text data. Combined with the intent-aware attention mechanism, obtain the modality alignment loss function:

[0063] Based on the timestamps of the text data, aligned audio and video clips are extracted from the original data, and specialized tools are used to extract features from the text, audio, and video data.

[0064] ;

[0065] ;

[0066] ;

[0067] In the formula, For the first Text feature vectors extracted from text fragments, , For the first The original audio data corresponding to each text segment For timestamp boundaries based on text fragments, The audio feature vector extracted from the original audio data. For the first The original video data corresponding to each text segment The video feature vector extracted from the original video data;

[0068] at this time, The dimension is 768. The dimension is 73. The dimension is 20;

[0069] Therefore, the intention-perception attention mechanism satisfies:

[0070] ;

[0071] In the formula, To perceive aligned features, For querying the matrix, The key matrix, For value matrices, , , These are all multimodal temporal features; This is a temporal mask matrix that only allows points with similar timings to pay attention to each other. Indicates the dimension of the key to prevent gradient explosion;

[0072] The query matrix and key matrix satisfy:

[0073] ;

[0074] ;

[0075] In the formula, The matrix is ​​the concatenation of the audio feature vector and the video feature vector, i.e.:

[0076] ;

[0077] The value matrix satisfies:

[0078] ;

[0079] The values ​​of the timing mask matrix satisfy:

[0080] ;

[0081] In the formula, , This represents the timestamps at the m-th and n-th time points. For attention time constraint window, =0.5s;

[0082] Construct the timing alignment objective function:

[0083] ;

[0084] In the formula, , To balance the hyperparameters, audio and video alignment weights are set, with values ​​ranging from [0.1, 1.0]. KL divergence is used to measure the difference in distribution. , , The temporal attention distributions for text, audio, and video are respectively, and satisfy the following:

[0085] ;

[0086] In the formula, , , Let be the temporal attention intensity vectors for text, audio, and video, and satisfy the following:

[0087] ;

[0088] ;

[0089] ;

[0090] In the formula, It is the query sequence length. This is a matrix slicing operation that extracts a range of columns from the j-th row. , , These represent the number of tokens in the text, the number of audio frames, and the number of video frames, respectively. A token is the smallest unit in the text, i.e., a word. Let be the attention matrix, and satisfy:

[0091] ;

[0092] Step 3, as follows Figure 2 As shown, after encoding the audio feature vector and video feature vector, late-stage fusion is performed to obtain the final intent probability distribution, specifically:

[0093] The MLP architecture is used for encoding audio and video feature vectors.

[0094] ;

[0095] ;

[0096] In the formula, It is the encoded audio feature vector. It is an audio feature vector. and It is the first layer weights and biases (73→128) in the audio MLP architecture. and It is the second layer of weights and biases (128→64) in the audio MLP architecture. It is the encoded video feature vector. It is a video feature vector. and These are the weights and biases of the first layer in the video MLP architecture (20→64). and It is the second layer weights and biases in the video MLP architecture (64→32).

[0097] Late-stage fusion involves concatenating text feature vectors, encoded audio feature vectors, and encoded video feature vectors.

[0098] ;

[0099] Thus, the probability distribution of the final intent is obtained:

[0100] ;

[0101] In the formula, This represents the probability distribution of the final intention. and These are the hidden layer parameters (864→256). and These are the classification layer parameters (256→K).

[0102] Step 4: Calculate the multi-objective joint optimization function using the classification loss function, modality alignment loss function, regularization loss function, and consistency loss function. The first objective is classification accuracy, which minimizes the difference between the predicted result and the high-confidence pseudo-label. The second objective is modality consistency, which minimizes the semantic distance and prediction discrepancies between different modal features. Finally, the third objective is model efficiency, which is achieved by using regularization to constrain model complexity and solving the gradient optimization algorithm to obtain the optimal model parameters. Specifically:

[0103] The classification loss function satisfies:

[0104] ;

[0105] In the formula, This represents the number of training samples;

[0106] By calculating the square of the Euclidean distance, text features and audio / video features are brought closer together in the semantic space, thus the modality alignment loss function satisfies:

[0107] ;

[0108] By calculating the square of the Frobenius norm, overfitting to potentially noisy pseudo-labels is prevented, thus the regularization loss function satisfies:

[0109] ;

[0110] The consistency loss function satisfies:

[0111] ;

[0112] In the formula, Let be the Jensen-Shannon divergence, and satisfy:

[0113] ;

[0114] In the formula, It is the final predicted distribution of the multimodal fusion model, i.e. ; It is the distribution of intent prediction inferred through a zero-shot text model, i.e. M is P and The intermediate distribution, i.e. ;

[0115] The multi-objective joint optimization function satisfies:

[0116] ;

[0117] In the formula, [0.05, 0.5] are the alignment loss weights. [0.001, 0.1] is the regularization weight. [0.01, 0.2] is the consistency weight;

[0118] Step 5: Train the multi-objective fusion model until the loss no longer decreases for 20 consecutive rounds, at which point training is complete;

[0119] In the training process of the multi-objective fusion model, preliminary intention understanding pseudo-labels are generated by labeling the training set data, providing initial supervision signals for the system.

[0120] ;

[0121] In the formula, It is the first Pseudo-labels for a text fragment (preliminary intent assessment) This serves as a prompt template for the k-th type of intent (e.g., the "sleep distress" template: "The patient exhibits symptoms such as difficulty sleeping and insomnia"). Used to calculate the semantic matching degree between text and template;

[0122] That is, during the model training process, by... Substitute To train the model;

[0123] After the model is trained, the final probability intent distribution is obtained by minimizing the multi-objective joint optimization function. The intent with the highest probability is selected as the final intent, which is the predicted intent classification result.

[0124] Step 3: To achieve the optimal balance between efficiency and accuracy, the system employs an adaptive inference strategy during the deployment phase, dynamically allocating multimodal data to obtain the predicted intent classification results.

[0125] ;

[0126] In the formula, For time The calculation of budget allocation, To perform complete multimodal alignment and fusion, To directly use the results of zero-shot text inference, It is the confidence threshold, and ;

[0127] In other words, when the confidence level of the full model is low during the initial inference or when the zero-sample module judges the confidence level (below the threshold τ), the full model path, which includes all alignment and fusion calculations, is used to ensure both accuracy and efficiency. When the zero-sample text inference itself has a high confidence level (reaching the threshold or above), the time-consuming multimodal processing is skipped, and the text intent judgment result obtained from the zero-sample pseudo-label generation stage is used directly, which significantly reduces inference latency and is suitable for scenarios with high real-time requirements.

[0128] For continuous dialogue stream analysis, to avoid jitter in instantaneous prediction results, the following approach is adopted: The sliding window majority voting mechanism smooths out the intent sequence:

[0129] ;

[0130] In the formula, The length of the sliding window. For the final intended stream output at time t, This represents the predicted intent classification result.

[0131] This real-time intent stream processing effectively filters out random errors, outputting a more stable and coherent intent stream that better reflects the persistence of intent in human conversation, thus improving user experience and clinical usability.

[0132] Step 4: Output interpretable results for the predicted intent classification results, key evidence fragments (text / audio / video), modality contribution analysis graphs, and time series evolution reports, providing transparent evidence for clinical decision-making.

[0133] The key evidence fragments (text / audio / video) refer to those obtained through temporal attention intensity vectors. , and Retrieve the text, audio, or video clips corresponding to the top 20% of high-weight values;

[0134] The contribution of each modality is obtained based on attention weights. First, the total attention of each modality is calculated:

[0135] ;

[0136] ;

[0137] ;

[0138] In the formula, , , These are the sums of attention for text, audio, and video modalities, respectively. , , These represent the number of tokens in the text, the number of audio frames, and the number of video frames, respectively.

[0139] Furthermore, the contribution of each modality satisfies:

[0140] ;

[0141] ;

[0142] ;

[0143] In the formula, , , These represent the contribution values ​​for text, audio, and video modalities, respectively.

[0144] In this embodiment, the number of intention categories is defined as 12 categories based on relevant psychological research, specifically including: expressing hopelessness or sadness, mentioning suicidal thoughts, expressing self-criticism or guilt, reporting sleep issues, reporting eating issues, reporting low energy or fatigue, mentioning trouble concentrating, discussing work or school stress, discussing family relationships, recounting traumatic events, positive social interaction, and neutral or non-clinical conversation.

[0145] In another embodiment, prompt learning or few-shot learning can be used instead of the current zero-shot method based on semantic similarity.

[0146] In another embodiment, audio features can also be extracted using GeMAPS, and pose or eye-tracking features can be added to the video features.

[0147] like Figure 3 As shown, to verify the method described in this invention, the DAIC-WOZ dataset was used to divide the training set, validation set, and test set into an 8:1:1 ratio. The multi-objective fusion model described in this invention was trained, validated, and tested respectively. In the figure, the rows represent the true labels, the columns represent the predicted labels, and the values ​​in the cells represent how many samples with the true label k were predicted as label j. The main diagonal represents the number of correctly predicted samples, and the values ​​off the diagonal represent the number of incorrectly predicted samples. As can be seen from the figure, there are a total of 4384 test samples, and the overall accuracy of the model is 73.2%. Among them, categories 4 and 10 performed exceptionally well, with an accuracy exceeding 94%, indicating that the model can achieve a high accuracy after sufficient learning on these categories.

[0148] This invention presents a multimodal psychological intent understanding method based on zero-shot and intent-aware alignment. Employing zero-shot learning and intent-aware alignment techniques, it can be started and run without relying on large-scale manually labeled data, fundamentally solving the bottleneck of high data labeling costs and difficulty in data acquisition in the field of psychological dialogue. Furthermore, through temporal alignment, it achieves accurate fusion of cross-modal information, significantly improving the accuracy of analysis. It possesses "out-of-the-box" rapid deployment and expansion capabilities; only new intent descriptions need to be defined to adapt to new scenarios without retraining the model. This allows the system to maintain an agile response and scenario adaptability that is difficult for traditional methods to achieve when facing sudden or niche psychological problems. The late-stage dynamic fusion architecture guided by intent semantics can intelligently balance multi-source evidence such as text, speech, and vision, effectively addressing common signal ambiguity, conflict, or partial missing signals in real-world scenarios, thus ensuring the system's robustness and practicality in complex real-world environments.

[0149] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and embodiments shown and described herein.

Claims

1. A multimodal psychological intent understanding method based on zero-shot and intent-aware alignment, characterized in that, Includes the following steps: Step 1: Collect time-synchronized dialogue data containing text, audio, and video to form multimodal data; Step 2: Construct a multi-objective fusion model. By combining time-aligned multimodal data with a late-stage fusion strategy, a multi-objective joint optimization function is obtained. Step 3: Using an adaptive reasoning strategy, multimodal data is dynamically allocated to obtain the predicted intent classification results.

2. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 1, characterized in that, Step two specifically includes the following steps: Step 1: Preprocess the text data to obtain a preliminary intent probability distribution and confidence score; Step 2: Extract features from the multimodal data, and perform temporal alignment on the extracted multimodal data based on the text data. Combine this with an intent-aware attention mechanism to obtain the modality alignment loss function. Step 3: After encoding the audio feature vector and video feature vector, perform late-stage fusion to obtain the final intent probability distribution; Step 4: Calculate the multi-objective joint optimization function using the classification loss function, modality alignment loss function, regularization loss function, and consistency loss function; Step 5: Train the multi-objective fusion model until the verification loss no longer decreases for 20 consecutive rounds. Once training is complete, select the intent with the highest probability from the final probability intent distribution as the predicted intent classification result.

3. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 2, characterized in that, The preliminary intent probability distribution is obtained by passing the text fragments sequentially through a pre-trained inference model, a projection layer, and a normalization layer.

4. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 3, characterized in that, The pre-trained inference model is The encoder, wherein the projection layer is a linear transformation function layer.

5. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 4, characterized in that, The intent-aware attention mechanism satisfies: ; In the formula, To perceive aligned features, For querying the matrix, The key matrix, For value matrices, It is a time sequence mask matrix. Indicates the dimension of the key.

6. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 5, characterized in that, The query matrix, key matrix, and value matrix satisfy: ; ; ; In the formula, For the first Text feature vectors extracted from text fragments, This is the matrix resulting from concatenating the audio and video feature vectors. To query the projection matrix, Let be the key projection matrix.

7. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 6, characterized in that, Both the audio feature vector and the video feature vector are obtained through a multilayer perceptron.

8. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 7, characterized in that, In the multi-objective fusion model training process in step 5, preliminary intent understanding pseudo-labels are generated by labeling the training set data, and training is carried out using the preliminary intent understanding pseudo-labels; The preliminary intent understanding pseudo-tag satisfies: ; In the formula, It is the first Preliminary intent pseudo-labels for a text fragment, For the first Intent-based prompt template, For the first in the text sequence A text fragment, Used to calculate the semantic matching degree between text and template.

9. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 8, characterized in that, The dynamic calculation allocation specifically includes: When the confidence score of the complete model is lower than the confidence threshold during the initial inference or when the confidence score of the text data is lower than the confidence threshold, the multi-objective fusion model is enabled to predict the intent classification results. When the confidence score of the text data reaches or exceeds the confidence threshold, the zero-sample text inference results are directly adopted.

10. The multimodal psychological intent understanding method based on zero-shot and intent-aware alignment according to claim 9, characterized in that, Also includes: Step 4: Output interpretability results for the predicted intent classification results, key evidence fragments, modality contribution analysis graphs, and time series evolution reports.

Images