A hierarchical prompt and multi-layer distillation method for multi-modal intent understanding

Abstract

This invention discloses a missing modality adaptive modeling method for multimodal dialogue intent understanding. It constructs a hierarchical prompting and multi-level distillation model for multimodal intent understanding. A scene-level prompt generator decomposes and unifies different modality missing scenarios into a single semantic space, and contrastive learning is used to improve alignment. An instance-level adaptive prompter utilizes a cross-attention mechanism to integrate sample-specific features into scene-level prompts. A multi-level knowledge distillation teacher-student network module achieves full-layer knowledge transfer, fully utilizing all available modality samples. The method's core design concept is "dynamic prompting adapts to missing modes, and multi-level distillation transfers cross-modal knowledge." Through the synergy of a modular architecture and standardized processes, it achieves robust semantic modeling of multimodal data such as text, images, and audio in modality missing scenarios.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing technology, specifically relating to a hierarchical prompting and multi-level distillation method for multimodal intent understanding. Background Technology

[0002] Intent understanding is a key technology in human-computer dialogue systems, intelligent assistants, and task-oriented chatbots. Its main goal is to identify the user's interaction intent from the user's input information, thereby driving corresponding business processing or service responses. With the rapid development of application scenarios such as intelligent customer service, in-vehicle interaction, and smart homes, user interaction forms have gradually expanded from single text to multimodal interaction methods that integrate multiple information such as text, voice, and images. Therefore, multimodal intent understanding technology has become an important research direction for improving system robustness and user experience.

[0003] Existing multimodal intent understanding technologies typically rely on multimodal feature fusion and deep learning models for modeling. By jointly learning representations from different modalities, they achieve feature alignment and intent discrimination across modalities. However, in real-world applications, factors such as device conditions, privacy restrictions, and network quality often result in missing modal data, such as acquiring only text while lacking speech or image information. This leads to a significant performance degradation of multimodal models under incomplete input conditions. To address the issue of missing modalities, related technologies mainly revolve around three core approaches: first, generation and reconstruction-based methods that reconstruct missing modalities from available modalities; second, knowledge distillation methods (such as MMANet and MMH) that transfer knowledge from teacher networks trained on complete data to student networks handling incomplete input; and third, prompt-based methods (such as MPVR and Fourier Prompt Tuning) that design prompts to adapt pre-trained models to handle incomplete data.

[0004] In summary, existing technologies still have shortcomings in dealing with complex and dynamic modality loss scenarios: on the one hand, methods for generating or reconstructing missing modalities are computationally expensive, and the reconstruction results may introduce semantic biases; on the other hand, it is difficult to fully characterize the semantic relationships between different missing modalities during knowledge distillation or feature fusion. Furthermore, when the modality loss situation differs between the training and real-world application phases, existing technologies often exhibit significant fluctuations in recognition performance. Therefore, it remains necessary to propose a technical solution that can achieve stable and effective intent understanding under modality loss conditions. Summary of the Invention

[0005] To address the shortcomings of the aforementioned technical issues, this invention provides a hierarchical prompting and multi-layer distillation method for multimodal intent understanding. It includes core modules that are functionally independent yet closely collaborative, and also covers the entire technical process from data input to result output, ensuring stable output of high-quality semantic modeling results in scenarios with different missing rates and different modal combinations.

[0006] The technical solution adopted in this invention is: a hierarchical prompting and multi-level distillation method for multimodal intent understanding, the specific steps of which are as follows:

[0007] S1. Construct a hierarchical prompting and multi-layer distillation model for multimodal intent understanding;

[0008] The model includes: a multimodal encoder, a hierarchical prompt generator module, and a multi-layer knowledge distillation teacher-student network module.

[0009] The hierarchical prompt generator module includes: a scenario-level prompt generator and an instance-level prompt generator. The multi-layered knowledge distillation teacher-student network module includes: a teacher network, a student network, an internally shared knowledge distillation module, and a task-specific knowledge distillation module.

[0010] S2. Based on step S1, input the multimodal data into the multimodal encoder and extract features from different modal data respectively;

[0011] Multimodal data is input into the hierarchical prompting and multi-level distillation model, and the multimodal data undergoes feature extraction via a dedicated multimodal encoder for each modality.

[0012] The multimodal data is divided into complete modalities and missing modalities, including text, images, and audio. For missing modalities, the hierarchical cueing and multi-level distillation model uses a standardized placeholder mechanism, that is, missing text and audio are represented by vectors with all zeros, and missing visual information is represented by images with all pixels having a value of 1.

[0013] S3. Based on step S2, the extracted features are input into the hierarchical prompt generator module. Scene-level prompts capture missing pattern associations, and instance-level prompts are adapted to each individual sample.

[0014] Among them, the scene-level prompt generator uses modality factorization strategy and contrastive learning constraints to enable prompts to adaptively match different missing patterns and capture semantic associations between patterns; the instance-level prompt generator generates prompt vectors that combine pattern adaptability and sample uniqueness by fusing sample-specific features with scene-level global priors.

[0015] S4. Based on step S3, input the prompts output by the hierarchical prompt generator and the modal features extracted by the multimodal encoder into the multi-layer knowledge distillation teacher-student network module. Through the collaborative design of modal sharing distillation and weighted task distillation, full-layer feature alignment and confidence weighted transfer are achieved.

[0016] The teacher-student network module adopts a dual-network architecture design, with ViLT as the basic backbone network. During training, the backbone network parameters are frozen, and only the learnable parameters related to the classification head and prompts are updated.

[0017] The learnable parameters include: the fundamental vector for each modality. Feature aggregation MLP, feature fusion MLP, cross-attention projection matrix Alignment weights between layers .

[0018] S5. Based on step S4, integrate the core loss, complete model training, output the specific prediction results of the intent recognition task, and realize adaptive modeling of missing modalities.

[0019] The classification loss uses the cross-entropy loss function; the cue regularization loss comes from the hierarchical cue generator module; and the distillation loss is a fusion of the modality sharing distillation loss between the teacher network and the student network. Weighted task distillation loss .

[0020] The three types of losses are combined into a unified training objective function, i.e., the total loss function, using preset weight coefficients. The Adam optimizer then iteratively optimizes the classification head parameters and cue-related learnable parameters of the student network until the loss function converges and the model achieves stable performance. After training, the semantic modeling results of the student network are received, the specific prediction results for the intent recognition task are output, and the model's performance metrics under different modality missing rate scenarios are statistically analyzed.

[0021] Furthermore, step S3 is specifically as follows:

[0022] S31. Input the extracted features into the scene-level prompt generator. Through modal factorization strategy and contrastive learning constraints, the prompts adaptively match different missing patterns and capture the semantic associations between patterns.

[0023] First, a learnable fundamental vector is assigned to each modality. Then, the basis vectors are initialized using a Xavier normal distribution. For any input sample, its observed modal features are first encoded using a feature aggregation MLP to obtain the fused observed feature vector. .

[0024] in, Indicates modal index, Represents the space of real numbers. Representing dimension, Indicates sample The set of observation modes Indicates sample The Primitive characteristics of a modality That is, for the current sample The feature input of the observable modality is aggregated with the embedded aggregated representation obtained by the MLP multilayer perceptron.

[0025] Then, the matching degree between the observed feature vector and the basis vector of each modality is calculated based on cosine similarity, and the attention weights of each basis vector are obtained through the Softmax function. and through temperature coefficient The sharpness of the attention weight is adjusted, and the specific calculation expression is as follows:

[0026] (1);

[0027] in, Indicates the first The degree to which the fundamental vectors of each modality contribute to the final combined cues. This indicates the calculation of the cosine similarity between two vectors.

[0028] After obtaining the attention weight coefficients, a weighted combination of the base vector and attention weight coefficients for each modality is obtained to determine the result at a temperature coefficient of [value missing]. Next, the Scene-level prompts for each sample The calculation expression is as follows:

[0029] (2);

[0030] in, This represents the learnable fundamental vector for each modality.

[0031] Then, a contrastive structure alignment loss is introduced. Strengthening the semantic structural associations between different missing patterns, loss This includes: contrast constraints based on sample labels and distance constraints based on modal overlap.

[0032] The comparison constraint based on sample labels is expressed as follows: This allows scene-level prompts with the same label to be placed close to each other in the semantic space; if the labels are not equal, then... Distance constraints based on modal overlap are expressed as follows: .

[0033] in, Indicates sample Label.

[0034] Then compare the structural alignment loss The expression is as follows:

[0035] (3);

[0036] in, This represents the weighting coefficient, balancing the contributions of the contrast constraint and the distance constraint. This means that the difference in scene-level cues is measured by calculating the square of the Euclidean distance between the scene-level cue vectors of two samples.

[0037] S32, based on step S31, input the extracted features into the instance-level cue generator, and generate a cue vector that combines pattern adaptability and sample uniqueness by fusing sample-specific features with scene-level global priors.

[0038] First, the features extracted in step S2 are concatenated, with missing modalities filled in, to form a fused feature vector. Then, this fused feature vector is used to generate initial instance hints through a feature fusion MLP. .

[0039] in, This indicates that the original features are input into the feature fusion MLP.

[0040] Then the scene-level prompts obtained in step S31 The input instance-level hint generator serves as the key and value vectors, while the initial instance hints serve as the query vector, facilitated by a learnable projection matrix. After dimensional mapping, the similarity between the query vector and the key vector is calculated, and attention weights are obtained through Softmax normalization. Then, instance content information is fused with scene-level cues for correction and enhancement to generate instance-level cues. The calculation expression is as follows:

[0041] (4);

[0042] in, Indicates the scaling factor. This indicates the transpose operation.

[0043] Then, instance-level hint regularization loss is introduced. Ensure semantic consistency of instance-level prompts based on modal overlap between samples. We calculate the Euclidean distance between instance-level cue vectors using weighted calculations, ensuring that instance cues for samples with high modal overlap remain semantically similar. (Lower loss) The specific calculation expression is as follows:

[0044] (5).

[0045] Furthermore, step S4 is specifically as follows:

[0046] S41, Internal shared knowledge distillation module;

[0047] The teacher network is trained using only complete modality samples and scene-level cues. Features are learned layer by layer through multiple semantic encoders within the teacher network: lower-level semantic encoders focus on capturing modality-shared features, while higher-level semantic encoders learn the category decision boundaries in the intent recognition task. The hidden features of each layer are expressed using the formula... calculate.

[0048] Among them, subscript This refers to a network of teachers. This represents the complete modal sample of the input teacher network. The original characteristics, Represents complete modal samples The features extracted by the multimodal encoder are used as the initial input to the teacher network. Indicates the number of semantic encoder layers. The first term representing the teacher network Layer semantic encoder.

[0049] The student network receives missing modality samples, scenario-level hints, and instance-level hints, and uses formulas... Calculate the hidden features of each layer.

[0050] Among them, subscript This refers to the student network. This represents the incomplete modality sample features after inputting the student network. Specifically, missing image modalities are filled with blank images containing all pixels with a value of 1, missing text modalities are filled with empty text, and missing audio modalities are filled with vectors containing all zeros. Indicates missing modal samples Features extracted by a multimodal encoder.

[0051] Teacher network and student network modal sharing distillation loss The core concept is full-layer feature alignment, specifically expressed as follows:

[0052] (6);

[0053] in, This represents the total number of layers in the ViLT semantic encoder. Represented as inter-layer alignment weight, i.e., the first layer... Distillation weight of the layer, This represents the Euclidean distance metric function. These represent the teacher network and the student network respectively in the [number]th [year]. The hidden state of the layer; The weight hyperparameters of the global pooling term are represented by MeanPool, which aligns the features of each layer after global mean pooling, thereby enhancing the overall consistency of the features across the entire layer.

[0054] S42, Task-Specific Knowledge Distillation Module;

[0055] First, the outputs of the teacher and student networks are smoothed using a temperature-controlled Softmax function to generate the softening probability, as shown in the following expression:

[0056] (7);

[0057] in, These represent the teacher and student networks for the sample, respectively. No. The original output of the class, This represents the traversal index in the summation operation, and indicates traversing all... Each of the categories. This represents the temperature hyperparameter. It is smoothed by introducing a temperature-scaled Softmax function to obtain the softening probability. The softening probabilities across all categories collectively constitute the predicted probability distribution for softening. This indicates that temperature scaling is being used.

[0058] Then, the KL divergence between the softening probability distributions of the teacher network and the student network is calculated, and the divergence values ​​of all samples are summed to obtain the task-specific distillation loss. The expression is as follows:

[0059] (8);

[0060] in, This indicates the teacher network and student network on the sample. In the The softening probability on the class.

[0061] Then, sample weights are introduced. That is, for the first For each training sample, the teacher network predicts the maximum probability of the sample category, assigning higher weights to high-confidence samples and lower weights to low-confidence samples. The KL divergence between the teacher model output and the student model output is calculated, and the difference is amplified by weights to finally obtain the weighted task distillation loss. The calculation expression is as follows:

[0062] (9);

[0063] in, These represent the teacher model and student model for the samples, respectively. The softening probability distribution.

[0064] Furthermore, in step S5, the total loss function is as follows:

[0065] In the construction of the loss function, the total loss includes four core components: 1) Intent classification loss ;2) Comparison of structural alignment loss Compared with instance-level hint regularization loss ;3) Distillation loss of teacher network and student network modal sharing ;4) Weighted task distillation loss .

[0066] Intended classification loss The cross-entropy loss function is used to directly optimize the basic classification performance of the student network in the intent recognition task, as shown in the following expression:

[0067] (10);

[0068] in, Indicates the first The true label of each sample Indicates the student network's response to the first The predicted probability of a sample.

[0069] Comparison of structural alignment loss Compared with instance-level hint regularization loss Semantic consistency and pattern adaptability of constraint hints;

[0070] Teacher network and student network modal sharing distillation loss Weighted task distillation loss These are used to align the feature representations and output distributions of each layer in the teacher and student networks, respectively.

[0071] Then the total loss function The expression is defined as follows:

[0072] (11);

[0073] in, Indicates the contrast structure alignment loss The weighting coefficients, Indicates instance-level hint regularization loss The weighting coefficients, Represents knowledge distillation loss The weighting coefficients; and the training of the hierarchical hints and multi-level distillation model uses the Adam optimizer.

[0074] The beneficial effects of this invention are: This invention constructs a hierarchical prompting and multi-level distillation model for multimodal intent understanding. A scene-level prompt generator decomposes and unifies different modalities lacking context into a single semantic space, and contrastive learning is used to improve alignment. An instance-level adaptive prompter utilizes a cross-attention mechanism to integrate sample-specific features into scene-level prompts. A multi-level knowledge distillation teacher-student network module achieves full-layer knowledge transfer, fully utilizing all available modal samples. The core design concept of this invention is "dynamic prompting adapts to missing modes, and multi-level distillation transfers cross-modal knowledge." Through the synergy of a modular architecture and standardized processes, it achieves robust semantic modeling of multimodal data such as text, images, and audio in modality-missing scenarios.

[0075] Existing static prompts cannot capture the correlation between missing patterns and sample-specific features. The method of this invention achieves accurate adaptation to diverse missing patterns and individual samples through dynamic construction of scene-level prompts and adaptive optimization of instance-level prompts, avoiding insufficient generalization ability caused by fixed prompts. Existing distillation only focuses on output layer or high-level feature alignment and relies on complete samples. The method of this invention achieves fine-grained knowledge transfer through multi-layer feature alignment and weighted task distillation, and supports student models to utilize all available modal samples, reducing dependence on complete modal samples. Existing generative methods require additional learning of modality generation logic, resulting in high computational costs and easy introduction of distribution bias. The method of this invention directly compensates for missing information through prompt guidance and knowledge distillation, reducing computational overhead while avoiding the impact of generation bias on semantic modeling. Existing technologies require designing a large number of fixed prompts to cover all missing combinations. The method of this invention uses a scene-level prompt dynamic combination strategy with modality factorization, which only needs to set a basic vector for each modality to generate prompts corresponding to all missing patterns, avoiding combination explosion and parameter redundancy. Existing methods exhibit significant performance fluctuations when training and testing missing rates are mismatched. The method of this invention improves the model's adaptability to different missing rate scenarios and reduces performance degradation caused by changes in missing rates by flexibly adapting to missing patterns through hierarchical prompts and comprehensively transferring knowledge through multi-level distillation. Existing incomplete multimodal learning methods are prone to overfitting under conditions of high missing rates or small sample sizes. The method of this invention introduces a regularization term for instance-level prompts based on modal overlap weighted Euclidean distance constraints, and combines this with a weighted task distillation mechanism to suppress noise in low-confidence samples, thus helping to improve the model's generalization ability. Attached Figure Description

[0076] Figure 1This is a flowchart of a missing modality adaptive modeling method for multimodal dialogue intent understanding according to the present invention.

[0077] Figure 2 This is a schematic diagram of the hierarchical prompting and multi-layer distillation model structure for multimodal intent understanding in an embodiment of the present invention.

[0078] Figure 3 This is a schematic diagram of the scene-level prompt generator structure in an embodiment of the present invention.

[0079] Figure 4 This is a schematic diagram of the instance-level prompt generator structure in an embodiment of the present invention.

[0080] Figure 5 This is a schematic diagram of the multi-layer knowledge distillation teacher-student network module structure in an embodiment of the present invention. Detailed Implementation

[0081] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0082] like Figure 1 The flowchart shown below illustrates a missing modality adaptive modeling method for multimodal dialogue intent understanding according to the present invention. The specific steps are as follows:

[0083] S1. Construct a hierarchical prompting and multi-layer distillation model for multimodal intent understanding;

[0084] like Figure 2 As shown, the model includes: a multimodal encoder, a hierarchical prompt generator module, and a multi-layer knowledge distillation teacher-student network module.

[0085] The hierarchical prompt generator module includes: a scenario-level prompt generator and an instance-level prompt generator. The multi-layered knowledge distillation teacher-student network module includes: a teacher network, a student network, an internally shared knowledge distillation module, and a task-specific knowledge distillation module.

[0086] S2. Based on step S1, input the multimodal data into the multimodal encoder and extract features from different modal data respectively;

[0087] The model's operation begins with the input of raw multimodal data. In real-world applications, data is often incomplete; for example, a social media post may contain text and images but lack audio, or a surveillance video may have lost some frames due to equipment malfunction. The method of this invention considers this uncertainty from the outset. When multimodal data is input into the hierarchical cueing and multi-level distillation model, the multimodal data undergoes feature extraction via a dedicated multimodal encoder for each modality.

[0088] The multimodal data is divided into complete modalities and missing modalities, including text, images, and audio. For missing modalities, the hierarchical cueing and multi-level distillation model employs a standardized placeholder mechanism: missing text and audio are represented by vectors containing all zeros, and missing visual information is represented by images with all pixels having a value of 1. This preprocessing step ensures that regardless of the changes in the input modal combinations, the model receives a structurally uniform embedding sequence. This sequence not only contains the semantic information of the available modalities but also implicitly marks the location of the missing modalities, providing a foundation for subsequent missing modality analysis.

[0089] S3. Based on step S2, the extracted features are input into the hierarchical prompt generator module. Scene-level prompts capture missing pattern associations, and instance-level prompts are adapted to each individual sample.

[0090] Among them, the scene-level prompt generator uses modality factorization strategy and contrastive learning constraints to enable prompts to adaptively match different missing patterns and capture semantic associations between patterns; the instance-level prompt generator generates prompt vectors that combine pattern adaptability and sample uniqueness by fusing sample-specific features with scene-level global priors.

[0091] The hierarchical prompt generator is the core of this invention, breaking through the limitations of traditional static prompts. It captures missing pattern associations through scene-level prompts and adapts instance-level prompts to each individual sample, constructing a two-layer adaptation system of global pattern modeling and local sample optimization. This enables dynamic responses to multiple missing modalities, improving the robustness and generalization ability of intent recognition. This module is further subdivided into two functional units: scene-level prompt generation and instance-level prompt generation. The core idea of ​​the scene-level prompt generator is to avoid designing fixed prompts for each missing pattern. Instead, it maintains a learnable base vector for each modality, such as text, image, and audio. Upon receiving preprocessed multimodal data, the unit first extracts the observed modal features of the sample, calculates the cosine similarity between the observed modal features and the base vectors of each modality through an attention mechanism, and then dynamically allocates weights based on the similarity results. The base vectors are then combined to generate scene-level prompts adapted to the current missing pattern. Meanwhile, to strengthen the semantic association of intent between different missing patterns, the unit also introduces contrastive structure alignment loss calculation. Through loss optimization, prompts corresponding to semantically similar missing patterns (such as "text missing - image present" and "text missing - image + audio present") are brought closer together in the semantic space, while prompts for missing patterns with large semantic differences are kept away from each other. This allows for more accurate capture of missing configurations that are common in intent understanding. The instance-level prompt generator focuses on "adapting to individual samples." It first fuses all available modal features of the sample through a multilayer perceptron to generate initial instance prompts. Then, with the help of a cross-attention mechanism, it injects the prior information of the global missing pattern contained in the scene-level prompts into the initial instance prompts, achieving a deep fusion of global scene knowledge and local sample-specific features. Finally, it generates instance-level prompts exclusive to each user interaction sample, ensuring that the prompts not only fit the missing pattern but also adapt to the unique way the user expresses their intent (e.g., for "complaint," some people use strong tones and dense keywords, while others only upload screenshots). In addition, the hierarchical prompt generator module also calculates the prompt regularization loss and calculates the Euclidean distance between scene-level and instance-level prompts based on the modal overlap (Jaccar coefficient) between samples, thereby ensuring the semantic consistency of prompts and providing high-quality prompt support for subsequent intent recognition network training.

[0092] S4. Based on step S3, input the prompts output by the hierarchical prompt generator and the modal features extracted by the multimodal encoder into the multi-layer knowledge distillation teacher-student network module. Through the collaborative design of modal sharing distillation and weighted task distillation, full-layer feature alignment and confidence weighted transfer are achieved.

[0093] The teacher-student network module is used to realize knowledge transfer and learning in the framework. It adopts a dual-network architecture design, with ViLT (Vision-and-Language Transformer) as the basic backbone network. During training, the backbone network parameters are frozen, and only the learnable parameters related to the classification head and prompts are updated. This can reduce training complexity while preserving the cross-modal alignment capability of the pre-trained model.

[0094] The learnable parameters include: the fundamental vector for each modality. Feature aggregation MLP, feature fusion MLP, cross-attention projection matrix Alignment weights between layers .

[0095] The teacher-student network module accurately transfers the complete modal knowledge of the teacher network to the student network, addressing the shortcomings of existing distillation techniques, such as coarse granularity and reliance on complete samples. This module comprises two key units: modality-shared distillation (an internal shared knowledge distillation module) and weighted task distillation (a task-specific knowledge distillation module). The core of the modality-shared distillation unit is to achieve full-layer feature alignment. It performs a dimension-by-dimensional comparison of the hidden features of each layer in both the teacher and student networks, ensuring consistency between the low-level modal-shared features of the student network and the teacher network by calculating Euclidean distance loss. Simultaneously, it aligns the global mean pooling results of features from each layer of both the teacher and student networks, further strengthening the complete transfer of low-level shared knowledge and avoiding the knowledge gaps caused by existing techniques that only align high-level features. The weighted task distillation unit is primarily responsible for task-related knowledge transfer. It first smooths the output prediction distribution of the teacher network using a temperature-scaling soft max function to reduce decision bias caused by hard labels, and then calculates the difference between the smoothed output distributions of the teacher and student networks using KL divergence to obtain the basic distillation loss. To enhance the effectiveness of knowledge transfer, this unit also assigns different weights to samples based on the maximum class probability of the teacher network's prediction for each sample, i.e., the teacher's confidence in the intention category. Samples with high confidence receive higher weights, thereby strengthening the transfer of reliable knowledge, weakening the noise interference that low-confidence samples may bring, and ensuring that the student network can learn the intention reasoning logic of the teacher network in the complete modality scenario.

[0096] S5. Based on step S4, integrate the core loss, complete model training, output the specific prediction results of the intent recognition task, and realize adaptive modeling of missing modalities.

[0097] Specifically, the classification loss uses the cross-entropy loss function to directly optimize the basic performance of the student network in the intent recognition task; the cue regularization loss comes from the hierarchical cue generator module to ensure the semantic consistency of the cue; and the distillation loss integrates the modality-sharing distillation loss of the teacher network and the student network. Weighted task distillation loss This ensures the effective transfer of cross-modal knowledge.

[0098] The three types of losses are combined into a unified training objective function, i.e., the total loss function, using preset weight coefficients. The Adam optimizer then iteratively optimizes the classification head parameters and cue-related learnable parameters of the student network until the loss function converges and the model reaches stable performance. After training, the semantic modeling results of the student network are received, and the specific prediction results for the intent recognition task are output. Performance metrics of the model under different modality missing rate scenarios are also statistically analyzed, including intent accuracy on the MintRec dataset (Macro-F1, SIMMC 2.1, MultiATIS++, and MELD datasets), providing quantitative evidence for evaluating the model's generalization ability.

[0099] In this embodiment, step S3 is specifically as follows:

[0100] S31. Input the extracted features into the scene-level prompt generator. Through modal factorization strategy and contrastive learning constraints, the prompts adaptively match different missing patterns and capture the semantic associations between patterns.

[0101] The core design idea of ​​scene-level prompts is to avoid designing fixed prompts for each missing pattern. Through modal factorization strategy and contrastive learning constraints, prompts can adaptively match different missing patterns and capture semantic relationships between patterns.

[0102] The scene-level prompt generator, such as Figure 3 As shown, a learnable fundamental vector is first assigned to each mode. Then, the base vectors are initialized using a Xavier normal distribution to ensure gradient stability during training. For any input sample, the observed modal features are first encoded using a feature aggregation MLP to obtain the fused observed feature vector. .

[0103] in, Indicates modal index, Represents the space of real numbers. Representing dimension, Indicates sample The set of observation modes Indicates sample The Primitive characteristics of a modality That is, for the current sample The feature input of the observable modality is aggregated with the embedded aggregated representation obtained by the MLP multilayer perceptron.

[0104] Then, the matching degree between the observed feature vector and the basis vector of each modality is calculated based on cosine similarity, and the attention weights of each basis vector are obtained through the Softmax function. and through temperature coefficient The sharpness of the attention weight is adjusted, and the specific calculation expression is as follows:

[0105] (1);

[0106] in, Indicates the first The degree to which the fundamental vectors of each modality contribute to the final combined cues. This indicates the calculation of the cosine similarity between two vectors.

[0107] After obtaining the attention weight coefficients, a weighted combination of the base vector and attention weight coefficients for each modality is obtained to determine the result at a temperature coefficient of [value missing]. Next, the Scene-level prompts for each sample This cue vector is used to characterize the modal availability structure of a sample, and can represent the presence of various modalities. The calculation expression is as follows:

[0108] (2);

[0109] in, This represents the learnable fundamental vector for each modality. Figure 3 middle This indicates the number of modalities in the input data, such as when there is only text and audio. It is 2. This represents the number of possible modal combinations, i.e., the maximum number of modal combinations (minus 1 to exclude the case where all modalities are missing). This represents a scene-level cue corresponding to a combination of modalities, and there are a total of This is one of the situations.

[0110] To enhance the semantic structural associations between different missing patterns, a contrastive structural alignment loss is introduced. Strengthening the semantic structural associations between different missing patterns, loss This includes: contrast constraints based on sample labels and distance constraints based on modal overlap.

[0111] The comparison constraint based on sample labels is expressed as follows: This allows scene-level prompts for samples with the same label to be close to each other in the semantic space, improving the feature consistency of similar samples. If the labels are not equal, then... Distance constraints based on modal overlap are expressed as follows: The Jaccard coefficient is used to measure the degree of overlap of the observed modal sets, so that the sample prompts with high modal overlap maintain semantic continuity, while the sample prompts with low modal overlap and different labels are kept apart from each other, thus ensuring the semantic distinguishability of different missing modes.

[0112] in, Indicates sample Label.

[0113] Then compare the structural alignment loss The expression is as follows:

[0114] (3);

[0115] in, This represents the weighting coefficient, balancing the contributions of the contrast constraint and the distance constraint. This means that the difference in scene-level cues is measured by calculating the square of the Euclidean distance between the scene-level cue vectors of two samples. Simultaneously considering both category similarity and modal structure similarity, cues for samples within the same category are more closely linked, while cues for samples with significant modal differences maintain clearer distinctions. The technical principle behind this mechanism lies in decomposing complex missing patterns into combinations of basic modal vectors through modal factorization, avoiding the redundancy issues associated with cues corresponding to fixed modal missing combinations. Contrast structure alignment, through dual constraints of label semantics and modal structure, allows scene-level cues to match the current missing pattern and learn the intrinsic relationships between different patterns. Its implementation is remarkably effective; in a three-modal scenario, only three basic vectors are needed to cover seven missing patterns. Even with a missing rate as high as 60%, this mechanism can still effectively model the similarity of missing structures, enhancing the model's generalization ability to similar missing scenarios.

[0116] S32, based on step S31, input the extracted features into the instance-level cue generator, and generate a cue vector that combines pattern adaptability and sample uniqueness by fusing sample-specific features with scene-level global priors.

[0117] Instance-level prompts focus on solving the problem that scene-level prompts are difficult to adapt to individual sample characteristics. By fusing sample-specific features with scene-level global priors, prompt vectors that combine pattern adaptability and sample uniqueness are generated.

[0118] The instance-level prompt generator, such as Figure 4 As shown, the features extracted in step S2 are first concatenated, with missing modalities padded to ensure input dimension consistency, forming a fused feature vector. This fused feature vector is then used to generate initial instance hints via a feature fusion MLP (2 hidden layers + ReLU activation function). .

[0119] in, This indicates that the original features are input into the feature fusion MLP.

[0120] The initial prompts only contain the feature information of the sample itself and do not consider the global patterns of missing modes. To inject scene-level priors, a cross-attention fusion mechanism is introduced to integrate the scene-level prompts obtained in step S31. The input instance-level suggestion generator serves as the key and value vectors, while the initial instance suggestion serves as the query vector. This is achieved through a learnable projection matrix. After dimensional mapping, the similarity between the query vector and the key vector is calculated, and attention weights are obtained through Softmax normalization. Then, instance content information is fused with scene-level cues for correction and enhancement to generate instance-level cues. (By adding the original) To achieve residual connectivity in a way that avoids gradient explosion and information overload, the calculation expression is as follows:

[0121] (4);

[0122] in, This represents a scaling factor used to limit the numerical range of attention scores, preventing gradient vanishing due to Softmax saturation and improving the numerical stability of attention calculations. This indicates the transpose operation.

[0123] To ensure semantic consistency of instance-level suggestions and avoid a decrease in generalization ability due to excessive personalization, instance-level suggestion regularization loss is introduced. Ensure semantic consistency of instance-level prompts based on modal overlap between samples. We calculate the Euclidean distance between instance-level cue vectors using weighted calculations, ensuring that instance cues for samples with high modal overlap remain semantically similar. (Lower loss) The specific calculation expression is as follows:

[0124] (5);

[0125] The core principle of this mechanism is to achieve a deep integration of global missing pattern knowledge and local sample features, so that the prompts can not only fit the overall pattern of the current missing scenario, but also adapt to the unique attributes of individual samples.

[0126] like Figure 5 As shown, in this embodiment, step S4 is specifically as follows:

[0127] S41, Internal shared knowledge distillation module;

[0128] Multi-layer knowledge distillation technology is key to achieving accurate transfer of complete modal knowledge to missing modal scenarios. Through the collaborative design of modal sharing distillation and weighted task distillation, it breaks through the limitations of existing distillation that only focuses on high-level features or output layers and has coarse knowledge transfer granularity, ensuring the full-layer transfer of modal sharing knowledge and intention decision knowledge.

[0129] The goal of modal shared distillation is to convey the low-level modal shared features learned by the teacher network, such as the basic association between speech intonation and text sentiment words, and the underlying semantic alignment between image content and service intent, so as to avoid the knowledge gaps caused by existing distillation neglecting low-level features.

[0130] The teacher network is trained using only complete modal samples and scene-level cues. It learns features layer by layer through multiple semantic encoders within the network: lower-level semantic encoders focus on capturing modal-shared features (such as the basic semantic association between text descriptions and corresponding images, and the matching logic between audio intonation and text sentiment); higher-level semantic encoders learn the category decision boundaries in intent recognition tasks (such as category decision boundaries in intent recognition tasks and sentiment scoring criteria in sentiment analysis tasks). This ultimately forms a complete cross-modal knowledge system, and the hidden features and output prediction distributions of each layer are passed to the subsequent distillation module.

[0131] Hidden features at each layer are expressed by formula calculate.

[0132] Among them, subscript This refers to a network of teachers. This represents the complete modal sample of the input teacher network. The original characteristics, Represents complete modal samples The features extracted by the multimodal encoder are used as the initial input to the teacher network. Indicates the number of semantic encoder layers. The first term representing the teacher network Layer semantic encoder.

[0133] The student network adapts to missing modal scenarios and receives more complex inputs, including incomplete modal samples, scene-level prompts, and instance-level prompts. It integrates these two levels of prompts into the feature learning process of the multi-layer encoder, guiding the encoder to prioritize the effective features of available modalities through prompts, compensating for the information loss caused by missing modalities. At the same time, it outputs the hidden features and prediction distributions of each layer to the distillation module, providing the feature foundation of the student model for knowledge distillation.

[0134] The student network receives missing modality samples, scenario-level hints, and instance-level hints, and uses formulas... Calculate the hidden features of each layer.

[0135] Among them, subscript This refers to the student network. This represents the incomplete modality sample features after inputting the student network. Specifically, missing image modalities are filled with blank images containing all pixels with a value of 1, missing text modalities are filled with empty text, and missing audio modalities are filled with vectors containing all zeros. Indicates missing modal samples Features extracted by a multimodal encoder.

[0136] Teacher network and student network modal sharing distillation loss The core concept is full-layer feature alignment, specifically expressed as follows:

[0137] (6);

[0138] in, This represents the total number of layers in the ViLT semantic encoder. Represented as inter-layer alignment weight, i.e., the first layer... Distillation weight of the layer, This represents the Euclidean distance metric function. These represent the teacher network and the student network respectively in the [number]th [year]. The hidden state of the layer; The weight hyperparameters representing the global pooling terms are aligned after applying MeanPool global mean pooling to the features of each layer, thereby enhancing the overall consistency of features across all layers. The technical principle behind this mechanism is that shared features of lower-level modalities are the foundation for cross-modal semantic modeling. Full-layer alignment ensures that the student network fully inherits the cross-modal association logic of the teacher network, avoiding incomplete learning caused by aligning only high-level features.

[0139] S42, Task-Specific Knowledge Distillation Module;

[0140] The goal of weighted task distillation is to transfer the high-level task intent decision-making knowledge of the teacher network (such as the category decision boundary of downstream classification tasks and the scoring criteria of sentiment analysis tasks) while filtering out noise interference from low-confidence samples, thereby improving the effectiveness of knowledge transfer.

[0141] First, the outputs of the teacher and student networks are smoothed using a temperature-controlled Softmax function to generate the softening probability, as shown in the following expression:

[0142] (7);

[0143] in, These represent the teacher and student networks for the sample, respectively. No. The original output of the class, This represents the traversal index in the summation operation, and indicates traversing all... Each of the categories. This represents the temperature hyperparameter. It is smoothed by introducing a temperature-scaled Softmax function to obtain the softening probability. The softening probabilities across all categories collectively constitute the predicted probability distribution for softening. This indicates the use of temperature scaling; this smoothing process allows student networks to learn not only the most likely correct category, but also the relative confidence relationships between different categories as determined by the teacher, rather than relying solely on hard labels.

[0144] Then, the KL divergence between the softening probability distributions of the teacher network and the student network is calculated, and the divergence values ​​of all samples are summed to obtain the task-specific distillation loss. The expression is as follows:

[0145] (8);

[0146] in, This indicates the teacher network and student network on the sample. In the The softening probability on the class.

[0147] To further improve the efficiency of knowledge transfer, sample weights are introduced. That is, for the first For each training sample, the teacher network predicts the maximum probability of the sample category, assigning higher weights to high-confidence samples and lower weights to low-confidence samples. The KL divergence between the teacher model output and the student model output is calculated, and the difference is amplified by weights to finally obtain the weighted task distillation loss. The calculation expression is as follows:

[0148] (9);

[0149] in, These represent the teacher model and student model for the samples, respectively. The mechanism involves a softened probability distribution. The principle behind this is that the teacher network is trained on complete modality data, and its high-confidence predictions typically exhibit stronger semantic consistency and reliability. Through weighted distillation, the student network can prioritize learning this high-quality, high-reliability knowledge while suppressing noise interference from low-confidence samples caused by modality loss or semantic ambiguity, thereby improving generalization ability in scenarios with missing modalities.

[0150] In this embodiment, the total loss function in step S5 is as follows:

[0151] Multi-loss collaborative optimization technology is the core support for integrating hierarchical prompt generation and multi-level knowledge distillation objectives and ensuring the overall performance of the model. By scientifically integrating classification loss, prompt regularization loss and distillation loss, it achieves a balance between basic task performance, prompt adaptability and knowledge transfer effectiveness.

[0152] In the construction of the loss function, the total loss includes four core components: 1) Intent classification loss ;2) Comparison of structural alignment loss Compared with instance-level hint regularization loss ;3) Distillation loss of teacher network and student network modal sharing ;4) Weighted task distillation loss .

[0153] Intended classification loss The cross-entropy loss function is used to directly optimize the basic classification performance of the student network in the intent recognition task, as shown in the following expression:

[0154] (10);

[0155] in, Indicates the first The true label of each sample Indicates the student network's response to the first The predicted probability of a sample.

[0156] Comparison of structural alignment loss Compared with instance-level hint regularization loss Semantic consistency and pattern adaptability of constraint hints;

[0157] Teacher network and student network modal sharing distillation loss Weighted task distillation loss These are used to align the feature representations and output distributions of each layer in the teacher and student networks, respectively.

[0158] Then the total loss function The expression is defined as follows:

[0159] (11);

[0160] In this embodiment, the optimal loss weight is determined through grid search in the total loss function, and compared with the structure alignment loss. Weighting coefficients for Instance-level hints for regularization loss Weighting coefficients for Knowledge distillation loss Weighting coefficients The learning rate is set to 1 to avoid reducing flexibility due to overly restrictive prompts while prioritizing the effectiveness of knowledge transfer. The hierarchical prompting and multi-level distillation model are trained using the Adam optimizer with a learning rate set to 10. -2 The weight decay is set to 2×10. -2 This technique balances training speed with the risk of overfitting. It effectively achieves a balance by rationally allocating loss weights, allowing cue adaptation, knowledge transfer, and intent recognition performance to form synergistic gains, thus avoiding performance defects caused by optimizing a single objective.

[0161] This embodiment also conducted further experimental verification. The specific performance improvement of the method of the present invention compared with the baseline method under different datasets, combined with experimental data, is shown in Table 1.

[0162] Table 1

[0163]

[0164] Table 1 shows that the proposed method is compared with representative baseline methods in the field of multimodal intent understanding under different modal missing settings. The experiments use the Macro-F1 score on the MintRec dataset, as well as the intent recognition accuracy on the SIMMC 2.1, MultiATIS++, and MELD datasets, as evaluation metrics. The proposed method consistently outperforms the comparison methods in intent recognition performance metrics, maintaining relatively reliable semantic modeling capabilities even with incomplete modal information. Specifically, in the image missing scenario of the MintRec dataset, the proposed method improves performance by 1.8 percentage points compared to the representative baseline method EPE-P; in the text missing scenario of the SIMMC 2.1 dataset, the performance improvement reaches 2.2 percentage points; in the audio missing scenario of MultiATIS++, it also achieves an improvement of 0.7 percentage points; and in the MELD scenario where both audio and image are missing, the proposed method still improves performance by 1.4 percentage points compared to the baseline method Ma. The experimental results demonstrate that the proposed method exhibits good robustness and generalization ability across different datasets and modal missing modes, effectively mitigating the impact of modal missingness on intent understanding performance.

[0165] In summary, the method of this invention is a unified technical system specifically designed for incomplete multimodal learning. With "dynamic prompting to adapt to missing modes and multi-layer distillation to transfer cross-modal knowledge" as its core design concept, it achieves robust semantic modeling of multimodal data such as text, images, and audio in modality-missing scenarios through the synergy of modular architecture and standardized processes. The entire framework includes both functionally independent yet closely collaborative core modules and covers the entire technical process from data input to result output, ensuring stable output of high-quality semantic modeling results under scenarios with different missing rates and different modality combinations.

[0166] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A hierarchical prompting and multi-level distillation method for multimodal intent understanding, the specific steps of which are as follows: S1. Construct a hierarchical prompting and multi-layer distillation model for multimodal intent understanding; The model includes: Multimodal encoder, hierarchical prompt generator module, multi-layer knowledge distillation teacher-student network module; The hierarchical prompt generator module includes: a scenario-level prompt generator and an instance-level prompt generator; the multi-layered knowledge distillation teacher-student network module includes: a teacher network, a student network, an internally shared knowledge distillation module, and a task-specific knowledge distillation module. S2. Based on step S1, input the multimodal data into the multimodal encoder and extract features from different modal data respectively; Multimodal data is input into the hierarchical prompting and multi-level distillation model, and the multimodal data undergoes feature extraction through a dedicated multimodal encoder for each modality; The multimodal data is divided into complete modalities and missing modalities, including text, images, and audio. For missing modalities, the hierarchical cueing and multi-level distillation model uses a standardized placeholder mechanism, that is, a vector with all zeros is used to represent missing text and audio, and an image with all pixels having a value of 1 is used to represent missing visual information. S3. Based on step S2, the extracted features are input into the hierarchical prompt generator module. Scene-level prompts capture missing pattern associations, and instance-level prompts are adapted to each individual sample. Among them, the scene-level prompt generator uses modality factorization strategy and contrastive learning constraints to enable prompts to adaptively match different missing patterns and capture semantic associations between patterns; the instance-level prompt generator generates prompt vectors that combine pattern adaptability and sample uniqueness by fusing sample-specific features with scene-level global priors. S4. Based on step S3, input the prompts output by the hierarchical prompt generator and the modal features extracted by the multimodal encoder into the multi-layer knowledge distillation teacher-student network module. Through the collaborative design of modal sharing distillation and weighted task distillation, full-layer feature alignment and confidence weighted transfer are achieved. The teacher-student network module adopts a dual-network architecture design, with ViLT as the basic backbone network. During training, the backbone network parameters are frozen, and only the learnable parameters related to the classification head and prompts are updated. The learnable parameters include: the fundamental vector for each modality. Feature aggregation MLP, feature fusion MLP, cross-attention projection matrix Alignment weights between layers ; S5. Based on step S4, integrate the core loss, complete model training, output the specific prediction results of the intent recognition task, and realize adaptive modeling of missing modalities. The classification loss uses the cross-entropy loss function; the cue regularization loss comes from the hierarchical cue generator module; and the distillation loss is a fusion of the modality sharing distillation loss between the teacher network and the student network. Weighted task distillation loss ; The three types of losses are combined into a unified training objective function, namely the total loss function, through preset weight coefficients. Then, the Adam optimizer iteratively optimizes the classification head parameters and cue-related learnable parameters of the student network until the loss function converges and the model reaches stable performance. After training, the semantic modeling results of the student network are received, the specific prediction results of the intent recognition task are output, and the performance indicators of the model under different modality missing rate scenarios are statistically analyzed.

2. The hierarchical prompting and multi-level distillation method for multimodal intent understanding according to claim 1, characterized in that, Step S3 is as follows: S31. Input the extracted features into the scene-level prompt generator. Through modal factorization strategy and contrastive learning constraints, the prompts adaptively match different missing patterns and capture the semantic associations between patterns. First, a learnable fundamental vector is assigned to each modality. Then, the basis vectors are initialized using the Xavier normal distribution; then, for any input sample, its observation modal features are encoded using a feature aggregation MLP to obtain the fused observation feature vector. ; in, Indicates modal index, Represents the space of real numbers. Representing dimension, Indicates sample The set of observation modes, Indicates sample The Primitive characteristics of a modality That is, for the current sample The embedded aggregated representation of the feature input of the observable modality is obtained by aggregating the features of the MLP multilayer perceptron; Then, the matching degree between the observed feature vector and the basis vector of each modality is calculated based on cosine similarity, and the attention weights of each basis vector are obtained through the Softmax function. and through temperature coefficient The sharpness of the attention weight is adjusted, and the specific calculation expression is as follows: （1）； in, Indicates the first The degree to which the fundamental vectors of each modality contribute to the final combined cues. This indicates the calculation of the cosine similarity between two vectors. After obtaining the attention weight coefficients, a weighted combination of the base vector and attention weight coefficients for each modality is obtained to determine the result at a temperature coefficient of [value missing]. Next, the Scene-level prompts for each sample The calculation expression is as follows: （2）； in, Represents the learnable fundamental vectors for each modality; Then, a contrastive structure alignment loss is introduced. Strengthening the semantic structural associations between different missing patterns, loss This includes: contrast constraints based on sample labels and distance constraints based on modal overlap; The comparison constraint based on sample labels is expressed as follows: This allows scene-level prompts with the same label to be placed close to each other in the semantic space; if the labels are not equal, then... Distance constraints based on modal overlap are expressed as follows: ; in, Indicates sample Label; Then compare the structural alignment loss The expression is as follows: （3）； in, This represents the weighting coefficient, balancing the contributions of the contrast constraint and the distance constraint. This means calculating the square of the Euclidean distance between two sample scene-level cue vectors to measure the difference in scene-level cue responses; S32, based on step S31, input the extracted features into the instance-level cue generator, and generate a cue vector that combines pattern adaptability and sample uniqueness by fusing sample-specific features with scene-level global priors. First, the features extracted in step S2 are concatenated, with missing modalities filled in, to form a fused feature vector. Then, this fused feature vector is used to generate initial instance hints through a feature fusion MLP. ; in, This indicates that the original features are input into the feature fusion MLP; Then the scene-level prompts obtained in step S31 The input instance-level hint generator serves as the key and value vectors, while the initial instance hints serve as the query vector, facilitated by a learnable projection matrix. After dimensional mapping, the similarity between the query vector and the key vector is calculated, and attention weights are obtained through Softmax normalization. Then, instance content information is fused with scene-level cues for correction and enhancement to generate instance-level cues. The calculation expression is as follows: （4）； in, Indicates the scaling factor. Indicates the transpose operation; Then, instance-level hint regularization loss is introduced. Ensure semantic consistency of instance-level prompts based on modal overlap between samples. We calculate the Euclidean distance between instance-level cue vectors using weighted calculations, ensuring that instance cues for samples with high modal overlap remain semantically similar. (Lower loss) The specific calculation expression is as follows: （5）。 3. The hierarchical prompting and multi-level distillation method for multimodal intent understanding according to claim 2, characterized in that, Step S4 is as follows: S41, Internal shared knowledge distillation module; The teacher network is trained using only complete modality samples and scene-level cues. It learns features layer by layer through multiple semantic encoders within the network: lower-level semantic encoders focus on capturing modality-shared features, while higher-level semantic encoders learn the category decision boundaries in the intent recognition task; the hidden features at each layer are expressed using a formula. calculate; Among them, subscript Indicates a teacher network, This represents the complete modal sample of the input teacher network. The original characteristics, Represents complete modal samples The features extracted by the multimodal encoder are used as the initial input to the teacher network. Indicates the number of semantic encoder layers. The first term representing the teacher network Layer semantic encoder; The student network receives missing modality samples, scenario-level hints, and instance-level hints, and uses formulas... Calculate the hidden features of each layer; Among them, subscript Indicates student network, This represents the incomplete modality sample features after inputting the student network. Specifically, missing image modalities are filled with blank images containing all pixels with a value of 1, missing text modalities are filled with empty text, and missing audio modalities are filled with vectors containing all zeros. Indicates missing modal samples Features extracted by a multimodal encoder; Teacher network and student network modal sharing distillation loss The core concept is full-layer feature alignment, specifically expressed as follows: （6）； in, This represents the total number of layers in the ViLT semantic encoder. Represented as inter-layer alignment weight, i.e., the first layer... Distillation weight of the layer, This represents the Euclidean distance metric function. These represent the teacher network and the student network respectively in the [number]th [year]. The hidden state of the layer; The weight hyperparameters of the global pooling term are represented by the alignment of the features across all layers after performing MeanPool global mean pooling. This enhances the overall consistency of the features across all layers. S42, Task-Specific Knowledge Distillation Module; First, the outputs of the teacher and student networks are smoothed using a temperature-controlled Softmax function to generate the softening probability, as shown in the following expression: （7）； in, These represent the teacher and student networks for the sample, respectively. No. The original output of the class, This represents the traversal index in the summation operation, and indicates traversing all... Each of the categories; This represents the temperature hyperparameter; it is smoothed by introducing a temperature-scaled Softmax function to obtain the softening probability. The softening probabilities across all categories collectively constitute the predicted probability distribution for softening. This indicates that temperature scaling is being used; Then, the KL divergence between the softening probability distributions of the teacher network and the student network is calculated, and the divergence values ​​of all samples are summed to obtain the task-specific distillation loss. The expression is as follows: （8）； in, This indicates the teacher network and student network on the sample. In the The softening probability on the class; Then, sample weights are introduced. That is, for the first For each training sample, the teacher network predicts the maximum probability of the sample category, assigning higher weights to high-confidence samples and lower weights to low-confidence samples. The KL divergence between the teacher model output and the student model output is calculated, and the difference is amplified by weights to finally obtain the weighted task distillation loss. The calculation expression is as follows: （9）； in, These represent the teacher model and student model for the samples, respectively. The softening probability distribution.

4. The hierarchical prompting and multi-level distillation method for multimodal intent understanding according to claim 3, characterized in that, In step S5, the total loss function is as follows: In the construction of the loss function, the total loss includes four core components: 1) Intent classification loss ; 2) Comparison of structural alignment losses Compared with instance-level hint regularization loss 3) Distillation loss due to modal sharing between teacher and student networks ;4) Weighted task distillation loss ; Intended classification loss The cross-entropy loss function is used to directly optimize the basic classification performance of the student network in the intent recognition task, as shown in the following expression: （10）； in, Indicates the first The true label of each sample Indicates the student network's response to the first The predicted probability of each sample; Comparison of structural alignment loss Compared with instance-level hint regularization loss Semantic consistency and pattern adaptability of constraint hints; Teacher network and student network modal sharing distillation loss Weighted task distillation loss These are used to align the feature representations and output distributions of each layer in the teacher and student networks, respectively. Then the total loss function The expression is defined as follows: （11）； in, Indicates the contrast structure alignment loss The weighting coefficients, Indicates instance-level hint regularization loss The weighting coefficients, Represents knowledge distillation loss The weighting coefficients; and the training of the hierarchical hints and multi-level distillation model uses the Adam optimizer.

Images