An adaptive knowledge transfer method based on semantic uncertainty measure knowledge

Abstract

This invention discloses an adaptive knowledge transfer method based on semantic uncertainty measurement knowledge, belonging to the field of large language models. The invention aims to dynamically adjust the knowledge distillation process according to the features of the input data, innovate an adaptive fusion mechanism for multimodal information, and intelligently allocate computing resources, thereby significantly improving training efficiency and inference speed while reducing deployment costs, while ensuring model performance.

Description

Technical Field

[0001] This invention relates to the field of large language models, and specifically to an adaptive knowledge transfer method based on semantic uncertainty measurement knowledge. Background Technology

[0002] With the deep application of Multimodal Large Language Models (MLLMs) in complex scene understanding, intelligent interaction, and content generation, the scale of model parameters is constantly increasing, leading to an exponential increase in training and inference costs. To alleviate this contradiction, knowledge distillation is widely used for model compression and capability transfer. However, most existing distillation techniques employ fixed strategies, such as fixed teacher model contributions, fixed modality fusion methods, or fixed training paths. When faced with the imbalance of multimodal data, the diversity of task requirements, and the dynamic changes in input complexity, these static designs are gradually revealing significant shortcomings.

[0003] Existing technologies typically employ a "fixed-weight multi-teacher distillation framework." This framework constructs multiple teacher models, simultaneously inputting their outputs or intermediate features into the distillation module, and then calculating the distillation loss of the student models using preset weights. Its multimodal processing methods mainly include:

[0004] Early fusion: Multimodal features are directly concatenated and then input into a unified model;

[0005] Late-stage fusion: Average the independent prediction results of each modality or use a fixed weighting;

[0006] Fixed distillation intensity: The model knowledge of each teacher is always involved in distillation at a fixed proportion during the training process.

[0007] The above scheme has a clear training structure and is easy to implement, but neither the distillation strategy nor the modality fusion strategy reflects dynamism and adaptability.

[0008] In practical applications, the importance of different modalities varies greatly across different scenarios. For example, in vision-dominated scenarios, textual information serves only as an aid; while in pure text tasks, visual modalities can even become distracting information. However, existing multimodal fusion structures often employ simple concatenation, linear weighting, or fixed gating strategies, failing to provide fine-grained modeling of cross-modal semantic relationships, resulting in coarse fusion effects. Furthermore, existing knowledge distillation schemes typically rely on manually preset weights or distillation strategies based on static sample features, unable to determine the reliability of teacher knowledge on the current samples in real time, thus affecting the learning efficiency of student models.

[0009] Furthermore, in the inference stage of large language models, all input data is often processed using a unified computation graph and a fixed number of inference layers, ignoring the significant differences in semantic complexity among different samples. This leads to a waste of computational resources on simple samples, while complex samples lack sufficient inference depth, resulting in performance bottlenecks. Existing technologies suffer from the following problems:

[0010] 1. The reliability of teachers' knowledge cannot be dynamically assessed.

[0011] Using fixed distillation weights means that the teacher model's knowledge will be forced to be learned regardless of whether the teacher model is "trustworthy" of the current sample, which may lead to noisy transfer or the injection of incorrect knowledge.

[0012] 2. Multimodal fusion methods lack structural expressive power.

[0013] Simple splicing or weighting methods only reflect the differences in intensity between modes, and cannot express deep semantic relationships such as "complementarity, conflict, and dependence". They are also difficult to handle complex problems such as unbalanced modal quality, missing modes, and modal noise.

[0014] 3. The computational costs of training and inference are uncontrollable.

[0015] Because the semantic complexity of different samples is not differentiated, all samples follow the same inference path, resulting in: wasted computational resources for simple samples; insufficient model capabilities for complex samples; and decreased overall inference efficiency.

[0016] Therefore, existing technologies have significant shortcomings in terms of the dynamism of knowledge transfer, the flexibility of multimodal fusion, and the allocation of reasoning resources, and new technical solutions are urgently needed.

[0017] Therefore, how to achieve joint optimization of "dynamic knowledge transfer + cross-modal structured fusion + adaptive scheduling of computing resources" has become a key technical problem that the industry urgently needs to solve. Summary of the Invention

[0018] To address the aforementioned shortcomings in existing technologies, this invention provides an adaptive knowledge transfer method based on semantic uncertainty measurement knowledge.

[0019] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0020] An adaptive knowledge transfer method based on semantic uncertainty measurement knowledge includes the following steps:

[0021] S1. Through an adaptive semantic consistency distillation mechanism, the alignment degree between the teacher model and the student model in the semantic space is calculated, and the distillation intensity is dynamically determined to optimize knowledge transfer.

[0022] S2. Through a multimodal self-tuning collaborative attention fusion mechanism, dynamic fusion of intermodal relationships is achieved based on the interdependent attention matrix and self-tuning coefficients to generate a unified representation;

[0023] S3. Through the adaptive computing resource allocation module, the inference path and computing resources are dynamically scheduled using the policy network and composite reward function to balance model latency, computational load and accuracy.

[0024] Furthermore, S1 includes the following steps:

[0025] S11. Select intermediate layer representations from the teacher model and student model as semantic vector sources, and configure a lightweight projector to project the intermediate layer representations onto a low-dimensional common semantic space.

[0026] S12. Construct a semantic consistency matrix based on the projected semantic vectors. ,in The degree of semantic alignment is quantified by calculating the outer product or dot product of normalized vectors.

[0027] S13. Calculate the semantic consistency matrix. With the identity matrix The deviation between them is used as semantic energy difference ,in , Denote the L1 norm, and for Exponential smoothing and range pruning are performed to enhance robustness;

[0028] S14. Based on the robust semantic energy difference Dynamically mapped distillation weights ,in Calculated using the softmax or sigmoid function, it is used to adjust the strength of knowledge transfer;

[0029] S15. Construct an adaptive semantic consistency distillation loss function Combining the distributed distillation loss and the representation alignment loss, where:

[0030]

[0031] In the formula, Output the combined distribution distillation loss for both teacher and student models. and Lightweight projectors for teachers and students respectively. The semantic vectors are projected for teachers and students, respectively.

[0032] Furthermore, S2 includes the following steps:

[0033] S21. Generate feature tensors for multimodal data through independent encoders and map them to a unified attention dimension, while supporting a masking mechanism for missing modalities.

[0034] S22. Calculate the intermodal dependency attention matrix A, where In the formula, For modal pairs The interdependence attention matrix, For modal pairs Generate feature tensors for synchronization time or synchronization semantics. This is the query and key weight matrix under the attention mechanism. This is the attention dimension, used to capture modal dependencies;

[0035] S23. Generate a collaborative response based on the interdependence attention matrix A, calculated as follows:

[0036]

[0037] In the formula, For the cooperative response of the feature tensor, This is the value weight matrix under the attention mechanism;

[0038] The merged vector is obtained by merging the residual connection with the original modal features. , represented as:

[0039]

[0040] For residual connections;

[0041] S24. Calculate the intermodal correlation by estimating the subnet through mutual information, and derive the self-adjustment coefficient. Perform weighted fusion;

[0042] S25. Based on the self-adjustment coefficient Generate the final fusion vector The task is adapted through a downstream network, and the final fusion vector is represented as follows:

[0043]

[0044] In the formula, For modality The self-adjustment coefficient.

[0045] Furthermore, S3 includes the following steps:

[0046] S31. Design a lightweight complexity feature extractor to quickly evaluate the complexity features of samples, including text length, visual scene density, and audio energy distribution.

[0047] S32. Construct a policy network to output the probability distribution of inference path selection. The action space includes a layer skipping mechanism and module activation control.

[0048] S33. Based on the sample complexity score, inference path selection is performed. A composite reward function R is used to balance immediate rewards and long-term rewards during training to optimize resource allocation strategies. The policy network is then trained through reinforcement learning. The composite reward function R is expressed as:

[0049]

[0050] In the formula, For reasoning delay, The number of floating-point operations. For model accuracy, This is the balance coefficient for the corresponding term.

[0051] Furthermore, the method also includes step S4 of performing multi-teacher and cross-modal extension, specifically:

[0052] S41. When multiple teacher models exist, calculate the semantic energy difference independently for each teacher. and distillation weight And perform normalized aggregation;

[0053] S42. Introduce a modal validity mask in cross-modal scenarios to dynamically shield distillation and fusion signals of irrelevant modes;

[0054] S43. Gradually adjust the distillation weights using a hyperparameter scheduling strategy. The weighting of the reward function is used to strengthen teacher guidance in the early stages of training and enhance student autonomy in the later stages.

[0055] Furthermore, the method also includes a training and inference optimization step S5, specifically:

[0056] S51. A phased training strategy is adopted, first optimizing the student model parameters, and then jointly fine-tuning the knowledge transfer and resource allocation modules.

[0057] S52. During the inference phase, redundant computational branches are dynamically skipped, and complete inference is performed only for high-complexity samples.

[0058] S53. Use gradient normalization to balance the backpropagation of multiple loss functions and avoid optimization bias.

[0059] The present invention has the following beneficial effects:

[0060] 1. Significantly improved training efficiency: The dynamic distillation mechanism avoids interference from irrelevant knowledge, enabling student models to learn key information more efficiently and accelerating model convergence.

[0061] 2. Model performance optimization: The adaptive multimodal fusion mechanism can handle the relationship between modes more finely, and is more robust, especially in scenarios with missing modes or conflicting information, thus improving the accuracy and generalization ability of the final model.

[0062] 3. Efficient utilization of computing resources: Reinforcement learning-based resource scheduling can achieve "on-demand allocation" during inference, significantly reducing average computing overhead, latency and energy consumption, making large models easier to deploy on edge devices.

[0063] 4. End-to-end optimization: This invention optimizes training and inference as a whole, forming a complete and efficient model production and deployment solution. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the knowledge transfer method of the present invention. Detailed Implementation

[0065] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0066] An adaptive knowledge transfer method based on semantic uncertainty measurement knowledge, such as Figure 1 As shown, it includes the following steps:

[0067] S1. Through an adaptive semantic consistency distillation mechanism, the alignment degree between the teacher model and the student model in the semantic space is calculated, and the distillation intensity is dynamically determined to optimize knowledge transfer.

[0068] In the knowledge transfer process, the teacher model output is not directly used as a guidance signal. Instead, the alignment degree between the teacher and student models in the semantic structure space is calculated through the semantic consistency matrix to determine the dynamic distillation intensity.

[0069] Unlike traditional dynamic weight adjustment based on sample similarity, this invention uses semantic energy difference as the driving force to achieve finer-grained knowledge transfer control.

[0070] The main steps include:

[0071] 1. Intermediate layer selection and lightweight projector configuration:

[0072] One or two intermediate representations from the teacher and student models (e.g., the output of the L / 2 layer of a transformer or the pooled output of a cross-modal unified encoder) are selected as sources of semantic vectors. A lightweight projection network P (e.g., two linear layers + LayerNorm + GELU) is designed to project the intermediate representations of the teacher and student models onto the same low-dimensional common semantic space (dimension d_s, typically 128–512) to reduce computational overhead and eliminate dimensionality inconsistency. This projector is updated or fixed during training (as a hyperparameter selection).

[0073] 2. Construction of the semantic consistency matrix M:

[0074] For a single sample, let Project semantic vectors for teachers and students respectively. (Teacher projects) (Students project the image). Calculate the normalized vector. Construct matrices using outer product or dot product methods. In terms of implementation, to save storage, the diagonal and principal components can be calculated directly, or a low-rank approximation (SVD-based) can be used to accelerate the process in large batches.

[0075] 3. Calculation and robustness of semantic energy difference ΔE:

[0076] use or Norm measurement With the identity matrix Deviation: To avoid numerical noise, a small constant ε is added and exponential smoothing (EMA) is applied to ΔE to reduce the drastic impact of intra-batch variance on λ. Optionally, interval clipping and normalization are performed on ΔE to ensure that λ varies within a reasonable range (e.g., restricting λ ∈ [0.05, 0.95]).

[0077] 4. Mapping strategy for dynamic distillation weight λ:

[0078] use Alternatively, a sigmoid mapping can be used, with the temperature τ (a hyperparameter) controlling the sensitivity, where... This is the balance coefficient. In a multi-teacher scenario, it is calculated separately for each teacher. And normalization was performed to obtain multiple teachers (softmax across teachers). A "minimum distillation intensity threshold" is retained in the implementation to prevent λ from dropping to 0 in the presence of extreme noise, causing students to completely lose teacher guidance.

[0079] 5. Distillation loss Construction and gradient flow:

[0080] The loss is calculated using a combination of distributed distillation loss (KL divergence between teacher logits and student logits, possibly preceded by temperature smoothing) and representation alignment loss (MSE between projected features). The loss weights are controlled by λ. In implementation, the scales of KL and MSE are normalized (e.g., scaled by batch-wise standard deviation) to avoid a dominant gradient. If the teacher network parameters are fixed (common), only the derivatives with respect to the student, projector, and several fusion layers need to be calculated; the projector can be set to trainable or frozen as needed for stable training.

[0081] If there are multiple teachers (T1, T2, ...), calculate separately for each teacher. and Finally, the losses of multiple teachers will be calculated according to... Weighted aggregation. To avoid conflicts between teachers, temperature or confidence adjustments can be made to teacher logits before aggregation (e.g., dynamic weighting based on teacher performance on the validation set). If teachers come from a specific modality (visual teachers focus on visual tasks), the ASCD projector must support cross-modal alignment (common semantic space) and, in conjunction with the SRCAF modality validity mask, avoid forced distillation of irrelevant modalities.

[0082] The training scheduling in this embodiment includes the following key parameters: projection dimension. Smoothing coefficient ,temperature , min / max The training strategy for the boundary, the base weights of KL and MSE, and the batch size is to first start training with a higher λ (more mimicking the teacher distribution), and then gradually increase the representation alignment ratio (decrease λ) to allow students to learn stronger internal representations; or to adopt periodic switching training (teacher-dominated period vs. representation alignment period).

[0083] The system includes a "semantic consistency evaluation submodule" for real-time assessment of the alignment between teachers and students in the feature space. During training, each sample generates feature representations using both teacher and student models, and the semantic consistency matrix M is computed in a lightweight projection space. The system obtains the semantic energy difference using a fast matrix energy difference evaluation algorithm (applying the L1 norm). The system dynamically feeds this information back into the distillation loss function. When the semantic structures of the teacher and student models are similar, the distillation weights automatically increase, strengthening knowledge transfer. When the semantics are inconsistent or the teacher's output contains noise, the system automatically weakens that part of the distillation signal to avoid overfitting or negative transfer. This mechanism achieves adaptive and interpretable knowledge distillation through dynamic adjustment of the semantic space, significantly improving the convergence speed and generalization stability of the student model.

[0084] S2. Through a multimodal self-tuning collaborative attention fusion mechanism, dynamic fusion of intermodal relationships is achieved based on the interdependent attention matrix and self-tuning coefficients to generate a unified representation;

[0085] Traditional multimodal fusion methods integrate information through simple weighting or gating functions, failing to reflect the complex dependencies between modalities. This invention proposes a multimodal self-adjusting collaborative attention fusion mechanism that achieves dynamic collaboration and weight self-adjustment between modalities through modal interdependence attention and self-adjustment layer-leveling, ensuring the semantic integrity and adaptability of information fusion.

[0086] Each modality (text, visual, audio) generates a feature tensor that is synchronized at the temporal or semantic scale through an independent encoder (e.g., text-transformer, visual-backbone+transformer, audio-conv+transformer). , , To facilitate attention calculation, a linear layer is used. , , Map each modality to a unified attention dimension d. Define a modality mask (0 for missing modalities) and mask the contribution of missing modalities during attention computation. These are the query weight matrix, key weight matrix, and value weight matrix under the attention mechanism, respectively.

[0087] For each pair of modes ,calculate , For modal pairs Generate feature tensors for synchronization time or synchronization semantics. In implementation, considering batch and parallel efficiency, all symmetric matrices are computed simultaneously followed by softmax. To control computational cost, sparse attention or local attention mechanisms are supported (e.g., interacting only with the top-k key tokens of each modality), adapting to long sequence scenarios. Subsequent computations... Connect it to the original modal features via residuals. Merge to obtain the merged vector , This represents the collaborative response of the feature tensors. Here, the residuals help preserve the modal characteristics and facilitate gradient propagation. Optionally, for... Add a learnable scaling factor (Initial value 1), the magnitude of the co-response can be limited in the early stages of training to prevent gradient explosion.

[0088] Design a differentiable mutual information estimation subnet. (For example, using InfoNCE loss based on contrastive learning or MINE-like networks) to estimate the correlation between original and co-modal features. In practice, a small MLP is used for splicing. It outputs relevant scores and uses them for training in conjunction with contrast tasks or negative sample sampling. It also calculates modalities. Self-adjustment coefficient To stabilize the value, a small constant can be added to the denominator and adjusted accordingly. Temperature scaling is applied. If a mode is missing, its I value is set to a minimum to automatically reduce the weight of that mode. The final fusion vector is generated and adapted downstream. Then, task adaptation (classification head or decoder) is performed by fusing subsequent networks (such as cross-modal transformer layers or MLPs).

[0089] In multi-task scenarios, task-specific fusion heads can be retained for different tasks (sharing the underlying SRCAF, but the output layer can be differentiated). During training, noise or occlusion can be randomly applied to some modalities, and the stability of SRCAF output can be supervised (by adding a robust loss term). Through negative sample comparison and cluster regularization, the mutual information estimator can be prevented from failing due to overfitting to a single modality pair.

[0090] In the actual training process, the mutual information estimator can be frozen in the initial stage, and the interdependent attention layer and cooperative response generator can be trained first. After they are basically stable, they can be trained together. Different learning rates are applied to the attention layer / mutual information estimator / fusion head (generally, the learning rate is smaller for the mutual information estimator).

[0091] S3. Through the adaptive computing resource allocation module, the inference path and computing resources are dynamically scheduled using the policy network and composite reward function to balance model latency, computational load and accuracy.

[0092] This embodiment designs a lightweight convolutional / MLP network to quickly extract sample complexity features (such as text length, visual scene density estimation, audio time-frequency energy distribution, etc.) before inference, outputting a low-dimensional state vector s. This extractor must operate under low latency conditions (e.g., within milliseconds). To improve stability, the extractor can output an additional uncertainty estimate for the policy network to reference during decision-making (conservative policies tend to favor more computation under high uncertainty).

[0093] The policy network can use a small transformer or MLP to output the action probability distribution π(a|s). Examples of action 'a' definitions include: skipping some layers from layers 3–6, reducing the head count of certain submodules, and choosing whether to enable fine-grained SRCAF cross-modal interaction. The action space needs to be discretized and constrained by system capabilities. To reduce the search space, actions can be defined as indices of a "preset set of computational paths" (e.g., Path A: full-depth, Path B: skip every-2 layers, Path C: shallow-only), facilitating the determination of feasible paths when deploying a real system.

[0094] The reward uses a composite function: ,in This is the balance coefficient.

[0095] During training, it is necessary to balance immediate rewards with long-term benefits (e.g., considering cumulative latency in continuous streaming scenarios), which can be achieved by introducing a discount factor. Using Pseudo-rewards: When accuracy cannot be evaluated online (e.g., task metrics cannot be calculated in real time), surrogate rewards (such as the similarity between student and teacher outputs) can be used as an approximation.

[0096] In practical reinforcement learning, batch-based policy optimization (PPO) is employed: N sample decision trajectories (state, action, reward) are collected in each training cycle, the advantage estimate (GAE) is calculated, and the policy network is optimized multiple times. To stabilize training, an entropy regularization term is introduced to promote policy exploration, and a normalized reward batch is used for standardization. To avoid the policy from sacrificing accuracy for low computational cost, a lower limit on accuracy can be set, or a large negative penalty can be imposed on accuracy drops exceeding the threshold. During inference, the policy network outputs action 'a', controlling the activation or skipping of predefined layers / modules by the model execution engine. A "safe fallback mechanism" must be designed: if an anomaly occurs on the execution path (e.g., skipping layers leads to excessively low output confidence), the system automatically falls back to a more conservative path and logs the information for online policy fine-tuning.

[0097] The entire system in this embodiment forms a closed-loop training-inference co-optimization structure, and the specific process is as follows:

[0098] Text / visual / audio encoders can be pre-trained or fine-tuned to baseline performance on their respective modality tasks or pre-training tasks, or publicly available pre-trained models can be reused to save resources.

[0099] By fixing or slightly fine-tuning the modal encoder, the dependent attention layer and collaborative response generator of SRCAF are trained to achieve the desired quality of the fused representation on data containing all modality combinations. Multimodal task losses (classification / retrieval / alignment) are used to guide training, and modality missing robustness is monitored. A contrastive learning task for the mutual information estimator is introduced to optimize computational stability.

[0100] The student model (which can be designed as a lightweight variant) is trained using ASCD distillation with the teacher pool and the already trained SRCAF. During training, joint minimization is performed. ,in, For the total loss, For knowledge distillation loss, For multimodal fusion loss, Multimodal fusion loss weights For the loss of downstream tasks, The target downstream task loss weights are determined. A course learning strategy is adopted, first distilling on simple samples and then gradually introducing complex samples, which helps to stabilize the projection alignment.

[0101] The policy network (PPO) is trained on the basis of the trained student model as the environment. Latency / FLOPs / accuracy data are collected using batch simulation inference as rewards. Online noise is introduced into the simulation environment to improve policy robustness, such as random network latency and node failure simulation.

[0102] Joint fine-tuning (ASCD, SRCAF, student agent and policy network joint) is performed to repair boundary conditions and make the system end-to-end optimal. A hierarchical test set containing modality missing, strong noise and high complexity samples is designed to evaluate comprehensive indicators, such as latency-accuracy curve, robustness and resource saving ratio, as shown in Tables 1, 2 and 3 below.

[0103] Table 1. Delay-Accuracy Curve Parameter Table

[0104]

[0105] Table 2 Robustness Parameter Table

[0106]

[0107] Table 3 Resource Saving Ratio Parameter Table

[0108]

[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0113] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An adaptive knowledge transfer method based on semantic uncertainty measurement knowledge, characterized in that, Includes the following steps: S1. Through an adaptive semantic consistency distillation mechanism, the alignment degree between the teacher model and the student model in the semantic space is calculated, and the distillation intensity is dynamically determined to optimize knowledge transfer. S2. Through a multimodal self-tuning collaborative attention fusion mechanism, dynamic fusion of intermodal relationships is achieved based on the interdependent attention matrix and self-tuning coefficients to generate a unified representation. This includes the following steps: S21. Generate feature tensors for multimodal data through independent encoders and map them to a unified attention dimension, while supporting a masking mechanism for missing modalities. S22. Calculate the intermodal dependency attention matrix A, where In the formula, For modal pairs The interdependence attention matrix, For modal pairs Generate feature tensors for synchronization time or synchronization semantics. This is the query and key weight matrix under the attention mechanism. This is the attention dimension, used to capture modal dependencies; S23. Generate a collaborative response based on the interdependence attention matrix A, calculated as follows: In the formula, For the cooperative response of the feature tensor, This is the value weight matrix under the attention mechanism; The merged vector is obtained by merging the residual connection with the original modal features. , represented as: For residual connections; S24. Calculate the intermodal correlation by estimating the subnet through mutual information, and derive the self-adjustment coefficient. Perform weighted fusion; S25. Based on the self-adjustment coefficient Generate the final fusion vector The task is adapted through a downstream network, and the final fusion vector is represented as follows: In the formula, For modality The self-adjustment coefficient; S3. Through the adaptive computing resource allocation module, the inference path and computing resources are dynamically scheduled using the policy network and composite reward function to balance model latency, computational load and accuracy. This includes the following steps: S31. Design a lightweight complexity feature extractor to quickly evaluate the complexity features of samples, including text length, visual scene density, and audio energy distribution. S32. Construct a policy network to output the probability distribution of inference path selection. The action space includes a layer skipping mechanism and module activation control. S33. Based on the sample complexity score, inference path selection is performed. A composite reward function R is used to balance immediate rewards and long-term rewards during training to optimize resource allocation strategies. The policy network is then trained through reinforcement learning. The composite reward function R is expressed as: In the formula, For reasoning delay, The number of floating-point operations. For model accuracy, This is the balance coefficient for the corresponding term.

2. The adaptive knowledge transfer method based on semantic uncertainty measurement knowledge according to claim 1, characterized in that, S1 includes the following steps: S11. Select intermediate layer representations from the teacher model and student model as semantic vector sources, and configure a lightweight projector to project the intermediate layer representations onto a low-dimensional common semantic space. S12. Construct a semantic consistency matrix based on the projected semantic vectors. ,in The degree of semantic alignment is quantified by calculating the outer product or dot product of normalized vectors. S13. Calculate the semantic consistency matrix. With the identity matrix The deviation between them is used as semantic energy difference ,in , Denote the L1 norm, and for Exponential smoothing and range pruning are performed to enhance robustness; S14. Based on the robust semantic energy difference Dynamically mapped distillation weights ,in Calculated using the softmax or sigmoid function, it is used to adjust the strength of knowledge transfer; S15. Construct an adaptive semantic consistency distillation loss function Combining the distributed distillation loss and the representation alignment loss, where: In the formula, Output the combined distribution distillation loss for both teacher and student models. and Lightweight projectors for teachers and students respectively. The semantic vectors are projected for teachers and students, respectively.

3. The adaptive knowledge transfer method based on semantic uncertainty measurement knowledge according to claim 1, characterized in that, The method further includes step S4, which involves multi-teacher and cross-modal extension, specifically: S41. When multiple teacher models exist, calculate the semantic energy difference independently for each teacher. and distillation weight And perform normalized aggregation; S42. Introduce a modal validity mask in cross-modal scenarios to dynamically shield distillation and fusion signals of irrelevant modes; S43. Gradually adjust the distillation weights using a hyperparameter scheduling strategy. The weighting of the reward function is used to strengthen teacher guidance in the early stages of training and enhance student autonomy in the later stages.

4. The adaptive knowledge transfer method based on semantic uncertainty measurement knowledge according to claim 1, characterized in that, The method further includes a training and inference optimization step S5, specifically: S51. A phased training strategy is adopted, first optimizing the student model parameters, and then jointly fine-tuning the knowledge transfer and resource allocation modules. S52. During the inference phase, redundant computational branches are dynamically skipped, and complete inference is performed only for high-complexity samples. S53. Use gradient normalization to balance the backpropagation of multiple loss functions and avoid optimization bias.

Images