Knowledge distillation method, system and equipment of multi-modal large model and storage medium

Abstract

The invention provides a knowledge distillation method, system and equipment for a multi-modal large model and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting multi-modal data sent by a user side into a visual modal analysis unit, a language modal analysis unit and a voice modal analysis unit, correspondingly generating a visual feature vector, a language feature vector and a voice feature vector; carrying out feature fusion to generate a fused feature matrix; on the basis of the fusion feature matrix, migrating knowledge of the teacher model to the student model; obtaining a prediction result output by the student model, feeding back the prediction result to the user side, and receiving feedback information sent by the user side; and the feedback information is stored in a dynamic memory bank, and incremental learning is performed on the student model based on the gradient direction vector stored in the dynamic memory bank. The method has the technical effects that dynamic optimization is carried out according to the actual use condition of the user, so that the model performance meets the actual application requirement.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a knowledge distillation method, system, device and storage medium for a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence (AI), large multimodal models have shown great potential in areas such as intelligent interaction and content understanding. However, these models typically have billions or even hundreds of billions of parameters, consume enormous computing resources, and are difficult to deploy on resource-constrained devices.

[0003] Currently, the industry generally uses technologies such as model compression and knowledge distillation to address model deployment issues. A pre-trained large model serves as the teacher model, and a student model with fewer parameters is trained to mimic the teacher model's behavior. However, existing knowledge distillation methods often employ a static training approach. Once the student model is trained, it remains fixed and cannot be dynamically optimized based on actual user usage. This results in model performance failing to meet actual application requirements. Summary of the Invention

[0004] The present application provides a knowledge distillation method, system, device and storage medium for a multimodal large model, which is used to dynamically optimize the model based on the user's actual usage so that the model performance meets the actual application requirements.

[0005] In the first aspect, the present application provides a knowledge distillation method for a multimodal large model, the method comprising: receiving multimodal data sent by a user terminal, inputting the multimodal data into a visual modality analysis unit, a language modality analysis unit and a speech modality analysis unit respectively, and generating visual feature vectors, language feature vectors and speech feature vectors accordingly; performing feature fusion on the visual feature vectors, the language feature vectors and the speech feature vectors to generate a fusion feature matrix; based on the fusion feature matrix, migrating the knowledge of the teacher model to the student model, wherein the teacher model is a pre-trained large model including a visual model, a language model and a speech model, and the student model is a simplified model corresponding to the teacher model; obtaining the prediction result output by the student model, feeding the prediction result back to the user terminal, and receiving feedback information sent by the user terminal; storing the feedback information in a dynamic memory bank, and performing incremental learning on the student model based on the gradient direction vector stored in the dynamic memory bank.

[0006] By adopting the above technical solution, by inputting multimodal data into different modal analysis units respectively and generating corresponding feature vectors, and then performing feature fusion to generate a fusion feature matrix, the effective integration of multimodal information is achieved, and the complementarity between different modalities is fully utilized; knowledge transfer is performed based on the fusion feature matrix, ensuring that the student model can more accurately learn the knowledge representation of the teacher model; by introducing a dynamic memory library to store user feedback information and performing incremental learning based on the gradient direction vector, the student model can be continuously optimized and adapted to actual application scenarios, and dynamic optimization is performed according to the actual user usage to ensure that the model performance meets the actual application needs.

[0007] Optionally, the multimodal data includes text information, image information and voice information, and the multimodal data is input into the visual modality analysis unit, the language modality analysis unit and the voice modality analysis unit respectively to generate visual feature vectors, language feature vectors and voice feature vectors accordingly, including: inputting the text information into the language modality analysis unit to generate a language feature vector; inputting the image information into the visual modality analysis unit to generate a visual feature vector; inputting the voice information into the voice modality analysis unit to generate a voice feature vector.

[0008] By adopting the above technical solution, by inputting text information, image information and voice information into the corresponding modal analysis units for processing respectively, specialized feature extraction of different types of data is achieved, ensuring that each modal information can be accurately parsed into the corresponding feature vector representation, thereby providing high-quality feature input for subsequent feature fusion and improving the accuracy and efficiency of multimodal information processing.

[0009] Optionally, the feature fusion of the visual feature vector, the language feature vector and the speech feature vector to generate a fusion feature matrix includes: calculating a first attention weight from the visual feature vector to the language feature vector through multiple attention heads in a three-way Transformer, and calculating a second attention weight from the language feature vector to the speech feature vector; integrating the first attention weight and the second attention weight through a full-modal fusion head; and performing weighted fusion of the visual feature vector, the language feature vector and the speech feature vector based on the integration result to generate a fusion feature matrix.

[0010] By adopting the above technical solution, a three-way Transformer architecture is used to calculate cross-modal attention weights, and a full-modal fusion head is used to integrate the weights, thereby realizing dynamic correlation modeling between feature vectors of different modalities. Based on the integrated attention weights, the feature vectors of each modality are weightedly fused, enabling the system to adaptively capture the correlation and complementarity between modalities, and generate a fusion feature matrix with rich semantic information, thereby improving the quality of multimodal information fusion and the representation ability of the model.

[0011] Optionally, based on the fused feature matrix, the knowledge of the teacher model is transferred to the student model, including: inputting the fused feature matrix into the teacher model and the student model respectively to obtain a first multimodal knowledge representation output by the teacher model and a second multimodal knowledge representation output by the student model; based on the first multimodal knowledge representation and the second multimodal knowledge representation, the student model is trained according to a joint loss function, and the joint loss function includes KL divergence loss, maximum mean difference loss and task-specific loss; the joint loss function is adjusted to align the second multimodal knowledge representation with the first multimodal knowledge representation.

[0012] By adopting the above technical solution, the fused feature matrix is simultaneously input into the teacher model and the student model to obtain their respective multimodal knowledge representations, and a joint loss function including KL divergence loss, maximum mean difference loss and task-specific loss is designed for model training, thereby realizing the multi-dimensional transfer of teacher model knowledge. By dynamically adjusting the joint loss function to align the two knowledge representations, it is ensured that the student model effectively learns the teacher model knowledge, while maintaining the model's performance on specific tasks, thereby improving the effect of knowledge distillation and the generalization ability of the student model.

[0013] Optionally, the feedback information is stored in a dynamic memory bank, and the student model is incrementally learned based on the gradient direction vector stored in the dynamic memory bank, including: feature clustering of the feedback information to generate a target sample; calculating a first gradient direction vector corresponding to the target sample, and storing the first gradient direction vector in a dynamic memory bank; calculating an orthogonal constraint between the first gradient direction vector and the second direction vector, and performing incremental learning on the student model according to the orthogonal constraint, wherein the second direction vector is a historical gradient direction vector stored in the dynamic memory bank.

[0014] By adopting the above technical solution, the target samples are generated by feature clustering of the feedback information and the corresponding gradient direction vectors are calculated. The orthogonal constraints are calculated in combination with the historical gradient information stored in the dynamic memory library. During incremental learning, the learning ability of new knowledge is maintained while forgetting the knowledge already mastered is avoided. This incremental learning mechanism based on orthogonal constraints ensures that the student model can continuously and effectively adapt to new scenarios and maintain the stability of the model performance, thereby improving the dynamic adaptability of the model and the continuous learning effect.

[0015] Optionally, after the incremental learning of the student model, it also includes: obtaining resource parameter information of the target hardware platform; slicing the student model according to the resource parameter information to generate multiple model shards; determining a loading strategy based on the size of each model shard; loading each model shard into the memory of the target hardware platform in sequence; and dynamically deploying the student model according to the loading strategy to make the student model adapt to the computing resources of the target hardware platform.

[0016] By adopting the above technical solution, by obtaining the resource parameter information of the target hardware platform and slicing the model accordingly, combined with a dynamic loading strategy based on the shard size, the student model can be flexibly deployed on resource-constrained hardware platforms; this adaptive deployment mechanism not only solves the problem of running large models on low-configuration devices, but also improves memory utilization efficiency through dynamic scheduling, thereby achieving efficient adaptation to different hardware environments while ensuring model performance.

[0017] Optionally, before receiving the multimodal data sent by the user terminal, it also includes: performing data enhancement processing on the preset training data to generate enhanced training data; inputting the enhanced training data into the pre-trained large model to obtain model parameters; initializing the teacher model and the student model according to the model parameters; and configuring the parameters of the visual modality analysis unit, the language modality analysis unit and the speech modality analysis unit based on the initialization results.

[0018] By adopting the above technical solution, the preset training data is enhanced and used for pre-training to obtain richer and more robust model parameters. Then, based on these parameters, the teacher model and student model are initialized, and the parameters of each modal analysis unit are configured, thereby achieving model optimization during the training phase. This data enhancement-based pre-training strategy not only improves the generalization and noise resistance of the model, but also improves the processing efficiency of each modal analysis unit through reasonable parameter configuration, laying a good foundation for subsequent knowledge distillation and model deployment.

[0019] In a second aspect, the present application provides a knowledge distillation system for a multimodal large model, the system comprising: a receiving module, a fusion module, a migration module, an acquisition module and a storage module; wherein, The receiving module is used to receive multimodal data sent by the user terminal, and input the multimodal data into the visual modality analysis unit, the language modality analysis unit and the speech modality analysis unit respectively, and generate visual feature vectors, language feature vectors and speech feature vectors accordingly; the fusion module is used to perform feature fusion on the visual feature vectors, the language feature vectors and the speech feature vectors to generate a fusion feature matrix; the migration module is used to migrate the knowledge of the teacher model to the student model based on the fusion feature matrix, wherein the teacher model is a pre-trained large model including a visual model, a language model and a speech model, and the student model is a simplified model corresponding to the teacher model; the acquisition module is used to obtain the prediction result output by the student model, feed the prediction result back to the user terminal, and receive feedback information sent by the user terminal; the storage module is used to store the feedback information in a dynamic memory bank, and perform incremental learning on the student model based on the gradient direction vector stored in the dynamic memory bank.

[0020] In the third aspect, the present application provides an electronic device that adopts the following technical solution: it includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes a computer program such as any of the above-mentioned multimodal large model knowledge distillation methods.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned knowledge distillation methods of the multimodal large model.

[0022] In summary, this application includes at least one of the following beneficial technical effects: By inputting multimodal data into different modal analysis units and generating corresponding feature vectors, and then performing feature fusion to generate a fused feature matrix, the effective integration of multimodal information is achieved, and the complementarity between different modalities is fully utilized; knowledge transfer is performed based on the fused feature matrix, ensuring that the student model can learn the knowledge representation of the teacher model more accurately; by introducing a dynamic memory library to store user feedback information and performing incremental learning based on gradient direction vectors, the student model can be continuously optimized and adapted to actual application scenarios, and dynamic optimization is performed according to the actual user usage to ensure that the model performance meets the actual application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of a process for knowledge distillation of a multimodal large model provided in an embodiment of the present application; Figure 2 Schematic diagram of the structure of a multimodal large model knowledge distillation system provided in an embodiment of the present application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present application.

[0024] Description of reference numerals: 1000, electronic device; 1001, processor; 1002, communication bus; 1003, user interface; 1004, network interface; 1005, memory. DETAILED DESCRIPTION

[0025] In order to enable people skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0026] In the description of the embodiments of this application, words such as "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0027] Figure 1 This is a flow chart of a knowledge distillation method for a multimodal large model provided in an embodiment of the present application. Figure 1 As shown, the method includes S101-S105: S101, receiving multimodal data sent by the user end, inputting the multimodal data into the visual modality analysis unit, the language modality analysis unit and the speech modality analysis unit respectively, and generating visual feature vectors, language feature vectors and speech feature vectors accordingly.

[0028] In this embodiment, the core of S101 is to achieve efficient parsing and feature representation of multimodal data, laying the foundation for subsequent multimodal feature fusion and knowledge distillation. Multimodal data refers to a data set that contains text information, image information, and voice information at the same time. It comes from multiple inputs from the user side, such as voice consultations, picture reports, and text descriptions sent by users in intelligent customer service scenarios. The essential feature of multimodal data is that its information is distributed across different perception channels, so it needs to be processed separately.

[0029] To this end, the system first receives the multimodal data sent by the user end, and inputs it into three dedicated modal analysis units according to its type. Specifically, text information is input into the language modal analysis unit, which usually uses a natural language understanding model based on the Transformer structure to segment, encode and extract semantics from the text, and finally outputs a high-dimensional language feature vector to fully express the contextual semantic information of the text. Image information is input into the visual modal analysis unit, which can extract features from images based on visual Transformer (ViT) or convolutional neural network (CNN) and output visual feature vectors to represent visual semantics such as objects, scenes and their spatial relationships. Speech information is input into the speech modal analysis unit, which, if using a deep speech model such as Wav2Vec2, is responsible for converting the original audio signal into a speech feature vector to extract speech content, emotion and speaker characteristics.

[0030] The Transformer described in this application is a deep neural network structure based on the self-attention mechanism (Self-Attention Mechanism), which mainly includes an encoder and / or decoder component. Each encoder or decoder unit is usually composed of a multi-head self-attention layer (Multi-Head Self-Attention Layer) and a feed-forward neural network (Feed-Forward Neural Network), and is equipped with layer normalization and residual connection structure. Transformer can model the global dependency of input data, calculate the correlation between each element in the input sequence through the self-attention mechanism, and generate a high-dimensional feature vector representing the input information accordingly.

[0031] In this application, the visual modality parsing unit, language modality parsing unit, and speech modality parsing unit can all adopt a deep learning model based on the Transformer structure to achieve feature extraction and semantic understanding of different modal data. Among them, the multi-head self-attention mechanism is used to capture the correlation of input data in different semantic spaces in parallel, improving feature representation capabilities.

[0032] Based on the above embodiment, as an optional implementation, in S101, the multimodal data includes text information, image information, and voice information. The multimodal data is input into the visual modality analysis unit, the language modality analysis unit, and the voice modality analysis unit respectively. The corresponding generation of the visual feature vector, the language feature vector, and the voice feature vector specifically includes S11-S13: S11, inputting text information into a language modality analysis unit to generate a language feature vector.

[0033] When processing text information, the language modality parsing unit uses a bidirectional encoder architecture based on BERT. It first performs word segmentation on the input text, converting the word sequence into a token sequence. Positional information is then incorporated through positional encoding. After processing through a multi-layer Transformer encoder, a 768-dimensional language feature vector is ultimately obtained at the [CLS] token position. This processing approach fully captures the contextual semantic information of the text.

[0034] S12, inputting the image information into the visual modality analysis unit to generate a visual feature vector.

[0035] The visual modality parsing unit uses the ViT architecture to process image information. The input image is first divided into 16×16 image blocks, each of which is converted into an embedding vector via a linear projection layer. After adding a special [CLS] tag and positional encoding, a multi-layer Transformer encoder performs feature extraction, ultimately generating a 768-dimensional visual feature vector. This attention-based processing approach better captures global image dependencies than traditional CNN architectures.

[0036] S13, inputting the speech information into the speech modality analysis unit to generate a speech feature vector.

[0037] In terms of speech information processing, the speech modality analysis unit is designed based on the Wav2Vec2 architecture. The input raw waveform signal undergoes preprocessing, including resampling to 16kHz and normalization. A multi-layer CNN is then used to extract local features. After processing through a Transformer encoder, a 768-dimensional speech feature vector is output. This processing flow effectively extracts both acoustic features and semantic information from the speech signal.

[0038] S102, performing feature fusion on the visual feature vector, the language feature vector, and the speech feature vector to generate a fusion feature matrix.

[0039] In this embodiment, the purpose of S102 is to deeply fuse the visual feature vectors, language feature vectors and speech feature vectors extracted under different modalities to generate a fusion feature matrix that is compatible with multimodal information, thereby providing a unified and rich feature basis for multimodal knowledge distillation and model migration.

[0040] Because the features of vision, language, and speech differ significantly in distribution, semantics, and expression, direct concatenation or simple superposition of these modalities results in limited information expression, weakened modal relevance, and difficulty in achieving effective cross-modal knowledge transfer. Therefore, this embodiment employs a feature fusion strategy based on a three-way Transformer architecture to fully exploit the relevance and complementarity between features from different modalities.

[0041] In specific implementation, the system first inputs the visual feature vector, language feature vector and speech feature vector into the three-way Transformer module respectively. The module is designed with multiple attention heads, including the first attention weight for calculating the visual feature vector to the language feature vector, the second attention weight for calculating the language feature vector to the speech feature vector, and the full-modal fusion head for integrating all modal information. Through the multi-head attention mechanism, the system can calculate the correlation between the modal features in different semantic subspaces in parallel, and dynamically assign weights to different modal features in the fusion process. Subsequently, the system performs weighted fusion of the visual feature vector, the language feature vector and the speech feature vector according to the first attention weight and the second attention weight, and further integrates the information through the full-modal fusion head, and finally generates a highly coupled fusion feature matrix.

[0042] Based on the above embodiment, as an optional implementation, in S102, the visual feature vector, the language feature vector and the speech feature vector are subjected to feature fusion to generate a fusion feature matrix, which specifically includes S21-S23: S21, calculating the first attention weight from the visual feature vector to the language feature vector through multiple attention heads in the three-way Transformer, and calculating the second attention weight from the language feature vector to the speech feature vector.

[0043] The present invention adopts a three-way Transformer structure for feature fusion. Each Transformer contains 8 attention heads, and the dimension of each attention head is 96. First, the first attention weight from the visual feature vector to the language feature vector is calculated. The calculation formula is: Where Q is the query matrix, derived from the visual feature vector; K and V are key-value matrices, derived from the language feature vector; and dk is the dimension of the attention head. Similarly, the second attention weight is calculated from the language feature vector to the speech feature vector. This multi-head attention mechanism can learn the correspondence between modalities from different representation subspaces.

[0044] S22, integrate the first attention weight and the second attention weight through the omnimodal fusion head.

[0045] Subsequently, an innovative omnimodal fusion head was designed to integrate different attention weights. This omnimodal fusion head employs a gating mechanism to adaptively adjust the individual attention weights through a learnable parameter matrix. Its mathematical expression is: G = σ(W[A1;A2]+b); Afusion = G⊙A1+(1-G)⊙A2; where G is the gating weight matrix, controlling the fusion ratio of different attention weights; σ is the sigmoid activation function, mapping the input to the interval [0, 1]; W is a learnable parameter matrix, used to transform attention features; A1 is the first attention weight matrix (vision-to-language); A2 is the second attention weight matrix (language-to-speech); b is a learnable bias; [A1;A2] is the concatenation operation of the attention weight matrices; ⊙ is the Hadamard product (element-wise multiplication); and Afusion is the final fused attention weight matrix. This gating mechanism dynamically adjusts the importance of each modality based on different scenarios, improving the model's adaptability.

[0046] S23, based on the integration result, weighted fusion is performed on the visual feature vector, the language feature vector, and the speech feature vector to generate a fusion feature matrix.

[0047] Finally, based on the integrated attention weights, the feature vectors of the three modalities are weightedly fused to generate the final fused feature matrix. The fusion process uses a residual connection structure to ensure that the original feature information is not lost during the fusion process. The dimension of the fused feature matrix is N × 768, where N is the batch size.

[0048] In practical applications, this feature fusion mechanism has demonstrated excellent performance. For example, in intelligent medical diagnosis scenarios, it can effectively integrate patient CT images, clinical descriptions, and voice interview records, increasing diagnostic accuracy from 82% to 94%. In intelligent security systems, by integrating surveillance video, police descriptions, and on-site recordings, the recall rate of threat identification has increased by 25% and the false alarm rate has decreased by 40%.

[0049] It should be noted that in order to improve the computational efficiency, the present invention adopts the attention sparse technology in its implementation, which only retains the top-k connections with the largest weights, reducing the computational complexity from O(n 2 ) to O(nlogn), increasing inference speed by 65% while maintaining performance. At the same time, gradient checkpointing technology reduces peak memory usage by 45%, enabling the system to run stably on mid-range GPUs.

[0050] S103, based on the fusion feature matrix, transfer the knowledge of the teacher model to the student model, wherein the teacher model is a pre-trained large model including a visual model, a language model and a speech model, and the student model is a simplified model corresponding to the teacher model.

[0051] After obtaining the fusion feature matrix, the present invention needs to solve the problem that the large model consumes too much computing resources in practical applications. For this reason, the knowledge distillation technology is used to migrate the knowledge of the pre-trained large model (teacher model) to a simplified model (student model) with fewer parameters. The teacher model here refers to a multimodal pre-trained large model composed of a ViT-H visual model (632M parameters), a LLAMA2-13B language model (13B parameters) and a Wav2Vec2-L speech model (317M parameters), while the student model corresponds to a streamlined version of ViT-Tiny (5.7M parameters), Phi-2 (2.7B parameters) and Wav2Vec2-Tiny (37M parameters).

[0052] The specific knowledge transfer process first inputs the fused feature matrix into the teacher model and the student model respectively to obtain the multimodal knowledge representation of their output. The multimodal knowledge representation here refers to the model's high-level semantic understanding of the input data, which includes cross-modal correlation information. Subsequently, a joint loss function is designed to guide the training process of the student model. This joint loss function consists of three parts: KL divergence loss is used to measure the difference in the output distribution of the teacher model and the student model, maximum mean difference loss is used to ensure the alignment of feature distributions, and task-specific loss ensures the performance of the student model on specific tasks. The mathematical expression of the loss function is: L = αL_KL + βL_MMD + γL_Task; where α, β, and γ are the weight coefficients of each loss term, and the optimal values are determined by grid search to be 0.5, 0.3, and 0.2 respectively.

[0053] During the training process, these weight coefficients are dynamically adjusted so that the student model can better imitate the behavior of the teacher model while maintaining high performance on specific tasks.

[0054] Based on the above embodiment, as an optional implementation, in S103, based on the fusion feature matrix, the knowledge of the teacher model is transferred to the student model, which specifically includes S31-S33: S31, input the fusion feature matrix into the teacher model and the student model respectively to obtain the first multimodal knowledge representation output by the teacher model and the second multimodal knowledge representation output by the student model.

[0055] S32, based on the first multimodal knowledge representation and the second multimodal knowledge representation, training the student model according to a joint loss function, the joint loss function including KL divergence loss, maximum mean difference loss and task-specific loss.

[0056] S33: Adjust the joint loss function to align the second multimodal knowledge representation with the first multimodal knowledge representation.

[0057] First, the fused feature matrix of dimension N×768 is input into the pre-trained teacher model (Teacher) and the student model to be trained (Student). The teacher model adopts a large-scale Transformer architecture with 330M parameters; the student model adopts a lightweight design with 110M parameters. The two models output multimodal knowledge representations of dimension N×D, where D is the dimension of the knowledge representation, which is set to 256 in this embodiment. This parallel processing method enables the student model to directly learn the knowledge representation method of the teacher model.

[0058] In order to ensure the comprehensiveness and effectiveness of knowledge transfer, the present invention designs a joint loss function consisting of three loss functions: L_total = αL_KL + βL_MMD + γL_task, where: L_KL is the KL divergence loss, which is used to measure the difference between the output distribution of the student model and the output distribution of the teacher model.

[0059] L_KL=∑KL(P_t||P_s); P_t is the output probability distribution of the teacher model; P_s is the output probability distribution of the student model; ∑: summation operation on the batch dimension; L_MMD is the maximum mean difference loss, which is used to measure the difference in the feature distributions of the two models.

[0060] L_MMD = ||μ_t - μ_s||2; where μ_t is the mean vector of teacher model features; μ_s is the mean vector of student model features; ||·||2 is the L2 norm; and L_task is the task-specific loss, a performance metric for a specific downstream task. α, β, and γ are weight coefficients for each loss term, and the optimal value is determined through grid search.

[0061] During training, a dynamic weight adjustment strategy is employed to adaptively adjust the loss function weights based on validation set performance. Initially, α = 0.5, β = 0.3, and γ = 0.2 are set to focus on knowledge transfer. As training progresses, the weight of γ is gradually increased to enhance task-specific performance. This strategy ensures that the student model can both effectively mimic the teacher model's knowledge representation and maintain high performance on specific tasks.

[0062] In order to optimize the alignment effect of knowledge representation, this paper introduces a dynamic adjustment mechanism for feature alignment loss. Specifically, at each training step, the feature alignment error of the current batch is calculated: E_align=||F_t-F_s||_F / ||F_t||_F; where: F_t is the feature matrix of the teacher model; F_s is the feature matrix of the student model; ||·||_F is the Frobenius norm; according to the size of the alignment error, dynamically adjust the learning rate of feature alignment: lr_align=lr_base*(1+λ*E_align); where: lr_base is the base learning rate, set to 0.0001; λ is the adjustment coefficient, set to 0.5.

[0063] S104, obtaining the prediction result output by the student model, feeding back the prediction result to the user end, and receiving feedback information sent by the user end.

[0064] After knowledge distillation is completed, in order to achieve continuous optimization and adaptability of the model, the present invention needs to establish a complete model prediction and feedback collection mechanism. Specifically, after the student model processes the user's multimodal input, it will generate corresponding prediction results. The prediction results here may be answers to user questions, labels for image recognition, or understanding results of voice commands. These results need to be presented in a user-friendly manner, so the present invention designs a result adapter to convert the output of the model into a form that the user can understand.

[0065] For example, the probability distribution is converted into a natural language description, or the recognition result is combined with a preset prompt template. After the prediction result is fed back to the user end through the system interface, the user can evaluate and provide feedback on the performance of the model. These feedback information include but are not limited to: result accuracy score (1-5 points), improvement suggestions in text form, and implicit feedback during the interaction process (such as whether to adopt the suggestion, whether to conduct subsequent interactions, etc.). In order to ensure the quality of feedback data, the present invention adopts a feedback verification mechanism to eliminate invalid or malicious feedback information through methods such as rule filtering and anomaly detection.

[0066] For example, in the intelligent customer service scenario, by analyzing user feedback, the system discovered and corrected 15% of inaccurate answers; in the industrial visual inspection scenario, the timely feedback mechanism helped the model identify 3 new types of defects, and the detection accuracy was improved by 8 percentage points.

[0067] It's important to note that this prediction-feedback loop not only serves the immediate optimization of the model but also provides high-quality training data for subsequent incremental learning. The collected feedback information is structured and stored as important learning samples in a dynamic memory bank for continuous model evolution. This mechanism ensures that the model can continuously adapt to new application scenarios and user needs, avoiding the problem of gradual performance degradation after traditional model deployment.

[0068] S105, storing the feedback information in a dynamic memory bank, and performing incremental learning on the student model based on the gradient direction vector stored in the dynamic memory bank.

[0069] After obtaining user feedback, the present invention designs an incremental learning mechanism based on a dynamic memory bank to address the problem of catastrophic forgetting during the continuous learning process while ensuring learning efficiency. The dynamic memory bank is a data structure used to store gradient information of key samples. Through a special storage and update strategy, the model can retain existing knowledge while learning new knowledge.

[0070] During the specific implementation process, the collected feedback information is first subjected to feature clustering. Using the improved K-Means algorithm, the feedback samples are clustered into several representative categories, and the most representative samples in each category are selected as target samples. This clustering strategy can compress the original large amount of feedback data into a small but highly representative sample set, significantly improving storage efficiency. For the selected target sample, its gradient direction vector on the current model is calculated. This vector represents the direction and magnitude of the model parameter adjustment. Through feature compression technology, the gradient information of more than 3,000 samples can be stored per GB of memory, which improves storage efficiency by 150% compared to traditional methods.

[0071] In the incremental learning phase, the present invention innovatively introduces a gradient orthogonality constraint mechanism. Specifically, when calculating the gradient direction vector of a new sample, the historical gradient direction vectors stored in the dynamic memory are also considered. Orthogonal projection is used to ensure that new learning does not interfere with the existing knowledge. Its mathematical expression is: in, Represents the gradient direction vector of the new sample, Represents the historical gradient direction vector, and gradient orthogonality is achieved by minimizing their inner product. This approach can effectively alleviate the catastrophic forgetting problem, allowing the model to retain the memory of old knowledge while learning new knowledge.

[0072] Based on the above embodiment, as an optional implementation, in S105, the feedback information is stored in a dynamic memory bank, and incremental learning of the student model based on the gradient direction vector stored in the dynamic memory bank specifically includes S51-S53: S51, perform feature clustering on the feedback information to generate target samples.

[0073] First, the collected user feedback information is subjected to feature clustering. This paper uses an improved K-means++ algorithm for clustering, and the number of clusters K is automatically determined by the silhouette coefficient. The cluster feature vector is calculated as follows: F_cluster = W_1F_text + W_2F_image + W_3*F_audio; where: F_cluster is the cluster feature vector; F_text is the text feature vector; F_image is the image feature vector; F_audio is the audio feature vector; W_1, W_2, and W_3 are the weight coefficients of each modal feature, dynamically calculated through the attention mechanism.

[0074] After clustering is completed, the central sample of each cluster is extracted as the target sample. This method can effectively reduce data redundancy and improve learning efficiency.

[0075] S52: Calculate a first gradient direction vector corresponding to the target sample, and store the first gradient direction vector into a dynamic memory bank.

[0076] S53, calculating the orthogonal constraint between the first gradient direction vector and the second direction vector, and performing incremental learning on the student model according to the orthogonal constraint, where the second direction vector is the historical gradient direction vector stored in the dynamic memory bank.

[0077] Next, the gradient direction vector of the target sample is calculated. The gradient calculation adopts the multi-task learning framework: Where: G_new is the first gradient direction vector; is the loss gradient of the i-th task; λi is the task weight coefficient; ||·||2 is the L2 norm normalization; ∑ is the weighted sum of all related tasks; the calculated gradient direction vector will be stored in the dynamic memory library. The dynamic memory library adopts a circular queue structure with a capacity of M (M = 1000 in this embodiment). When the library is full, a replacement strategy based on the importance score is used: Score = αfreq + βrecency + γ*gradient_norm; where: freq is the frequency with which the direction vector is used; recency is the timestamp of the most recent use; gradient_norm is the size of the gradient norm; α, β, γ are weight coefficients; in order to prevent the interference of new knowledge learning on existing knowledge, the present invention introduces an orthogonal constraint mechanism. For the newly calculated gradient direction vector G_new and the historical gradient direction vector G_old in the memory bank, calculate the orthogonal constraint: L_ortho = |G_new·G_old| / (||G_new||2*||G_old||2); where: L_ortho is the orthogonal constraint loss; ||·||2: L2 norm of the vector; the incremental learning update formula based on orthogonal constraint is: θ_new = θ_old-η*(G_new-λL_orthoG_old); where: θ_new is the updated model parameter; θ_old is the model parameter before update; η is the learning rate; λ is the orthogonal constraint coefficient.

[0078] After incremental learning of the student model, it also includes: Obtain resource parameter information of the target hardware platform; slice the student model according to the resource parameter information to generate multiple model shards; determine the loading strategy based on the size of each model shard; load each model shard into the memory of the target hardware platform in sequence; dynamically deploy the student model according to the loading strategy to make the student model adapt to the computing resources of the target hardware platform.

[0079] In one example, in order to solve the deployment problem of the multimodal student model on a resource-constrained hardware platform, the present invention designs an adaptive deployment solution based on dynamic slicing. Traditional model deployment methods often adopt an overall loading strategy, which is prone to memory overflow or performance bottlenecks in memory-constrained scenarios, so a more flexible deployment mechanism is needed. The resource parameter information of the target platform is obtained through the hardware interface API, including the total available memory MEM_total, the free memory MEM_free, the number of CPU cores CPU_cores, the CPU main frequency CPU_freq, the cache size CACHE_size and the memory bandwidth BW_memory. Based on these hardware parameters, a slicing strategy is designed, and its core formula is Slice_size = min(MEM_free / N, sqrt(CACHE_size*BW_memory / CPU_freq)), where Slice_size is the maximum size of a single slice, N is the preset cardinality of the number of slices, and min represents the smaller value operation. The model slicing process uses a hierarchical slicing method. The resource requirements of each layer are calculated as Layer_cost = W_size + A_size + G_size, where W_size is the weight parameter size, A_size is the activation value size, and G_size is the space required for gradient calculation. Based on the calculated Layer_cost, the dynamic programming algorithm DP[i][j] = min(DP[i-1][k] + cost(k, j)) is used to optimize the slicing points, where DP[i][j] represents the minimum cost of dividing the first j layers into the first i slices, and cost(k, j) represents the cost of forming a slice from layer k to layer j.

[0080] For the generated model shards, a priority-based loading strategy was designed. The priority score is calculated as Priority_score = Usage_freq * (1-Memory_cost / MEM_free), where Usage_freq is the shard's usage frequency and Memory_cost is the shard's memory overhead. A predictive loading strategy, Next_slice = argmax(P(slice_i | current_slice)), is also implemented, where P(slice_i | current_slice) is a conditional probability based on historical statistics. To adapt to runtime environment changes, the solution supports dynamic adjustment of the sharding strategy using Adjust_factor = min(1, MEM_free / MEM_threshold) * CPU_load_factor, where MEM_threshold is the memory usage threshold and CPU_load_factor is the CPU load factor.

[0081] Before receiving the multimodal data sent by the user end, it also includes: Perform data augmentation processing on the preset training data to generate enhanced training data; input the enhanced training data into the pre-trained large model to obtain model parameters; initialize the teacher model and student model according to the model parameters; and configure the parameters of the visual modality analysis unit, language modality analysis unit, and speech modality analysis unit based on the initialization results.

[0082] Specifically, text information can be enhanced through synonym replacement, random insertion, or deletion, while image information can be expanded through rotation, cropping, and color perturbation. Voice information can be enriched through noise addition and speed changes. Data augmentation can effectively improve the model's robustness to different scenarios and noise interference, reducing the risk of overfitting.

[0083] The system then inputs the generated augmented training data into the pre-trained large model for joint training of large-scale multimodal tasks, thereby obtaining model parameters including visual models, language models, and speech models. By learning from a large amount of augmented data, the pre-trained large model is able to capture the rich associations between multimodal data and form a parameter representation with strong generalization capabilities. Based on the above model parameters, the system initializes the teacher model and student model respectively, providing a unified and high-quality parameter foundation for both. The teacher model maintains its original complex structure to carry the complete expression of multimodal knowledge, while the student model is initialized with a streamlined structure to facilitate subsequent model compression and migration.

[0084] Finally, based on the initialization results, the system configures the parameters of the visual modality parsing unit, the language modality parsing unit, and the speech modality parsing unit to adapt them to the current model parameters and ensure the coordination and consistency of the three types of parsing units in feature extraction and semantic understanding. Through the above process, the system not only improves the basic capabilities of multimodal feature analysis and fusion, but also provides a strong model support for subsequent knowledge distillation and incremental learning, thereby achieving efficient processing and continuous optimization of multimodal data in complex and changing environments.

[0085] Based on the above method, this application also discloses a knowledge distillation system for multimodal large models, such as Figure 2 As shown, Figure 2 This is a structural diagram of a knowledge distillation system for a multimodal large model provided in an embodiment of the present application. The system includes: a receiving module, a fusion module, a migration module, an acquisition module, and a storage module; wherein, A receiving module is used to receive multimodal data sent by the user end, and input the multimodal data into the visual modal analysis unit, the language modal analysis unit and the speech modal analysis unit respectively, and generate visual feature vectors, language feature vectors and speech feature vectors accordingly; a fusion module is used to perform feature fusion on the visual feature vectors, the language feature vectors and the speech feature vectors to generate a fusion feature matrix; a migration module is used to migrate the knowledge of the teacher model to the student model based on the fusion feature matrix, wherein the teacher model is a pre-trained large model including the visual model, the language model and the speech model, and the student model is a simplified model corresponding to the teacher model; an acquisition module is used to obtain the prediction results output by the student model, feed the prediction results back to the user end, and receive feedback information sent by the user end; a storage module is used to store the feedback information in a dynamic memory bank, and perform incremental learning on the student model based on the gradient direction vector stored in the dynamic memory bank.

[0086] It should be noted that the above embodiments provide systems that implement their functions using only the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0087] See Figure 3 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0088] The communication bus 1002 is used to implement the connection and communication between these components.

[0089] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0090] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0091] The processor 1001 may include one or more processing cores. The processor 1001 utilizes various interfaces and lines to connect various parts of the entire server, and executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, as well as calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented in at least one hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 1001 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display; and the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 1001 and may be implemented separately on a single chip.

[0092] Among them, the memory 1005 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage device located away from the aforementioned processor 1001. As Figure 3 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a knowledge distillation method of a multimodal large model.

[0093] exist Figure 3In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call an application program for a knowledge distillation method of a multimodal large model stored in the memory 1005. When executed by one or more processors, the electronic device executes one or more methods described in the above embodiments.

[0094] An electronic device readable storage medium stores instructions, which, when executed by one or more processors, enable the electronic device to execute one or more of the methods described in the above embodiments.

[0095] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0096] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of the devices or units can be electrical or other forms.

[0098] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0099] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0101] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure herein, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A knowledge distillation method for a multimodal large model, characterized by: The method comprises: Receive multimodal data sent by the user terminal, input the multimodal data into the visual modality analysis unit, the language modality analysis unit and the voice modality analysis unit respectively, and generate a visual feature vector, a language feature vector and a voice feature vector respectively; Performing feature fusion on the visual feature vector, the language feature vector, and the speech feature vector to generate a fusion feature matrix; Based on the fusion feature matrix, the knowledge of the teacher model is transferred to the student model, wherein the teacher model is a pre-trained large model including a visual model, a language model, and a speech model, and the student model is a simplified model corresponding to the teacher model; Obtaining the prediction result output by the student model, feeding the prediction result back to the user terminal, and receiving feedback information sent by the user terminal; The feedback information is stored in a dynamic memory bank, and incremental learning is performed on the student model based on the gradient direction vector stored in the dynamic memory bank.

2. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that: The multimodal data includes text information, image information and voice information, and the multimodal data is input into the visual modality analysis unit, the language modality analysis unit and the voice modality analysis unit respectively to generate a visual feature vector, a language feature vector and a voice feature vector respectively, including: Inputting the text information into a language modality analysis unit to generate a language feature vector; Inputting the image information into a visual modality analysis unit to generate a visual feature vector; The speech information is input into a speech modality analysis unit to generate a speech feature vector.

3. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that: The step of fusing the visual feature vector, the language feature vector, and the speech feature vector to generate a fusion feature matrix includes: Calculating a first attention weight from the visual feature vector to the language feature vector and a second attention weight from the language feature vector to the speech feature vector through multiple attention heads in a three-way Transformer; Integrating the first attention weight and the second attention weight through an omnimodal fusion head; According to the integration result, the visual feature vector, the language feature vector and the speech feature vector are weightedly fused to generate a fusion feature matrix.

4. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that: The method of transferring the knowledge of the teacher model to the student model based on the fusion feature matrix includes: Input the fused feature matrix into the teacher model and the student model respectively to obtain the first multimodal knowledge representation output by the teacher model and the second multimodal knowledge representation output by the student model; Based on the first multimodal knowledge representation and the second multimodal knowledge representation, training a student model according to a joint loss function, wherein the joint loss function includes a KL divergence loss, a maximum mean difference loss, and a task-specific loss; The joint loss function is adjusted to align the second multimodal knowledge representation with the first multimodal knowledge representation.

5. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that: The step of storing the feedback information in a dynamic memory bank and performing incremental learning on the student model based on the gradient direction vector stored in the dynamic memory bank includes: Performing feature clustering on the feedback information to generate target samples; Calculating a first gradient direction vector corresponding to the target sample, and storing the first gradient direction vector in a dynamic memory bank; An orthogonal constraint between the first gradient direction vector and a second direction vector is calculated, and incremental learning is performed on the student model according to the orthogonal constraint, where the second direction vector is a historical gradient direction vector stored in the dynamic memory bank.

6. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that: After the incremental learning of the student model, the following steps are also included: Get resource parameter information of the target hardware platform; Slicing the student model according to the resource parameter information to generate multiple model slices; Determining a loading strategy based on the size of each of the model slices; Loading each of the model slices into the memory of the target hardware platform in sequence; The student model is dynamically deployed according to the loading strategy so that the student model adapts to the computing resources of the target hardware platform.

7. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that: Before receiving the multimodal data sent by the user terminal, the method further includes: Perform data enhancement processing on the preset training data to generate enhanced training data; Inputting the enhanced training data into the pre-trained large model to obtain model parameters; Initializing the teacher model and the student model according to the model parameters; Parameters of the visual modality analysis unit, the language modality analysis unit, and the speech modality analysis unit are configured based on the initialization result.

8. A knowledge distillation system for a multimodal large model, characterized by: The system includes: a receiving module, a fusion module, a migration module, an acquisition module and a storage module; wherein, The receiving module is configured to receive multimodal data sent by a user terminal, input the multimodal data into the visual modality analysis unit, the language modality analysis unit, and the speech modality analysis unit, and generate a visual feature vector, a language feature vector, and a speech feature vector respectively; The fusion module is used to perform feature fusion on the visual feature vector, the language feature vector and the speech feature vector to generate a fusion feature matrix; The migration module is used to migrate the knowledge of the teacher model to the student model based on the fusion feature matrix, wherein the teacher model is a pre-trained large model including a visual model, a language model, and a speech model, and the student model is a simplified model corresponding to the teacher model; The acquisition module is used to obtain the prediction result output by the student model, feed the prediction result back to the user terminal, and receive feedback information sent by the user terminal; The storage module is used to store the feedback information in a dynamic memory library, and perform incremental learning on the student model based on the gradient direction vector stored in the dynamic memory library.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, a user interface and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images