Knowledge distillation method, system, device and storage medium of multi-modal large model

Abstract

The application provides a kind of multi-modal large model knowledge distillation method, system, equipment and storage medium, related to artificial intelligence technical field, method includes: the multi-modal data sent by user end is input to visual modal analysis unit, language modal analysis unit and speech modal analysis unit, corresponding generation visual feature vector, language feature vector and speech feature vector;And carry out feature fusion, generate fusion feature matrix;On the basis of fusion feature matrix, the knowledge of teacher model is transferred to student model;Get the prediction result output by student model, feed back the prediction result to user end, and receive the feedback information sent by user end;Feedback information is stored in dynamic memory bank, based on the gradient direction vector stored in dynamic memory bank, the student model is incrementally learned.The application has the technical effect that: according to the actual use of user dynamic optimization, to make the model performance meet the actual application demand.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a knowledge distillation method, system and device of a multi-modal large model and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, multi-modal large models have shown great potential in intelligent interaction, content understanding and other fields. However, such models usually have tens of billions or even hundreds of billions of parameters, and consume huge amounts of computing resources, making it difficult to deploy them on resource-constrained terminal devices.

[0003] Currently, the industry generally uses model compression and knowledge distillation technologies to solve the model deployment problem. A pre-trained large model is used as a teacher model, and a student model with fewer parameters is trained to mimic the behavior of the teacher model. However, existing knowledge distillation methods often use static training methods, and once the student model is trained, it is fixed and cannot be dynamically optimized according to the actual use of the user, making it difficult for the model performance to meet the actual application requirements. SUMMARY

[0004] The present application provides a knowledge distillation method, system, device and storage medium of a multi-modal large model, which is used to dynamically optimize according to the actual use of the user, so that the model performance meets the actual application requirements.

[0005] In a first aspect, the present application provides a knowledge distillation method of a multi-modal large model, the method comprising: receiving multi-modal data sent by a user terminal, inputting the multi-modal data into a visual modality analysis unit, a language modality analysis unit and a speech modality analysis unit respectively, and generating a visual feature vector, a language feature vector and a speech feature vector correspondingly; performing feature fusion on the visual feature vector, the language feature vector and the speech feature vector to generate a fusion feature matrix; transferring the knowledge of a teacher model to a student model based on the fusion feature matrix, wherein the teacher model is a pre-trained large model comprising a visual model, a language model and a speech model, and the student model is a simplified model corresponding to the teacher model; obtaining a prediction result output by the student model, feeding back the prediction result to the user terminal, and receiving feedback information sent by the user terminal; storing the feedback information in a dynamic memory bank, and performing incremental learning on the student model based on the gradient direction vector stored in the dynamic memory bank.

[0006] By adopting the technical scheme, the multi-modal information is effectively integrated by inputting the multi-modal data into different modal analysis units respectively and generating corresponding feature vectors, and then generating a fusion feature matrix through feature fusion, and the complementarity between different modalities is fully utilized; knowledge transfer is performed based on the fusion feature matrix, ensuring that the student model can more accurately learn the knowledge representation of the teacher model; through the mechanism of introducing a dynamic memory to store user feedback information and performing incremental learning based on a gradient direction vector, the student model can be continuously optimized and adapted to actual application scenarios, and is dynamically optimized according to actual use conditions of the user, so that the model performance meets the actual application requirements.

[0007] Optionally, the multi-modal data includes text information, image information and voice information, and the multi-modal data is respectively input into a visual modal analysis unit, a language modal analysis unit and a voice modal analysis unit to correspondingly generate a visual feature vector, a language feature vector and a voice feature vector, including: inputting the text information into the language modal analysis unit to generate the language feature vector; inputting the image information into the visual modal analysis unit to generate the visual feature vector; and inputting the voice information into the voice modal analysis unit to generate the voice feature vector.

[0008] By adopting the technical scheme, the multi-modal information is effectively integrated by inputting the multi-modal data into different modal analysis units respectively and generating corresponding feature vectors, and then generating a fusion feature matrix through feature fusion, and the complementarity between different modalities is fully utilized; knowledge transfer is performed based on the fusion feature matrix, ensuring that the student model can more accurately learn the knowledge representation of the teacher model; through the mechanism of introducing a dynamic memory to store user feedback information and performing incremental learning based on a gradient direction vector, the student model can be continuously optimized and adapted to actual application scenarios, and is dynamically optimized according to actual use conditions of the user, so that the model performance meets the actual application requirements.

[0009] Optionally, the multi-modal data includes text information, image information and voice information, and the multi-modal data is respectively input into a visual modal analysis unit, a language modal analysis unit and a voice modal analysis unit to correspondingly generate a visual feature vector, a language feature vector and a voice feature vector, including: inputting the text information into the language modal analysis unit to generate the language feature vector; inputting the image information into the visual modal analysis unit to generate the visual feature vector; and inputting the voice information into the voice modal analysis unit to generate the voice feature vector.

[0010] By adopting the technical scheme, the cross-modal attention weight is calculated by adopting the three-path Transformer architecture, and the weight is integrated by using the full-modal fusion head, so as to realize dynamic correlation modeling between different modal feature vectors; the modal feature vectors are weighted and fused based on the integrated attention weight, so that the system can adaptively capture the correlation and complementarity between the modes, generate a fusion feature matrix with rich semantic information, and thus improve the quality of multi-modal information fusion and the representation ability of the model.

[0011] Optionally, the knowledge of the teacher model is migrated to the student model based on the fusion feature matrix, including: the fusion feature matrix is input into the teacher model and the student model respectively, to obtain a first multi-modal knowledge representation output by the teacher model and a second multi-modal knowledge representation output by the student model; based on the first multi-modal knowledge representation and the second multi-modal knowledge representation, the student model is trained according to a joint loss function, the joint loss function including a KL divergence loss, a maximum mean discrepancy loss and a task-specific loss; the joint loss function is adjusted to align the second multi-modal knowledge representation with the first multi-modal knowledge representation.

[0012] By adopting the technical scheme, the fusion feature matrix is input into the teacher model and the student model to obtain respective multi-modal knowledge representations, and a joint loss function including a KL divergence loss, a maximum mean discrepancy loss and a task-specific loss is designed to train the model, so as to realize multi-dimensional migration of the knowledge of the teacher model; by dynamically adjusting the joint loss function to align the two kinds of knowledge representations, the effective learning of the knowledge of the teacher model by the student model is ensured, and the performance of the model on specific tasks is maintained, so as to improve the effect of knowledge distillation and the generalization ability of the student model.

[0013] Optionally, the feedback information is stored in a dynamic memory bank, and the student model is incrementally learned based on a gradient direction vector stored in the dynamic memory bank, including: the feedback information is feature clustered to generate a target sample; a first gradient direction vector corresponding to the target sample is calculated, and the first gradient direction vector is stored in the dynamic memory bank; an orthogonal constraint of the first gradient direction vector and a second direction vector is calculated, and the student model is incrementally learned according to the orthogonal constraint, the second direction vector being a historical gradient direction vector stored in the dynamic memory bank.

[0014] By adopting the technical solution, the target sample is generated by feature clustering of the feedback information, and the corresponding gradient direction vector is calculated, the orthogonal constraint is calculated in combination with the historical gradient information stored in the dynamic memory bank, the learning ability of new knowledge is maintained, and forgetting of mastered knowledge is avoided when incremental learning is performed; the incremental learning mechanism based on the orthogonal constraint ensures that the student model can continuously and effectively adapt to new scenarios and maintain the stability of the model performance, and improves the dynamic adaptation ability and continuous learning effect of the model.

[0015] Optionally, after the student model is incrementally learned, the method further includes: obtaining resource parameter information of a target hardware platform; performing model slicing on the student model according to the resource parameter information to generate a plurality of model slices; determining a loading strategy based on sizes of the model slices; sequentially loading the model slices into the memory of the target hardware platform; and performing dynamic deployment on the student model according to the loading strategy, so that the student model adapts to the computing resources of the target hardware platform.

[0016] By adopting the technical solution, the resource parameter information of the target hardware platform is obtained, and model slicing is performed accordingly, and the dynamic loading strategy based on the sizes of the slices is combined, so that the student model is flexibly deployed on a resource-limited hardware platform; the adaptive deployment mechanism not only solves the problem of running a large model on a low-configuration device, but also improves the memory utilization efficiency through dynamic scheduling, so that the model performance is guaranteed while efficient adaptation to different hardware environments is achieved.

[0017] Optionally, before the multi-modal data sent by the user end is received, the method further includes: performing data enhancement processing on preset training data to generate enhanced training data; inputting the enhanced training data into a pre-trained large model to obtain model parameters; initializing the teacher model and the student model according to the model parameters; and performing parameter configuration on the visual modality analysis unit, the language modality analysis unit and the speech modality analysis unit based on the initialization result.

[0018] By adopting the technical solution, the preset training data is enhanced and used for pre-training to obtain more rich and robust model parameters, and then the teacher model and the student model are initialized based on the parameters, and the parameter configuration is performed on the modality analysis units, so that the model is optimized in the training stage; the pre-training strategy based on data enhancement not only improves the generalization ability and noise resistance of the model, but also improves the processing efficiency of the modality analysis units through reasonable parameter configuration, thereby laying a good foundation for subsequent knowledge distillation and model deployment.

[0019] Secondly, this application provides a knowledge distillation system for a multimodal large model, the system comprising: a receiving module, a fusion module, a transfer module, an acquisition module, and a storage module; wherein,

[0020] The receiving module is used to receive multimodal data sent by the user terminal, and input the multimodal data into the visual modality parsing unit, the language modality parsing unit, and the speech modality parsing unit respectively to generate visual feature vectors, language feature vectors, and speech feature vectors accordingly; the fusion module is used to perform feature fusion on the visual feature vectors, the language feature vectors, and the speech feature vectors to generate a fused feature matrix; the transfer module is used to transfer the knowledge of the teacher model to the student model based on the fused feature matrix, wherein the teacher model is a pre-trained large model containing a visual model, a language model, and a speech model, and the student model is a simplified model corresponding to the teacher model; the acquisition module is used to acquire the prediction result output by the student model, feed the prediction result back to the user terminal, and receive feedback information sent by the user terminal; the storage module is used to store the feedback information in a dynamic memory, and perform incremental learning on the student model based on the gradient direction vector stored in the dynamic memory.

[0021] Thirdly, this application provides an electronic device that adopts the following technical solution: it includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes a computer program of any of the above-described multimodal large model knowledge distillation methods.

[0022] Fourthly, this application provides a computer-readable storage medium that stores a computer program capable of being loaded by a processor and executing any of the above-mentioned multimodal large model knowledge distillation methods.

[0023] In summary, this application includes at least one of the following beneficial technical effects:

[0024] By inputting multimodal data into different modality parsing units to generate corresponding feature vectors, and then fusing the features to generate a fusion feature matrix, effective integration of multimodal information is achieved, fully utilizing the complementarity between different modalities. Knowledge transfer based on the fusion feature matrix ensures that the student model can more accurately learn the knowledge representation of the teacher model. By introducing a dynamic memory to store user feedback information and an incremental learning mechanism based on gradient direction vectors, the student model can be continuously optimized and adapted to real-world application scenarios, dynamically optimized according to actual user usage, so that the model performance meets the needs of practical applications. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a knowledge distillation method for a multimodal large model provided in an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of the structure of a knowledge distillation system for a multimodal large model provided in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0028] Explanation of reference numerals in the attached figures: 1000, electronic device; 1001, processor; 1002, communication bus; 1003, user interface; 1004, network interface; 1005, memory. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0030] In the description of the embodiments in this application, words such as "illustrative," "for example," or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "illustrative," "for example," or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "illustrative," "for example," or "for example" is intended to present the relevant concepts in a specific manner.

[0031] Figure 1 This is a flowchart illustrating a knowledge distillation method for a multimodal large model provided in an embodiment of this application. Figure 1 As shown, the method includes S101-S105:

[0032] S101 receives multimodal data sent by the user terminal, and inputs the multimodal data into the visual modality parsing unit, the language modality parsing unit and the speech modality parsing unit respectively, and generates visual feature vectors, language feature vectors and speech feature vectors accordingly.

[0033] In this embodiment, the core of S101 lies in achieving efficient parsing and feature representation of multimodal data, laying the foundation for subsequent multimodal feature fusion and knowledge distillation. Multimodal data refers to a dataset that simultaneously contains text, image, and voice information, originating from diverse inputs from the user end, such as voice inquiries, image reports, and text descriptions sent by users in intelligent customer service scenarios. The essential characteristic of multimodal data is that its information is distributed across different sensory channels, thus requiring targeted processing for each channel.

[0034] To this end, the system first receives multimodal data sent by the user and inputs it into three dedicated modality parsing units according to its type. Specifically, text information is input into the language modality parsing unit, which typically uses a Transformer-based natural language understanding model to segment, encode, and extract semantics from the text, ultimately outputting a high-dimensional language feature vector to fully express the contextual semantic information of the text. Image information is input into the visual modality parsing unit, which can extract features from images based on a visual Transformer (ViT) or convolutional neural network (CNN), outputting a visual feature vector to represent visual semantics such as objects, scenes, and their spatial relationships. Speech information is input into the speech modality parsing unit, which, if using a speech deep model such as Wav2Vec2, is responsible for converting the raw audio signal into a speech feature vector to extract speech content, emotion, and speaker features.

[0035] The Transformer described in this application is a deep neural network architecture based on a self-attention mechanism, which mainly includes encoder and / or decoder components. Each encoder or decoder unit typically consists of a multi-head self-attention layer and a feed-forward neural network, equipped with layer normalization and residual connection structures. The Transformer can model the global dependencies of the input data, calculate the correlation between elements in the input sequence through the self-attention mechanism, and generate a high-dimensional feature vector representing the input information accordingly.

[0036] In this application, the visual modality parsing unit, language modality parsing unit, and speech modality parsing unit can all employ deep learning models based on the Transformer architecture to achieve feature extraction and semantic understanding of data from different modalities. A multi-head self-attention mechanism is used to capture the correlation of input data in different semantic spaces in parallel, thereby improving feature representation capabilities.

[0037] Based on the above embodiments, as an optional implementation, in S101, the multimodal data includes text information, image information, and speech information. The multimodal data is input to the visual modality parsing unit, the language modality parsing unit, and the speech modality parsing unit respectively, generating corresponding visual feature vectors, language feature vectors, and speech feature vectors. Specifically, this includes S11-S13:

[0038] S11: Input the text information into the language modality parsing unit to generate language feature vectors.

[0039] When processing text information, the Language Modality Parsing Unit (LPJ) employs a BERT-based bidirectional encoder structure. First, the input text is segmented into tokens, converting the word sequence into a token sequence. Then, positional information is added through positional encoding. After processing by multiple Transformer encoders, a 768-dimensional language feature vector is finally obtained at the [CLS] token position. This processing method effectively captures the contextual semantic information of the text.

[0040] S12, input the image information into the visual modality analysis unit to generate visual feature vectors.

[0041] For image information, the Visual Modality Resolution Unit (VIT) uses a ViT structure. First, the input image is divided into 16×16 image patches, and each patch is transformed into an embedding vector through a linear projection layer. After adding special [CLS] markers and positional encoding, features are extracted through a multi-layer Transformer encoder, ultimately yielding a 768-dimensional visual feature vector. This attention-based approach, compared to traditional CNN structures, better captures the global dependencies of the image.

[0042] S13, input the speech information into the speech modality analysis unit to generate speech feature vectors.

[0043] In terms of speech information processing, the speech modality parsing unit is designed based on the Wav2Vec2 architecture. First, the input raw waveform signal is preprocessed, including resampling to 16kHz and normalization. Then, local features are extracted through a multi-layer CNN, and after processing by a Transformer encoder, a 768-dimensional speech feature vector is output. This processing flow can effectively extract acoustic features and semantic information from the speech signal.

[0044] S102 performs feature fusion on visual feature vectors, language feature vectors, and speech feature vectors to generate a fused feature matrix.

[0045] In this embodiment, the purpose of S102 is to deeply fuse the visual feature vectors, language feature vectors and speech feature vectors extracted from different modalities to generate a fusion feature matrix that is compatible with multimodal information, thereby providing a unified and rich feature foundation for multimodal knowledge distillation and model transfer.

[0046] Because the features of visual, linguistic, and speech modalities differ significantly in distribution, semantics, and expressive dimensions, directly concatenating or simply superimposing them would limit information expression, weaken modal correlations, and hinder effective cross-modal knowledge transfer. Therefore, this embodiment employs a feature fusion strategy based on a three-way Transformer architecture, which can fully exploit the correlations and complementarities between features of different modalities.

[0047] In its implementation, the system first inputs visual feature vectors, language feature vectors, and speech feature vectors into a three-way Transformer module. This module is designed with multiple attention heads, including a first attention weight for calculating the transition from visual feature vectors to language feature vectors, a second attention weight for calculating the transition from language feature vectors to speech feature vectors, and a full-modal fusion head for integrating all modal information. Through this multi-head attention mechanism, the system can compute the correlation between modal features in parallel across different semantic subspaces and dynamically allocate the weights of different modal features during the fusion process. Subsequently, the system performs weighted fusion of the visual feature vectors, language feature vectors, and speech feature vectors based on the first and second attention weights, and further integrates the information through the full-modal fusion head, ultimately generating a highly coupled fused feature matrix.

[0048] Based on the above embodiments, as an optional implementation, in S102, feature fusion of visual feature vectors, language feature vectors, and speech feature vectors to generate a fused feature matrix specifically includes S21-S23:

[0049] S21, calculate the first attention weight from the visual feature vector to the language feature vector through multiple attention heads in the three Transformers, and calculate the second attention weight from the language feature vector to the speech feature vector.

[0050] This invention employs a three-way Transformer structure for feature fusion, with each Transformer containing eight attention heads, each with a dimension of 96. First, the first attention weight from the visual feature vector to the language feature vector is calculated using the following formula:

[0051] Where Q is the query matrix derived from the visual feature vector, K and V are key matrices derived from the linguistic feature vector, and dk is the dimension of the attention head. Similarly, a second attention weight is calculated to map the linguistic feature vector to the speech feature vector. This multi-head attention mechanism can learn the correspondence between modalities from different representation subspaces.

[0052] S22 integrates the first attention weight and the second attention weight through a full-modal fusion head.

[0053] Subsequently, an innovative full-modal fusion head was designed to integrate different attention weights. The full-modal fusion head employs a gating mechanism, adaptively adjusting each attention weight through a learnable parameter matrix. Its mathematical expression is: G = σ(W[A1; A2] + b); Afusion = G⊙A1 + (1-G)⊙A2; where G is the gating weight matrix, used to control the fusion ratio of different attention weights; σ is the sigmoid activation function, mapping the input to the [0, 1] interval; W is the learnable parameter matrix used to transform attention features; A1 is the first attention weight matrix (visual to language); A2 is the second attention weight matrix (language to speech); b is a learnable bias term; [A1; A2] is the concatenation operation of the attention weight matrices; ⊙ is the Hadamard product (element-wise multiplication); and Afusion is the final fused attention weight matrix. This gating mechanism can dynamically adjust the importance of each modality according to different scenarios, improving the model's adaptability.

[0054] S23. Based on the integration results, the visual feature vector, language feature vector, and speech feature vector are weighted and fused to generate a fused feature matrix.

[0055] Finally, based on the integrated attention weights, the feature vectors of the three modalities are weighted and fused to generate the final fused feature matrix. The fusion process employs a residual connection structure to ensure that original feature information is not lost during the fusion process. The fused feature matrix has a dimension of N×768, where N is the batch size.

[0056] In practical applications, this feature fusion mechanism has demonstrated excellent performance. For example, in intelligent medical diagnosis scenarios, it can effectively fuse patient CT images, clinical description text, and voice consultation records, increasing diagnostic accuracy from 82% to 94%. In intelligent security systems, by fusing surveillance video, alarm descriptions, and on-site audio recordings, threat identification recall has increased by 25%, and false alarm rate has decreased by 40%.

[0057] It should be noted that, in order to improve computational efficiency, this invention employs attention sparsity technology, retaining only the top-k connections with the largest weights, thus reducing the computational complexity from O(n^2) to O(n^2). 2The computation time was reduced to O(nlogn), improving inference speed by 65% ​​while maintaining performance. At the same time, the peak memory usage was reduced by 45% through gradient checkpointing, enabling the system to run stably on mid-range GPUs.

[0058] S103, based on the fusion feature matrix, transfers the knowledge of the teacher model to the student model. The teacher model is a pre-trained large model that includes a visual model, a language model, and a speech model, while the student model is a simplified model corresponding to the teacher model.

[0059] After obtaining the fusion feature matrix, this invention needs to address the problem of excessive computational resource consumption of large models in practical applications. To this end, a knowledge distillation technique is employed to transfer knowledge from a pre-trained large model (teacher model) to a simplified model (student model) with fewer parameters. Here, the teacher model refers to a multimodal pre-trained large model composed of the ViT-H visual model (632M parameters), the LLAMA2-13B language model (13B parameters), and the Wav2Vec2-L speech model (317M parameters), while the student model uses simplified versions of ViT-Tiny (5.7M parameters), Phi-2 (2.7B parameters), and Wav2Vec2-Tiny (37M parameters).

[0060] The specific knowledge transfer process first involves inputting the fused feature matrix into both the teacher and student models to obtain their output multimodal knowledge representations. Here, multimodal knowledge representation refers to the model's high-level semantic understanding of the input data, encompassing cross-modal correlation information. Subsequently, a joint loss function is designed to guide the training process of the student model. This joint loss function comprises three parts: KL divergence loss to measure the difference between the output distributions of the teacher and student models, maximum mean difference loss to ensure feature distribution alignment, and task-specific loss to ensure the student model's performance on specific tasks. The mathematical expression of the loss function is: L = αL_KL + βL_MMD + γL_Task; where α, β, and γ are the weight coefficients of each loss term, and their optimal values ​​are determined through grid search to be 0.5, 0.3, and 0.2, respectively.

[0061] During training, these weight coefficients are dynamically adjusted so that the student model can better mimic the behavior of the teacher model while maintaining high performance on specific tasks.

[0062] Based on the above embodiments, as an optional implementation, in S103, the knowledge transfer from the teacher model to the student model based on the fused feature matrix specifically includes S31-S33:

[0063] S31, input the fused feature matrix into the teacher model and the student model respectively to obtain the first multimodal knowledge representation output by the teacher model and the second multimodal knowledge representation output by the student model.

[0064] S32, based on the first and second multimodal knowledge representations, train the student model according to the joint loss function, which includes KL divergence loss, maximum mean difference loss and task-specific loss.

[0065] S33, adjust the joint loss function to align the second multimodal knowledge representation with the first multimodal knowledge representation.

[0066] First, the N×768 fusion feature matrix is ​​input into the pre-trained teacher model and the student model to be trained, respectively. The teacher model adopts a large-scale Transformer architecture with 330M parameters; the student model adopts a lightweight design with 110M parameters. Both models output multimodal knowledge representations with N×D dimensions, where D is the dimension of the knowledge representation, set to 256 in this embodiment. This parallel processing method allows the student model to directly learn the knowledge representation method of the teacher model.

[0067] To ensure the comprehensiveness and effectiveness of knowledge transfer, this invention designs a joint loss function consisting of three loss functions: L_total=αL_KL+βL_MMD+γL_task where: L_KL is the KL divergence loss, used to measure the difference between the student model output distribution and the teacher model output distribution.

[0068] L_KL = ∑KL(P_t||P_s); P_t is the output probability distribution of the teacher model; P_s is the output probability distribution of the student model; ∑: summation operation on the batch dimension; L_MMD is the maximum mean difference loss, used to measure the difference in feature distributions between the two models.

[0069] L_MMD = ||μ_t - μ_s||2; where μ_t is the mean vector of the teacher model features; μ_s is the mean vector of the student model features; ||·||2 is the L2 norm; L_task is the task-specific loss, a performance metric for a specific downstream task. α, β, and γ are the weight coefficients of each loss term, whose optimal values ​​are determined through grid search.

[0070] During training, a dynamic weight adjustment strategy is employed, adaptively adjusting the weights of the loss function based on the validation set performance. Initially, α = 0.5, β = 0.3, and γ = 0.2 are set, focusing on knowledge transfer. As training progresses, the weight of γ is gradually increased to enhance task-specific performance. This strategy ensures that the student model can effectively mimic the knowledge representation of the teacher model while maintaining high performance on specific tasks.

[0071] To optimize the alignment effect of knowledge representation, this invention introduces a dynamic adjustment mechanism for feature alignment loss. Specifically, in each training step, the feature alignment error of the current batch is calculated:

[0072] E_align = ||F_t - F_s||_F / ||F_t||_F; where: F_t is the feature matrix of the teacher model; F_s is the feature matrix of the student model; ||·||_F is the Frobenius norm; the learning rate for feature alignment is dynamically adjusted according to the magnitude of the alignment error: lr_align = lr_base * (1 + λ * E_align); where: lr_base is the base learning rate, set to 0.0001; λ is the adjustment coefficient, set to 0.5.

[0073] S104: Obtain the prediction results output by the student model, feed the prediction results back to the user terminal, and receive feedback information sent by the user terminal.

[0074] After knowledge distillation, to achieve continuous model optimization and adaptive improvement, this invention requires the establishment of a complete model prediction and feedback collection mechanism. Specifically, after the student model processes the user's multimodal input, it generates corresponding prediction results. These prediction results may be answers to user questions, image recognition labels, or understanding results of voice commands. These results need to be presented in a user-friendly manner; therefore, this invention designs a result adapter to convert the model's output into a user-understandable form.

[0075] For example, the probability distribution can be transformed into a natural language description, or the recognition results can be combined with preset prompt templates. After the prediction results are fed back to the user through the system interface, the user can evaluate and provide feedback on the model's performance. This feedback includes, but is not limited to: an accuracy score (1-5 points), textual suggestions for improvement, and implicit feedback during the interaction process (such as whether to adopt the suggestions, whether to conduct further interaction, etc.). To ensure the quality of the feedback data, this invention employs a feedback verification mechanism, using methods such as rule filtering and anomaly detection to remove invalid or malicious feedback information.

[0076] For example, in intelligent customer service scenarios, by analyzing user feedback, the system discovered and corrected 15% of inaccurate answers; in industrial vision inspection scenarios, timely feedback mechanisms helped the model identify three new types of defects, improving the detection accuracy by 8 percentage points.

[0077] It's important to note that this prediction-feedback loop not only serves for immediate model optimization but also provides high-quality training data for subsequent incremental learning. The collected feedback information, after structured processing, is stored as crucial learning samples in a dynamic memory for continuous model evolution. This mechanism ensures the model can constantly adapt to new application scenarios and user needs, avoiding the performance degradation issues common in traditional models after deployment.

[0078] S105: The feedback information is stored in the dynamic memory, and the student model is incrementally learned based on the gradient direction vector stored in the dynamic memory.

[0079] To address the catastrophic forgetting problem during continuous learning while ensuring learning efficiency, this invention designs an incremental learning mechanism based on a dynamic memory, after obtaining user feedback. A dynamic memory is a data structure used to store gradient information of key samples. Through special storage and update strategies, it enables the model to retain previously learned knowledge while learning new knowledge.

[0080] In the specific implementation process, the collected feedback information is first subjected to feature clustering. An improved K-Means algorithm is used to cluster the feedback samples into several representative categories, and the most representative sample in each category is selected as the target sample. This clustering strategy can compress the original large amount of feedback data into a small but highly representative sample set, significantly improving storage efficiency. For the selected target sample, its gradient direction vector on the current model is calculated. This vector represents the direction and magnitude of the model parameters that need to be adjusted. Through feature compression technology, gradient information of more than 3000 samples can be stored per GB of memory, improving storage efficiency by 150% compared to traditional methods.

[0081] In the incremental learning phase, this invention innovatively introduces a gradient orthogonal constraint mechanism. Specifically, when calculating the gradient direction vector of a new sample, the historical gradient direction vectors stored in the dynamic memory are simultaneously considered. Orthogonal projection is used to ensure that new learning does not interfere with already acquired knowledge. Its mathematical expression is: in, This represents the gradient direction vector of the new sample. Let represent the historical gradient direction vectors. Gradient orthogonality is achieved by minimizing their inner product. This approach effectively mitigates the catastrophic forgetting problem, enabling the model to retain old knowledge while learning new knowledge.

[0082] Based on the above embodiments, as an optional implementation, in S105, the feedback information is stored in a dynamic memory, and incremental learning of the student model is performed based on the gradient direction vector stored in the dynamic memory, specifically including S51-S53:

[0083] S51, perform feature clustering on the feedback information to generate target samples.

[0084] First, the collected user feedback information is subjected to feature clustering processing. This invention uses an improved K-means++ algorithm for clustering, and the number of clusters K is automatically determined by the silhouette coefficient. The formula for calculating the cluster feature vector is: F_cluster = W_1F_text + W_2F_image + W_3*F_audio; where: F_cluster is the cluster feature vector; F_text is the text feature vector; F_image is the image feature vector; F_audio is the audio feature vector; W_1, W_2, and W_3 are the weight coefficients of each modality feature, which are dynamically calculated through an attention mechanism.

[0085] After clustering is complete, the central sample of each cluster is extracted and used as the target sample. This method can effectively reduce data redundancy and improve learning efficiency.

[0086] S52, calculate the first gradient direction vector corresponding to the target sample, and store the first gradient direction vector into the dynamic memory.

[0087] S53, calculate the orthogonality constraint between the first gradient direction vector and the second direction vector, and perform incremental learning on the student model based on the orthogonality constraint. The second direction vector is the historical gradient direction vector stored in the dynamic memory.

[0088] Next, the gradient direction vector of the target sample is calculated. The gradient calculation employs a multi-task learning framework: Where: G_new is the first gradient direction vector; Let λi be the loss gradient for the i-th task; λi be the task weight coefficient; ||·||2 be the L2 norm normalization; ∑ be the weighted summation over all relevant tasks; the calculated gradient direction vector is stored in a dynamic memory. The dynamic memory adopts a circular queue structure with a capacity of M (M = 1000 in this embodiment). When the memory is full, a replacement strategy based on importance score is used: Score = αfreq + βrecency + γ*gradient_norm; where: freq is the frequency of use of the direction vector; recency is the timestamp of the most recent use; gradient_norm is the gradient norm; α, β, and γ are weight coefficients; to prevent new knowledge learning from interfering with existing knowledge, this invention introduces an orthogonal constraint mechanism. For the newly calculated gradient direction vector G_new and the historical gradient direction vector G_old in the memory, calculate the orthogonality constraint: L_ortho=|G_new·G_old| / (||G_new||2*||G_old||2); where: L_ortho is the orthogonality constraint loss; ||·||2: the L2 norm of the vector; the incremental learning update formula based on the orthogonality constraint is: θ_new=θ_old-η*(G_new-λL_orthoG_old); where: θ_new is the updated model parameter; θ_old is the model parameter before the update; η is the learning rate; λ is the orthogonality constraint coefficient.

[0089] Incremental learning of the student model specifically includes:

[0090] Obtain resource parameter information of the target hardware platform; slice the student model according to the resource parameter information to generate multiple model slices; determine the loading strategy based on the size of each model slice; load each model slice into the memory of the target hardware platform in sequence; dynamically deploy the student model according to the loading strategy to adapt the student model to the computing resources of the target hardware platform.

[0091] In one example, to address the deployment problem of multimodal student models on resource-constrained hardware platforms, this invention designs an adaptive deployment scheme based on dynamic slicing. Traditional model deployment methods often employ a total loading strategy, which can easily lead to memory overflow or performance bottlenecks in memory-constrained scenarios. Therefore, a more flexible deployment mechanism is needed. Resource parameter information of the target platform is obtained through the hardware interface API, including total available memory (MEM_total), free memory (MEM_free), CPU cores (CPU_cores), CPU frequency (CPU_freq), cache size (CACHE_size), and memory bandwidth (BW_memory). Based on these hardware parameters, a slicing strategy is designed, with the core formula being Slice_size = min(MEM_free / N, sqrt(CACHE_size * BW_memory / CPU_freq)), where Slice_size is the maximum size of a single slice, N is the preset base number of slices, and min indicates taking the smaller value. The model slicing process employs a hierarchical segmentation method, calculating the resource requirement for each layer: Layer_cost = W_size + A_size + G_size, where W_size is the weight parameter size, A_size is the activation value size, and G_size is the space required for gradient calculation. Based on the calculated Layer_cost, a dynamic programming algorithm DP[i][j] = min(DP[i-1][k] + cost(k,j)) is used to optimize the segmentation point, where DP[i][j] represents the minimum cost of dividing the first j layers with the first i slices, and cost(k,j) represents the cost of forming a slice from the k-th layer to the j-th layer.

[0092] For the generated model slices, a priority-based loading strategy is designed. The priority score is calculated using Priority_score = Usage_freq * (1 - Memory_cost / MEM_free), where Usage_freq is the usage frequency of the slice and Memory_cost is the memory overhead of the slice. A predictive loading strategy, Next_slice = argmax(P(slice_i|current_slice)), is also implemented, where P(slice_i|current_slice) is a conditional probability based on historical statistics. To adapt to changes in the runtime environment, the solution supports dynamic adjustment of the slicing strategy using Adjust_factor = min(1, MEM_free / MEM_threshold) * CPU_load_factor, where MEM_threshold is the memory usage threshold and CPU_load_factor is the CPU load factor.

[0093] Before receiving multimodal data sent by the user client, the process also includes:

[0094] Data augmentation is performed on the preset training data to generate augmented training data; the augmented training data is input into the pre-trained large model to obtain model parameters; the teacher model and student model are initialized according to the model parameters; and the parameters of the visual modality parsing unit, language modality parsing unit and speech modality parsing unit are configured based on the initialization results.

[0095] Specifically, text information can be enhanced through synonym replacement, random insertion, or deletion; image information can be expanded using methods such as rotation, cropping, and color perturbation; and speech information can be enriched by adding noise and adjusting the speed. Data augmentation can effectively improve the model's robustness to different scenarios and noise interference, reducing the risk of overfitting.

[0096] Subsequently, the system inputs the generated augmented training data into the pre-trained large model for joint training on large-scale multimodal tasks, thereby obtaining model parameters including visual, language, and speech models. Through learning from a large amount of augmented data, the pre-trained large model can capture the rich correlations between multimodal data and form parameter representations with strong generalization ability. Based on these model parameters, the system initializes the teacher and student models separately, providing a unified and high-quality parameter foundation for both. The teacher model retains its original complex structure to carry the complete expression of multimodal knowledge, while the student model is initialized with a simplified structure to facilitate subsequent model compression and transfer.

[0097] Finally, based on the initialization results, the system configures the parameters of the visual modality parsing unit, language modality parsing unit, and speech modality parsing unit to adapt them to the current model parameters, ensuring the synergistic consistency of the three types of parsing units in feature extraction and semantic understanding. Through the above process, the system not only improves the basic capabilities of multimodal feature parsing and fusion, but also provides robust model support for subsequent knowledge distillation and incremental learning, thereby achieving efficient processing and continuous optimization of multimodal data in complex and ever-changing environments.

[0098] Based on the above method, this application also discloses a knowledge distillation system for multimodal large models, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a knowledge distillation system for a multimodal large model provided in an embodiment of this application. The system includes: a receiving module, a fusion module, a transfer module, an acquisition module, and a storage module; wherein,

[0099] The receiving module receives multimodal data sent by the user terminal and inputs the multimodal data into the visual modality parsing unit, language modality parsing unit, and speech modality parsing unit respectively, generating visual feature vectors, language feature vectors, and speech feature vectors accordingly. The fusion module fuses the visual feature vectors, language feature vectors, and speech feature vectors to generate a fused feature matrix. The transfer module transfers the knowledge of the teacher model to the student model based on the fused feature matrix. The teacher model is a pre-trained large model containing a visual model, a language model, and a speech model, while the student model is a simplified model corresponding to the teacher model. The acquisition module acquires the prediction results output by the student model, feeds the prediction results back to the user terminal, and receives feedback information sent by the user terminal. The storage module stores the feedback information in a dynamic memory and performs incremental learning on the student model based on the gradient direction vectors stored in the dynamic memory.

[0100] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0101] Please see Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0102] The communication bus 1002 is used to realize the connection and communication between these components.

[0103] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0104] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0105] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1001 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 1001 and may be implemented as a separate chip.

[0106] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 3 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a multimodal large-scale model knowledge distillation method.

[0107] exist Figure 3In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 1001 can be used to call the application program storing a knowledge distillation method for a multimodal large model in the memory 1005. When executed by one or more processors, the electronic device performs one or more of the methods described in the above embodiments.

[0108] An electronic device readable storage medium stores instructions that, when executed by one or more processors, cause the electronic device to perform one or more of the methods described in the above embodiments.

[0109] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0110] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some service interfaces; indirect couplings or communication connections between devices or units may be electrical or other forms.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0114] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0115] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A knowledge distillation method for a multimodal large model, characterized in that, The method includes: The system receives multimodal data sent by the user terminal and inputs the multimodal data into the visual modality parsing unit, the language modality parsing unit, and the speech modality parsing unit respectively, generating visual feature vectors, language feature vectors, and speech feature vectors accordingly. The visual feature vector, the language feature vector, and the speech feature vector are fused to generate a fused feature matrix. Based on the fused feature matrix, the knowledge of the teacher model is transferred to the student model, wherein the teacher model is a pre-trained large model that includes a visual model, a language model and a speech model, and the student model is a simplified model corresponding to the teacher model; Obtain the prediction results output by the student model, feed the prediction results back to the user terminal, and receive the feedback information sent by the user terminal; The feedback information is stored in a dynamic memory, and the student model is incrementally learned based on the gradient direction vectors stored in the dynamic memory. This process includes: performing feature clustering on the feedback information to generate target samples; calculating the first gradient direction vector corresponding to the target sample and storing the first gradient direction vector in the dynamic memory; calculating the orthogonality constraint between the first gradient direction vector and a second direction vector, and incrementally learning the student model based on the orthogonality constraint, wherein the second direction vector is a historical gradient direction vector stored in the dynamic memory.

2. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that, The multimodal data includes text information, image information, and speech information. The step of inputting the multimodal data into the visual modality analysis unit, the language modality analysis unit, and the speech modality analysis unit respectively, and generating corresponding visual feature vectors, language feature vectors, and speech feature vectors, includes: The text information is input into the language modality parsing unit to generate language feature vectors; The image information is input into the visual modality analysis unit to generate a visual feature vector; The speech information is input into the speech modality parsing unit to generate a speech feature vector.

3. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that, The step of fusing the visual feature vector, the language feature vector, and the speech feature vector to generate a fused feature matrix includes: The first attention weight from the visual feature vector to the language feature vector is calculated using multiple attention heads in the three Transformers, and the second attention weight from the language feature vector to the speech feature vector is calculated. The first attention weight and the second attention weight are integrated through a full-modal fusion head; Based on the integration results, the visual feature vector, the language feature vector, and the speech feature vector are weighted and fused to generate a fused feature matrix.

4. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that, Based on the fused feature matrix, the knowledge transfer from the teacher model to the student model includes: The fusion feature matrix is ​​input into the teacher model and the student model respectively to obtain the first multimodal knowledge representation output by the teacher model and the second multimodal knowledge representation output by the student model. Based on the first multimodal knowledge representation and the second multimodal knowledge representation, the student model is trained according to the joint loss function, which includes KL divergence loss, maximum mean difference loss and task-specific loss. The joint loss function is adjusted to align the second multimodal knowledge representation with the first multimodal knowledge representation.

5. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that, After incremental learning of the student model, the process also includes: Obtain resource parameter information of the target hardware platform; Based on the resource parameter information, the student model is sliced ​​to generate multiple model slices; The loading strategy is determined based on the size of each model fragment. Each of the model fragments is loaded sequentially into the memory of the target hardware platform; The student model is dynamically deployed according to the loading strategy so that the student model is adapted to the computing resources of the target hardware platform.

6. The knowledge distillation method for a multimodal large model according to claim 1, characterized in that, Before receiving the multimodal data sent by the user terminal, the method further includes: Data augmentation processing is performed on the preset training data to generate augmented training data; The enhanced training data is input into the pre-trained large model to obtain the model parameters; The teacher and student models are initialized based on the model parameters. Based on the initialization results, the parameters of the visual modality parsing unit, the language modality parsing unit, and the speech modality parsing unit are configured.

7. A knowledge distillation system for a multimodal large model, characterized in that, The system includes: a receiving module, a fusion module, a migration module, an acquisition module, and a storage module; wherein, The receiving module is used to receive multimodal data sent by the user terminal, and input the multimodal data into the visual modality parsing unit, the language modality parsing unit and the speech modality parsing unit respectively, and generate visual feature vectors, language feature vectors and speech feature vectors accordingly. The fusion module is used to fuse the visual feature vector, the language feature vector and the speech feature vector to generate a fused feature matrix; The transfer module is used to transfer knowledge from the teacher model to the student model based on the fused feature matrix. The teacher model is a pre-trained large model that includes a visual model, a language model, and a speech model, and the student model is a simplified model corresponding to the teacher model. The acquisition module is used to acquire the prediction results output by the student model, feed the prediction results back to the user terminal, and receive feedback information sent by the user terminal. The storage module is used to store the feedback information into a dynamic memory, and to perform incremental learning on the student model based on the gradient direction vectors stored in the dynamic memory. The process of storing the feedback information into the dynamic memory and performing incremental learning on the student model based on the gradient direction vectors stored in the dynamic memory includes: performing feature clustering on the feedback information to generate target samples; calculating the first gradient direction vector corresponding to the target sample and storing the first gradient direction vector into the dynamic memory; calculating the orthogonality constraint between the first gradient direction vector and a second direction vector, and performing incremental learning on the student model according to the orthogonality constraint, wherein the second direction vector is a historical gradient direction vector stored in the dynamic memory.

8. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1-6.

Images