Heterogeneous multi-teacher distillation deployment method for multimodal large models on low-resource devices

Abstract

The present invention discloses a multimodal large-model heterogeneous multi-teacher distillation deployment method for low-resource devices, which relates to the field of multimodal knowledge distillation technology. The method includes taking image-text pairs as input, extracting image embeddings and text embeddings, and determining the contrast distributions of image to text and text to image; calculating and determining image modality weights and text modality weights; weighting the image and text embeddings of the teacher model to obtain fused image embeddings and fused text embeddings; and weighting the contrast distributions of the teacher model to obtain fused image to text and text to image distributions; thereby performing feature distillation and contrast relationship distillation. Complementary multimodal information is extracted from aligned images and text, and distilled into the student model, achieving efficient cross-modal knowledge transfer, enhancing the ability to understand cross-modal semantics, and improving the accuracy of image recognition and classification.

Description

Technical Field

[0001] The present invention relates to the field of multimodal knowledge distillation technology, and in particular to a multimodal large model heterogeneous multi-teacher distillation deployment method for low-resource devices. Background Art

[0002] Compared to traditional visual representation models that rely solely on visual annotations, Contrastive Language-Image Pre-Training (CLIP) introduces textual supervision, effectively injecting rich semantic knowledge from natural language into the visual channel. By jointly training on large-scale image-text pairs, CLIP achieves outstanding performance in tasks such as zero-shot classification, cross-modal retrieval, and dense prediction.

[0003] There is currently research applying contrastive learning methods to medical scenarios. Improving image-text alignment using global-local objectives relies on large-scale paired datasets, which are often scarce in clinical settings. To address this issue, MedCLIP (a multimodal deep learning framework based on model improvement) introduced pre-training on unpaired image-text data to improve the performance of downstream tasks such as disease classification. IMITATE (a machine learning method that trains intelligent agents by imitating expert behavior) incorporates clinical priors and hierarchical supervision to enhance semantic representation. UniChest (a model based on the Transformer model architecture) decouples global and local semantics to better generalize across datasets.

[0004] The aforementioned methods aim to learn robust feature representations during pre-training through large-scale multimodal alignment. However, deploying CLIP models in real-world clinical settings remains challenging. Full-scale CLIP models using ViT-B or ViT-L as backbone networks contain hundreds of millions of parameters and require enormous memory and computational resources, which limits their application in resource-limited edge devices.

[0005] To address this issue, research has introduced knowledge distillation (KD) to transfer knowledge from a large teacher model to a smaller student model. Multi-teacher distillation methods have been introduced to improve the generalization ability of student models by aggregating the knowledge of multiple experts. DistillVLM (a distillation model based on a visual language model) and contrastive relation distillation extend knowledge distillation to multimodal tasks by aligning cross-modal features or relational structures.

[0006] Research has also explored distilling CLIP models for efficient deployment. TinyCLIP (a lightweight version of CLIP compressed using knowledge distillation) introduces affinity imitation, which preserves relational structure but requires architectural alignment between the teacher and student. CLIP-KD overcomes this limitation through a flexible multi-teacher framework, enabling distillation between heterogeneous models. MedCLIP applies distillation in a cross-modal setting, jointly learning images and text for improved efficiency and semantic alignment.

[0007] However, current approaches primarily focus on distilling knowledge from a single teacher model and specialize in single-modal tasks. Distillation for large, multimodal models like CLIP remains ineffective, and they overlook the potential of collaborative guidance from multiple teachers to enhance the cross-domain generalization capabilities of student models. Furthermore, these approaches fail to leverage the cross-modal complementarity of CLIP's dual-encoder architecture and fail to consider the synchronized text-image semantic patterns of different knowledge streams. Consequently, CLIP distillation for heterogeneous, multi-teacher models remains a relatively understudied topic. Summary of the Invention

[0008] To address the above problems, the present invention proposes a multimodal large-model heterogeneous multi-teacher distillation deployment method for low-resource devices. The method extracts complementary multimodal information from aligned images and texts, distills the multimodal information into the student model, realizes efficient cross-modal knowledge transfer, enhances the ability to understand cross-modal semantics, and improves the accuracy of image recognition and classification.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a multi-modal large model heterogeneous multi-teacher distillation deployment method for low-resource devices, comprising:

[0011] In the target cloud computing cluster, image-text pairs are retrieved from a preset database based on the acquired problem domain, and a student model is determined based on the target device, using a preset large language model as the teacher model; the target device is a device whose computing power meets the preset low-resource determination criteria;

[0012] The obtained image-text pairs are used as inputs to the student model and multiple teacher models, and the image embedding and text embedding of each model are obtained. The image embedding and text embedding are used as anchors to determine the image-to-text contrast distribution and text-to-image contrast distribution of each model.

[0013] Based on the image embedding and text embedding of the student model, the similarity scores between the student model and each teacher model are calculated to determine the image modality weight and text modality weight;

[0014] According to the image modality weight and text modality weight, the image embeddings and text embeddings of all teacher models are weighted summed to obtain the fused image embedding and fused text embedding. The contrast distributions of all teacher models are weighted aggregated to obtain the fused distribution of image to text and the fused distribution of text to image.

[0015] Feature distillation is performed based on the image embedding and text embedding, as well as the fused image embedding and fused text embedding of the student model. Contrastive relationship distillation is performed based on the contrast distribution of the student model, as well as the fused distribution of image to text and the fused distribution of text to image, to complete the training of the student model and deploy the trained student model to the target device.

[0016] As an optional embodiment, Image embedding As anchor point, Image-to-text contrastive distribution of teacher models Comparative distribution of image to text with the student model They are:

[0017] ; ;

[0018] First Text embedding As anchor point, Text-to-image contrastive distribution of teacher models Comparative distribution of text to image with the student model They are:

[0019] ; ;

[0020] in, represents the dot product; and All are Teacher Model and student models Temperature scaling factor in ; For the Teacher Model No. text embeddings; For the Teacher Model No. text embeddings; Model for students No. text embeddings; Model for students No. text embeddings; For the Teacher Model No. Image embedding; For the Teacher Model No. Image embedding; Model for students No. Image embedding; Model for students No. Image embedding.

[0021] As an optional implementation, an independent projection layer is set in each teacher model to pass the projection function and Map image embedding and text embedding to shared dimensions respectively The feature space of the student model is normalized to a unit vector after projection, and the similarity score is calculated after the projection layer is processed.

[0022] As an alternative implementation, image embedding based on the student model or text embedding , calculate the student model and the Similarity scores between teacher models for: ;

[0023] Determined modal weights for: ;

[0024] in, For the first image, the image embedding or text embedding generated by the student model is consistent with the Similarity scores between image embeddings or text embeddings generated by the teacher models; For the first image, the image embedding or text embedding generated by the student model is consistent with the The similarity scores between the image embeddings or text embeddings generated by the teacher models, is the number of teacher models; is the sigmoid activation function; is the intermediate parameter; is the similarity weight matrix; is a scaling vector; m represents image modality or text modality; M represents image embedding or text embedding.

[0025] As an optional implementation, in the process of determining the modal weight, a weight regularization loss is introduced for: ;in, is an ideal uniform weight; is the dataset size;

[0026] The weighted regularization loss of the image model and the weighted regularization loss of the text modality are weighted to obtain the weighted regularization total loss.

[0027] As an optional implementation, during the training of the student model, the total training loss function includes CLIP loss, contrastive relationship distillation loss, feature distillation loss and weight regularization total loss;

[0028] Among them, the contrast relationship distillation loss for: ;

[0029] ; ;

[0030] in, It is the fusion distribution from image to text; It is the fusion distribution from text to image; and The contrastive distribution of image to text and text to image for the student model;

[0031] Feature Distillation Loss for: ;in, To fuse image embedding; To fuse text embeddings; and Image embedding and text embedding for the student model.

[0032] In a second aspect, the present invention provides a multimodal large-model heterogeneous multi-teacher distillation deployment system for low-resource devices, comprising:

[0033] A deployment module is configured to obtain image-text pairs from a preset database based on the acquired problem domain in a computing cluster of a target cloud, determine a student model based on a target device, and use a preset large language model as a teacher model; the target device is a device whose computing power meets a preset low resource determination condition;

[0034] a feature extraction module configured to use the acquired image-text pairs as inputs of a student model and a plurality of teacher models, obtain image embeddings and text embeddings of each model, and determine image-to-text contrast distributions and text-to-image contrast distributions of each model using the image embeddings and text embeddings as anchors, respectively;

[0035] a weight calculation module configured to calculate similarity scores between the student model and each teacher model based on the image embedding and text embedding of the student model, thereby determining the image modality weight and the text modality weight;

[0036] A fusion module is configured to perform weighted summation of the image embeddings and text embeddings of all teacher models according to the image modality weight and the text modality weight to obtain a fused image embedding and a fused text embedding, and to perform weighted aggregation on the contrast distributions of all teacher models to obtain a fused image-to-text distribution and a fused text-to-image distribution;

[0037] The distillation module is configured to perform feature distillation based on the image embedding and text embedding of the student model and the fused image embedding and fused text embedding; perform contrastive relationship distillation based on the contrast distribution of the student model and the fused distribution of image to text and the fused distribution of text to image, thereby completing the training of the student model and deploying the trained student model to the target device.

[0038] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the method described in the first aspect when executed by a processor.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention proposes a multimodal large-model heterogeneous multi-teacher distillation deployment method for low-resource devices. The method uses image-text pairs as input to the student model and multiple teacher models, thereby extracting multimodal feature embeddings and image-to-text and text-to-image contrast distributions. Based on the multimodal feature embeddings of the student model, the image modality weight and text modality weight are determined, respectively. The image embeddings and text embeddings of all teacher models are weighted summed to obtain fused image embeddings and fused text embeddings, respectively. The contrast distributions of all teacher models are weighted aggregated to obtain fused image-to-text and fused text-to-image distributions. Feature distillation and contrast relationship distillation are then performed to complete the training of the student model. The present invention extracts complementary multimodal information from aligned images and texts, distills the multimodal information into the student model, and achieves efficient cross-modal knowledge transfer, enhancing its ability to understand cross-modal semantics and improving the accuracy of image recognition and classification.

[0043] This paper designs an adaptive heterogeneous multi-teacher knowledge distillation framework. It utilizes multimodal knowledge distillation to distill heterogeneous multi-teacher models, integrating knowledge from multiple teacher models through instance-based weighting while ensuring lightweight deployment. Compared to existing methods, this paper adaptively learns the importance of each teacher model to each instance, thereby generating an integrated soft target. It also decouples the student model from the teacher architecture, enabling it to operate independently of the teacher model architecture and unconstrained by the teacher model's structure, providing greater model design applicability and deployment flexibility. It outperforms single-teacher and multi-teacher average distillation baselines across multiple classification benchmarks, particularly in resource-constrained environments.

[0044] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 Flowchart of the multimodal large model heterogeneous multi-teacher distillation deployment method for low-resource devices provided in Example 1 of the present invention;

[0047] Figure 2 Schematic diagram of the multimodal large-model heterogeneous multi-teacher distillation deployment method for low-resource devices provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "comprise" and any variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0052] Example 1

[0053] like Figure 1 As shown, this embodiment proposes a multi-modal large model heterogeneous multi-teacher distillation deployment method for low-resource devices, including:

[0054] In the target cloud computing cluster, image-text pairs are retrieved from a preset database based on the acquired problem domain, and a student model is determined based on the target device, using a preset large language model as the teacher model; the target device is a device whose computing power meets the preset low-resource determination criteria;

[0055] The obtained image-text pairs are used as inputs to the student model and multiple teacher models, and the image embedding and text embedding of each model are obtained. The image embedding and text embedding are used as anchors to determine the image-to-text contrast distribution and text-to-image contrast distribution of each model.

[0056] Based on the image embedding and text embedding of the student model, the similarity scores between the student model and each teacher model are calculated to determine the image modality weight and text modality weight;

[0057] According to the image modality weight and text modality weight, the image embeddings and text embeddings of all teacher models are weighted summed to obtain the fused image embedding and fused text embedding. The contrast distributions of all teacher models are weighted aggregated to obtain the fused distribution of image to text and the fused distribution of text to image.

[0058] Feature distillation is performed based on the image embedding and text embedding, as well as the fused image embedding and fused text embedding of the student model. Contrastive relationship distillation is performed based on the contrast distribution of the student model, as well as the fused distribution of image to text and the fused distribution of text to image, to complete the training of the student model and deploy the trained student model to the target device.

[0059] The method proposed in this example includes three core parts: (1) feature extraction from multiple teacher models to capture complementary cross-modal patterns from different expert models; (2) adaptive feature fusion to integrate heterogeneous knowledge through dynamic weight calibration; and (3) dual-objective knowledge distillation to simultaneously optimize image-text alignment and domain-specific semantic preservation.

[0060] The method described in this embodiment achieves efficient cross-modal knowledge transfer through formalized heterogeneous feature extraction, instance-based teacher fusion and dual-objective knowledge distillation mechanism. Multiple pre-trained CLIP models are used as teacher models to extract complementary multimodal information from aligned images and texts, and the multimodal information is distilled into the student model to enhance its ability to understand cross-modal semantics. At the same time, in order to address the heterogeneity of structure and representation between teacher models, an adaptive knowledge fusion Adapter module is proposed. This module evaluates the similarity between the student model and each teacher model in the image and text space, and dynamically assigns importance weights to different teacher models. The feature embeddings of the teacher models are projected into a unified representation space and fused in a weighted manner to form the final distillation signal.

[0061] In this embodiment, in the computing cluster of the target cloud, a target data set including image-text pairs is determined from a preset database based on the acquired problem domain, and the student model is determined in a preset student model library using the configuration information of the target device, and a distillation model training framework is deployed; the target device can be a device whose computing power meets the preset low computing power (low resource) judgment condition.

[0062] It's important to note that the target cloud computing cluster consists of multiple nodes, each equipped with multiple GPUs (Graphics Processing Units) and one or more high-performance CPUs. Before starting a distillation deployment task, the system automatically collects information about the cluster's computing resources, including key parameters such as each node's CPU utilization, GPU memory usage, and network bandwidth. This information provides valuable insights for the subsequent model distillation process.

[0063] When the target cloud receives a distillation deployment instruction, it first determines the problem domain targeted by this distillation task. This problem domain can be determined in various ways. For example, when initiating a distillation deployment request, the user can directly enter a specific problem domain label in the user interface. Upon receiving the label, it is matched against domain classifications in a pre-set database to locate the target dataset. Simultaneously, the target device's configuration information must be obtained. A target device is one whose computing power meets the pre-set low-computing power criteria. This device configuration information can be collected by installing a lightweight information collection tool on the device. This tool automatically collects hardware parameters such as the target device's processor model, memory size, and storage capacity, and uploads this information to the cloud in real time. After receiving the configuration information, the target cloud matches it with the model configuration requirements in the pre-set student model library to determine the target student model. The pre-set student model library can store a variety of small-scale and large-scale models of varying sizes and complexities. These small-scale and large-scale models are optimized for different low-computing power device configurations to ensure they can run on the target device.

[0064] During framework deployment, computing resources can be appropriately allocated to the distillation model training framework based on the current cluster's computing resource availability. For example, on nodes with limited computing resources, the system will prioritize the deployment of lightweight distillation model training modules to prevent training failures due to insufficient resources. Furthermore, the distillation model training framework can be initialized and configured, including setting training parameters and loading necessary dependencies, to ensure proper operation.

[0065] The following combination Figure 2 The method of this embodiment is described in detail by taking the image of the electric power work scene and the corresponding description text of wearing a safety helmet as an example.

[0066] In this example, paired images of power work scenes and corresponding descriptions of helmet wearing were extracted from an existing dataset. Preliminary data cleaning and preprocessing were performed, and the images were uniformly cropped to a uniform size. The corresponding descriptions were extracted, irrelevant information removed, and format errors corrected to ensure the quality and consistency of the input data. Finally, the dataset was randomly divided into training, validation, and test sets in a specific ratio (e.g., 8:1:1). The validation set was used for hyperparameter tuning, and the test set was used to evaluate the model's generalization performance.

[0067] In this example, a network architecture based on the CLIP model is established using multiple pre-trained CLIP models as teacher models. The core of each student model and each teacher model consists of a bimodal encoder: an image encoder and a text encoder. The image encoder extracts visual features from images, while the parallel text encoder extracts semantic information embedded in the corresponding text.

[0068] Specifically, given a set of images including paired image-text pairs As input, it is input into the image encoder and text encoder of the student model and multiple teacher models respectively. All models independently generate feature embeddings of specific modalities, namely The image embeddings and text embeddings generated by the teacher model and student model are and ;

[0069] in, For the images; For the The text corresponding to the image; For image sets The amount of data; For the Teacher Model For the first Image embedding generated from the image; For the Teacher Model For the first Text embedding generated by the text corresponding to the image; Model for students For the first Image embedding generated from the image; Model for students For the first The text embedding generated by the text corresponding to the image.

[0070] To establish semantic alignment between images and text, all models employ the contrastive learning strategy introduced in CLIP, optimizing the similarity of matching pairs while simultaneously widening the distance between mismatched pairs. The underlying principle is to maximize the similarity between matching image-text pairs while minimizing the similarity between mismatched pairs, effectively synchronizing multimodal representations into a unified representation domain.

[0071] The first Image embedding As an anchor point, Image-to-text contrastive distribution of teacher models Comparative distribution of image to text with the student model They are defined as:

[0072] ;

[0073] ;

[0074] in, represents the dot product, which is used to measure similarity; and It is Teacher Model and student models A learnable temperature scaling factor in ; Used to index the contrast distribution; in practice, negative samples are from Extraction; For the Teacher Model No. text embeddings; For the Teacher Model No. text embeddings; Model for students No. text embeddings; Model for students No. Text embedding.

[0075] Similarly, Text embedding As an anchor point, Text-to-image contrastive distribution of teacher models Comparative distribution of text to image with the student model They are defined as:

[0076] ;

[0077] ;

[0078] in, For the Teacher Model No. Image embedding; For the Teacher Model No. Image embedding; Model for students No. Image embedding; Model for students No. Image embedding.

[0079] Multimodal representations and contrastive models from multiple sources drive two complementary distillation paths: Feature Distillation (FD) for embedding alignment and Contrastive Representation Distillation (CRD) for distribution matching. The implementation of dual-objective knowledge distillation requires strategic fusion through the Adapter module for adaptive knowledge fusion.

[0080] To effectively integrate image and text knowledge from multiple heterogeneous teacher models, we propose an Adapter module. The main goal of this module is to dynamically aggregate multimodal features from multiple teacher models on a per-sample basis while maintaining semantic consistency with the student model representation. The resulting fused features serve as a unified supervisory signal for distilling knowledge into the student model.

[0081] Specifically include the following:

[0082] (1) Feature projection and normalization.

[0083] Given the differences in feature dimensions and distributions among different teacher models, the Adapter sets up an independent projection layer for each teacher model to map its image embedding and text embedding into the feature space of the student model.

[0084] Specifically:

[0085] make and Respectively represent The image embeddings and text embeddings generated by the teacher model are For the The output dimension of the teacher model is obtained by the corresponding projection function in the projection layer. and Map image embedding and text embedding to a shared dimension The projected image is embedded in the space and the projected text embedding :

[0086] ;

[0087] ;

[0088] in, ;if , then the projection function degenerates into the identity mapping.

[0089] All projected features are Normalized to unit vector for scale-invariant similarity calculation.

[0090] (2) Adaptive weight calculation.

[0091] In order to evaluate the contribution of each teacher model to a given sample, a weighting mechanism based on the similarity of the output features of the teacher model and the student model is designed.

[0092] For the image modality, a set of learnable parameters is introduced: the similarity weight matrix and scale vector ,in, is the number of teacher models, Represents the dimension of the student model feature space.

[0093] Image embedding given the normalized student model , student model and Similarity scores between teacher models Calculated as:

[0094] ;

[0095] ;

[0096] in, Represents the sigmoid activation function, which is used to stabilize the scaling factor; is the intermediate parameter.

[0097] Finally, the image modality weights are obtained by applying the softmax activation function to all teacher models :

[0098] ;

[0099] in, For the first image, the image embedding generated by the student model is the same as the Similarity scores between image embeddings generated by the teacher models; For the first image, the image embedding generated by the student model is the same as the Similarity scores between image embeddings generated by two teacher models.

[0100] Similarly, for the text modality, a set of learnable parameters is introduced: the similarity weight matrix and scale vector ,in, is the number of teacher models, Represents the dimension of the student model feature space.

[0101] Given the normalized text embedding of the student model , student model and Similarity scores between teacher models Calculated as:

[0102] ;

[0103] ;

[0104] in, Represents the sigmoid activation function, which is used to stabilize the scaling factor; is the intermediate parameter.

[0105] Finally, the text modality weights are obtained by applying the softmax activation function to all teacher models :

[0106] ;

[0107] in, For the first Image, the text embedding generated by the student model is the same as the Similarity scores between text embeddings generated by the teacher models; For the first Image, the text embedding generated by the student model is the same as the Similarity scores between text embeddings generated by two teacher models.

[0108] Therefore, the contribution of the teacher model is dynamically adjusted according to the similarity between the student model and the teacher model, achieving sample-based adaptive fusion.

[0109] (3) Weighted feature fusion.

[0110] According to the calculated image modality weight and text modality weight , the Adapter performs a weighted summation of the image embeddings and text embeddings generated by all teacher models to generate a unified fused image embedding and fused text embedding :

[0111] ;

[0112] .

[0113] right and Normalized and compared with the one generated by the student model and The alignment is then used in the feature distillation process.

[0114] Secondly, in addition to feature embedding, the Adapter also performs weighted aggregation on the contrast distribution (logits) generated by each teacher model to obtain the fused distribution of image to text and text-to-image fusion distribution , for contrastive relation distillation.

[0115] (4) Regularization weight calculation.

[0116] In order to prevent the student model from over-relying on a specific teacher model during the feature fusion process, weight regularization constraints are introduced to encourage uniform weight distribution among all teacher models, promote comprehensive knowledge utilization, and reduce the risk of biased fusion dominated by some teacher models.

[0117] For image modality, the weight regularization loss It is calculated by the mean squared error between the learned weights and the uniform target:

[0118] ;

[0119] in, is the number of teacher models, is an ideal uniform weight.

[0120] Similarly, we get the weight regularization loss of text modality .

[0121] Therefore, the adapter weight regularization loss Defined as:

[0122] ;

[0123] in, and They are the weighted coefficients of the image modality weight regularization loss and the text modality weight regularization loss, respectively, which control the strength of regularization.

[0124] By introducing weight regularization, the weight generation process of the Adapter module becomes more stable, providing a more balanced guidance signal for the student model, thereby improving the overall effect of multimodal knowledge distillation.

[0125] In this example, we introduce two distillation strategies: feature distillation and contrastive relationship distillation. Combined with the proposed Adapter module, we utilize weighted teacher embeddings and contrastive distributions to perform multi-teacher knowledge fusion and distillation. The following describes the design details of each loss function.

[0126] (1) Contrast relationship distillation.

[0127] The CLIP model is based on maximizing the similarity between paired image-text embeddings through contrastive similarity distributions. For contrastive relation distillation, the primary knowledge type is the output-oriented contrastive distribution, which effectively captures the structured relationships between feature embeddings. Using contrastive relation distillation in multi-teacher distillation enables the student model to better replicate the structured semantic relationships from multiple teacher models, thereby improving the quality of feature representation.

[0128] In this example, the proposed contrastive relationship distillation method uses the sample-adaptive fusion weights generated by the Adapter module to align the contrast distribution of the student model with the weighted fusion distributions of image-to-text and text-to-image. By using the Kullback-Leibler (KL) divergence, the student model is ensured to learn robust cross-modal representations from the aggregated teacher knowledge.

[0129] Specifically: No. The contrast distributions of image-to-text and text-to-image generated by the teacher model are expressed as and ,in It is a mini-batch dataset of paired image-text samples. Image modality weights using the Adapter module and text modality weight , the calculated image to text fusion distribution and text-to-image fusion distribution They are:

[0130] ;

[0131] .

[0132] By calculating the contrast distribution of the Student model and After fusion with and The KL divergence of the contrast relationship distillation is used to align the contrast distribution of the student model with the contrast distribution of the fused teacher model. Therefore, the contrast relationship distillation loss is constructed by the KL divergence. It is calculated by averaging the losses in two directions, and the expression is:

[0133] ;

[0134] ;

[0135] ;

[0136] in, is the KL divergence of the contrast distribution of image to text and the fusion distribution of image to text of the student model; It is the KL divergence between the contrast distribution of text to image and the fusion distribution of text to image of the student model.

[0137] (2) Feature distillation.

[0138] Feature Distillation (FD) is introduced to further align the feature embeddings of the student model with those of the fused teacher model, thereby enhancing the student model's fine-grained alignment capabilities in the representation space. FD complements CRD: while CRD focuses on aligning cross-modal semantic relationships, FD focuses on directly aligning image and text feature embeddings. Intuitively, if the features of the student model can be closely aligned with those of the teacher model, the performance gap between the student and teacher models can be significantly narrowed.

[0139] Specifically: The image embedding and text embedding of the student model are expressed as and , and has been carried out Normalized. The fused image embedding and fused text embedding generated by the Adapter module are represented as and , and normalized accordingly.

[0140] It is worth noting that since the feature dimensions of different teacher models may not match those of the student model, the features of the teacher model have been mapped into the feature space of the student model through the projection layer to ensure the consistency of the dimensions.

[0141] By computing the image embedding of the student model and text embedding and fused image embedding and fused text embedding The mean square error loss is used to perform feature distillation; thus, the feature distillation loss Defined as:

[0142] .

[0143] By minimizing the feature distillation loss , the student model can directly learn fine-grained information from the fused teacher model features, thereby enhancing its multimodal representation ability.

[0144] (3) Total loss function.

[0145] To jointly train the student model, CLIP loss is combined with knowledge distillation loss to form the total loss function. CLIP loss is used to train the student model so that it can learn the alignment between image and text representations.

[0146] Contrastive loss from image embedding to text embedding For example, it is defined as:

[0147] ;

[0148] in, is a learnable temperature parameter used to scale the similarity; Indicates the mini-batch dataset size.

[0149] Similarly, we get the contrast loss of text embedding to image embedding .

[0150] Therefore, CLIP loss Combining the contrastive loss of image embedding to text embedding and text embedding to image embedding, it is expressed as follows: .

[0151] Knowledge distillation loss includes contrastive relationship distillation, feature distillation and adapter weight regularization loss, which are: contrastive relationship distillation loss Used to align cross-modal semantic relations, feature distillation loss Feature embedding and adapter weight regularization loss for aligning student models and fusion teacher models Used to regularize the weight generation process. Knowledge distillation loss It is constructed by weighted combination of the above losses, defined as:

[0152] ;

[0153] in, and are hyperparameters that adjust the weights of the contrastive relationship distillation loss and the feature distillation loss, respectively, to balance the contribution of different loss terms to the student model training.

[0154] By jointly optimizing the above losses, the student model can effectively learn multimodal knowledge while maintaining the stability of feature fusion.

[0155] The Adam optimizer is used to update the student model parameters, iterating until the model converges or reaches a preset number of epochs. At the end of each epoch, the model performance is evaluated on the validation set, and the best-performing model parameters are saved for subsequent testing.

[0156] In this embodiment, the process of deploying the trained student model includes:

[0157] (1) Model encapsulation: Encapsulate the trained student model into a standardized prediction interface and provide necessary documentation.

[0158] (2) System integration: Integrate the model into the power system and connect it with other modules such as database and Web front-end display to ensure smooth data flow.

[0159] (3) Online testing: Test the actual performance of the model in a small-scale real-world scenario, and fine-tune and optimize the model based on the feedback.

[0160] (4) Model online: The model is officially deployed online to assist operation and maintenance personnel in image recognition and improve analysis efficiency. At the same time, the model's online recognition results are continuously monitored to ensure their stability and rationality.

[0161] (5) Model update: Regularly retrain the model using newly collected power work scenario data to adapt to new demands and changes in data distribution.

[0162] It should be noted that all data is obtained in compliance with laws and regulations and with user consent, and the data is used legally.

[0163] Example 2

[0164] This embodiment provides a multimodal large-model heterogeneous multi-teacher distillation deployment system for low-resource devices, including:

[0165] A deployment module is configured to obtain image-text pairs from a preset database based on the acquired problem domain in a computing cluster of a target cloud, determine a student model based on a target device, and use a preset large language model as a teacher model; the target device is a device whose computing power meets a preset low resource determination condition;

[0166] a feature extraction module configured to use the acquired image-text pairs as inputs of a student model and a plurality of teacher models, obtain image embeddings and text embeddings of each model, and determine image-to-text contrast distributions and text-to-image contrast distributions of each model using the image embeddings and text embeddings as anchors, respectively;

[0167] a weight calculation module configured to calculate similarity scores between the student model and each teacher model based on the image embedding and text embedding of the student model, thereby determining the image modality weight and the text modality weight;

[0168] A fusion module is configured to perform weighted summation of the image embeddings and text embeddings of all teacher models according to the image modality weight and the text modality weight to obtain a fused image embedding and a fused text embedding, and to perform weighted aggregation on the contrast distributions of all teacher models to obtain a fused image-to-text distribution and a fused text-to-image distribution;

[0169] The distillation module is configured to perform feature distillation based on the image embedding and text embedding of the student model and the fused image embedding and fused text embedding; perform contrastive relationship distillation based on the contrast distribution of the student model and the fused distribution of image to text and the fused distribution of text to image, thereby completing the training of the student model and deploying the trained student model to the target device.

[0170] It should be noted that the above modules correspond to the steps described in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0171] In further embodiments, there is also provided:

[0172] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed by the processor, wherein when the computer instructions are executed by the processor, the method described in Example 1 is performed. For the sake of brevity, no further details are given here.

[0173] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0174] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0175] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in Example 1 is performed.

[0176] The method in Example 1 can be directly implemented as a hardware processor, or can be implemented using a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, it will not be described in detail here.

[0177] A computer program product includes a computer program, which implements the method described in embodiment 1 when executed by a processor.

[0178] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions contained in program modules, which are executed in a device on a real or virtual processor of a target to perform the process / method described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided between program modules as needed. The machine-executable instructions for the program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0179] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the computer or other programmable data processing device, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0180] In the context of the present invention, computer program code or related data can be carried by any appropriate carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc. Examples of signals include electrical, optical, radio, acoustic, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0181] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0182] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A multi-modal large-model heterogeneous multi-teacher distillation deployment method for low-resource devices, characterized by: include: In the target cloud computing cluster, image-text pairs are retrieved from a preset database based on the acquired problem domain, and a student model is determined based on the target device, using a preset large language model as the teacher model; the target device is a device whose computing power meets the preset low-resource determination criteria; The obtained image-text pairs are used as inputs to the student model and multiple teacher models, and the image embedding and text embedding of each model are obtained. The image embedding and text embedding are used as anchors to determine the image-to-text contrast distribution and text-to-image contrast distribution of each model. Based on the image embedding and text embedding of the student model, the similarity scores between the student model and each teacher model are calculated to determine the image modality weight and text modality weight; According to the image modality weight and text modality weight, the image embeddings and text embeddings of all teacher models are weighted summed to obtain the fused image embedding and fused text embedding. The contrast distributions of all teacher models are weighted aggregated to obtain the fused distribution of image to text and the fused distribution of text to image. Perform feature distillation based on the image embedding and text embedding of the student model, as well as the fused image embedding and fused text embedding. Perform contrastive relationship distillation based on the contrast distribution of the student model, as well as the fused distribution of image to text and the fused distribution of text to image, to complete the training of the student model and deploy the trained student model to the target device. Set up an independent projection layer in each teacher model to pass the projection function and Map image embedding and text embedding to shared dimensions respectively The feature space of the student model is normalized to a unit vector after projection, and the similarity score is calculated after the projection layer is processed. Image embedding according to the student model or text embedding , calculate the student model and the Similarity scores between teacher models for: ; Determined modal weights for: ; in, For the first image, the image embedding or text embedding generated by the student model is consistent with the Similarity scores between image embeddings or text embeddings generated by the teacher models; For the first image, the image embedding or text embedding generated by the student model is consistent with the The similarity scores between the image embeddings or text embeddings generated by the teacher models, is the number of teacher models; is the sigmoid activation function; is the intermediate parameter; is the similarity weight matrix; is a scaling vector; m represents image modality or text modality; M represents image embedding or text embedding.

2. The multimodal large model heterogeneous multi-teacher distillation deployment method for low-resource devices according to claim 1 is characterized in that: First Image embedding As anchor point, Image-to-text contrastive distribution of teacher models Comparative distribution of image to text with the student model They are: ； ； First Text embedding As anchor point, Text-to-image contrastive distribution of teacher models Comparative distribution of text to image with the student model They are: ； ； in, represents the dot product; and All are Teacher Model and student models Temperature scaling factor in ; For the Teacher Model No. text embeddings; For the Teacher Model No. text embeddings; Model for students No. text embeddings; Model for students No. text embeddings; For the Teacher Model No. Image embedding; For the Teacher Model No. Image embedding; Model for students No. Image embedding; Model for students No. Image embedding.

3. The multimodal large model heterogeneous multi-teacher distillation deployment method for low-resource devices according to claim 1 is characterized in that: In the process of determining the modal weight, the weight regularization loss is introduced for: ;in, is an ideal uniform weight; is the dataset size; The weighted regularization loss of the image model and the weighted regularization loss of the text modality are weighted to obtain the weighted regularization total loss.

4. The multimodal large model heterogeneous multi-teacher distillation deployment method for low-resource devices according to claim 3 is characterized in that: During the training of the student model, the total training loss function includes CLIP loss, contrastive relationship distillation loss, feature distillation loss and weight regularization total loss; Among them, the contrast relationship distillation loss for: ; ； ； in, It is the fusion distribution from image to text; It is the fusion distribution from text to image; and The contrastive distribution of image to text and text to image for the student model; Feature Distillation Loss for: ;in, To fuse image embedding; To fuse text embeddings; and Image embedding and text embedding for the student model.

5. A multi-modal large-model heterogeneous multi-teacher distillation deployment system for low-resource devices, characterized by: include: A deployment module is configured to obtain image-text pairs from a preset database based on the acquired problem domain in a computing cluster of a target cloud, determine a student model based on a target device, and use a preset large language model as a teacher model; the target device is a device whose computing power meets a preset low resource determination condition; a feature extraction module configured to use the acquired image-text pairs as inputs of a student model and a plurality of teacher models, obtain image embeddings and text embeddings of each model, and determine image-to-text contrast distributions and text-to-image contrast distributions of each model using the image embeddings and text embeddings as anchors, respectively; a weight calculation module configured to calculate similarity scores between the student model and each teacher model based on the image embedding and text embedding of the student model, thereby determining the image modality weight and the text modality weight; Set up an independent projection layer in each teacher model to pass the projection function and Map image embedding and text embedding to shared dimensions respectively The feature space of the student model is normalized to a unit vector after projection, and the similarity score is calculated after the projection layer is processed. Image embedding according to the student model or text embedding , calculate the student model and the Similarity scores between teacher models for: ; Determined modal weights for: ; in, For the first image, the image embedding or text embedding generated by the student model is consistent with the Similarity scores between image embeddings or text embeddings generated by the teacher models; For the first image, the image embedding or text embedding generated by the student model is consistent with the The similarity scores between the image embeddings or text embeddings generated by the teacher models, is the number of teacher models; is the sigmoid activation function; is the intermediate parameter; is the similarity weight matrix; is a scaling vector; m represents image modality or text modality; M represents image embedding or text embedding; A fusion module is configured to perform weighted summation of the image embeddings and text embeddings of all teacher models according to the image modality weight and the text modality weight to obtain a fused image embedding and a fused text embedding, and to perform weighted aggregation on the contrast distributions of all teacher models to obtain a fused image-to-text distribution and a fused text-to-image distribution; The distillation module is configured to perform feature distillation based on the image embedding and text embedding of the student model and the fused image embedding and fused text embedding; perform contrastive relationship distillation based on the contrast distribution of the student model and the fused distribution of image to text and the fused distribution of text to image, thereby completing the training of the student model and deploying the trained student model to the target device.

6. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 4 is completed.

7. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 4 when executed by a processor.

Images