A text-guided knowledge distillation method based on attribute-driven fusion

Abstract

The present invention discloses a text-guided knowledge distillation method based on attribute-driven fusion. The method first constructs a multimodal dataset and hard attribute strings of each modality, and inputs the multimodal dataset and hard attribute strings into an attribute-driven cross-modal dynamic fusion module to achieve multimodal image adaptive fusion. Then, based on the combination of hard attribute strings, a learnable alternating prompt text sequence is generated, and the fused image features are feature modulated via a multimodal large model text encoder and a multi-layer perceptron. Finally, a teacher model is trained based on the above fusion and modulation features, and the student model is guided to complete knowledge distillation. The present invention effectively improves the robustness of the features constructed by the teacher model and the comprehensiveness of the knowledge it transfers, and significantly enhances the learning efficiency, generalization ability and performance of the student model in a complex practical environment with multiple noise interferences.

Description

Technical Field

[0001] The present invention relates to the field of knowledge distillation based on deep learning, and in particular to a text-guided knowledge distillation method based on attribute-driven fusion. Background Art

[0002] With the rapid development of deep learning technology, model structures are becoming increasingly complex. While this has improved model accuracy and generalization capabilities to a certain extent, it has also led to a sharp increase in computing resource requirements, resulting in significant challenges for deployment in resource-constrained scenarios such as edge computing. Knowledge distillation has been proposed as an important model optimization technique. Its ability to preserve model performance while significantly improving computational efficiency has garnered widespread attention in both research and practice. Knowledge distillation can transfer knowledge from a large teacher model to a smaller student model, improving both model efficiency and generalization.

[0003] In complex practical application environments, the teacher model of traditional knowledge distillation methods is usually trained based on single-modal input. However, the limited information of single-modal RGB images limits the learning effect of the student model. First, because the information dimension contained in the single-modal RGB image itself is relatively single and lacks effective redundant information or complementary clues, the features extracted by the model learning are generally low in signal-to-noise ratio. The model shows poor robustness to the widespread and diverse noise interference in the environment and is easily affected by negative interference. Secondly, the quality of single-modal RGB images is highly dependent on lighting conditions. This dependence makes it particularly fragile and unstable in extreme or non-ideal lighting scenes such as backlight and dark light. At the same time, in these scenes, single-modal RGB images are prone to degradation phenomena such as underexposure, overexposure, contrast reduction and noise increase. The combined effect of multiple degradation factors leads to serious loss or significant distortion of key visual features such as texture, edge, color, structure, etc. that are originally required to define and identify objects in the image, greatly weakening the reliability of model feature extraction and recognition. At the same time, single-modal RGB has obvious limitations in providing sufficient semantic expression.

[0004] In some scenarios, it is difficult to stably, reliably, and fully express and convey the required precise semantics. This makes the performance of the teacher model itself limited and unstable in complex real-world scenarios, and ultimately degrades the feature extraction and recognition performance of the student model. Summary of the Invention

[0005] In response to the problems of traditional knowledge distillation mentioned above, the present invention proposes a text-guided knowledge distillation method based on attribute-driven fusion. This method is mainly divided into three stages: attribute-driven cross-modal dynamic fusion, combination-like prompt modulation and joint distillation. First, based on the idea of ​​modal complementarity, edge images and saliency images are introduced to complete the information missing of the original RGB images in real tasks, and the model is optimized to adapt to a wider range of application scenarios, so as to solve the problem that the model is easily affected by noise interference in actual application, resulting in a decrease in accuracy. At the same time, the present invention also introduces attribute text information for fusion weight generation and feature modulation, aiming to give semantic-level interpretation and guidance to different visual modalities, so that the input features are more in line with the needs of the real environment. Finally, through a joint distillation strategy, the present invention efficiently and robustly transfers the teacher model knowledge, which contains multi-modal complementary information and semantic attribute alignment after attribute-driven fusion and combination-like prompt modulation optimization, to a lightweight student model.

[0006] The specific steps of the method of the present invention include:

[0007] S1. Constructing a multimodal dataset: For the RGB image dataset, we generate a multimodal image dataset using edge detection and saliency detection image processing techniques. At the same time, we construct hard attribute strings for each modality. The specific steps are as follows:

[0008] S11. Get RGB image dataset, RGB image dataset consists of RGB images Composition, of which is the dimension of the RGB image, is the number of channels of the RGB image, and is the height and width of the RGB image.

[0009] S12. Perform edge detection and saliency detection on the RGB dataset and generate corresponding edge images respectively and saliency images ,in, is the number of channels of the edge image, is the number of channels of the saliency image. The multimodal image dataset consists of RGB images , edge image , saliency image Three types of images.

[0010] S13. Construct hard attribute strings for each modality, including RGB image feature attributes, edge image feature attributes, and saliency image feature attributes; each category in each modality corresponds to a universal hard attribute string, wherein the RGB image feature attributes are composed of an object subject description and an RGB core attribute description, wherein the RGB core attribute description is a description of the RGB image that focuses on displaying color information, geometric information, and environmental information; the edge image feature attributes are composed of an object subject description, an edge detection algorithm description, and an edge core attribute description, wherein the edge core attribute description is a description of the edge image that focuses on displaying edge contour information and edge line strength; the saliency image feature attributes are composed of an object subject description, a saliency detection algorithm description, and a saliency core attribute description, wherein the saliency core attribute description is a description of the saliency image that focuses on displaying background contour information and the degree of separation between the subject and the background.

[0011] S2. Construct an attribute-driven cross-modal dynamic fusion module: Input the multimodal image dataset constructed in step S12 and the hard attribute strings of each modality constructed in step S13 into the attribute-driven cross-modal dynamic fusion module to guide the dynamic generation of weights, realize the adaptive fusion of the multimodal image dataset, and obtain the fusion features. The designed attribute-driven cross-modal dynamic fusion module consists of four sub-modules: image shallow feature extraction, attribute text feature extraction and folding expansion, attribute-driven weight matrix generation, and feature dynamic fusion. The specific steps are as follows:

[0012] S21. Input the multimodal image dataset obtained in step S12 into the shallow feature extraction network for feature extraction, and obtain RGB image features, edge image features, and saliency image features respectively. The shallow feature extraction network is a convolution module used to extract shallow features. The extracted shallow feature dimensions are , is the number of channels of shallow features.

[0013] S22. Input the most representative hard attribute character string of each modality constructed in step S13 into the multimodal large model text encoder for text feature extraction, and then perform feature folding and expansion to obtain RGB image attribute text features. , edge image attribute text features , salient image attribute text features , as follows:

[0014] The hard attribute string RGB image feature attributes, edge image feature attributes, and saliency image feature attributes of each modality are input into the multimodal large model text encoder for text feature extraction. The text features corresponding to each modality are extracted. The dimension of the text features is .

[0015] The text features corresponding to each modality are folded and expanded in spatial dimensions respectively, and the text features are transformed from After the dimension is folded into a single channel, it is expanded to Dimension, get RGB image attribute text features , edge image attribute text features , salient image attribute text features .

[0016] S23. Input the RGB image features, edge image features, saliency image features and RGB image attribute text features, edge image attribute text features, and saliency image attribute text features into the convolution layer to obtain the RGB image weights , edge image weight , saliency image weight , as follows:

[0017] The RGB image features, edge image features and saliency image features are spliced ​​with the corresponding RGB image attribute text features, edge image attribute text features and saliency image attribute text features in the channel dimension to obtain the three-modal image-text splicing features. The three spliced ​​features are then spliced ​​again in the channel dimension to obtain the overall feature vector containing all modal image information. The splicing feature dimensions of each modality image are , the overall eigenvector The dimension is .

[0018] The overall feature vector Input to the convolution layer to generate convolution output, and then pass The activation function obtains the weight matrix The weight matrix contains the weights of the three different modal images.

[0019] The weight matrix generated Split into 3 different weights, take The 0th channel is used as the RGB image weight ,Pick The first channel of is used as the edge image weight ,Pick The second channel of is used as the saliency image weight , each weight corresponds to an image modality.

[0020] S24. Weight the RGB image , edge image weight , saliency image weight Perform weighted dynamic feature fusion with the RGB image features, edge image features, and saliency image features obtained in step S21 to obtain the fusion feature , as follows:

[0021] The RGB image features, edge image features, and saliency image features are Function mapping to obtain RGB image activation features, edge image activation features, and saliency image activation features

[0022] Weight the generated RGB image , edge image weight , saliency image weight The RGB image activation features, edge image activation features, and saliency image activation features are weighted element by element, and then normalized to generate intermediate weighted features of each modality. Finally, the three types of intermediate features are weighted and fused to obtain the fusion feature. .

[0023] S3. Combination prompt modulation module: Combine the most representative hard attribute strings of each modality constructed in step S13 with the corresponding learnable soft prompts to form a learnable alternating prompt text sequence , will be able to learn alternating prompt text sequences Input to the multimodal large model text encoder to generate a learnable alternating prompt embedding vector , and then embed the learnable alternating prompt vector Through multi-layer perceptron Generate parameters to fusion features Perform personalized feature modulation to obtain modulation fusion features The specific steps are as follows:

[0024] S31. Alternately combine the RGB image feature attributes, edge image feature attributes, saliency image feature attributes, and corresponding learnable soft prompts constructed in step S13 to form a learnable alternating prompt sequence. .

[0025] S32. Input learnable alternating prompt sequence through pre-trained multimodal large model text encoder Converted to the corresponding learnable alternating prompt embedding vector .

[0026] S33. Embedding the learnable alternating prompt into vectors Input to a multilayer perceptron In the module, two key parameters of feature modulation are obtained: scaling factor and bias .

[0027] Multilayer Perceptron The output dimension is ,forward Dimension as a scaling factor ,back Dimension as a bias term .

[0028] S34. By scaling factor and bias Fusion features Each channel of the feature map is dynamically scaled and translated to obtain the modulated fusion feature The formula is:

[0029]

[0030] S4. Knowledge distillation: Construct the teacher model and the student model, and combine the modulated fusion features obtained in step S3 Used to pre-train the teacher model to obtain the teacher's predicted category distribution ; The RGB image dataset obtained in step S11 Input into the shallow feature extraction module, extract shallow features and input into the backbone network of the student model for training, and output the student's predicted category distribution The knowledge of the teacher model is transferred to the student model through the knowledge distillation technology, and the classification loss and distillation loss are combined to the student model and the learnable alternating prompt sequence obtained in step S31. Optimize to improve its inference speed and performance. The specific steps are as follows:

[0031] S41. The modulation fusion feature obtained in step S34 Input to the teacher model, using modulation fusion features Train the teacher model to obtain the teacher's predicted category distribution .

[0032] S42. The RGB image dataset obtained in step S11 Input shallow feature extraction network Perform feature extraction to obtain a single RGB image feature, and then input the single RGB image feature into the student model for training to obtain the student prediction category distribution .

[0033] S43. Calculate the difference between the output distribution of the student model and the output distribution of the teacher model as the distillation loss ;

[0034] S44. Calculate the image classification loss of the student model and output the predicted category of the student model and the true label Combined with cross entropy loss Calculate and get the classification loss of the student model ;in is the number of image categories.

[0035] S45. Total Loss of Text Dynamics-Guided Knowledge Distillation Method for Attribute-Driven Cross-Modal Fusion Distillation loss With classification loss The weighted sum of , the parameters of the student model's backbone network, learnable prompt text embedding, and feature modulation module are all updated synchronously, thus achieving end-to-end training.

[0036] Beneficial effects of the present invention:

[0037] A text-guided knowledge distillation method based on attribute-driven fusion expands the data input of the teacher model by combining an attribute-driven cross-modal dynamic fusion module and a combined class cue modulation module, achieving robust feature construction and more comprehensive knowledge transfer. The method first processes three images with different information: RGB images, edge images, and saliency images. Then, by combining attribute-driven weight matrices, an attribute-driven cross-modal dynamic fusion module is generated based on the features of the three images. A combined class cue modulation module is introduced to modulate the features of the dynamically fused cross-modal features. From these modules, a dynamic adjustment mechanism is used to continuously refine the teacher feature input that is more suitable for knowledge distillation, thereby continuously adjusting the different proportions of information provided by the teacher model in knowledge distillation and making the distillation more generalizable. The present invention has the following benefits: 1. It introduces diversified data processing to expand the input. By performing edge detection and saliency detection on the original three-channel RGB dataset, additional input data is generated, increasing the information content and diversity of the model input. This enables the model to maintain high performance even in complex or changing environments. 2. It designs an attribute-driven cross-modal dynamic fusion module. This design achieves adaptive fusion by combining attribute-driven weight matrix generation to achieve dynamic weight allocation. By adjusting the contribution of each image type at different locations, the model can more accurately capture key information in the image. 3. A combined cue modulation module was designed: This module leverages a pre-trained multimodal large-scale text encoder to extract learnable text semantic information and dynamically modulate the fused features. Simultaneously, backpropagation continuously updates the learnable text cues during training, enabling the model to more flexibly represent features based on specific task requirements, thereby improving its adaptability and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a diagram of the architecture of the present invention;

[0039] Figure 2 Schematic diagram of the attribute-driven cross-modal dynamic fusion module;

[0040] Figure 3 This is a schematic diagram of the modulation module for the combination type prompt. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] As a preferred implementation form of the present invention, a text-guided knowledge distillation method based on attribute-driven fusion is provided, and its architecture is as follows: Figure 1 As shown, the following steps are included:

[0043] S1. Constructing a multimodal dataset: For the RGB image dataset, we generate a multimodal image dataset using image processing techniques such as edge detection and saliency detection. At the same time, we construct the most representative hard attribute strings for each modality. The specific steps are as follows:

[0044] S11. Obtain RGB image dataset , RGB image dataset A series of RGB images Composition, of which is the dimension of the RGB image, 3 is the number of channels of the RGB image, and 32 and 32 are the height and width of the RGB image.

[0045] S12. For RGB dataset Perform edge detection and saliency detection on all RGB images in the image and generate corresponding edge images respectively. and saliency images ,in is the dimension of the edge image and the dimension of the saliency image, is the number of channels of edge image and saliency image, 32 and 32 are the height and width of edge image and saliency image. , edge image , saliency image Three types of images.

[0046] S13. Construct the most representative hard attribute string set of each modality, including RGB image feature attributes , edge image feature attributes , salient image feature attributes Each category in each modality corresponds to a universal hard attribute string. Among them, the RGB image feature attribute It consists of object main body description and RGB core attribute description, where the RGB core attribute description is a description of the RGB image that focuses on showing color information, geometric information, and environmental information; the edge image feature attribute It consists of object subject description, edge detection algorithm description, and edge core attribute description. The edge core attribute description is a description of the edge image that focuses on showing edge contour information and edge line intensity; the saliency image feature attribute It consists of the description of the object body, the description of the saliency detection algorithm, and the description of the saliency core attributes. The description of the saliency core attributes focuses on showing the background contour information and the separation degree between the subject and the background for the saliency image. The specific structure is as follows:

[0047] ]

[0048] ]

[0049] ]

[0050] The object body description is provided by the corresponding category label, and the core attribute description of each modality is the core visual feature description of each modality.

[0051] S2. Attribute-driven cross-modal dynamic fusion module, such as Figure 2 As shown in the figure: the multimodal image dataset constructed in step S12 and the most representative hard attribute string of each modality constructed in step S13 are input into the attribute-driven cross-modal dynamic fusion module to guide the dynamic generation of weights to achieve adaptive fusion of the multimodal image dataset and obtain the fusion feature The designed attribute-driven cross-modal dynamic fusion module consists of four submodules: image shallow feature extraction, attribute text feature extraction and folding expansion, attribute-driven weight matrix generation, and feature dynamic fusion. The specific steps are as follows:

[0052] S21. Input the multimodal image dataset obtained in step S12 into the shallow feature extraction network Perform feature extraction to obtain RGB image features , edge image features , salient image features . Among them, the shallow feature extraction network Is a convolutional module consisting of a The convolutional layer, a Batch normalization layer, a The nonlinear activation layer is used to extract shallow features. The extracted shallow feature dimensions are , is the number of channels of shallow features, are the height and width of the image.

[0053] S22. Input the most representative hard attribute character string of each modality constructed in step S13 into the multimodal large model text encoder for text feature extraction, and fold and expand the features to obtain RGB image attribute text features. , edge image attribute text features , salient image attribute text features , the specific sub-steps are as follows:

[0054] S221. The most representative hard attribute string RGB feature attributes of each modality constructed in step S13 , edge image feature attributes , salient image feature attributes The data are input into the multimodal large model text encoder for text feature extraction, and the text features corresponding to each modality are extracted. The dimension of the text features is 512.

[0055] S222. The text features corresponding to each modality obtained in step S221 are folded and expanded in spatial dimensions respectively, and the text features are converted from After the dimension is folded into a single channel, it is expanded to Dimension, get RGB image attribute text features , edge image attribute text features , salient image attribute text features ,in are the height and width of the image.

[0056] S23. The RGB image features obtained in step S21 , edge image features , salient image features and the RGB image attribute text feature obtained in step S222 , edge image attribute text features , salient image attribute text features Input to the convolutional layer Get RGB image weights , edge image weight , saliency image weight , the specific steps are as follows:

[0057] S231. The corresponding type of RGB image features obtained in step S21 , edge image features and salient image features The corresponding RGB image attribute text feature obtained in step S222 , edge image attribute text features , salient image attribute text features The three modal image and text splicing features are spliced ​​in the channel dimension respectively, and then the three spliced ​​features are spliced ​​again in the channel dimension to obtain the overall feature vector of all image type information. The splicing feature dimensions of each modality image are ,, the overall eigenvector The dimension is , 32 and 32 are the height and width of the image, the formula is:

[0058]

[0059] in, Represents a splicing operation.

[0060] S232. The overall eigenvector obtained in S231 Input The convolution output is generated in the convolution layer and then passes through The activation function obtains the weight matrix The weight matrix contains the weights of three different modal images. The dimension represents the weight based on the spatial position mapping, the formula is:

[0061]

[0062] in, The convolution kernel is The convolutional layer. represent The activation function is expressed as , the output range of this function is .

[0063] S233. The weight matrix generated in step S232 Further split into 3 different weights, take The 0th channel is used as the RGB image weight ,Pick The first channel of is used as the edge image weight ,Pick The second channel of is used as the saliency image weight , the specific formula is:

[0064]

[0065] Each weight corresponds to an image modality. These weights are used to dynamically adjust the contribution of each image type at different positions, and the value range is (0,1).

[0066] S24. Weight the RGB image obtained in step S23 , edge image weight , saliency image weight Respectively with the RGB image features obtained in step S21 , edge image features , salient image features Input is dynamically fused with weights to obtain fusion features , the specific steps are as follows:

[0067] S241. The RGB image features obtained in step S21 , edge image features , salient image features go through Functions are mapped to Get the RGB image activation feature , edge image activation features , salient image activation features . Represents an activation function with an output range of (-1, 1), which can enhance the contrast of features and suppress noise, expressed as .

[0068] S242. Weight the RGB image generated in step S233 , edge image weight , saliency image weight Respectively activate the RGB image features obtained in step S241 , edge image activation features , salient image activation features Perform element-by-element weighting, then normalize to generate intermediate weighted features of each modality, and finally weight the three types of intermediate features to obtain the fusion feature , the specific formula is:

[0069]

[0070] in, Is a normalization operation used to adjust the feature distribution, Represents the dot product of weights and features.

[0071] S3. Combination of prompt modulation modules, such as Figure 3 As shown: the most representative hard attribute character strings of each modality constructed in step S13 and the corresponding learnable soft prompts are combined into a learnable alternating prompt text sequence , will be able to learn alternating prompt text sequences Input to pre-trained Text encoder generates learnable alternating hint embedding vectors , and through a multi-layer perceptron Fusion features Perform personalized feature modulation to obtain modulation fusion features The specific steps are as follows:

[0072] S31. The RGB image attributes constructed in step S13 , edge image feature attributes , salient image feature attributes and the corresponding learnable soft prompts to form a learnable alternating prompt sequence The specific splicing structure is as follows:

[0073]

[0074] The hard hint attribute text is used to make the learnable soft hint more focused on learning the relevant attributes of various modalities. The embedding vector of the learnable soft hint is optimized by back propagation, and its update is subject to the total loss function (Described in step S45) Drive.

[0075] S32. Through pre-training Text Encoder A series of learnable alternating prompt sequences will be input Converted to the corresponding learnable alternating prompt embedding vector ,in The text feature dimension is 512.

[0076] S33. Input learnable alternating hint embedding vector To a multi-layer perceptron Module, get two key parameters of feature modulation: scaling factor and bias , the role of these two parameters is to make personalized adjustments to the feature map, so as to achieve more flexible feature expression. It contains a nonlinear hidden module and a linear output module. The hidden module consists of a fully connected layer and an activation function Composition. Multilayer Perceptron The output dimension is 96, and the first 48 dimensions are used as scaling factors , the last 48 dimensions are used as bias terms To maintain training stability, The initialization value is often set to a vector of all 1s. Initialized to an all-zero vector, this makes the modulation module equivalent to the identity transformation in the early stages of training.

[0077] S34. By scaling factor and bias Fusion features Each channel of the feature map is dynamically scaled and translated to obtain the modulated fusion feature The core idea of ​​this module is to use and Dynamic scaling and translation operations are performed on each channel of the feature map to enhance the expressiveness of the model and achieve a more personalized fusion effect. The formula is:

[0078]

[0079] S4. Knowledge distillation: Construct the teacher model and the student model, and combine the modulated fusion features obtained in step S3 Used to pre-train the teacher model to obtain the teacher's predicted category distribution ; The RGB image dataset obtained in step S11 Input into the shallow feature extraction module, extract shallow features and input into the student model for training, and output the student's predicted category distribution The knowledge of the teacher model is transferred to the student model through the knowledge distillation technology, and the classification loss and distillation loss are combined to the student model and the learnable alternating prompt sequence obtained in step S31. Optimize to improve its inference speed and performance. The specific steps are as follows:

[0080] S41. The modulation fusion feature obtained in step S34 Input to the teacher model, using modulation fusion features Train the teacher model to obtain the pre-trained teacher model; then modulate the fusion features Input into the pre-trained teacher model to obtain the teacher's predicted category distribution .

[0081] S42. The RGB image dataset obtained in step S11 Input shallow feature extraction network Perform feature extraction to obtain a single RGB image feature, and then input the single RGB image feature into the student model for training to obtain the student prediction category distribution , where the category corresponding to the item with the highest predicted probability is the predicted category.

[0082] S43. The teacher prediction category distribution obtained in step S41 The student prediction category distribution obtained in step S42 Perform knowledge distillation: Calculate the difference between the output distribution of the student model and the output distribution of the teacher model as the distillation loss , and optimize the performance of the student model by minimizing this difference;

[0083] S44. Calculate the image classification loss of the student model and output the predicted category of the student model and the true label Combined with cross entropy loss Calculate to get the classification loss of the student model .in is the number of image categories.

[0084] S45. Total Loss of Text-Guided Knowledge Distillation Method Based on Attribute-Driven Fusion Distillation loss With classification loss The weighted sum of is expressed as:

[0085]

[0086] Among them, according to the actual task dynamic adjustment, the two losses are weighed as equally important for experimentation. At the same time, through gradient descent optimization , the parameters of the student model's backbone network, learnable prompt text embedding, and feature modulation module are all updated synchronously, thus achieving end-to-end training.

[0087] Example:

[0088] The steps of this embodiment are the same as those of the specific implementation method, and will not be repeated here. The implementation process and results are shown below.

[0089] The experiment of this invention uses the dataset CIFAR100 (a 100-category dataset developed by the Canadian Institute for Advanced Research (CIFAR)), a large dataset of 60,000 32x32 color images, including 50,000 training images and 10,000 test images, which provides a challenging and ideal dataset for complex machine learning and computer vision tasks.

[0090] The experimental planning model of the present invention has undergone a total of 240 rounds of training, with a batch size of 64 in each round. The student model is optimized using the SGD optimizer, with settings including weight decay The momentum is 0.9, and the learning rate starts at 0.05. In the learning rate scheduling strategy, the learning rate is proportionally reduced to one-tenth of its original level at three key nodes: 150th, 180th, and 210th epochs.

[0091] For the text-guided knowledge distillation method with attribute-driven fusion, the experimental setup primarily compares different knowledge distillation frameworks. The experiment primarily involves distillation comparisons between networks with different architectures, as well as between student and teacher models. The network architectures used in the experiment include ResNet, Wide Residual Network (WRN), VGG, and MobileNet.

[0092] The comparative experiments comparing different knowledge distillation frameworks mainly focus on other classic distillation methods. Specifically, they include KD (a knowledge distillation method based on neural networks); DKD (a knowledge distillation method based on target decoupling); FitNet (a knowledge distillation method for narrow and deep neural networks); AT (a knowledge distillation method based on attention transfer); RKD (a knowledge distillation method based on relation construction); CRD (a knowledge distillation method based on representation information comparison); ReviewKD (a knowledge distillation method based on knowledge review); and OFD (a feature knowledge distillation method based on comprehensive reconstruction).

[0093] The experimental results are shown in Table 1. Model knowledge distillation comparison experiment.

[0094] According to the experimental results, the main innovative method proposed in this paper outperforms most model results when compared with other classic knowledge distillation methods, and is an effective optimization of knowledge distillation.

[0095] Table 1. Model knowledge distillation comparison experiment

[0096]

Claims

1. A text-guided knowledge distillation method based on attribute-driven fusion, characterized by: The following steps are involved: S1. For the RGB image dataset, a multimodal image dataset is generated by performing edge detection and saliency detection image processing. At the same time, hard attribute strings for each modality are constructed. The specific implementation process is as follows: S11. Get a series of RGB images The RGB image dataset is composed of is the number of channels of the RGB image, and is the height and width of the RGB image; S12. Perform edge detection and saliency detection on all RGB images and generate corresponding edge images respectively and saliency images ,in is the number of channels of the edge image, is the number of channels of the saliency image. The multimodal image dataset consists of three types of images: RGB images, edge images, and saliency images. S13. Construct a set of hard attribute strings for each modality, including RGB image feature attributes, edge image feature attributes, and saliency image feature attributes. Each category in each modality corresponds to a common hard attribute string. The RGB image feature attributes consist of an object body description and an RGB core attribute description. The edge image feature attributes consist of an object body description, an edge detection algorithm description, and an edge core attribute description. The saliency image feature attributes consist of an object body description, a saliency detection algorithm description, and a saliency core attribute description. S2. Input the multimodal image dataset and the hard attribute strings of each modality into the attribute-driven cross-modal dynamic fusion module to achieve adaptive fusion of the multimodal image dataset and obtain fusion features; S3. Based on the hard attribute string, a learnable alternating prompt sequence is formed, and parameters are generated to perform personalized feature modulation on the fusion feature to obtain the modulated fusion feature; S4. Train the teacher model with the modulation fusion features, train the student model based on the RGB image dataset, transfer the knowledge of the teacher model to the student model, and optimize the student model and the learnable alternating prompt sequence to complete knowledge distillation.

2. The text-guided knowledge distillation method based on attribute-driven fusion according to claim 1 is characterized in that: The step S2 is specifically implemented as follows: S21. Input the multimodal image dataset into a shallow feature extraction network for feature extraction to obtain RGB image features, edge image features, and saliency image features; The shallow feature extraction network consists of a convolutional layer, a batch normalization layer, and a nonlinear activation layer. The feature dimensions extracted by the shallow feature extraction network are , is the number of channels; S22. Input the hard attribute strings of each modality into the multimodal large model text encoder for text feature extraction, and then perform feature folding and expansion to obtain RGB image attribute text features, edge image attribute text features, and saliency image attribute text features; S23. The RGB image features, edge image features, saliency image features and RGB image attribute text features, edge image attribute text features, and saliency image attribute text features are input into the convolutional layer to obtain RGB image weights, edge image weights, and saliency image weights; S24. Perform weighted dynamic feature fusion on the RGB image weight, edge image weight, and saliency image weight with the RGB image features, edge image features, and saliency image features, respectively, to obtain fused features.

3. The text-guided knowledge distillation method based on attribute-driven fusion according to claim 2 is characterized in that: The step S22 is specifically implemented as follows: inputting the hard attribute string RGB image feature attributes, edge image feature attributes, and saliency image feature attributes of each modality into the multimodal large model text encoder for text feature extraction, and extracting the text features corresponding to each modality. The dimension of the text features is ; The text features corresponding to each modality are After the dimension is folded into a single channel, it is expanded to The dimensions of are used to obtain RGB image attribute text features, edge image attribute text features, and saliency image attribute text features respectively.

4. The text-guided knowledge distillation method based on attribute-driven fusion according to claim 3 is characterized in that: The specific implementation process of step S23 is as follows: The RGB image features, edge image features, and saliency image features are concatenated with the corresponding RGB image attribute text features, edge image attribute text features, and saliency image attribute text features in the channel dimension, and then the three concatenated features are concatenated again in the channel dimension to obtain an overall feature vector containing all modal image information; The overall feature vector is input to the convolutional layer and then The activation function obtains the weight matrix ; The weight matrix contains the weights of the three different modality images; The weight matrix Split into 3 different weights, take The 0th channel of is used as the RGB image weight, and The first channel of is taken as the edge image weight, The second channel of is used as the saliency image weight, and each weight corresponds to an image modality.

5. The text-guided knowledge distillation method based on attribute-driven fusion according to claim 4 is characterized in that: The S24 is specifically implemented as follows: RGB image features, edge image features, and saliency image features are processed Function mapping to obtain RGB image activation features, edge image activation features, and saliency image activation features; The RGB image weight, edge image weight, and saliency image weight are weighted element-by-element with the RGB image activation feature, edge image activation feature, and saliency image activation feature, respectively. Then, they are normalized to generate intermediate weighted features of each modality. Finally, the three types of intermediate features are weightedly fused to obtain the fused features.

6. The text-guided knowledge distillation method based on attribute-driven fusion according to claim 5 is characterized in that: The specific implementation process of step S3 is as follows: S31. Alternately combining the RGB image feature attributes, the edge image feature attributes, the saliency image feature attributes, and the corresponding learnable soft prompts to form a learnable alternating prompt sequence; The embedding vector of the learnable soft hint is optimized through back-propagation; S32. Convert the input learnable alternating prompt sequence into the corresponding learnable alternating prompt embedding vector through the pre-trained multimodal large model text encoder ; S33. Input the learnable alternating prompt embedding vector into the multilayer perceptron, and the output feature dimension of the multilayer perceptron is ,forward Dimension as a scaling factor ,back Dimension as a bias term ; S34. By scaling factor and bias Fusion features Each channel of the feature map is dynamically scaled and translated to obtain the modulated fusion feature .

7. The text-guided knowledge distillation method based on attribute-driven fusion according to claim 6 is characterized in that: The specific implementation process of step S4 is as follows: S41. Input the modulation fusion features into the teacher model for training to obtain the teacher's predicted category distribution ; S42. Input the RGB image dataset into the shallow feature extraction network for feature extraction, and then input the extracted features into the student model for training to obtain the student prediction category distribution ; S43. Calculate the output distribution of the student model Output distribution of the teacher model The difference is taken as distillation loss; S44. Calculate the image classification loss of the student model and output the predicted category of the student model Combined with the true label to calculate the cross entropy loss, the classification loss of the student model is obtained; S45. The total loss is the weighted sum of the distillation loss and the classification loss. The total loss is optimized by gradient descent. The parameters of the student model's backbone network, learnable prompt text embedding, and feature modulation module are all updated synchronously to achieve end-to-end training.

Images