Knowledge distillation methods, electronic devices, and computer-readable storage media

Abstract

This application discloses a knowledge distillation method, electronic device, and computer-readable storage medium, relating to the field of artificial intelligence technology. The method involves grouping the attention graph matrix of the teacher network based on the number of attention heads in the initial network structure of the student network. This divides the high-dimensional attention space of the teacher network into subspaces of the same dimension as the student network. By concatenating the attention graph matrices within each group, each concatenated matrix aligns with the attention heads of the initial network structure, achieving a relatively regular correspondence between the attention graphs of the teacher and student networks. This allows for the calculation of distillation loss, enabling the accurate transfer of knowledge from the complex teacher network to the lightweight student network. Therefore, it addresses the problem of the difference in the number of attention heads between the teacher and student networks in multi-head attention mechanisms, which hinders the student network from fully learning the knowledge from the teacher network, thus improving the knowledge distillation effect and enhancing the performance of the student network.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to knowledge distillation methods, electronic devices, and computer-readable storage media. Background Technology

[0002] In the context of the rapid development of artificial intelligence, Transformer-based neural network models have become a core technology for intelligent transformation in various fields due to their superior feature extraction and processing capabilities. However, Transformer models typically require a large amount of storage and computing resources, limiting their application in resource-constrained scenarios.

[0003] To address these issues, related technologies employ knowledge distillation to transfer knowledge from complex teacher networks to lightweight student networks, reducing overhead while maintaining performance. However, when applying knowledge distillation to the Transformer model, the difference in the number of heads between the teacher and student networks in the multi-head attention mechanism makes it difficult to construct the distillation loss, hindering the accurate transfer of knowledge from the teacher network to the student network and impeding the student network's full learning of the teacher network's knowledge. Summary of the Invention

[0004] This application provides a knowledge distillation method, electronic device, and computer-readable storage medium to at least address the problem in related technologies where the difference in the number of heads between the teacher network and the student network in multi-head attention mechanisms hinders the student network from fully learning the knowledge of the teacher network.

[0005] This application provides a knowledge distillation method, including:

[0006] Construct the initial network structure of the student network based on the network structure of the teacher network;

[0007] The multiple first attention map matrices of the teacher network are grouped; the number of groups is the same as the number of attention heads in the initial network structure.

[0008] The first attention map matrices within each group are concatenated to obtain the concatenation matrix corresponding to each group.

[0009] The concatenation matrix and the second attention map matrix of the initial network structure are calculated based on the distillation loss function. The network parameters of the initial network structure are updated based on the calculation results to obtain the student network.

[0010] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described knowledge distillation methods when executing the computer program.

[0011] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described knowledge distillation methods.

[0012] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described knowledge distillation methods.

[0013] This application also provides an image retrieval system, including an image encoding module and a text encoding module;

[0014] The image encoding module is a student network obtained by using the image encoder in the pre-trained model as the teacher network and any of the above-mentioned knowledge distillation methods.

[0015] The text encoding module is a student network obtained by using the text encoder in the pre-trained model as the teacher network and any of the above-mentioned knowledge distillation methods.

[0016] This application addresses the issue of grouping the teacher network's attention graph matrix based on the number of attention heads in the initial network structure of the student network. This divides the high-dimensional attention space of the teacher network into subspaces of the same dimension as the student network. By concatenating the attention graph matrices within each group, aligning each concatenated matrix with the attention heads of the initial network structure, a more regular correspondence is achieved between the teacher and student network attention graphs. This makes the attention graph matrices of the teacher and student networks comparable in form and structure, enabling the calculation of distillation loss and accurately transferring the knowledge of the complex teacher network to the lightweight student network. Therefore, it solves the problem of the difference in the number of attention heads between the teacher and student networks in multi-head attention mechanisms, which hinders the student network from fully learning the knowledge of the teacher network, thus improving the knowledge distillation effect and enhancing the performance of the student network. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the principle of the single-head attention mechanism provided in the embodiments of this application;

[0019] Figure 2 This is a schematic flowchart of the knowledge distillation method provided in an embodiment of this application;

[0020] Figure 3This is a schematic diagram illustrating the principle of the knowledge distillation method in the embodiments of this application;

[0021] Figure 4 This is a schematic diagram illustrating the training principle of the image retrieval system according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the knowledge distillation apparatus provided in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0025] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0026] The core idea of ​​the attention mechanism is to compute the output based on the query, key, and value. The input is a given query matrix. Key matrix Sum matrix ,in It is the number of queries, keys, and values. It is the dimension of query and key. It is the dimension of the value. express OK, A column of real numbers.

[0027] During the operation of the attention mechanism, the similarity between the query and the key is first calculated to obtain the attention score matrix S:

[0028]

[0029] In the above formula, superscript This represents the transpose of a matrix. This represents a real matrix with n rows and n columns. To avoid the dot product result being too large and affecting gradient stability during training, a scaling operation is usually performed, dividing by... , obtain the scaled attention score :

[0030]

[0031] Next, the scaled attention scores are converted into a probability distribution using the Softmax function, resulting in the attention map matrix A:

[0032]

[0033] in, The function represents converting a row of elements into probability values.

[0034] Finally, the attention map matrix AND-value matrix Multiply to obtain the output of the attention mechanism. :

[0035]

[0036] Figure 1 This is a schematic diagram of a single-head self-attention mechanism. Figure 1 In the middle, assuming the current layer has The input embedding vector, the th The input embedding vectors are , Input embedding vector Input to query value transformation matrix Get output query Input embedding vector Input to key-value transformation matrix Get the output key Input embedding vector Input to feature embedding value transformation matrix Get value The query matrix is ​​represented as follows: Key matrix Value matrix In a self-attention mechanism, the inputs to the three transformation matrices are the same; in a non-self-attention mechanism, the inputs to the three transformation matrices are different.

[0037] The similarity calculation layer between the query and the key calculates the similarity between them, resulting in an attention score matrix. This matrix is ​​then scaled in a scaling layer to obtain scaled attention scores. Next, the Softmax layer uses the Softmax function to convert the scaled attention scores into a probability distribution, yielding an attention map matrix. Finally, the output calculation layer multiplies the attention map matrix by the value matrix to obtain the output of the attention mechanism, namely the feature embedding vector. .here, , , , , , Both are column vectors.

[0038] Multi-head attention mechanisms compute attention map matrices in parallel using multiple different attention heads, enabling the model to capture information from different representation subspaces.

[0039] Assume there is Each attention head. For the input query matrix Key matrix Sum matrix through respectively Different transformation matrices , and , Perform the transformation, where, It is the dimension of the model. It is the dimension of query and key. It is the dimension of the value. .

[0040] No. A query matrix for each attention head Key matrix Sum matrix They are respectively:

[0041]

[0042]

[0043]

[0044] For each attention head, calculate the output following the steps of the basic single-head attention mechanism described above:

[0045]

[0046]

[0047] in, Indicates the first Attention map matrix of attention heads, superscript Representative matrix Transpose Indicates the first The output of each attention head.

[0048] Will The output of each attention head By concatenating the parts, we obtain the concatenated matrix. :

[0049]

[0050] in, Represents n rows, column real matrix

[0051] Finally, through a linear transformation matrix The concatenated matrix is ​​transformed to obtain the final output of the multi-head attention mechanism. :

[0052]

[0053] Knowledge distillation is a model compression technique that aims to transfer the knowledge contained in a complex and powerful teacher network to a lightweight student network. This allows the student network to maintain performance close to that of the teacher network while significantly reducing the number of parameters and computational complexity, making it particularly suitable for resource-constrained scenarios.

[0054] In the knowledge distillation framework, the teacher network typically consists of a large number of parameters, possessing excellent feature extraction and processing capabilities, and is able to learn complex knowledge representations. Teacher networks are usually well-trained deep neural networks with deep hierarchical structures, such as ResNet (Residual Network) and BERT (Bidirectional Encoder Representations from Transformers). Due to their large number of parameters and complex structure, they can learn rich feature information and advanced representations. The teacher network provides "guidance" to the student network by outputting soft labels or intermediate features, enabling the student network to better understand the underlying structure of the data.

[0055] Student networks are relatively simple and lightweight models, such as MobileNet (Mobile Network-based models) and TinyBERT (Tiny Bidirectional Encoder Representations from Transformers). Their goal is to mimic the behavior of the teacher network and efficiently complete tasks under limited computational resources. Student networks have fewer parameters and layers, and by absorbing the knowledge inherent in the teacher network, they can achieve satisfactory accuracy while maintaining low computational cost. During training, the student network learns the soft labels or intermediate features output by the teacher network, gradually approximating the teacher network's performance, ultimately achieving model optimization and compression.

[0056] In the knowledge distillation process based on the Transformer architecture, the difference in the number of heads in the multi-head attention mechanism between the teacher network and the student network leads to an inconsistency in the number of attention graph matrices. This makes it difficult to directly apply the distillation loss based on a one-to-one comparison of attention graph matrices. If forced to use it, information loss will occur due to the mismatch in the number of attention graphs, which will seriously affect the effective learning and absorption of knowledge from the teacher network by the student network, and will fail to achieve the goal of guiding knowledge distillation or improving the performance of the student network.

[0057] To address the aforementioned problems, embodiments of this application provide a knowledge distillation method.

[0058] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] The specific application environment architecture or specific hardware architecture on which the knowledge distillation method depends is described here.

[0060] The embodiments of this application provide a knowledge distillation method, and the method is described in detail below in conjunction with the execution flow of the knowledge distillation method.

[0061] Specifically, Figure 2 This is a flowchart of a knowledge distillation method provided according to an embodiment of this application.

[0062] like Figure 2 As shown, the knowledge distillation method includes steps 210, 220, 230 and 240.

[0063] Step 210: Construct the initial network structure of the student network based on the network structure of the teacher network.

[0064] The network structure of a teacher network comprises several elements, such as the number of network layers, the number of neurons per layer, the choice of activation functions, and the number of attention heads in a multi-head attention mechanism. When constructing the initial network structure of a student network, the teacher network's structure can be analyzed and evaluated. For example, a teacher network might have a deep number of layers and a large number of attention heads to achieve complex feature extraction and information interaction. The student network, on the other hand, needs to reduce the number of network layers and attention heads while maintaining a certain level of performance, thereby reducing computational complexity and memory usage.

[0065] When constructing the initial network structure of the student network, the topological relationships and connection logic of the teacher network can be retained. For example, if the teacher network adopts a basic block structure of "multi-head attention layer + feedforward network + residual connection", the initial network structure of the student network can also use the same basic block structure, ensuring that the feature processing flow of the initial network structure is consistent with that of the teacher network. Based on the lightweight objective, the number of attention heads in the initial network structure of the student network can be reduced, allowing the student network to inherit the capabilities of the teacher network while meeting the lightweight requirement.

[0066] Step 220: Group the multiple first attention map matrices of the teacher network; the number of groups is the same as the number of attention heads in the initial network structure.

[0067] In this embodiment, based on the above description of the multi-head attention mechanism, each attention head is an independent computational unit within the mechanism. Each head captures the features and relationships of the input data from different perspectives, enhancing the model's understanding of information through parallel computation. The attention map matrix records the distribution of attention scores at various positions during the computation process. Each attention head generates a corresponding attention map matrix, and these matrices collectively constitute the multi-head attention mechanism's comprehensive understanding of the input data.

[0068] In this embodiment, the attention graph matrix of the teacher network is referred to as the first attention graph matrix. In a knowledge distillation scenario based on the Transformer architecture, the number of attention heads in the teacher network is greater than the number of attention heads in the student network, and the number of attention graph matrices generated by the teacher network is also greater than the number generated by the student network. Grouping can be performed according to the number of attention heads in the initial network structure, ensuring that the number of groups matches the number of attention heads in the initial network structure. This grouping operation reorganizes the high-dimensional attention information of the teacher network, transforming it into a form adapted to the student network structure, laying the foundation for information alignment between the teacher and student networks in subsequent knowledge distillation processes.

[0069] For example, the initial network structure has 4 attention heads, the teacher network has 12 attention heads, and correspondingly, the teacher network also has 12 attention graph matrices. Based on the number of attention heads in the initial network structure, the 12 attention graph matrices of the teacher network can be divided into 4 groups, so that the teacher network structure is compatible with the student network structure, that is, one attention head of the student network corresponds to one group of attention graph matrices of the teacher network.

[0070] Step 230: Concatenate the first attention map matrices within each group to obtain the concatenation matrix corresponding to each group.

[0071] Since each group contains multiple first attention graph matrices of the teacher network, these attention graph matrices, although containing rich attention information, are independent and have dispersed dimensions, making it difficult to directly compare and transfer knowledge with the attention graph matrices of the student network.

[0072] Therefore, the first attention map matrices within each group can be concatenated to obtain the concatenated matrix corresponding to each group. Concatenation can be performed column-wise, element-weighted, or using other methods; this embodiment does not limit the specific methods used.

[0073] In this embodiment, by matrix concatenation, multiple attention map matrices that were originally scattered in each group are merged into a concatenated matrix with a specific dimension. This not only compresses the dimensionality of the teacher network's attention information, making the concatenated matrix more consistent with the dimension of the attention map matrix corresponding to a single attention head in the initial network structure, but also fuses attention information from different angles and levels within the same group. In this way, each concatenated matrix is ​​equivalent to condensing the information from multiple attention heads in the teacher network into a feature representation that can directly correspond to a single attention head in the initial network structure, enabling the rearranged attention knowledge of the teacher network to be presented in a more regular and comparable form.

[0074] In some embodiments, the dimension of the splicing matrix is ​​the same as the dimension of the attention map matrix of the initial network structure, so that the splicing matrix can be compared one by one with the dimension of the attention map matrix corresponding to a single attention head of the initial network structure. This makes the attention map matrix of the teacher and the initial network structure comparable in form and structure, thereby enabling the calculation of distillation loss and realizing the accurate transfer of knowledge from the complex teacher network to the lightweight student network.

[0075] Step 240: Calculate the splicing matrix and the second attention map matrix of the initial network structure based on the distillation loss, and update the network parameters of the initial network structure based on the calculation results to obtain the student network.

[0076] In the embodiments of this application, distillation loss is a quantitative indicator that measures the knowledge difference between the teacher network and the student network. Its function is to guide the student network to learn the knowledge of the teacher network, so that the output of the student network is as close as possible to that of the teacher network.

[0077] In the knowledge distillation scenario of the Transformer model, due to the difference in the number of heads in the multi-head attention mechanism between the teacher and student networks, directly calculating the loss function between the attention graph matrices of the teacher and student networks would be difficult to accurately transfer the knowledge of the teacher network to the student network due to dimensionality mismatch and structural inconsistency. However, after grouping and concatenating the attention graph matrix of the teacher network, the concatenated matrix of the teacher network and the second attention graph matrix of the initial network structure achieve uniformity in dimensionality and structure, making them directly comparable. Based on this, distillation loss can be constructed using KL (Kullback-Leibler) divergence, mean squared error (MSE), and cross-entropy loss function to measure the difference between the concatenated matrix and the second attention graph matrix of the initial network structure. For example, using KL divergence to construct the distillation loss can capture the semantic deviation in the attention weight allocation pattern between the teacher concatenated matrix and the student attention graph by calculating the difference in probability distribution.

[0078] By calculating the difference between the splicing matrix and the second attention map matrix of the initial network structure, the knowledge of attention distribution, feature association, etc. contained in the teacher network is quantified in the form of distillation loss. This loss value intuitively reflects the gap between the current state of the initial network structure and the expected state of the teacher network.

[0079] In this embodiment of the application, during the training process, the constructed distillation loss is used as the optimization target. The initial network structure adjusts its own parameters according to the magnitude of the distillation loss through the backpropagation algorithm, continuously narrowing the gap with the teacher network in terms of attention features, and gradually learning the attention mechanism knowledge of the teacher network. This enables the effective transfer of knowledge from the complex teacher network to the lightweight student network, reducing model resource overhead while ensuring the performance of the student network.

[0080] Specifically, during training, the training dataset can be simultaneously fed into the teacher network and the initial network structure. The training objectives include two parts: first, to measure the difference between the initial network structure and the teacher network's attention map using distillation loss; and second, to ensure the initial network structure's learning ability for the original task using task loss. The distillation loss and task loss can be weighted and summed to form the total loss, which serves as the basis for guiding the update of the initial network structure parameters.

[0081] In practical applications, student networks can be used in different scenarios to achieve various tasks. For example, in the field of computer vision, visual models pre-trained on large-scale image datasets can be used as teacher networks, such as ResNet (Residual Network) and EfficientNet (Efficient Convolutional Neural Network). Student networks trained through knowledge distillation can be applied to tasks such as image classification, object detection, and image retrieval. For instance, in medical image diagnosis, student networks can distinguish between normal and diseased tissues; in security monitoring, student networks can identify targets such as people and vehicles in the monitored images; and in autonomous driving applications, student networks can also identify pedestrians and traffic signs on the road through object detection.

[0082] In the field of natural language processing, pre-trained language models, such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), can be used as teacher networks. Student networks, trained through knowledge distillation, can be deployed in semantic understanding, text classification, sentiment analysis, or question answering systems. For example, in text classification tasks, student networks can classify news articles by topic, helping news platforms organize and recommend relevant content; in machine translation tasks, student networks can understand the semantics of source language text and translate it into target language text.

[0083] In multimodal interaction scenarios, multimodal pre-trained models, such as CLIP (Contrastive Language-Image Pre-training) and ALBEF (Aligning Bidirectional Encoder Representations from Transformers with Fine-tuning), can be used as teacher networks. Student networks trained through knowledge distillation can then be applied to multimodal interaction tasks. For example, in the image-text question-and-answer function of intelligent assistants, the student network can process user-inputted images and text questions, combining visual and textual information to generate answers. In the product retrieval scenario of e-commerce platforms, it can accurately match products that meet user needs from a product image library based on the user's input text description.

[0084] By distilling the attention mechanism learned by the teacher network, the student network can inherit the capabilities of the teacher network while reducing the computational burden, making it suitable for running on mobile or edge devices.

[0085] The training datasets used for student networks vary depending on the application scenario. For natural language tasks, text datasets can be used, such as multilingual corpora, labeled sentiment analysis datasets, or question-answer pair datasets. These can also include news articles or domain-specific corpora (e.g., medical or legal texts) to help student networks learn deeper semantic and attention allocation mechanisms. In image processing tasks, image datasets, such as ImageNet, or detection datasets labeled with target locations and categories, such as the COCO (Common Objects in Context) dataset, can be used. Continuous video frame sequences are also possible.

[0086] The goal of knowledge distillation is to enable a lightweight initial network structure while maintaining good task performance. For example, in image classification tasks, the initial network structure needs to correctly identify the object categories in an image. The task loss is calculated by comparing the predicted category probability distribution of the initial network structure with the true labels, allowing the initial network structure to mimic the knowledge of the network while also achieving the actual task objective. The task loss can be cross-entropy loss, mean squared error, etc.

[0087] During training, the backpropagation algorithm transmits the total loss to each parameter layer of the initial network structure, calculates the gradient of each parameter, and adjusts the weights accordingly. For example, when distillation loss indicates a significant difference between the output of a certain attention head in the initial network structure and the concatenation matrix of the teacher network, gradient updates will focus on adjusting the weight parameters corresponding to that attention head, making its attention distribution align with the teacher network. Conversely, when the task loss is high, it indicates a significant deviation between the initial network structure's predictions and the true labels when performing actual tasks (such as classification or regression). In this case, the backpropagation algorithm will focus on adjusting the parameters affecting the task output. Taking image classification as an example, if the initial network structure misclassifies a cat as a dog, the task loss will calculate the gradient based on the difference between the prediction and the true label, and then significantly update the parameters of convolutional layers, fully connected layers, and other layers responsible for feature extraction and final decision-making, enabling the initial network structure to more accurately distinguish between different categories in subsequent predictions. Through the synergistic effect of distillation loss and task loss, the initial network structure can continuously strengthen its ability to perform actual tasks while optimizing the attention mechanism features.

[0088] This application utilizes a method to group the attention graph matrix of the teacher network based on the number of attention heads in the initial network structure of the student network. This divides the high-dimensional attention space of the teacher network into subspaces of the same dimension as the student network. By concatenating the attention graph matrices within each group, each concatenated matrix aligns with the attention heads of the initial network structure, achieving a more regular correspondence between the attention graphs of the teacher and student networks. This makes the attention graph matrices of the teacher and student networks comparable in form and structure, enabling the calculation of distillation loss and accurately transferring the knowledge of the complex teacher network to the lightweight student network. Therefore, it addresses the problem of the difference in the number of attention heads between the teacher and student networks in multi-head attention mechanisms, which hinders the student network from fully learning the knowledge of the teacher network, thus improving the knowledge distillation effect and enhancing the performance of the student network.

[0089] In some embodiments, grouping multiple first attention map matrices of the teacher network includes:

[0090] Calculate the ratio of the number of attention heads in the teacher network to the number of attention heads in the student network;

[0091] Grouping strategy is determined based on ratio;

[0092] The teacher network's multiple first attention map matrices are grouped according to a grouping strategy.

[0093] In this embodiment, the architectural differences between the teacher network and the student network in terms of multi-head attention mechanisms can be quantified by the number of attention heads in the teacher network and the initial network structure.

[0094] Taking a certain layer of the teacher network and the initial network structure as an example, the multi-head attention mechanism of the teacher network in this layer includes... Each attention point, correspondingly generating First attention map matrix , , ..., In the initial network structure of the same layer, the number of attention heads in the multi-head attention mechanism is... Correspondingly, The second attention map matrix is ​​as follows: , , ..., ,generally, It can calculate the ratio. .

[0095] After obtaining the ratio, a grouping strategy can be dynamically designed based on the different ratio values. For example, when the ratio is an integer, equal-number grouping can be used directly, that is, every 3 heads of the teacher network corresponds to 1 head of the student network; if the ratio is not an integer, a mixed grouping strategy can be introduced: some groups contain 3 heads and some groups contain 2 heads.

[0096] In some embodiments, during the grouping process, consecutive first attention map matrices in the teacher network can be grouped together. For example, the 12 first attention map matrices of the teacher network can be numbered sequentially, and every three consecutive first attention map matrices can be grouped together, forming four groups. This continuous grouping method ensures that the first attention map matrices within each group maintain the integrity of the local features of the teacher network.

[0097] In some embodiments, the grouping strategy can also combine the semantic relevance of attention map matrices, prioritizing the grouping of attention map matrices with complementary features into the same group. For example, if one attention map matrix focuses on local syntax and another focuses on global semantics, it means that the two can be complementary.

[0098] Specifically, the similarity between the various first attention map matrices can be calculated, such as cosine similarity or Euclidean distance. Higher similarity indicates a stronger semantic relevance between the two matrices, meaning they focus on similar information. Conversely, lower similarity indicates a weaker semantic relevance, meaning they focus on different information and can complement each other. Therefore, first attention map matrices with low similarity can be grouped together.

[0099] In this embodiment, by calculating the similarity between the attention map matrices of the teacher network, attention map matrices with low similarity are preferentially grouped into the same group, so that each group includes heterogeneous features. This feature complementarity allows each group to capture fine-grained grammatical structures and macro-semantic relationships, improving the richness of knowledge representation. Furthermore, by reducing redundant combinations of high-similarity attention map matrices, the distillation signal dilution caused by information repetition within the group is reduced, allowing subsequent distillation loss calculations to more accurately focus on key feature differences.

[0100] In this embodiment, by calculating the ratio of the number of attention heads in the teacher network to the number of attention heads in the initial network structure, and using a dynamic grouping strategy based on the ratio, adaptive compression of the teacher network's attention knowledge and its structured alignment with the student network are achieved. This enables the construction of distillation loss, thereby accurately transferring the knowledge of the complex teacher network to the lightweight student network.

[0101] In some embodiments, grouping multiple first attention map matrices of the teacher network according to a grouping strategy includes:

[0102] When the ratio is an integer, the grouping strategy includes: dividing the multiple first attention map matrices into equal parts. Group; among which, It equals the number of attention heads in the student network.

[0103] In this embodiment, when the calculated ratio of the number of attention heads of the teacher network to that of the student network is an integer, the teacher network's... Each first attention map matrix is ​​evenly divided into Groups, each containing exactly one group A first attention map matrix. In this way, a relatively regular correspondence can be achieved between the attention maps of the teacher network and the student network.

[0104] For example, if the teacher network has 12 attention heads and the student network has 4 attention heads, with a ratio of 3, an equal-number partitioning strategy can be adopted. The 12 first attention map matrices of the teacher network are numbered sequentially, and every three consecutive first attention map matrices are grouped together to form four groups, which is the same as the number of attention heads in the student network. Each group can correspond to one attention head of the student network, so that each group of first attention map matrices can generate a distillation target aligned with the student network through simple concatenation or weighted averaging, without the need for additional dimensional transformations or complex mappings.

[0105] In this embodiment, a knowledge mapping from a teacher-dominated network to a student-dominated network is achieved through an equal-number grouping strategy under integer ratio conditions. This simple logic makes the grouped matrix comparable to the student network in terms of structural distribution, thereby enabling the construction of distillation loss and accurately transferring the knowledge of the complex teacher network to the lightweight student network.

[0106] In some embodiments, grouping multiple first attention map matrices of the teacher network includes:

[0107] When the ratio is not an integer, the multiple first attention map matrices are divided into... Groups; of which, at most The number of first attention maps in group -1 is the ratio rounded down; It equals the number of attention heads in the student network.

[0108] In this embodiment, the ratio is not an integer, i.e. no Multiples of integers can be used to expand the teacher network. Each first attention map matrix is ​​also divided into... Group. Among them, the first -1 group, each group contains The last group contains feature maps, while the last group contains Each feature map. Among them, ,in This indicates the floor operation.

[0109] In this embodiment, a hybrid grouping strategy with non-integer ratios achieves a balance between elastic compression and dimension alignment. Specifically, the problem of non-divisible grouping is solved by using rounding down and remainder allocation mechanisms, ensuring consistency between the number of groups and the number of attention heads in the student network. This enables the construction of distillation loss, allowing for the accurate transfer of knowledge from complex teacher networks to lightweight student networks.

[0110] In some embodiments, the second attention map matrix of the splicing matrix and the initial network structure is calculated based on the distillation loss, including:

[0111] The concatenation matrix is ​​dimensionality reduced so that the dimension of the dimensionality-reduced concatenation matrix is ​​the same as the dimension of the second attention map matrix;

[0112] The calculation results are obtained by using the dimensionality-reduced concatenated matrix and the second attention map matrix.

[0113] In this embodiment, after grouping and splicing the teacher network attention map matrix, although the spliced ​​matrix structurally corresponds to the attention head of the initial network structure, the spliced ​​matrix and the second attention map matrix of the initial network structure may still not match in dimension due to the architectural differences between the teacher network and the initial network structure. For example, the spliced ​​matrix of the teacher network may differ from the second attention map matrix of the student network in terms of the number of channels, sequence length, or feature dimension. For instance, the spliced ​​matrix may have a dimension of 128×128, while the second attention map matrix is ​​64×64, making it difficult to directly construct the distillation loss.

[0114] In this embodiment, a dimensionality reduction operation can be performed on the splicing matrix so that the dimension of the dimensionality-reduced splicing matrix is ​​consistent with the dimension of the second attention map matrix.

[0115] The core objective of dimensionality reduction is to adjust the dimension of the concatenated matrix through specific mathematical transformations, while preserving key feature information, so that the dimension of the dimensionality-reduced concatenated matrix is ​​consistent with the dimension of the second attention map matrix of the student network. In this embodiment, the dimensionality reduction methods employed include: compressing the spatial dimension of the concatenated matrix using pooling operations (such as average pooling and max pooling); or mapping and adjusting the feature dimensions of the concatenated matrix through neural network layers such as fully connected layers and convolutional layers. For example, if the sequence length of the concatenated matrix is ​​greater than that of the second attention map matrix of the student network, the sequence length can be reduced by average pooling; if the feature dimensions of the two are different, the feature dimensions of the concatenated matrix can be mapped to the same dimension as that of the student network using fully connected layers.

[0116] In this embodiment, the splicing matrix of the teacher network is adjusted by dimensionality reduction, so that the dimension of the splicing matrix after dimensionality reduction is the same as that of the second attention map matrix of the initial network structure. This allows for comparison in a unified dimensional space, and the difference between the teacher network and the initial network structure can be accurately quantified by distillation loss. The attention knowledge contained in the teacher network is fed back to the initial network structure in the form of loss value, further improving the efficiency and accuracy of knowledge distillation.

[0117] In some embodiments, dimensionality reduction of the splicing matrix includes:

[0118] The concatenated matrix is ​​reduced in dimensionality using the target matrix; where the target matrix is ​​a learnable matrix that is updated via backpropagation during the student network training process.

[0119] In this embodiment, before training begins, a learnable target matrix can be initialized for the concatenated matrix. The target matrix dimension is designed to be the number of rows in the concatenated matrix × the number of columns in the student attention map matrix (e.g., 128×64). During training, real-time dimensionality reduction can be achieved through matrix multiplication: multiplying the concatenated matrix (e.g., 128×128 dimension) with the target matrix (e.g., 128×64 dimension) yields the dimension-aligned output matrix (128×64).

[0120] For example, for the first Group( ), by group First attention map , , ..., Concatenate columns to form a concatenation matrix. The dimensions of the spliced ​​matrix are , represent OK, The columns are real matrices. The second attention map matrix of the initial network structure is... , .because, and Since the dimensions are different, the distillation loss cannot be directly constructed. Therefore, a learnable objective matrix is ​​introduced. The dimension of the target matrix is Through matrix multiplication ,Will Dimensional transformation to They have the same dimensions.

[0121] After each forward computation, the distillation loss between the dimensionality-reduced concatenated matrix and the second attention map matrix of the initial network structure can be calculated, and the gradient is passed to the target matrix through backpropagation. The update rule can employ an adaptive learning rate optimizer, such as Adam, which dynamically adjusts the weights of the target matrix based on the gradient direction. It's important to note that the update of the target matrix and the update of the network parameters of the initial network structure are performed simultaneously. On one hand, the target matrix learns the optimal projection direction; on the other hand, the student network learns how to match the projected feature distribution, thus creating a closed loop between the dimensionality reduction process and knowledge transfer.

[0122] Figure 3 This is a schematic diagram illustrating the principle of the knowledge distillation method according to an embodiment of this application. Figure 3 The diagram shows the attention map matrix of a certain layer in the teacher network and the corresponding attention map matrix of a certain layer in the initial network structure. First, the attention map matrices of the teacher network are grouped according to the number of attention heads in the initial network structure. Then, the concatenation of each group of attention map matrices is reduced to the same dimension as the attention map matrix of the initial network structure using the target matrix. Finally, the distillation loss is calculated using the reduced-dimensional concatenation matrix and the attention map matrix of the initial network structure.

[0123] In this embodiment, by using a learnable matrix that can be updated with the propagation algorithm as the target matrix for dimensionality reduction, the projection direction can be continuously optimized. Compared with fixed dimensionality reduction methods, more key knowledge of the teacher network can be preserved, and the dimensionality reduction process can be dynamically adjusted according to the learning progress of the initial network structure, further improving the efficiency of knowledge transfer.

[0124] In some embodiments, calculations are performed based on the dimensionality-reduced concatenation matrix and the second attention map matrix, including:

[0125] The objective diagonal matrix is ​​used as a constraint, and the calculation results are obtained by calculating the dimensionality-reduced concatenated matrix and the second attention map matrix based on the objective function; the objective function represents the difference between the teacher network and the initial network structure.

[0126] In this embodiment, after reducing the dimensionality of the spliced ​​matrix, it is necessary to construct an objective function that can accurately measure the differences between the teacher network and the student network. Various distance metrics can be used to construct the objective function, such as mean squared error and KL (Kullback-Leibler) divergence.

[0127] To further enhance the distillation effect, a target diagonal matrix is ​​introduced as a constraint when constructing the distillation loss. The target diagonal matrix is ​​a matrix whose diagonal elements are learnable, and all other elements are zero. The purpose of the target diagonal matrix is ​​to weight and adjust the objective function, ensuring that features of different dimensions have different importance in the knowledge transfer process.

[0128] In some embodiments, when yes When it is an integer multiple of, the first Group( The distillation loss of ) can be defined as:

[0129]

[0130] in, Indicates the first Layer, number Distillation loss of the group Describe the objective function. Indicates the first Group splicing matrix, Represents the target matrix. This represents the second attention map matrix. and Representing different target diagonal matrices, Indicates hyperparameters, This means taking the main diagonal elements of the matrix to form a column vector. This represents the zero norm of a matrix, and its function is to enable selective activation of the target diagonal matrix.

[0131] The formula for KL divergence is expressed as:

[0132]

[0133] Where P is the attention distribution of the teacher network, and Q is the attention distribution of the initial network structure.

[0134] exist When the divergence is KL:

[0135]

[0136] in, This means converting the value of each element in a row of a matrix into a probability value.

[0137] In some embodiments, when no When the value is an integer multiple, for the teacher network grouping obtained from the previous... Groups, each group contains A first attention map matrix, according to... yes In the same way as integers, the attention maps within each group are concatenated into a new matrix. ( The distillation loss is constructed by adjusting the dimension through the target matrix and then combining it with the target diagonal matrix.

[0138] As for the last group, it includes The first attention map is denoted as... , , ..., Similarly, these attention maps are first concatenated column-wise to form a concatenation matrix. ,at this time Because its dimension corresponds to the feature map of the initial network structure. Inconsistent, but still using the target matrix right Perform a dimensional transformation. Then, introduce the target diagonal matrix. and .use Constraints enable us to more effectively filter out the feature columns most valuable to students' online learning while retaining key information, ultimately constructing the distillation loss function:

[0139]

[0140] in, Indicates the first Layer, number Distillation losses of the group; The objective function can be represented by the KL divergence. The first term representing the teacher network Group 1 attention map matrix concatenation matrix It is the target matrix; and These are two target diagonal matrices, in which all elements except the main diagonal elements have a value of zero; This means taking the main diagonal elements of the matrix to form a column vector. This represents the zero norm of a matrix, and its function is to enable selective activation of the target diagonal matrix. It's a hyperparameter.

[0141] The above method addresses the processing of a specific layer in the teacher network and the initial network structure, and it is equally applicable to other layers in the Transformer architecture. This is because the multi-head attention mechanism structures of each layer in the Transformer architecture are similar. Although the input and output features of different layers differ, the mismatch problem of the attention map matrix caused by the difference in the number of heads in the multi-head attention mechanism is essentially the same.

[0142] In this embodiment, the objective function can accurately capture the differences between the teacher network and the initial network structure, providing a clear optimization target for knowledge transfer. The introduction of the objective diagonal matrix realizes differentiated weighting of different feature dimensions, which can selectively activate effective columns, enabling the training process to focus on key knowledge and improve the pertinence and efficiency of knowledge transfer.

[0143] In some embodiments, the target diagonal matrix comprises multiple diagonal matrices that are updated via backpropagation during the initial network structure training process.

[0144] In this embodiment, before distillation training begins, multiple independent diagonal matrices can be initialized. The dimensions of these diagonal matrices are the same as those of the concatenation matrix and the second attention map matrix. These diagonal matrices can be initialized as identity matrices, with initial weights assigned to the main diagonal elements. Each diagonal matrix is ​​labeled as a learnable parameter, allowing it to be updated via backpropagation during training.

[0145] Specifically, when the initial network structure begins training, the backpropagation algorithm is initiated after each round of forward propagation calculates the distillation loss based on the dimensionality-reduced concatenated matrix and the second attention map matrix. At this point, the gradient is not only propagated and updated along the parameter path of the initial network structure but also extended to these diagonal matrices. Since the diagonal elements of the diagonal matrices correspond to weights of different feature dimensions, the gradients calculated by backpropagation adjust the weights of each diagonal element according to the network differences reflected by the distillation loss. For example, if the feature dimension corresponding to a certain diagonal matrix contributes little to reducing network differences during knowledge transfer, the gradient of the diagonal element will guide a decrease in its weight, thereby achieving dynamic adaptive updates to the diagonal matrices.

[0146] In this embodiment, by designing the target diagonal matrix as multiple updatable diagonal matrices, the importance of knowledge in different dimensions of the teacher network can be adaptively mined, guiding the initial network structure to learn key knowledge. The synergistic effect of multiple diagonal matrices can more comprehensively and accurately measure the differences between the teacher network and the initial network structure, enabling the initial network structure to not only learn the overall knowledge distribution of the teacher network during training, but also deeply absorb the core features of each dimension, thus improving the efficiency and quality of knowledge distillation.

[0147] In some embodiments, the second attention map matrix of the splicing matrix and the initial network structure is calculated based on the distillation loss, and the network parameters of the initial network structure are updated based on the calculation results to obtain the student network, including:

[0148] Obtain the labeled training dataset;

[0149] With the network parameters of the teacher network fixed, the training dataset is input into the teacher network and the initial network structure. Based on the overall loss function, the network parameters of the initial network structure are updated through the backpropagation algorithm. The overall loss function is constructed based on the distillation loss and the task loss. The task loss represents the difference between the predicted values ​​of the initial network structure and the true values ​​corresponding to the labels.

[0150] In this embodiment, the training dataset forms the basis for learning the initial network structure. The training dataset may include input samples and their corresponding ground truth labels. For example, in an image classification task, the training dataset consists of image samples, and the labels are the categories to which the images belong (such as cat, dog, etc.).

[0151] The overall loss function includes distillation loss and task loss. Distillation loss measures the difference in attention mechanism between the initial network structure and the teacher network. It is constructed through operations such as grouping, concatenating, and reducing the dimensionality of the attention map matrix, allowing the initial network structure to mimic the feature extraction method of the teacher network. Task loss, on the other hand, evaluates the predictive ability of the student network by calculating the difference between the output of the initial network structure and the true labels. Task loss can be cross-entropy loss, mean squared error, etc.

[0152] Taking the initial network structure task of predicting the next word given a sequence of tokens as an example, the task loss function is... Cross-entropy loss function can be used.

[0153] Suppose there is a containing A sequence of n words, for each position The student network needs to predict which word belongs to the vocabulary. The probability distribution of each word. Let... It is A one-hot vector of dimension 1 represents a position. The category corresponding to the real word element at that location, It is a student network prediction If we have a probability distribution vector, then the cross-entropy loss function can be expressed as:

[0154]

[0155] in, Indicates mission loss. Indicates the total number of lexical units. Indicates the size of the vocabulary. yes The first vector One element, Vector is The dimension heat vector represents the true category corresponding to position i. yes The first vector One element, Let V be the probability distribution vector predicted by the initial network structure, representing the probability that the lexical at position i in the initial network structure belongs to each word in the vocabulary. This formula calculates the average difference between the probability distribution predicted by the initial network structure and the true label. By minimizing this loss function, the parameters of the student network are adjusted so that the initial network structure can better predict the next lexical, thereby improving the performance of the language model.

[0156] In this embodiment, by combining distillation loss and task loss, the dual goals of knowledge transfer and task optimization are achieved. Distillation loss enables the student network to continuously absorb the higher-order attention patterns of the teacher network during parameter updates, while the supervision of task loss ensures that the transferred knowledge serves the real task objectives, thereby improving the student network's ability to handle actual tasks.

[0157] In some embodiments, the overall loss function is constructed as follows:

[0158] The overall distillation loss is constructed based on the distillation loss; the overall distillation loss represents the overall difference between different network layers of the teacher network and the initial network structure.

[0159] Construct an overall loss function based on the overall distillation loss and the task loss.

[0160] In this embodiment, the distillation loss constructed above Indicates the first Layer, number Distillation loss of the group Indicates the first Layer, number The distillation loss of the group. Both the teacher network and the student network are L-layered, and can be determined based on the distillation loss. and distillation loss Construct the overall distillation loss.

[0161] Specifically, assuming the teacher network is number one The number of attention heads per layer is The attention map matrix is , ,…, Initial network structure The number of attention heads per layer is The attention map matrix is , ,…, ,in For the first Layer, calculation and in accordance with the rules mentioned above, the teacher network Group the attention map matrix.

[0162] When the ratio of the number of attention heads in the teacher network to the number of attention heads in the initial network structure is an integer, the overall distillation loss function... Defined as:

[0163]

[0164] in, Representing the Layer, number Distillation loss of the group.

[0165] The above expression can be expanded into the following formula:

[0166]

[0167] in, The objective function can be the KL divergence; It is the first teacher network The first layer A concatenated matrix consisting of a set of first attention map matrices. It is the first The layer can be a target matrix, used to... Dimensional transformation to The same dimensions It is the first student network The first layer A second attention map matrix; and It is the target diagonal matrix; This represents taking the diagonal elements of the matrix to form a column vector.

[0168] When the ratio of the number of attention heads in the teacher network to the number of attention heads in the initial network structure is not an integer, the overall distillation loss function... Defined as:

[0169]

[0170] Because the last group requires special handling, the above formula can also be expanded as follows:

[0171]

[0172] More specifically, when At that time:

[0173]

[0174] in, Representing the Layer, number Distillation loss of the group The objective function can be the KL divergence; It is the first teacher network The first layer A concatenated matrix consisting of a set of first attention map matrices. It is the first The target matrix of the layer is used to... Dimensional transformation to The same dimensions It is the first in the initial network structure The first layer A second attention map matrix; and These are target diagonal matrices, in which elements other than the main diagonal elements have a value of zero; This represents taking the diagonal elements of the matrix to form a column vector.

[0175] also,

[0176]

[0177] in, Indicates the first Layer, number The distillation loss function of the group The objective function can be the KL divergence; It is the first teacher network The first layer A concatenated matrix consisting of a set of first attention map matrices. It is the first The goal of the layer is to... Dimensional transformation to The same dimensions It is the first in the initial network structure The first layer A second attention map matrix; and These are target diagonal matrices, in which elements other than the main diagonal elements have a value of zero; This represents taking the diagonal elements of the matrix to form a column vector.

[0178] Based on the above-mentioned total distillation loss and task loss, define the total loss function. . Therefore, the total distillation loss and mission losses It is a function of the independent variable. In this embodiment, Using a linear function, as follows:

[0179]

[0180] in, These are constant coefficients used for balancing. and Contributions, for example, By adjusting It can achieve better accuracy.

[0181] In this embodiment, the overall distillation loss is constructed by calculating the differences between different network layers of the teacher network and the initial network structure. The differences between different network layers are taken into account, enabling the initial network structure to learn richer feature representations and hierarchical knowledge structures from the teacher network. The overall distillation loss is combined with the task loss to construct an overall loss function that further guides the initial network structure to inherit the generalization ability of the teacher network. During the learning process, the initial network structure can better balance the inheritance of knowledge from the teacher network and the fitting of training data, thereby improving the performance of the student network.

[0182] In some embodiments, the teacher network is a pre-trained model, and the student network is a lightweight target model built based on the pre-trained model; wherein the target model and the pre-trained model have the same number of network layers, the same number of neurons in the network layers, and the same activation function; the target model has fewer attention heads than the pre-trained model.

[0183] In this embodiment, the pre-trained model is typically trained on large-scale datasets for a long time, possessing strong feature extraction capabilities and generalization performance. For example, in the field of natural language processing, large pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) can serve as teacher networks. Teacher networks typically have a complex structure, containing many network layers, a large number of neurons, and a large number of attention heads.

[0184] The student network is used as a lightweight target model. It is a lightweight modification based on the teacher network. The number of network layers, neurons in each layer and activation functions are kept exactly the same. The number of parameters is compressed by reducing the number of attention heads (e.g., from 12 to 4).

[0185] Let's take a training dataset consisting of text data as an example. The text data is input into a pre-trained model, generating intermediate state values ​​during model operation, such as query embeddings, key embeddings, and value embeddings, as well as the final output text. In a pre-trained model used for question answering, the input is the question, and the output is the answer to the question.

[0186] The following example illustrates the training process of knowledge distillation. In this example, a large-scale text dataset containing hundreds of millions of sentences was collected, of which Chinese sentences accounted for 30%, English sentences for 60%, and other languages ​​for 10%. During knowledge distillation, text data was sampled from the large-scale text dataset using a random sampling method. The language percentages mentioned above are exemplary values; using other percentages will not affect the effectiveness of this invention.

[0187] To avoid the order of samples in the training dataset D affecting model performance, all samples in the training dataset D are randomly arranged at the start of knowledge distillation. During each round of training, the training algorithm reads a batch of samples, for example, 1024 samples.

[0188] Read data from the training dataset D, assuming the data has undergone necessary preprocessing and its format meets the model input requirements; initial training cycle. Total training cycle The stochastic gradient descent algorithm is the Adam algorithm, and the relevant parameters are: learning rate is... momentum coefficient , , The learning rate strategy is a cosine strategy, specifically adjusted dynamically within the training period based on the cosine function. The number of samples in a mini-batch is B = 1024.

[0189] Target Model With pre-trained models The structures are basically the same, differing only in the number of heads in the multi-head attention mechanism. In each layer of the target model... The number of attention heads should not be greater than the number of attention heads in the pre-trained model.

[0190] The overall loss function for training is Training steps:

[0191] Step 1: The pre-trained model All parameter values ​​are fixed; based on the source model Network structure construction target model Network structure; random initialization of the target model The element values ​​of the weight matrix, and other parameters;

[0192] Step 2: When When this happens, perform the following operations:

[0193] Step 2.1: ;

[0194] Step 2.2: Prepare the training dataset The order of the samples in the sample is randomly shuffled;

[0195] Step 2.3: From the training dataset Select a training sample from batch B;

[0196] Step 2.4: Based on the training samples and parameter settings described above, calculate the loss value, and then update the target model using the stochastic gradient descent algorithm (Adam algorithm). The parameters are set. Before updating the parameters, the gradient is clipped, and a gradient threshold of 5 is set (which can be adjusted according to the actual situation) to prevent gradient explosion.

[0197] Step 2.5: Repeat steps 2.3 and 2.4 until the training dataset is complete. All samples were utilized.

[0198] Step 3: End training and output the target model. Save the target model M' to the specified file path for later use.

[0199] In this embodiment, by keeping the number of network layers, neurons, and activation functions unchanged, the student network can inherit the basic structure and information processing capabilities of the teacher network. During the knowledge distillation process, it can learn the knowledge representation of the teacher network. The design of reducing the number of attention heads significantly reduces computational complexity and resource consumption while maintaining the similarity of model structure, making the model more suitable for deployment in resource-constrained environments.

[0200] This application also provides an image retrieval system, including an image encoding module and a text encoding module;

[0201] The image encoding module is a student network obtained by using the image encoder in the pre-trained model as the teacher network and any of the above knowledge distillation methods.

[0202] The text encoding module is a student network obtained by using the text encoder in the pre-trained model as the teacher network and any of the above knowledge distillation methods.

[0203] In this embodiment, the image encoder transforms input image data into semantically meaningful feature vectors. Through a deep learning network architecture, the image encoder abstracts and extracts pixel information layer by layer, capturing visual features such as shape, texture, and color. For example, when recognizing a picture of a cat, the image encoder can extract features such as the cat's outline, fur texture, and eye shape, transforming this complex visual information into image feature vectors that a computer can understand and process. These image feature vectors can then be used for various visual tasks such as image classification, object detection, and image generation.

[0204] A text encoder processes text data, transforming it into a vector representation that contains semantic information. A text encoder can understand the meaning of words, phrases, sentences, and even entire passages within a text. Taking a text describing scenery as an example, a text encoder can identify keywords such as "mountains," "rivers," and "beautiful," understand the relationships between these words, and generate a text feature vector that reflects the overall semantics of the text. Text feature vectors can be used for natural language processing tasks such as text classification, sentiment analysis, and machine translation, enabling computers to understand and process text content like humans.

[0205] In some embodiments, the pre-trained model can be a CLIP (Contrastive Language-Image Pre-training) model. The CLIP model is a powerful multimodal pre-trained model that includes an image encoder and a text encoder, possessing excellent image and text encoding capabilities. In the CLIP model, the image encoder and text encoder are jointly trained to learn the semantic relationships between images and text. For example, given an image of "sunrise" and a text describing "the sun rises above the horizon, shining brightly," the CLIP model's image encoder and text encoder can extract features from the image and text respectively, and determine a high degree of correlation between them through internal model mechanisms.

[0206] In the embodiments of this application, such as Figure 4 As shown, the image encoder in the CLIP model can be used as the teacher network to construct a student network. The student network can be trained using the knowledge distillation method described above to obtain the image encoding module. Furthermore, by using the text encoder in the CLIP model as the teacher network, a student network is constructed. The student network is then trained using the aforementioned knowledge distillation method to obtain the text encoding module. .

[0207] This application addresses the issue of the high-dimensional attention space of the teacher network being divided into subspaces of the same dimension as the student network by grouping the teacher network's attention graph matrix based on the number of attention heads in the student network. By concatenating the attention graph matrices within each group, aligning each concatenated matrix with the attention heads of the student network, a more regular correspondence is achieved between the teacher and student network attention graphs. This makes the teacher and student network attention graph matrices comparable in form and structure, enabling the construction of a distillation loss and accurately transferring the knowledge of the complex teacher network to the lightweight student network. Therefore, it solves the problem of the difference in the number of attention heads between the teacher and student networks in multi-head attention mechanisms, which hinders the student network from fully learning the teacher network's knowledge, thus improving the knowledge distillation effect and enhancing the performance of the student network.

[0208] In some embodiments, the image encoding module is used to extract features from images in the image database and store the extracted image features in the image feature database;

[0209] The text encoding module is used to extract features from the query text to obtain text features; calculate the similarity between the text features and each image feature in the image feature database; determine the target image features whose similarity to the text features is greater than a threshold; match the target image features corresponding to the target image features from the image database; and output the target image.

[0210] In this embodiment, the image retrieval system can perform the following functions: by using a text query, it retrieves one or more images from an image database that match the text description. For example, when the input query is "a cat," the image retrieval system will return images related to "cat," such as an orange cat napping in the sunlight or a black and white cat playing in the grass. Similarly, when the input query is "beautiful scenery," the image retrieval system can return images of magnificent mountains and blue skies, or images of stunning sunsets. Likewise, when the input query is "city night view," the image retrieval system can return images of twinkling city lights and skyscrapers, or images of a busy street at night. The image retrieval system helps users quickly find images that match their descriptions through concise text queries, providing a convenient image search and browsing experience.

[0211] In this embodiment, the image database This is the image database to be queried. The image database contains a large number of unlabeled images. The content of these images is unknown, that is, the categories of objects in the images are unknown. Image database The image database is vast, encompassing images from multiple fields and scenarios. Each image in the database is an independent instance, without prior classification or labeling. For ease of subsequent description, the image database is defined as follows: The total number of images is .

[0212] Image database Each image in Input to image encoding module This yields the output feature embedding vector for each image. , .exist In the middle, take As an image Features. Image feature database. include The set of image features is as follows Image features correspond one-to-one with images.

[0213] In this embodiment, the query text can be a piece of natural language text, such as "generate an image of a cat". It should be noted that the language type of the query text should be consistent with the language type of the training dataset used by the training text encoding module. For example, if the text in the training dataset is Chinese, then the query text should also be Chinese; if the text in the training dataset covers multiple languages, then the input query text can contain multiple languages.

[0214] Query text Input to text encoding module The obtained feature embedding vector is , .exist In the middle, take Textual features as query text.

[0215] Image feature database include The set of image features is as follows The text features of query text T are: Query the similarity between the text features of text T and each feature in the image feature data. It can be calculated as follows:

[0216]

[0217] in, ,in and They represent two column vectors respectively. represent The transpose of , Represents the magnitude of the vector.

[0218] In this embodiment, the task of the image query system is to find one or more images corresponding to the content of the input query text from the image database. To this end, a similarity threshold can be set. ,For example, Of course, the similarity threshold Other values ​​may also be used, and the embodiments of this application are not limited.

[0219] Based on similarity threshold The resulting image index set is , represents returning all The set of indices, that is, the set of features of the target image.

[0220] Based on the image index set From image database The system matches the corresponding target image and returns it.

[0221] The knowledge distillation method provided in this application can be executed by a knowledge distillation device. This application uses a knowledge distillation device to execute the knowledge distillation method as an example to illustrate the knowledge distillation device provided in this application.

[0222] like Figure 5 As shown, the knowledge distillation apparatus includes:

[0223] Module 510 is used to construct the initial network structure of the student network based on the network structure of the teacher network.

[0224] Grouping module 520 is used to group the multiple first attention map matrices of the teacher network; the number of groups is the same as the number of attention heads in the initial network structure.

[0225] The splicing module 530 is used to splice the first attention map matrices within each group to obtain the splicing matrix corresponding to each group.

[0226] The update module 540 is used to calculate the second attention map matrix of the splicing matrix and the initial network structure based on the distillation loss, and update the network parameters of the initial network structure based on the calculation results to obtain the student network.

[0227] This application addresses the issue of grouping the teacher network's attention graph matrix based on the number of attention heads in the initial network structure of the student network. This divides the high-dimensional attention space of the teacher network into subspaces of the same dimension as the student network. By concatenating the attention graph matrices within each group, aligning each concatenated matrix with the attention heads of the initial network structure, a more regular correspondence is achieved between the teacher and student network attention graphs. This makes the attention graph matrices of the teacher and student networks comparable in form and structure, enabling the calculation of distillation loss and accurately transferring the knowledge of the complex teacher network to the lightweight student network. Therefore, it solves the problem of the difference in the number of attention heads between the teacher and student networks in multi-head attention mechanisms, which hinders the student network from fully learning the knowledge of the teacher network, thus improving the knowledge distillation effect and enhancing the performance of the student network.

[0228] In some embodiments, the grouping module 520 is further configured to:

[0229] Calculate the ratio of the number of attention heads in the teacher network to the number of attention heads in the initial network structure;

[0230] Grouping strategy is determined based on ratio;

[0231] The teacher network's multiple first attention map matrices are grouped according to a grouping strategy.

[0232] In some embodiments, the grouping module 520 is further configured to:

[0233] When the ratio is an integer, the grouping strategy includes: dividing the multiple first attention map matrices into equal parts. Group; among which, It equals the number of attention heads in the student network.

[0234] In some embodiments, the grouping module 520 is further configured to:

[0235] When the ratio is not an integer, the multiple first attention map matrices are divided into... Groups; of which, at most The number of first attention maps in group -1 is the ratio rounded down; It equals the number of attention heads in the student network.

[0236] In some embodiments, the update module 540 is further configured to:

[0237] The concatenation matrix is ​​dimensionality reduced so that the dimension of the dimensionality-reduced concatenation matrix is ​​the same as the dimension of the second attention map matrix;

[0238] The calculation results are obtained by using the dimensionality-reduced concatenated matrix and the second attention map matrix.

[0239] In some embodiments, the update module 540 is further configured to:

[0240] The concatenated matrix is ​​reduced in dimensionality using the target matrix; where the target matrix is ​​a learnable matrix that is updated via backpropagation during the initial network structure training process.

[0241] In some embodiments, the update module 540 is further configured to:

[0242] The objective diagonal matrix is ​​used as a constraint, and the calculation results are obtained by calculating the dimensionality-reduced concatenated matrix and the second attention map matrix based on the objective function; the objective function represents the difference between the teacher network and the initial network structure.

[0243] In some embodiments, the update module 540 is further configured to:

[0244] Obtain the labeled training dataset;

[0245] With the network parameters of the teacher network fixed, the training dataset is input into the teacher network and the initial network structure. Based on the overall loss function, the network parameters of the initial network structure are updated through the backpropagation algorithm. The overall loss function is constructed based on the distillation loss and the task loss. The task loss represents the difference between the predicted values ​​of the initial network structure and the true values ​​corresponding to the labels.

[0246] The descriptions of the embodiments corresponding to the knowledge distillation apparatus can be found in the relevant descriptions of the embodiments corresponding to the knowledge distillation method, and will not be repeated here.

[0247] Embodiments of this application also provide an electronic device, such as... Figure 6 As shown, it includes a memory 602 and a processor 601. The memory 602 stores a computer program, and the processor 601 is configured to run the computer program to perform the steps in any of the knowledge distillation method embodiments described above.

[0248] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the knowledge distillation method embodiments described above when it is run.

[0249] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0250] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the knowledge distillation method embodiments described above.

[0251] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the knowledge distillation method embodiments described above.

[0252] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0253] The foregoing has provided a detailed description of the knowledge distillation method, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A knowledge distillation method, characterized in that, include: Construct the initial network structure of the student network based on the network structure of the teacher network; The multiple first attention map matrices of the teacher network are grouped; the number of groups is the same as the number of attention heads in the initial network structure. The first attention map matrices within each group are concatenated to obtain the concatenation matrix corresponding to each group. The process involves calculating the concatenation matrix and the second attention map matrix of the initial network structure based on the distillation loss, updating the network parameters of the initial network structure based on the calculation results, and obtaining the student network. This includes: reducing the dimensionality of the concatenation matrix so that the dimension of the reduced concatenation matrix is ​​the same as the dimension of the second attention map matrix; and reducing the dimensionality of the concatenation matrix using a target matrix, wherein the target matrix is ​​a learnable matrix updated via backpropagation during the training process of the initial network structure; and calculating the result based on the reduced concatenation matrix and the second attention map matrix.

2. The method according to claim 1, characterized in that, The grouping of multiple first attention map matrices of the teacher network includes: Calculate the ratio of the number of attention heads in the teacher network to the number of attention heads in the initial network structure; The grouping strategy is determined based on the ratio; The teacher network's multiple first attention map matrices are grouped according to the grouping strategy described above.

3. The method according to claim 2, characterized in that, The step of grouping multiple first attention map matrices of the teacher network according to the grouping strategy includes: When the ratio is an integer, the grouping strategy includes: dividing the plurality of first attention map matrices into equal parts. Group; among which, This is equal to the number of attention heads in the student network.

4. The method according to claim 2, characterized in that, The grouping of multiple first attention map matrices of the teacher network includes: When the ratio is not an integer, the plurality of first attention map matrices are divided into Groups; of which, at most The number of the first attention maps in group -1 is the ratio rounded down; This is equal to the number of attention heads in the student network.

5. The method according to claim 1, characterized in that, The calculation based on the dimensionality-reduced concatenation matrix and the second attention map matrix includes: The objective diagonal matrix is ​​used as a constraint, and the calculation result is obtained by calculating the dimensionality-reduced splicing matrix and the second attention map matrix based on the objective function; the objective function characterizes the difference between the teacher network and the initial network structure.

6. The method according to claim 5, characterized in that, The target diagonal matrix comprises multiple diagonal matrices that are updated via backpropagation during the training process of the initial network structure.

7. The method according to claim 5, characterized in that, The distillation loss function is expressed as: in, Indicates the first Layer, number The distillation loss function of the group Describe the objective function. Indicates the first Group splicing matrix, Represents the target matrix. This represents the second attention map matrix. and Representing different target diagonal matrices, Indicates hyperparameters, This means taking the main diagonal elements of the matrix to form a column vector. This represents the zero norm of a matrix.

8. The method according to claim 1, characterized in that, The process of calculating the second attention map matrix of the splicing matrix and the initial network structure based on the distillation loss, and updating the network parameters of the initial network structure based on the calculation results to obtain the student network includes: Obtain the labeled training dataset; The network parameters of the teacher network are fixed, and the training dataset is input into the teacher network and the initial network structure. Based on the overall loss function, the network parameters of the initial network structure are updated through backpropagation algorithm. The overall loss function is constructed based on the distillation loss and the task loss. The task loss characterizes the difference between the predicted value of the initial network structure and the true value corresponding to the label.

9. The method according to claim 8, characterized in that, The task loss is expressed as: in, Indicates mission loss. Indicates the total number of lexical units. Indicates the size of the vocabulary. yes The first vector One element, vector is A one-hot vector represents the true class corresponding to position i. yes The first vector One element, Let V be the probability distribution vector predicted for the initial network structure, representing the probability that the lexical at position i in the initial network structure belongs to each word in the vocabulary.

10. The method according to claim 8, characterized in that, The overall loss function is constructed as follows: The overall distillation loss is constructed based on the distillation loss; the overall distillation loss represents the overall difference between different network layers of the teacher network and the initial network structure. Construct the overall loss function based on the overall distillation loss and the task loss; The total distillation loss is expressed as: in, Indicates the total distillation loss. Representing the Layer, number Distillation loss of the group This indicates the number of attention heads in the initial network structure; The overall loss function is expressed as: in, Represents the overall loss function. Indicates mission loss. Indicates the total distillation loss. These are constant coefficients.

11. The method according to claim 1, characterized in that, The teacher network is a pre-trained model, and the student network is a lightweight target model built based on the pre-trained model; wherein, the target model has the same number of network layers, the same number of neurons in the network layers, and the same activation function as the pre-trained model; the target model has fewer attention heads than the pre-trained model.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the knowledge distillation step as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the knowledge distillation method as described in any one of claims 1 to 11.

Images