Knowledge distillation method and electronic equipment

Abstract

The invention provides a knowledge distillation method and electronic equipment, which can be applied to the technical field of artificial intelligence. The knowledge distillation method comprises the following steps: respectively determining a plurality of first attention maps for a teacher network and a plurality of second attention maps for a student network; dimensionality normalization is carried out on a first splicing matrix obtained based on splicing of the multiple first attention maps and a second splicing matrix obtained based on splicing of the multiple second attention maps, and a first attention matrix and a second attention matrix with the same dimensionality are obtained; according to the matrix features of the first attention matrix and the matrix features of the second attention matrix, multiple distillation losses used for conducting knowledge distillation from the first attention map to the second attention map are determined; and in the training process of the student network, according to the multiple distillation loss and the task loss of the student network, adjusting model parameters of the student network until samples of the sample set for training the student network are polled, and obtaining a target student network.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a knowledge distillation method and electronic device. Background Technology

[0002] In today's era of booming artificial intelligence, neural network models based on the self-attention mechanism have been widely used in many fields such as natural language processing, computer vision, speech recognition, and image processing due to their feature extraction and processing capabilities. However, the operation of this neural network model requires a large amount of storage and computing resources. Knowledge distillation can solve these problems. Knowledge distillation aims to extract knowledge from a high-performance but structurally complex teacher network and pass it to a relatively simple student network, thereby reducing the model's storage and computing overhead while maintaining certain performance. Summary of the Invention

[0003] In view of the above problems, the present invention provides a knowledge distillation method and electronic device to improve the accuracy of student network prediction and the effect of knowledge distillation.

[0004] One aspect of the present invention provides a knowledge distillation method, comprising: determining multiple first attention maps for a teacher network and multiple second attention maps for a student network; performing dimension normalization on a first concatenation matrix obtained by concatenating the multiple first attention maps and a second concatenation matrix obtained by concatenating the multiple second attention maps to obtain first attention matrices and second attention matrices with the same dimension; determining a multiple distillation loss for knowledge distillation from the first attention maps to the second attention maps based on the matrix characteristics of the first attention matrices and the matrix characteristics of the second attention matrices; and adjusting the model parameters of the student network during the training process of the student network according to the multiple distillation loss and the task loss of the student network until all samples in the sample set used to train the student network are polled to obtain the target student network, wherein the sample set includes training samples and sample labels, and the task loss is used to describe the difference between the output result of the student network based on the training samples and the sample labels.

[0005] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the knowledge distillation method described above.

[0006] According to an embodiment of the present invention, by determining a first attention map for the teacher network and a second attention map for the student network, and performing dimensionality normalization on a first concatenation matrix obtained by concatenating multiple first attention maps and a second concatenation matrix obtained by concatenating multiple second attention maps, first and second attention matrices with the same dimensions are obtained. Based on the matrix characteristics of the first and second attention matrices, a multiple distillation loss for knowledge distillation from the first attention map to the second attention map is determined. Based on the multiple distillation loss and the task loss for training the student network, the model parameters of the student network are adjusted to obtain the target student network. Since the concatenation matrix dimensions of the teacher network and the student network are unified during knowledge distillation, resulting in attention matrices with the same dimensions, and the multiple distillation loss is calculated based on the matrix characteristics of the dimension-unified attention matrix, and the student network is then trained using both the multiple distillation loss and the task loss, the problem of knowledge loss caused by the inconsistency in the number of heads in the multi-head attention mechanism between the image retrieval models of the teacher network and the student network is effectively solved. This allows the student network to fully learn and absorb knowledge from the teacher network, thereby improving the retrieval accuracy of the student network's image retrieval model. Attached Figure Description

[0007] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0008] Figure 1 An application scenario diagram of the knowledge distillation method according to an embodiment of the present invention is shown;

[0009] Figure 2 A flowchart of a knowledge distillation method according to an embodiment of the present invention is shown.

[0010] Figure 3 An architecture diagram of knowledge distillation according to an embodiment of the present invention is shown.

[0011] Figure 4 An architecture diagram of an image retrieval model according to an embodiment of the present invention is shown.

[0012] Figure 5 A structural block diagram of a knowledge distillation apparatus according to an embodiment of the present invention is shown.

[0013] Figure 6 A block diagram of an electronic device suitable for implementing a knowledge distillation method according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0014] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0016] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0017] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0018] Currently, Transformer-based neural network models are applied in numerous fields, such as text generation and machine translation in natural language processing, and image classification and object detection in computer vision. These applications have driven the intelligent development of various industries and improved production efficiency and service quality. Given the significant storage and computational resource consumption of Transformer-based neural network models, knowledge distillation can address this issue. However, during knowledge distillation, the number of heads in the multi-head attention mechanism of the teacher network is often greater than that of the student network. The multi-head attention mechanism is a core component of the Transformer model, with different numbers of heads playing different roles in capturing features and representing information. The difference in the number of attention heads makes it difficult to directly construct the distillation loss because it is not possible to simply match and compare the multi-head attention outputs of the teacher network with the corresponding outputs of the student network one-to-one. Furthermore, current technologies do not focus on the core multi-head attention mechanism of the Transformer architecture, do not consider the inconsistency in the number of attention graphs caused by the difference in the number of attention heads between the teacher and student networks, and do not design grouping strategies to establish accurate correspondences between attention graphs. They cannot solve the problem of "many to few" or "few to many" attention graph matching in this scenario, and are prone to losing key knowledge at the attention level during distillation, making it difficult to carry out the knowledge distillation process effectively and hindering the student network from fully learning and absorbing the knowledge of the teacher network.

[0019] Therefore, a technical solution is urgently needed to overcome the aforementioned technical problems. The knowledge distillation method provided in this invention can unify the dimensions of the splicing matrix of the teacher network and the student network, and combine multiple distillation losses and task losses to jointly train the student network, solving the knowledge loss problem, improving the training accuracy of the student network and the knowledge distillation effect, and providing a new solution for the efficient application of Transformer-based neural network models in resource-constrained scenarios.

[0020] Figure 1 A diagram illustrating an application scenario of the knowledge distillation method according to an embodiment of the present invention is shown.

[0021] like Figure 1 As shown, the application scenario of this embodiment can include a teacher network and a student network. The teacher network includes multiple first attention maps 101-1, and the student network includes multiple second attention maps 102-1. The application scenario can also include a first splicing matrix 101-2, a second splicing matrix 102-2, a first attention matrix 101-3, a second attention matrix 102-3, a multiple distillation loss 103, training samples 104, output results 105, sample labels 106, a task loss 107, and a target loss 108 obtained based on the multiple distillation loss 103 and the task loss 107 of the student network.

[0022] The teacher network includes multiple first attention maps 101-1, and the student network includes multiple second attention maps 102-1. The first concatenation matrix 101-2 obtained by concatenating multiple first attention maps 101-1 and the second concatenation matrix 102-2 obtained by concatenating multiple second attention maps 102-1 are dimensionally normalized to obtain first attention matrices 101-3 and 102-3 with the same dimensions. Based on the matrix characteristics of the first attention matrices 101-3 and 102-3, a multiple distillation loss 103 can be determined for knowledge distillation from the first attention maps to the second attention maps. By inputting training samples 104 into the attention maps of the student network, an output result 105 is obtained. Based on the output result 105 and the sample labels 106, the task loss 107 of the student network can be calculated. The target loss 108 obtained from the multiple distillation loss 103 and the task loss 107 can be used to adjust the student network through feedback until all training samples 104 have been polled, resulting in the target student network.

[0023] It should be understood that Figure 1 The number of attention graphs shown is merely illustrative. Depending on implementation requirements, any number of attention graphs can be used.

[0024] The following will be based on Figure 1 The described scene, through Figures 2-4 The knowledge distillation method of this invention will be described in detail.

[0025] Figure 2 A flowchart of a knowledge distillation method according to an embodiment of the present invention is shown.

[0026] like Figure 2 As shown, the knowledge distillation method of this embodiment includes operations S210 to S240.

[0027] In operation S210, multiple first attention maps are determined for the teacher network and multiple second attention maps are determined for the student network.

[0028] In operation S220, the dimensions of the first splicing matrix obtained by splicing multiple first attention maps and the second splicing matrix obtained by splicing multiple second attention maps are normalized to obtain first attention matrices and second attention matrices with the same dimensions.

[0029] In operation S230, based on the matrix characteristics of the first attention matrix and the matrix characteristics of the second attention matrix, a multiple distillation loss is determined for knowledge distillation from the first attention graph to the second attention graph.

[0030] In operation S240, during the training of the student network, the model parameters of the student network are adjusted according to the multiple distillation loss and the task loss of the student network until all samples in the sample set used to train the student network are polled to obtain the target student network. The sample set includes training samples and sample labels. The task loss is used to describe the difference between the output of the student network based on the training samples and the sample labels.

[0031] In some embodiments, the teacher network may include (n can represent the number, L can be an abbreviation for layer, and t can be an abbreviation for teacher.) The student network can include a first attention layer and may include... (s can be an abbreviation for Student) There are two second attention layers. Multiple first attention maps can be determined from the first attention layers. Multiple second attention maps can be determined from the second attention layers. The number of first attention maps can be greater than the number of second attention maps.

[0032] In some embodiments, the same number of students may be selected from both the teacher network and the student network. () When performing "one-to-one" knowledge distillation using the first and second attention layers (which can represent a one-to-one relationship between the teacher network and the student network), the number of these first or second attention layers selected must not exceed the total number of layers in the student network. It also does not exceed the total number of layers in the teacher network. ,Right now , .

[0033] In the knowledge distillation process based on the Transformer architecture, due to the difference in the number of heads in the multi-head attention mechanism between the teacher network and the student network during the one-to-one knowledge distillation between the first and second attention layers, the number of first attention graphs in the first attention layer is inconsistent with the number of second attention graphs in the second attention layer. This leads to a discrepancy between the number of first attention graphs in the first attention layer and the number of second attention graphs in the second attention layer, thus affecting the knowledge distillation process based on the Transformer architecture. Figure 1 The distillation loss in a one-to-one comparison is difficult to apply directly. If forced to use it, information loss will occur due to the mismatch in the number of attention maps, severely impacting the effective learning and absorption of knowledge from the teacher network by the student network, making it difficult to achieve the goal of knowledge distillation guidance or improving student network performance. Therefore, this invention provides a method for achieving knowledge distillation even when the number of attention maps between the teacher network and the student network is inconsistent.

[0034] Next, in order to elaborate on the technical details of the present invention, we will refer to the foregoing selection. Analyze any one of the "one-to-one" correspondence layers. Define this correspondence layer as... The range of values ​​for j is . (That is, j can be 1 to 1) Any integer between these values ​​corresponds to a different "one-to-one" correspondence layer. It should be noted that since all "one-to-one" correspondence layers follow the same processing logic and principles during distillation, therefore... The derived conclusions, methods, and formulas are applicable to all other selected "one-to-one" corresponding layers, without the need for repeated derivation for each layer individually. By analyzing a single layer, the results can be generalized to all "one-to-one" layers participating in the distillation process.

[0035] For any of the "one-to-one" corresponding layers selected above The multi-head attention mechanism of the layers in the teacher network is defined as including Size, correspondingly producing First attention map , , ..., In student networks, the number of heads in a multi-head attention mechanism is... Correspondingly generated The second attention map is .

[0036] Then, the aforementioned teacher network First attention map and student network The second attention map is divided into the same number of ( The group. In knowledge distillation, the teacher network needs to be included. First attention map and student network The second attention map is divided into the same number of... Grouping is used to establish a one-to-one correspondence between groups. Specifically, attention maps can be distributed evenly in order of their numbers, so that each group contains as many attention maps as possible, allowing the last group to have slightly fewer.

[0037] For example: when the teacher network has There are 1 first attention maps, numbered 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6 respectively. The student network has... The two second attention maps are numbered 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6, and are divided into... When grouping, the teacher network is grouped as [1.1,1.2,1.3] and [1.4,1.5,1.6], and the student network is grouped as [2.1,2.2,2.3] and [2.4,2.5,2.6]; when the teacher network has There are 1 first attention maps, numbered 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, and 1.7 respectively. The student network has... The two second attention maps are numbered 2.1, 2.2, 2.3, 2.4, and 2.5, and are divided into... When grouping, the teacher network is grouped as [1.1,1.2,1.3], [1.4,1.5], and [1.6,1.7], while the student network is grouped as [2.1,2.2], [2.3,2.4], and [2.5]. The core of this grouping method is to ensure the number of groups for both the teacher and student networks. Consistency is achieved through intra-group alignment (such as calculating distillation losses) to facilitate knowledge transfer. The specific implementation of group partitioning does not affect the process for determining multiple distillation losses in this embodiment of the invention, and this embodiment is also applicable to... Extreme situations.

[0038] Based on the grouping logic described above, the following section will use a corresponding group from the teacher network and the student network (denoted as...). The range of values ​​for i is 1. Let's take ) as an example for analysis. Since the processing methods for all corresponding groups are completely identical, therefore, for The analytical conclusions can be directly generalized to other groups.

[0039] In the corresponding group In the diagram, the subgroup of the teacher network contains p first attention maps (i.e., from that layer of the teacher network). The first attention map is assigned to The number is p); the subgroups of the student network contain q second attention maps (i.e., from this layer of the student network). The second attention map is assigned to The quantity is q). The quantities of p and q can be the same or different.

[0040] It needs to be clarified that the dimensions of all attention maps in both the teacher and student networks can be uniform: each attention map can be dimensional. The matrix is ​​a square matrix (n represents the number of tokens input to the training samples of the current layer), because the core function of the attention map is to characterize the correlation strength between n tokens, and the rows and columns of the matrix correspond to different tokens.

[0041] To integrate attention information within a group and align features between the teacher and student networks, the first and second attention maps within the group can be stitched together.

[0042] For the p first attention graphs of the teacher network, they are concatenated sequentially along the column direction (i.e., all columns of the first first attention graph are immediately followed by all columns of the second first attention graph, and so on), forming a new matrix, namely the first concatenation matrix. Since each first attention map is The matrix, after concatenating p first attention maps column-wise, is the first concatenated matrix. The dimension is ,Right now Similarly, for the q second attention graphs of the student network, a second concatenation matrix is ​​formed by concatenating them in the same column-wise manner. Its dimensions are ,Right now Through this splicing operation, information from multiple attention maps within the group is integrated into a matrix, laying the data foundation for subsequent solutions to dimensionality discrepancies and the construction of distillation loss.

[0043] In knowledge distillation, the construction of the loss function requires that the two feature vectors involved in the loss function calculation have the same dimension; otherwise, it is impossible to directly compare the similarity or measure the difference between the two feature vectors. In the intra-group alignment stage of this embodiment, this problem is specifically manifested in the first concatenation matrix of the teacher network subgroup. The second splicing matrix of the student network subgroup The dimensions do not match, the first concatenated matrix The dimension is (n is the number of input tokens, p is the number of first attention maps within the teacher network subgroup); second concatenation matrix The dimension is ( (The number of second attention maps within the student network subgroup). This addresses the issue of differing attention heads between teachers and students. and In the case of inequality, the first concatenation matrix Second splicing matrix number of columns and The differences between the two attention maps prevent them from being directly used to construct the distillation loss. Therefore, it is necessary to normalize the dimensions of the first concatenation matrix obtained by concatenating multiple first attention maps and the second concatenation matrix obtained by concatenating multiple second attention maps to obtain first and second attention matrices with the same dimensions.

[0044] For a first attention matrix and a second attention matrix with uniform dimensions, multiple distillation losses can be determined based on their respective matrix characteristics to perform knowledge distillation from the first attention graph to the second attention graph. These losses include a first distillation loss describing the differences in the core feature vectors of the first and second attention matrices; a second distillation loss quantifying the statistical differences in graph distribution features between the teacher and student networks; and a third distillation loss describing the differences in attention distribution between the normalized first and second attention graphs. These three distillation losses complement each other, capturing the knowledge differences between the teacher and student networks from different levels, ensuring the comprehensiveness of knowledge transfer.

[0045] During the training of the student network, a training sample set can be utilized. The training dataset used will vary depending on the application scenario of the student network. For natural language related tasks, a text dataset can be used, which can be a multilingual corpus, a labeled sentiment analysis dataset, or a question-and-answer pair dataset, etc. In image processing tasks, an image dataset can be used, such as a detection image dataset labeled with target location and category, a continuous video frame sequence, or a labeled image-text pair or video-audio pair.

[0046] Taking an image-text pair training sample set as an example, the training sample set can include training samples and sample labels. For instance, a training sample could be an image query text, such as text about a cat, beautiful scenery, or a city night scene. Sample labels could be images related to cats, such as an image of an orange cat napping in the sunlight, an image of a black and white cat playing in the grass, or images of magnificent mountains and blue skies, or images of a stunning sunset. They could also be images of flashing city lights and skyscrapers, or images of a busy street at night. The task loss can be determined based on the difference between the student network's output (i.e., the output image of a cat, beautiful scenery, or city night scene) and the sample labels. The model parameters of the student network can be adjusted based on the task loss and multiple distillation loss until all training samples have been queried, resulting in the target student network.

[0047] In some embodiments, the teacher network may include a first text encoding model and a first image encoding model, and the student network may include a second text encoding model and a second image encoding model obtained by knowledge distillation of the first text encoding model and the first image encoding model serving as the teacher network. The difference between the first text encoding model and the second text encoding model, and between the first image encoding model and the second image encoding model, lies in the fact that the first text encoding model has a larger number of heads in its multi-head attention mechanism than the second text encoding model, and the first image encoding model has a larger number of heads in its multi-head attention mechanism than the second image encoding model. The text encoding model can be used to process text data, transforming it into a vector representation containing semantic information. The text encoding model can understand the meaning of words and phrases in the text. For example, in a text describing a cat, the text encoding model can identify keywords such as "cat" and "orange." The image encoding model can transform input image data into feature vectors with semantic meaning. For example, when recognizing a landscape image, the image editing model can extract features such as mountains, rivers, and trees.

[0048] In some embodiments, an image retrieval model can be constructed based on an image encoding model and a text encoding model. This image retrieval model can perform the following functions: retrieving one or more images from an image library that match the text description by using a text as a query. For example, when the input query is "a cat," the image retrieval model will return images related to "cat," such as an orange cat napping in the sunlight or a black and white cat playing in the grass. Similarly, when the input query is "beautiful scenery," the image retrieval model can return images of magnificent mountains and blue skies, or images of stunning sunsets. Likewise, when the input query is "city night view," the image retrieval model can return images of twinkling city lights and skyscrapers, or images of a busy street at night. The image retrieval model obtained through the knowledge distillation method of this invention helps users quickly and accurately find images that match their descriptions through concise text queries, providing a convenient image search and browsing experience.

[0049] According to an embodiment of the present invention, by determining a first attention map for the teacher network and a second attention map for the student network, and performing dimensionality normalization on a first concatenation matrix obtained by concatenating multiple first attention maps and a second concatenation matrix obtained by concatenating multiple second attention maps, first and second attention matrices with the same dimensions are obtained. Based on the matrix characteristics of the first and second attention matrices, a multiple distillation loss for knowledge distillation from the first attention map to the second attention map is determined. Based on the multiple distillation loss and the task loss for training the student network, the model parameters of the student network are adjusted to obtain the target student network. Since the concatenation matrix dimensions of the teacher network and the student network are unified during knowledge distillation, resulting in attention matrices with the same dimensions, and the multiple distillation loss is calculated based on the matrix characteristics of the dimension-unified attention matrix, and the student network is then trained using both the multiple distillation loss and the task loss, the problem of knowledge loss caused by the inconsistency in the number of heads in the multi-head attention mechanism between the image retrieval models of the teacher network and the student network is effectively solved. This allows the student network to fully learn and absorb knowledge from the teacher network, thereby improving the retrieval accuracy of the student network's image retrieval model.

[0050] In some embodiments, during the training of the student network, a training sample set matching the image retrieval model can be retrieved from the database based on the model identifier of the student network, such as the image retrieval model. The training sample set is synchronously input into the teacher network and the student network. The difference between the attention maps of the student network and the teacher network is measured by distillation loss. The student network's learning ability for the original task is guaranteed by the task loss of the student network. The student network trained based on distillation loss and task loss, as well as the model parameters of the student network, are stored in a storage medium such as a database so that the image retrieval model of the student network can be directly retrieved from the storage medium such as the database when processing image retrieval tasks.

[0051] In some embodiments, when processing image retrieval tasks, users can send image retrieval requests through terminal devices such as smartphones, tablets, laptops, and desktop computers, for example, inputting the text "a picture of a cat" into the terminal device. Upon receiving the image retrieval request, the server retrieves a trained image retrieval model from a storage medium such as a database, inputs the image retrieval request into the image retrieval model, and then sends the cat-related images output by the image retrieval model to the terminal device, allowing the user to browse cat-related images through the terminal device.

[0052] In some embodiments, the student network can be applied in the field of computer vision. For example, a visual model pre-trained on an image dataset can be used as the teacher network. The student network obtained by knowledge distillation of the visual model can be applied to tasks such as image classification, object detection, and image retrieval. For example, an autonomous vehicle can determine whether it is a tunnel or a parking lot based on the current road image. Another example is in the field of product quality inspection, where product images can be used to detect whether there are scratches or stains on the product. Yet another example is in the field of item recommendation, where similar images can be retrieved based on the input item image.

[0053] Student networks can also be applied in the field of natural language processing. For example, a natural language model pre-trained on a text dataset can be used as a teacher network. The student network, obtained by knowledge distillation of the natural language model, can then be used for semantic understanding, text classification, sentiment analysis, or question-answering systems. For instance, in the field of intelligent customer service, a student network can perform semantic understanding based on user questions and provide automatic responses. Or, in the field of article recommendation, a student network can classify and recommend articles.

[0054] Student networks can also be applied to multimodal interaction scenarios. For example, a multimodal pre-trained model, pre-trained on an image-text dataset, can be used as a teacher network. The student network, obtained by knowledge distillation of the multimodal pre-trained model, can then be used for multimodal interaction tasks. For instance, in the field of image retrieval, based on the user's input text description "a cat," the system can return "an image of an orange cat napping in the sunlight."

[0055] By distilling the attention mechanism of the teacher network, the student network can inherit the capabilities of the teacher network while reducing the consumption of computing resources, making it suitable for operation on edge devices.

[0056] Figure 3 An architecture diagram of knowledge distillation according to an embodiment of the present invention is shown.

[0057] like Figure 3 As shown, a corresponding group is formed in the teacher network and the student network. For example, in this group The subgroups of the teacher network can include p first attention graphs (i.e., from the first attention layer of the teacher network). The first attention map is assigned to The number of attention layers is p), and the subgroups of the student network can include q second attention layers (i.e., from the second attention layer of the student network corresponding to the first attention layer). The second attention map is assigned to The quantity is q). Figure 3It may include a first splicing matrix 101-2 based on a first attention map 101-1, a second splicing matrix 102-2 based on a second attention map 102-1, a first weight matrix 301, a second weight matrix 302, a first attention matrix 101-3, a second attention matrix 102-3, a first redistillation loss 303, a second redistillation loss 304, a probability normalization layer 305, a normalized first attention matrix 101-4, a normalized second attention matrix 102-4, and a third distillation loss 306.

[0058] like Figure 3 As shown, the first weight matrix 301 can be a matrix used to perform dimensional transformation on the first concatenated matrix 101-2, and the second weight matrix 302 can be a matrix used to perform dimensional transformation on the second concatenated matrix 102-2.

[0059] Continue to refer to Figure 3 In the above operation S220, the process of normalizing the dimensions of the first concatenation matrix obtained by concatenating multiple first attention maps and the second concatenation matrix obtained by concatenating multiple second attention maps to obtain first and second attention matrices of the same dimension may include the following operations: reducing the dimension of the first concatenation matrix using the first weight matrix, and reducing the dimension of the second concatenation matrix using the second weight matrix, so that the first attention matrix obtained after reducing the dimension of the first concatenation matrix and the second attention matrix obtained after reducing the dimension of the second concatenation matrix are square matrices of a predetermined dimension, and the first weight matrix and the second weight matrix are learnable matrices updated by the backpropagation algorithm as the student network is trained. The predetermined dimension can be n rows and n columns, where n represents the number of tokens in the training samples input to the current layer.

[0060] Embodiments of the present invention introduce a learnable first weight matrix. Second weight matrix First weight matrix , dimension Through matrix multiplication , concatenate the first matrix Transform the first attention matrix with n rows and n columns Similarly, the second weight matrix The dimension of Through matrix multiplication operations , concatenate the second matrix Transform the second attention matrix with n rows and n columns First weight matrix Second weight matrix The parameters are continuously optimized during model training. Their role is to learn the most representative features from the concatenated matrix, achieving an effective mapping from high-dimensional concatenated information to a low-dimensional matrix, rather than simple dimensionality compression. The parameters of the first and second weight matrices are dynamically optimized during student network training using the backpropagation algorithm, achieving adaptive feature mapping from the concatenated matrix to the target dimension feature matrix.

[0061] According to an embodiment of the present invention, by reducing and unifying the dimensions of the first and second concatenation matrices, the dimensions of the attention matrix can be aligned, reducing computational complexity and memory usage, and improving the training efficiency and accuracy of the model.

[0062] In some embodiments, the multiple knowledge distillation loss for the knowledge distillation process from the first attention graph to the second attention graph can be determined based on the dimensionally normalized first and second attention matrices. For example, principal component analysis is used to determine the eigenvectors of the first and second attention matrices, respectively, and the first re-distillation loss of the first and second attention graphs is determined based on the eigenvectors. The first re-distillation loss describes the difference in attention patterns between the first and second attention graphs. Based on the graph distribution characteristics of the first and second attention matrices, the second re-distillation loss of the first and second attention graphs is determined. The second re-distillation loss describes the statistical difference between the first and second attention graphs. Based on the attention distribution characteristics of the first and second attention matrices, the third re-distillation loss of the first and second attention graphs is determined. The third re-distillation loss describes the difference in attention distribution between the first and second attention graphs.

[0063] In some embodiments, although the first attention matrix Second attention matrix All can be While direct comparison of the similarity between two square matrices can yield similar results, it still suffers from high computational complexity and sensitivity to noise. Therefore, this invention introduces Principal Component Analysis (PCA) to extract the core features of the first and second attention matrices, and determines the first redistillation loss based on the eigenvectors of these core features.

[0064] In some embodiments, the feature vectors include a first feature vector of a first attention matrix and a second feature vector of a second attention matrix, wherein the first feature vector is related to the maximum eigenvalue of the first attention map matrix and the second feature vector is related to the maximum eigenvalue of the second attention matrix; determining the first redistillation loss of the first attention map and the second attention map based on the feature vectors includes: processing the first feature vector and the second feature vector based on cosine similarity to obtain the first redistillation loss.

[0065] For example, the first attention matrix Perform PCA analysis to solve for the first attention matrix. The eigenvalues ​​are determined, and the first eigenvector corresponding to the largest eigenvalue is selected. (n-dimensional column vector); for the second attention matrix Perform the same operation to obtain the second feature vector. (n-dimensional column vector). The core function of PCA is to map high-dimensional data to a low-dimensional space through linear transformation, preserving the most discriminative information in the data (i.e., the direction of the largest eigenvalue). The first eigenvector extracted here. Second eigenvector It can be regarded as a condensed representation of the core patterns of the attention maps of the teacher group and the student group, avoiding the interference of redundant information when directly comparing matrices.

[0066] After the above steps, the first feature vector Second eigenvector The first feature vector has become a feature vector with the same dimension (n-dimensional), and can be quantified by cosine similarity. Second eigenvector The difference between the two, as a group The first redistillation loss, i.e. Figure 3 The first redistillation loss shown can be represented by formula (1).

[0067] (1)

[0068] in, It is the first eigenvector The transpose of , Represents the first eigenvector Length of the mold Represents the second eigenvector The length of the module.

[0069] By maximizing and The cosine similarity is used to minimize the first redistillation loss. This facilitates the alignment of student network attention patterns with teacher network attention patterns, achieving knowledge transfer at the attention level. The first feature vector is quantified using the complement of cosine similarity. Second eigenvector The smaller the complement number, the higher the alignment of the core features of the teacher network and the student network.

[0070] According to an embodiment of the present invention, by determining the first redistillation loss of knowledge distillation from the first attention map to the second attention map based on the first feature vector and the second feature vector, the difference of the core feature vectors extracted by PCA can be focused on, which can reflect the consistency of the macro-feature direction of the attention pattern, so as to determine the knowledge difference between the teacher and student networks from different levels, ensure the comprehensiveness of knowledge transfer, and improve the knowledge distillation effect.

[0071] In some embodiments, Figure 3 The second redistillation loss shown can be obtained based on the graph distribution characteristics of the first and second attention matrices. For example, graph distribution characteristics may include skewness and kurtosis, where skewness represents the degree of asymmetry of the attention graph in the attention matrix, and kurtosis represents the steepness of the peaks in the attention graph distribution in the attention matrix. Determining the second redistillation loss of the first and second attention graphs based on the graph distribution characteristics of the first and second attention matrices includes: determining the second redistillation loss based on the difference between the first skewness of the first attention matrix and the second skewness of the second attention matrix, and the difference between the first kurtosis of the first attention matrix and the second kurtosis of the second attention matrix.

[0072] The second redistillation loss can be based on higher-order statistics. This invention further introduces higher-order statistics such as skewness and kurtosis. By quantifying the differences between the teacher network and the student network in these higher-order statistics, the second redistillation loss term can be constructed.

[0073] For the first attention matrix Second attention matrix Calculate the first attention matrix respectively First skewness and the first peak And calculate the second attention matrix Second skewness Second kurtosis Both skewness and kurtosis can be calculated using corresponding statistical formulas. For example, the formula for skewness is... The kurtosis formula is (in The first attention matrix Or the second attention matrix The elements in The first attention matrix Or the second attention matrix mean The first attention matrix Or the second attention matrix Standard deviation The first attention matrix Or the second attention matrix Expected value). Calculate the first skewness. and the first peak In this case, the input matrix is ​​the first attention matrix. Calculate the second skewness Second kurtosis In this case, the input matrix is ​​the second attention matrix. .

[0074] According to the first skewness Second skewness First peak Second kurtosis The determined second redistillation loss can be shown in Equation (2).

[0075] (2)

[0076] in, and This is a hyperparameter; the default value is [value]. , The weights used to balance skewness and kurtosis differences are used. The second redistillation loss can uncover deeper differences in the distribution characteristics of the attention map. Compared to using first-order (e.g., mean) and second-order statistics (e.g., variance and covariance), higher-order statistics (i.e., skewness and kurtosis) can capture the non-Gaussian nature of the distribution, providing richer learning information to the student network and helping it to more comprehensively mimic the attention patterns of the teacher network.

[0077] According to an embodiment of the present invention, by determining the second distillation loss of knowledge distillation from the first attention map to the second attention map based on skewness and kurtosis, the differences between the teacher network and the student network in these higher-order statistics can be quantified to determine the knowledge differences between the teacher and student networks from different levels, ensuring the comprehensiveness of knowledge transfer and improving the knowledge distillation effect.

[0078] In some embodiments, Figure 3The third redistillation loss shown can be obtained based on the attention distribution characteristics of the first and second attention matrices. For example, the first and second attention matrices can be normalized to obtain normalized first and second attention matrices; the third redistillation loss can be determined based on the relative entropy between the attention distributions of the normalized first and second attention matrices.

[0079] In some embodiments, based on the first and second redistillation losses, this invention further introduces a third redistillation loss to capture the knowledge differences between teacher and student networks from different perspectives. The first step in determining the third redistillation loss is to perform a first attention matrix on the dimensionally unified matrix. Second attention matrix Perform probability normalization (Softmax) operation to obtain the normalized first attention matrix. and normalized second attention matrix .

[0080] The core function of the Softmax function is to transform the element values ​​of each row in the first or second attention matrix into a probability distribution (i.e., the sum of the elements in each row is 1, and each element value is in the range [0,1]). The significance of this transformation lies in the fact that the essence of the attention map is the correlation strength between tokens (the larger the value, the stronger the correlation). Softmax can normalize this strength into "attention allocation probability", making the matrix elements more intuitively represent the "distribution of the model's attention weights to different tokens", laying the foundation for subsequent loss calculation based on distribution differences.

[0081] The third redistillation loss aims to quantize and normalize the first attention matrix. and normalized second attention matrix The difference further constrains the attention patterns of students' online learning and teachers' online learning. The third redistillation loss function can be shown in formula (3).

[0082] (3)

[0083] Among them, the function This is a loss function used to measure the difference between two feature matrices (or distributions). Its core function is to quantify the deviation between the teacher network and the student network in attention distribution, and to guide the student network to adjust its parameters through backpropagation to reduce this deviation. Specifically, it can be implemented in the following ways: function It is the relative entropy (Kullback-Leibler, KL divergence), used to quantify the difference between the feature maps of the teacher network and the student network to guide the learning process of the student network; function Mean squared error loss, cosine similarity, etc. can also be used. KL divergence can be expressed as shown in formula (4).

[0084] (4)

[0085] Where z is the index, if there are Z (Z is a positive integer) source models and Z target models respectively, P(z) can be the attention distribution of the source model, such as the attention distribution of a teacher network, and Q(z) can be the attention distribution of the target model, such as the attention distribution of a student network. When the function... When the divergence is Kullback-Leibler, the loss of the third redistillation can be expressed as shown in Equation (5).

[0086] (5)

[0087] According to an embodiment of the present invention, the third distillation loss for knowledge distillation from the first attention map to the second attention map is determined by normalizing the first attention matrix and the second attention matrix. This allows attention to be paid to the difference in attention distribution between the first and second attention maps after Softmax transformation, reflecting the micro-details of attention allocation. This enables the determination of knowledge differences between teacher and student networks from different levels, ensuring the comprehensiveness of knowledge transfer and improving the knowledge distillation effect.

[0088] In some embodiments, as described above, the teacher network includes a first attention layer, and the student network includes a second attention layer. The first attention map originates from the first attention layer, and the second attention map originates from the second attention layer. The first, second, and third redistillation losses described above, combined with the task loss of the student network, can be used to adjust the model parameters of the student network. For example, the layer distillation loss between the first and second attention layers can be determined based on the multiple distillation loss; the target distillation loss from the teacher network to the student network can be determined based on the layer distillation loss; the target loss during the student network training process can be determined based on the target distillation loss and the task loss; and the model parameters of the student network can be adjusted based on the target loss.

[0089] In one embodiment, the total loss within a group can be determined by fusing complementary information from the first redistillation loss, the second redistillation loss, and the third redistillation loss. .

[0090] Because a single loss function has limitations in capturing the knowledge differences between teacher and student networks and cannot comprehensively cover knowledge features across different dimensions, this embodiment of the invention uses a first redistillation loss function. Second redistillation loss With the loss of the third redistillation To achieve more comprehensive knowledge alignment, integration is needed. Total loss The calculation process can be shown in formula (6).

[0091] (6)

[0092] in, It is a coefficient, by default. , , .

[0093] The advantage of the above-mentioned integration method is that... Focusing on the differences in the core feature vectors extracted by PCA reflects the consistency of the macro-feature direction of the attention pattern; Quantify the differences between teacher networks and student networks on these higher-order statistics; and This focuses on the differences in attention distribution after Softmax transformation, reflecting the micro-details of attention allocation. These three approaches complement each other, capturing the knowledge differences between teacher and student networks at different levels, ensuring the comprehensiveness of knowledge transfer.

[0094] In some embodiments, in formula (6) Both can be dynamically adjusted. For example... The training can be dynamically adjusted based on the polling progress of the training samples. For example, if the ratio of the number of polls of the training samples to the predetermined total number of polls of the training samples is less than a first predetermined ratio, the current training stage of the student network is determined to be the initial training stage. In the initial training stage, the parameters of the student network are random and differ greatly from those of the teacher network. In the initial training stage, more attention can be paid to the first and third distillation losses. and All can be greater than ,For example The values ​​are 0.4, 0.2, and 0.4, respectively. When the ratio of the number of times the training samples are polled to the predetermined total number of polls for the training samples is greater than or equal to a first predetermined ratio, and less than a second predetermined ratio, the current training stage of the student network is determined to be the mid-training stage. In the mid-training stage, the student network has initially grasped the teacher's attention pattern and requires more refined adjustments. The three can be distributed in a near-equilibrium manner, for example... The values ​​are 0.3, 0.3, and 0.4, respectively. When the ratio of the number of times the training samples are polled to the predetermined total number of polls for the training samples is greater than or equal to a second predetermined ratio, the current training stage of the student network is determined to be the late training stage. In the late training stage, the student network and the teacher network can be highly aligned, and it is necessary to break through the bottleneck and capture deeper differences. More attention can be paid to differences in higher-order statistics (skewness, kurtosis). and All can be less than ,For example The values ​​are 0.2, 0.5, and 0.3 respectively.

[0095] According to an embodiment of the present invention, by means of Dynamic adjustments in stages can improve the training accuracy and efficiency of student networks, enhance their robustness, reduce the workload of manual parameter tuning, adapt to different application scenarios, and improve knowledge distillation effects.

[0096] Determine the total loss within the group In this case, the "one-to-one" layer selected above can be determined. Layer distillation loss .

[0097] For any of the "one-to-one" corresponding layers selected above (This layer contains a total of) (one corresponding group), the layer distillation loss of the corresponding layer The average loss of all groups in the corresponding layer can be expressed as shown in formula (7).

[0098] (7)

[0099] Within the same floor Each group corresponds to a different subset of the attention map, and the knowledge information carried by each group differs to some extent. By averaging, we can avoid the extreme loss values ​​of a particular group from excessively affecting the overall layer loss, effectively balancing the knowledge contributions of each group. After this processing, the single-layer loss can more stably and objectively reflect the knowledge alignment effect of that layer, providing a reliable basis for the subsequent calculation of the overall distillation loss.

[0100] The target distillation loss for the entire knowledge distillation process is the sum of all selected values. The average loss of each "one-to-one" corresponding layer is calculated by averaging the loss across all attention layers involved in the distillation process. This integrates the knowledge from different attention layers (such as basic features at lower levels and semantic features at higher levels), enabling the student network to learn from the multi-level knowledge of the teacher network and ultimately improve performance. The target distillation loss for the entire knowledge distillation process can be expressed as shown in formula (8).

[0101] (8)

[0102] Note that in the calculation Formulas and calculations A constant coefficient can also be added to the formula.

[0103] In some embodiments, the process of determining the task loss function is described below. In this embodiment of the invention, knowledge distillation can be performed on a large language model. The task is to predict the next word given a sequence of tokens. Therefore, the task loss function... The cross-entropy loss function is used.

[0104] If there is a contained Given a sequence of N words, for each position e, the model needs to predict the probability distribution of the next word belonging to V words in the vocabulary. Let... It is a V-dimensional one-hot vector representing the category corresponding to the true word at position e. If is the V-dimensional probability distribution vector predicted by the model, then the cross-entropy loss function can be expressed as shown in formula (9).

[0105] (9)

[0106] in, yes The j-th element of the vector, yes The j-th element of the vector. Formula (9) calculates the average difference between the probability distribution predicted by the student network model and the real label. By minimizing this loss function, the parameters of the model are adjusted so that the model can better predict the next word, thereby improving the performance of the language model.

[0107] According to an embodiment of the present invention, by adjusting the model parameters of the student network based on multiple knowledge distillation, layer-by-layer and refined knowledge transfer is achieved, which helps the student network to better imitate the reasoning path and feature evolution process of the teacher network and improves the knowledge distillation effect.

[0108] In some embodiments, the process of determining the target loss of the student network training process based on the target distillation loss and the task loss described above may include the following operations: adjusting the target distillation loss using predetermined weights to obtain an updated distillation loss, wherein the predetermined weights are determined based on the accuracy of the student network's output results and are used to describe the importance of the target distillation loss to the target loss; and determining the target loss based on the updated distillation loss and the task loss.

[0109] In some embodiments, based on the target distillation loss and task loss described above, embodiments of the present invention define a target loss. , The target distillation loss and mission losses It is a function of the independent variable. In this embodiment of the invention, A linear function can be used, and the target loss can be as shown in formula (10).

[0110] (10)

[0111] in, It is a predetermined weight, a coefficient used for balancing. and The contribution. Through dynamic adjustment during training. It can achieve better accuracy.

[0112] In some embodiments, the predetermined weights are determined as follows: a first feature value is obtained based on the difference between the accuracy of the student network's output and a predetermined accuracy threshold; a second feature value is obtained based on the difference between a first training constant and the predetermined accuracy threshold; a first candidate weight is determined based on the ratio of the first feature value and the second feature value; and a predetermined weight is determined based on the maximum value between the first candidate weight and the second training constant, wherein the second training constant is less than the first training constant.

[0113] In some embodiments, a dynamic adjustment function can be defined. ,in This could be the accuracy of the student network in the current training batch. It refers to the accuracy of the student network's output in the current training batch. When used as input, (in This is the preset accuracy threshold; the default value is [value missing]. , 1 is the first feature value, and 1 is the first training constant. The second eigenvalue, (where 0.5 is the first candidate weight and 0.5 is the second training constant). This way, when the accuracy of the student network is low, for example, below a predetermined value, the weight can be appropriately reduced. The weight can be appropriately increased when the accuracy of the student network is high, for example, higher than the predetermined value. The weights can be preset to a certain accuracy threshold.

[0114] According to embodiments of the present invention, by providing dynamic predetermined weights and using these dynamic predetermined weights to dynamically adjust the target distillation loss, intelligent dynamic switching of the training focus can be achieved, automated tuning of model parameters can be realized, and the stability and efficiency of model training can be improved.

[0115] In this embodiment of the invention, the core feature vectors of the attention feature matrices of the teacher and student networks in the Transformer architecture, after unifying their dimensions, are extracted using Principal Component Analysis (PCA). A first loss based on the complement of cosine similarity is calculated to align the macroscopic feature directions. Then, a second redistillation loss is calculated for the teacher and student networks using skewness and kurtosis. Softmax probability normalization is then performed on the dimension-unified matrix, and a second loss based on the difference in attention distribution is calculated using KL divergence to match microscopic attention patterns. By fusing multiple losses, the comprehensive transfer of attention knowledge in the teacher network is achieved, avoiding the limitations of a single loss in knowledge capture.

[0116] The embodiments of this invention focus on establishing precise correspondences between attention maps in the Transformer architecture, using learnable matrices to achieve adaptive dimensional transformation and unification, and combining multiple loss to transfer attention knowledge. First, the attention maps of teachers and students are divided into equal groups according to their sequence numbers to establish a correspondence. Then, the different dimensional matrices within each group are unified into a square matrix of the same dimension using learnable matrices. Finally, knowledge distillation is achieved across differences in the number of attention heads by combining multiple loss calculations. This effectively addresses the problem of knowledge loss caused by the mismatch in the number of attention heads between teacher and student networks and the inconsistency in dimensions. Simultaneously, this scheme can accurately capture the core knowledge at the attention level under the Transformer architecture, significantly improving the efficiency and effectiveness of knowledge distillation. This helps student networks fully absorb the attention patterns of teacher networks, enhances student network performance, and further expands the adaptability and application value of knowledge distillation methods in Transformer-based neural network models, improving student network performance. The above method is not only applicable to knowledge distillation in scenarios where the number of attention maps in the teacher network and student network of the Transformer architecture is inconsistent, but can also be applied to deep learning tasks based on the Transformer architecture, such as natural language processing, computer vision, or multimodal understanding, to improve the inference performance and efficiency of the student network in the task.

[0117] Based on the training process of the student network combining multiple distillation loss and task loss described above, the following provides a specific embodiment to describe the training process of the student network.

[0118] For the training dataset D, the source model (teacher network) and target model (student model) in this embodiment are large language models. The data in the training dataset is text data. The text data is input into the large language model to generate intermediate state values ​​during model operation (e.g., query embeddings, key embeddings, and value embeddings), as well as the final output text. In the large language model used for question answering, the input is the question, and the output is the answer to the question. This invention collects a large-scale text dataset containing hundreds of millions of sentences, of which: Chinese sentences account for 30%, English sentences account for 60%, and other languages ​​account for 10%. During model training, text data is sampled from the large-scale text dataset using random sampling. The above language proportions are exemplary values; using other proportions does not affect the effectiveness of this embodiment.

[0119] To avoid the order of samples in the training dataset D affecting model performance, all samples in the training dataset D are randomly arranged at the start of model training. During each training round, the model training algorithm reads a batch of samples, for example, 1024 samples.

[0120] A model training algorithm can include training input, training output, loss function settings, and model training steps.

[0121] The training input can be an input source model (teacher network). For example, a large language model based on Transformer. Data is read from the training dataset D, assuming the data has undergone necessary preprocessing and its format meets the model input requirements; initial training cycle... Total training cycle The stochastic gradient descent algorithm is an adaptive moment estimator optimizer (Adam) algorithm. Relevant parameters include the learning rate. momentum coefficient , , The learning rate strategy is a cosine annealing strategy, specifically adjusted dynamically based on the cosine function during the training period. The number of samples in a mini-batch is B = 1024.

[0122] Training output, target model (student network) Among them, the target model The structure is basically the same as the source model, with the only difference being the number of heads in the multi-head attention mechanism. In each layer of the target model... The number of heads in the multi-head attention mechanism is no greater than the number of heads in the source model's multi-head attention mechanism.

[0123] The loss function is set as shown in formula (10). .

[0124] The model training steps are as follows:

[0125] Step 1: Convert the source model All parameter values ​​are fixed; based on the source model Network structure construction target model Network structure; random initialization of the target model The element values ​​of the weight matrix, and other parameters.

[0126] Step 2: When When this happens, perform the following operations:

[0127] Step 2.1: .

[0128] Step 2.2: Randomly shuffle the order of the samples in dataset D.

[0129] Step 2.3: Select a batch B of training samples from dataset D.

[0130] Step 2.4: Based on the above training samples and parameter settings, calculate the loss value, and then update the target model using the stochastic gradient descent algorithm (Adam algorithm). The parameters are set. Before updating the parameters, the gradient is clipped, and a gradient threshold of 5 is set (which can be adjusted according to the actual situation) to prevent gradient explosion.

[0131] Step 2.5: Repeat steps 2.3 and 2.4 until all samples of dataset D have been utilized.

[0132] Step 2.6: End the current training cycle.

[0133] Step 3: End training and output the target model. . Target model Save it to the specified file path for later use.

[0134] Based on the model training method described above, this embodiment of the invention also provides a method for applying the target model trained above. Specifically, this embodiment of the invention provides an image retrieval model.

[0135] Figure 4 An architecture diagram of an image retrieval model according to an embodiment of the present invention is shown.

[0136] like Figure 4As shown, the architecture of this image retrieval model may include a first image encoding model 401 and a first text encoding model 402 as the teacher network, and a second image encoder 403 and a second text encoding model 404 as the student network, obtained by distilling the first image encoding model 401 and the first text encoding model 402 respectively using the knowledge distillation method described above. Both the second image encoding model 403 and the second text encoding model 404 can be Transformer-based neural networks.

[0137] Image coding models and text coding models can be contrastive language-image pre-training (CLIP) models trained on datasets of images and text.

[0138] Before applying the image retrieval model, a database of images to be queried, Db, can be pre-created. This database contains a large number of unlabeled images. The content of these images is unknown; that is, the categories of objects within them are unknown. The image database Db is large, covering images from multiple domains and scenes. Each image in the database is an independent instance, without prior classification or labeling. For ease of explanation later, the total number of images in the image database Db is defined as... .

[0139] Create an image feature database based on the image database to be queried. Each image in the image database Db to be queried The input is fed into the second image coding model to obtain the output feature embedding vector for each image. , .exist In the middle, take As an image The characteristics of the image feature database. include The set of image features is as follows Image features correspond one-to-one with images.

[0140] Once the image database Db is created, the application process for the image retrieval model can be as follows.

[0141] The input query text T is a piece of natural language text, and its language type should be consistent with the language type of the text in the training dataset of the jointly trained image and text model. In the training dataset of the jointly trained image and text model, images and their corresponding texts constitute image-text pairs. The language type of the text determines the language type of the input text that the second text encoding model can process. In other words, if the text in the training dataset of the jointly trained image and text model is Chinese, then the input query text should also be Chinese; if the text in the training dataset is English, then the input query text should also be English; if the text in the training dataset covers multiple languages, then the input query text can contain multiple languages. Ensuring that the language type of the input query text is consistent with the text in the training dataset guarantees the accuracy and reliability of image retrieval.

[0142] The features of the input query text T are obtained, and the feature embedding vector obtained by inputting the query text T into the second text encoding model is: , .exist In the middle, take Features of the input query text.

[0143] Calculate the features of the input query text T and the image feature database. Image feature similarity in image database include The set of image features is as follows The characteristics of the input query text T are: Therefore, the similarity between the features of the input query text T and each feature in the image feature data is shown in formula (11).

[0144] (11)

[0145] in," The operation process represented by " For example, ,in, and These represent two column vectors, for example , represent The transpose of , Represents the magnitude of the vector.

[0146] The image retrieval model in this embodiment aims to obtain image indexes by finding one or more images from the image database that correspond to the input text. To this end, the invention sets a similarity threshold. In this embodiment .

[0147] The similarity between the features of the input query text T and the image features is as follows:

[0148]

[0149] So, based on the similarity threshold The resulting image index set is as follows:

[0150] , represents returning all A set of indices.

[0151] Returns a set of retrieved images, based on the image index set. It retrieves the corresponding image from the image database Db and returns it.

[0152] Based on the above-described knowledge distillation method, this invention also provides a knowledge distillation apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0153] Figure 5 A structural block diagram of a knowledge distillation apparatus according to an embodiment of the present invention is shown.

[0154] like Figure 5 As shown, the knowledge distillation apparatus 500 of this embodiment includes a first determining module 510, a normalization module 520, a second determining module 530, and an adjustment module 540.

[0155] The first determining module 510 is used to determine multiple first attention maps for the teacher network and multiple second attention maps for the student network, respectively.

[0156] The normalization module 520 is used to normalize the dimensions of the first splicing matrix obtained by splicing multiple first attention maps and the second splicing matrix obtained by splicing multiple second attention maps, so as to obtain the first attention matrix and the second attention matrix with the same dimensions.

[0157] The second determining module 530 is used to determine the multiple distillation loss for knowledge distillation from the first attention map to the second attention map based on the matrix characteristics of the first attention matrix and the matrix characteristics of the second attention matrix.

[0158] The adjustment module 540 is used to adjust the model parameters of the student network during the training process of the student network based on the multiple distillation loss and the task loss of the student network, until all samples in the sample set used to train the student network are polled to obtain the target student network. The sample set includes training samples and sample labels, and the task loss is used to describe the difference between the output results of the student network based on the training samples and the sample labels.

[0159] According to embodiments of the present invention, any plurality of modules among the first determining module 510, normalization module 520, second determining module 530, and adjustment module 540 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the first determining module 510, normalization module 520, second determining module 530, and adjustment module 540 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first determining module 510, the normalization module 520, the second determining module 530, and the adjustment module 540 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0160] It should be noted that the knowledge distillation apparatus part in the embodiments of the present invention corresponds to the knowledge distillation method part in the embodiments of the present invention. The description of the knowledge distillation apparatus part is specifically referred to in the knowledge distillation method part, and will not be repeated here.

[0161] Figure 6 A block diagram of an electronic device suitable for implementing a knowledge distillation method according to an embodiment of the present invention is shown schematically.

[0162] like Figure 6 As shown, an electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0163] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0164] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0165] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0166] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0167] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the knowledge distillation method provided in the embodiments of the present invention.

[0168] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0169] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0170] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0171] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0173] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0174] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A knowledge distillation method, characterized in that, The method includes: Multiple first attention maps are identified for the teacher network and multiple second attention maps are identified for the student network. The dimensions of the first concatenation matrix obtained by concatenating the multiple first attention maps and the second concatenation matrix obtained by concatenating the multiple second attention maps are normalized to obtain first attention matrices and second attention matrices with the same dimensions. Based on the matrix features of the first attention matrix and the matrix features of the second attention matrix, a multiple distillation loss is determined for knowledge distillation from the first attention graph to the second attention graph; During the training of the student network, the model parameters of the student network are adjusted according to the multiple distillation loss and the task loss of the student network until all samples in the sample set used to train the student network are polled to obtain the target student network. The sample set includes training samples and sample labels. The task loss is used to describe the difference between the output results of the student network based on the training samples and the sample labels.

2. The method according to claim 1, characterized in that, The step of determining the multiple distillation loss for knowledge distillation from the first attention map to the second attention map based on the matrix features of the first attention matrix and the matrix features of the second attention matrix includes: The eigenvectors of the first attention matrix and the second attention matrix are determined by principal component analysis, and the first redistillation loss of the first attention map and the second attention map is determined based on the eigenvectors. The first redistillation loss is used to describe the difference in attention patterns between the first attention map and the second attention map. Based on the graph distribution characteristics of the first attention matrix and the second attention matrix, a second redistillation loss is determined for the first attention graph and the second attention graph. The second redistillation loss is used to describe the statistical difference between the first attention graph and the second attention graph. Based on the attention distribution characteristics of the first attention matrix and the second attention matrix, a third redistillation loss is determined for the first attention map and the second attention map. The third redistillation loss is used to describe the difference in attention distribution between the first attention map and the second attention map.

3. The method according to claim 2, characterized in that, The feature vector includes a first feature vector of a first attention matrix and a second feature vector of a second attention matrix. The first feature vector is related to the maximum eigenvalue of the first attention matrix, and the second feature vector is related to the maximum eigenvalue of the second attention matrix. The step of determining the first redistillation loss of the first attention map and the second attention map based on the feature vector includes: The first redistillation loss is obtained by processing the first feature vector and the second feature vector based on cosine similarity.

4. The method according to claim 2, characterized in that, The graph distribution features include skewness and kurtosis. Skewness is used to represent the degree of asymmetry of the attention matrix, and kurtosis is used to represent the steepness of the peak value of the attention matrix. The step of determining the second redistillation loss of the first attention map and the second attention map based on the graph distribution characteristics of the first attention matrix and the second attention matrix includes: The second redistillation loss is determined based on the difference between the first skewness of the first attention matrix and the second skewness of the second attention matrix, and the difference between the first kurtosis of the first attention matrix and the second kurtosis of the second attention matrix.

5. The method according to claim 2, characterized in that, The step of determining the third redistillation loss of the first attention and the second attention based on the attention distribution characteristics of the first attention matrix and the second attention matrix includes: The first attention matrix and the second attention matrix are normalized respectively to obtain the normalized first attention matrix and the normalized second attention matrix; The third redistillation loss is determined based on the relative entropy between the attention distribution of the normalized first attention matrix and the attention distribution of the normalized second attention matrix.

6. The method according to claim 1, characterized in that, The teacher network includes a first attention layer, and the student network includes a second attention layer. The first attention graph originates from the first attention layer, and the second attention graph originates from the second attention layer. The step of adjusting the model parameters of the student network based on the multiple distillation loss and the task loss of the student network includes: Based on the multiple distillation loss, determine the layer distillation loss between the first attention layer and the second attention layer; Based on the layer distillation loss, determine the target distillation loss from the teacher network to the student network; The target loss of the student network training process is determined based on the target distillation loss and the task loss. The model parameters of the student network are adjusted based on the target loss.

7. The method according to claim 6, characterized in that, Determining the target loss of the student network training process based on the target distillation loss and the task loss includes: The target distillation loss is adjusted using predetermined weights to obtain an updated distillation loss. The predetermined weights are determined based on the accuracy of the output results of the student network and are used to describe the importance of the target distillation loss to the target loss. The target loss is determined based on the updated distillation loss and the task loss.

8. The method according to claim 7, characterized in that, The predetermined weights are determined in the following manner: The first feature value is obtained based on the difference between the accuracy of the output result of the student network and a predetermined accuracy threshold. The second feature value is obtained based on the difference between the first training constant and the predetermined accuracy threshold; The first candidate weight is determined based on the ratio of the first feature value to the second feature value; The predetermined weight is determined based on the maximum value between the first candidate weight and the second training constant, wherein the second training constant is less than the first training constant.

9. The method according to claim 1, characterized in that, The step of normalizing the dimensions of the first concatenation matrix obtained by concatenating the multiple first attention maps and the second concatenation matrix obtained by concatenating the multiple second attention maps to obtain first attention matrices and second attention matrices with the same dimensions includes: The first concatenated matrix is ​​reduced in dimension using a first weight matrix, and the second concatenated matrix is ​​reduced in dimension using a second weight matrix, so that the first attention matrix obtained after reducing the dimension of the first concatenated matrix and the second attention matrix obtained after reducing the dimension of the second concatenated matrix are square matrices of a predetermined dimension. The first weight matrix and the second weight matrix are learnable matrices that are updated by the backpropagation algorithm as the student network is trained.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

Images