An image retrieval method

Abstract

The application discloses an image retrieval method, and relates to the technical field of knowledge distillation, comprising processing a first attention map set generated by a teacher model and converting the first attention map set into a third attention map set with the same number of attention heads as a student model, so that direct alignment of attention maps between models of different sizes is realized. A loss function constructed based on the difference between the two is used to guide the learning of the student model, breaking through the dimensional barrier caused by the inconsistency in the number of heads and improving the efficiency and effect of knowledge transfer. The student model obtained by distillation greatly reduces the computing and storage overhead while maintaining high model accuracy, solving the problem that the attention maps of the teacher model and the student model cannot be directly aligned due to the difference in the number of attention heads, and the effect of knowledge transfer is limited, achieving the beneficial effects of efficiently deploying high-precision lightweight models on terminal devices and effectively balancing inference efficiency and model accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge distillation, and particularly relates to an image retrieval method. BACKGROUND

[0002] At present, the Transformer model is widely applied in the field of natural language processing, but its high computing and storage overhead seriously limits the deployment on terminal devices. To solve this problem, the knowledge distillation technology is often used to compress the Transformer model, and the student model with fewer parameters learns the output or intermediate layer features of the teacher model, so as to maintain high performance while significantly reducing the inference cost. Since such a model is based on the structure design of multi-head self-attention, and the number of attention heads of the teacher model and the student model is often inconsistent, the generated attention map is difficult to directly align. This dimension mismatch problem makes the distillation method based on attention transfer cannot be directly applied, which limits the knowledge transfer effect, resulting in the precision loss of the lightweight model obtained by distillation when deployed on terminal devices, and it is difficult to achieve effective balance between efficiency and precision.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. SUMMARY

[0004] The present application provides an image retrieval method to at least solve the problem that the attention map of the teacher model and the student model is difficult to directly align due to the different number of attention heads, and the knowledge transfer effect is limited in the related art.

[0005] The present application provides a knowledge distillation method, comprising: determining a first attention map set generated by a teacher model based on a training sample, and a second attention map set generated by a student model based on the training sample; the multi-head attention mechanism of the teacher model has a first number of attention heads, and the multi-head attention mechanism of the student model has a second number of attention heads, and the first number is greater than or equal to the second number; processing the first attention map set to obtain a third attention map set matched in number with the second attention map set; updating the model parameters of the student model by using a loss function; wherein the loss function is established based on the difference between the third attention map set and the second attention map set; and deploying the student model that meets the deployment condition on a terminal device.

[0006] The application also provides an image retrieval method, comprising: acquiring a query text; inputting the query text into a text encoding module to obtain a text feature embedding vector; the text encoding module is a student model obtained based on the knowledge distillation method described above; calculating the similarity between the text feature embedding vector and each image feature embedding vector in an image feature database; the image feature database comprises a plurality of image feature embedding vectors, each image feature embedding vector is obtained by inputting a corresponding image in an image database into an image encoding model, and the image encoding model is a student model obtained by using any one of the knowledge distillation methods described above; based on the comparison result of the calculated similarity and a preset similarity threshold, all images with a similarity greater than the preset similarity threshold are retrieved from the image database as a retrieval result and returned.

[0007] Through the application, by processing the first attention map set generated by the teacher model and converting it into a third attention map set with the same number of attention heads as the student model, direct alignment of attention maps between models of different scales is realized. The loss function constructed based on the difference between the two guides the learning of the student model, enabling it to efficiently mimic the attention distribution pattern extracted by the teacher model, breaking through the dimensional barrier caused by inconsistent number of heads, and improving the efficiency and effectiveness of knowledge transfer. Finally, the distilled student model significantly reduces the computational and storage overhead while maintaining high model accuracy, solving the technical problems of difficulty in direct alignment of attention maps and limited knowledge transfer effect caused by the difference in the number of attention heads between the teacher model and the student model, achieving the beneficial effects of efficient deployment of high-precision lightweight models on terminal devices and effective balance between inference efficiency and model accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application, the drawings required in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0009] Figure 1 A single-head self-attention mechanism diagram provided for the embodiments of the present application.

[0010] Figure 2 A step flowchart of a knowledge distillation method provided for the embodiments of the present application.

[0011] Figure 3 A structural schematic diagram of a knowledge distillation device provided for the embodiments of the present application.

[0012] Figure 4A step flowchart of an image retrieval method provided by an embodiment of the present application.

[0013] Figure 5 A model distillation schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0015] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] In order to enable a person skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0017] First, the key modules of the Transformer architecture neural network model are introduced.

[0018] For the basic single-head attention mechanism, the core idea of the attention mechanism is to calculate the output according to the query (Query), key (Key) and value (Value).

[0019] The input is given query matrix , key matrix and value matrix , where n is the number of queries, keys and values, is the dimension of the query and key, is the dimension of the value. In this embodiment, , represents an n-row, m-column real matrix.

[0020] When calculating the attention score, first calculate the similarity between the query and the key to obtain the attention score matrix S, wherein, The superscript T represents matrix transposition. is an n-by-n real matrix. To avoid the dot product result being too large to affect the gradient stability in the training process, scaling operation is performed, divided by , that is , is the scaled attention score.

[0021] In the calculation of the attention distribution, the Softmax function (normalized exponential function) is used to convert the scaled attention score into a probability distribution, and the attention graph matrix A is obtained ; where The function represents converting a row of elements into a probability value.

[0022] Finally, the attention graph matrix A is multiplied by the value matrix V to obtain the output of the attention mechanism , .

[0023] As shown in the single-head self-attention diagram, assuming that the current layer has Figure 1 input embedding vectors, the th input embedding vector is , , , the th input embedding vector is input into the query value transformation matrix to obtain the output query , the th input embedding vector is input into the key value transformation matrix to obtain the output key , the th input embedding vector is input into the feature embedding value transformation matrix to obtain the value .

[0024] Then the query matrix is represented as , the key matrix , and the value matrix . In the self-attention mechanism, the inputs of the three transformation matrices are the same, and in the non-self-attention mechanism, the inputs of the three transformation matrices are different. Among them, , , , , , are column vectors.

[0025] For multi-head attention mechanism, the multi-head attention mechanism calculates the attention graph matrix in parallel through multiple different attention heads (heads), so that the model can capture information from different representation subspaces.

[0026] Assuming there are h attention heads, for the input query matrix Q, key matrix K and value matrix V, respectively through h groups of different transformation matrices , and , , is the dimension of the model, is the dimension of the query and key, is the dimension of the value, .

[0027] The query matrix, key matrix and value matrix of the h-th attention head are , , , .

[0028] For the h-th attention head, the calculation is performed according to the steps of the above basic single-head attention mechanism, , , , wherein is the attention map matrix corresponding to the h-th attention head, is the output corresponding to the h-th attention head. The outputs of the h attention heads are spliced to obtain the spliced matrix ,

[0029] , , , wherein , wherein represents an n-row, column real matrix.

[0030] Finally, the spliced matrix is transformed by a linear transformation matrix to obtain the final output of the multi-head attention mechanism , .

[0031] In the knowledge distillation process based on the above Transformer architecture, the difference in the number of multi-head attention mechanism heads between the teacher network and the student network leads to inconsistent attention map numbers between the two, making it difficult to directly apply the traditional attention map comparison-based distillation loss. If forced to use, it will cause information loss due to the mismatch of the number of attention maps, seriously affecting the effective learning and absorption of the student network to the knowledge of the teacher network, and failing to achieve the goal of guiding knowledge distillation or improving the performance of the student network. To solve this problem, the embodiments of the present application provide a knowledge distillation method, which is described in detail in combination with the execution process of the knowledge distillation method as shown in Figure 2 .

[0032] S101: determine a first attention map set generated by a teacher model based on a training sample, and a second attention map set generated by a student model based on the training sample; the multi-head attention mechanism of the teacher model has a first number of attention heads, and the multi-head attention mechanism of the student model has a second number of attention heads, the first number being greater than or equal to the second number.

[0033] In this embodiment, the teacher model and the student model are both models based on the Transformer architecture, where the teacher model is a source model, that is, a model to be distilled, and the student model is a model obtained after knowledge distillation of the teacher model. The multi-head attention mechanisms of the teacher model and the student model have the same or different number of attention heads. Specifically, the first number of attention heads of the teacher model is greater than or equal to the second number of attention heads of the student model. By simultaneously inputting the same training sample into the teacher model and the student model, it is ensured that both of them process completely consistent data, thereby extracting attention information in the same semantic context and providing consistent and comparable data basis for subsequent attention map alignment and knowledge distillation.

[0034] It can be understood that the teacher model and the student model each include L layers. For ease of illustration, this embodiment analyzes a certain layer. Suppose that the multi-head attention mechanism of a certain layer of the teacher network includes attention heads. After inputting the training sample into the teacher model, the teacher network at this layer will correspondingly generate a first attention map set wherein is the attention map generated by the first attention head of the teacher network at this layer, is the attention map generated by the second attention head of the teacher network at this layer, is the attention map generated by the last attention head of the teacher network at this layer, and the multi-head attention mechanism of the student network at the same layer includes attention heads. After inputting the same training sample into the student model, the student network at this layer will correspondingly generate a second attention map set wherein is the attention map generated by the first attention head of the student network at this layer, is the attention map generated by the second attention head of the student network at this layer, is the attention map generated by the last attention head of the student network at this layer.

[0035] This parallel processing mode not only improves the training efficiency, but also enables the teacher model and the student model to generate attention maps under the same input condition, thereby avoiding noise caused by input differences and facilitating more accurate implementation of attention knowledge migration.

[0036] S102: Process the first set of attention maps to obtain a third set of attention maps that matches the second set of attention maps in number.

[0037] In this embodiment, considering that the first number of attention heads of the teacher model is greater than or equal to the second number of attention heads of the student model , there is a difficulty in directly aligning the attention maps and transferring knowledge. Therefore, this step processes the first set of attention maps (containing attention maps) generated by the teacher model to convert it into a completely new third set of attention maps (containing attention maps) that is the same number as the second set of attention maps. The processing here can include any mathematical operation or neural network mapping that can achieve attention map number compression or transformation, thereby achieving structural adaptation of attention knowledge from the teacher to the student.

[0038] As an optional embodiment, the processing method includes but is not limited to aggregating the attention maps of multiple attention heads of the teacher model through average pooling, weighted summation, linear projection, or mapping based on small neural networks, etc., to generate a third set of attention maps that is consistent with the number of attention heads of the student model and rich in semantic of the teacher's multiple heads. This embodiment refines the high-dimensional attention representation scattered in multiple attention heads in the teacher model into a low-dimensional but higher information density attention representation that matches the number of heads of the student model.

[0039] By integrating and compressing the rich multi-head attention information of the teacher network to align it with the student network in the number dimension, the loss of knowledge caused by directly discarding the attention head information of the teacher network is avoided, and the student model can more efficiently and stably learn the core attention patterns and decision logic of the teacher model, thereby significantly improving the final effect and generalization performance of knowledge distillation.

[0040] S103: Update the model parameters of the student model using a loss function; wherein the loss function is established based on the difference between the third set of attention maps and the second set of attention maps.

[0041] In this embodiment, after obtaining the third attention map set with aligned numbers, the model parameters are updated. Specifically, a pre-established loss function is used to update the student model's parameters. This loss function measures the difference between the corresponding attention maps in the third and second attention map sets. Its specific form can be any function that quantifies the distribution or numerical difference between two attention maps, such as mean squared error, Kullback-Leibler divergence, or cosine similarity. The error value calculated by this loss function is used to update the student model's parameters through a backpropagation algorithm, thereby driving the student model's attention map output to approximate the teacher model's attention map output.

[0042] By minimizing this loss function, the high-level semantic relationships and feature importance patterns captured by the teacher model, after being condensed and structurally adapted, can be effectively and continuously transferred to the student model. This optimization process enables the student model not only to learn the final task output but also to deeply learn the internal working mechanisms and decision-making basis of the teacher model. It solves the problem of direct alignment of attention maps, and because it is built on the third attention map set already matched in S102, loss calculation becomes feasible and effective. It ensures that even with a lighter student model structure (fewer attention heads), its attention behavior remains consistent with the teacher model through loss constraints, thus overcoming the knowledge transfer bottleneck caused by differences in the number of attention heads, improving the performance and generalization ability of the student model, and achieving effective knowledge transfer.

[0043] S104: Deploy the student model that meets the deployment conditions on the terminal device.

[0044] It is understood that the knowledge distillation method provided in this embodiment is an iterative training process, that is, S101~S103 above are iterative processes. Through multiple iterations, the model parameters of the student model are continuously optimized, so that while maintaining its lightweight nature, its attention mechanism and performance gradually approach those of the teacher model. This iterative optimization mechanism ensures that the student model can gradually absorb and integrate the multi-head attention knowledge of the teacher model, overcoming the initial alignment difficulties caused by the difference in the number of heads.

[0045] In the iteration process, it is determined whether the student model of the current iteration meets the deployment condition, which is a comprehensive performance index for evaluating whether the student model has reached a deployable state. The deployment condition in this embodiment can be that the number of iterations reaches a preset number, or the model accuracy reaches a preset accuracy, or the performance of the model on the validation set converges and meets the deployment requirements. Finally, the student model that meets the deployment condition is deployed on the terminal device, thereby completing the goal of successfully migrating the knowledge of the large teacher model and applying it to the actual scene with limited resources. In this embodiment, the attention knowledge in the teacher model that is difficult to directly transfer due to the complex structure (large number of heads) is aligned, condensed, and iteratively learned, and is completely migrated to the lightweight student model, thereby fundamentally solving the technical problems of the related art that the attention graph is difficult to directly align and the knowledge transfer effect is limited.

[0046] This embodiment enables the small-scale student model after distillation to inherit the core capabilities of the teacher model and successfully deploy on terminal devices (such as mobile phones, embedded devices, or Internet of Things devices) with limited computing power, storage space, and power consumption, so that the lightweight model can also have similar attention modeling capabilities and performance on the terminal device as the large teacher model.

[0047] In an example embodiment, the first set of attention graphs is processed to obtain a third set of attention graphs that matches the second set of attention graphs in quantity, including: determining a linear mapping matrix, the dimension of the linear mapping matrix is , is the first number, is the second number; the first set of attention graphs containing the first number of attention graphs is linearly transformed using the linear mapping matrix to map the third set of attention graphs containing the second number of attention graphs.

[0048] This embodiment requires the student network to learn knowledge from all the attention graphs of the teacher network. By representing the attention graphs of the student network as a linear combination of the attention graphs of the teacher network, the knowledge transfer from the teacher network to the student network is realized. The linear mapping matrix is a trainable parameter matrix, which functions to fuse and compress the rich attention information of multiple heads of the teacher model into a representation that matches the number of heads of the student model, and through the linear conversion matrix, the attention graphs of the student network are as close as possible to the knowledge patterns contained in the teacher network.

[0049] Assuming that the conversion matrix is W, the attention graphs of the teacher network are respectively generated into a new set of attention graphs, i.e., the third set of attention graphs If That is, the attention map of the student network can be expressed by the linear combination of the attention maps of the teacher network. The training objective of this embodiment is to minimize the distance between and .

[0050] This embodiment provides a lightweight and efficient way to realize knowledge transfer across different head structures, which not only preserves the multi-head attention information of the teacher model, but also avoids complex matching or heuristic rules, so that the student model can more accurately mimic the attention distribution of the teacher, thereby significantly improving the efficiency and effectiveness of knowledge transfer.

[0051] In this embodiment, the i-th attention map in the third attention map set is expressed as the linear combination of the attention maps of the teacher network , that is , where , , and written in matrix form as .

[0052] where the matrix is the attention map matrix of the teacher network, the matrix is the attention map matrix to be calculated, and the matrix is the coefficient matrix. As an optional embodiment, the coefficient matrix can be learned by a neural network backpropagation algorithm.

[0053] In an exemplary embodiment, the knowledge distillation method further comprises: constructing a loss function, the loss function comprising a distillation loss and a task loss, the distillation loss comprising a regularization term and a contrastive loss term determined based on the difference between the third attention map set and the second attention map set.

[0054] In this embodiment, by constructing a composite loss function, the learning process of the student model is guided to ensure that it can learn the knowledge of the teacher model and complete the specific task. Among them, the distillation loss is mainly responsible for aligning the intermediate representations (such as attention maps) of the teacher and student models, and the task loss (such as cross-entropy loss) ensures the performance of the student model on the final task (such as classification). The regularization term introduced in the distillation loss of this embodiment helps to prevent overfitting and improve the generalization ability of the model; and the contrastive loss term narrows the distance between the teacher model and the student model in the attention pattern through a contrastive learning method, while pushing away the mismatched patterns, thereby more finely promoting knowledge transfer.

[0055] The design of this composite loss function, through multi-angle and multi-target constraints, enables the student model to more comprehensively and stably absorb the knowledge of the teacher model, especially in scenarios where the teacher and student model structures differ greatly (such as different attention head numbers), effectively improving the efficiency and final performance of distillation.

[0056] The calculation process of the regularization term and the contrastive loss term in the distillation loss of a certain layer is described below.

[0057] In an example embodiment, the contrastive loss term is determined based on the difference between the third set of attention maps and the second set of attention maps, including: converting each row in the student network attention map matrix corresponding to the third set of attention maps into an independent probability distribution to obtain a first matrix, the sum of the elements in each row of the first matrix being 1; converting each row in the student network attention map matrix corresponding to the second set of attention maps into an independent probability distribution to obtain a second matrix, the sum of the elements in each row of the second matrix being 1; and taking the structural attention similarity between the first matrix and the second matrix as the contrastive loss term, wherein the structural attention similarity is obtained by weighted combination of element-level similarity, row-level distribution similarity and column-level distribution similarity.

[0058] In this embodiment, the contrastive loss term is defined as .

[0059] In the above formula represents that the attention matrix is first subjected to Softmax operation. The following benefits can be brought about by first performing Softmax operation: first, the probability distribution is normalized, the attention weight in the Transformer is essentially a probability distribution calculated by Softmax (each row represents the attention allocation to other positions), and first performing Softmax on the matrix can ensure that the sum of the elements in each row is 1, which conforms to the probability interpretation of the attention mechanism; second, the numerical range is unified, Softmax will map the elements to the [0, 1] interval, avoiding the interference of the difference in the numerical range of the original weight on the similarity calculation; third, the distribution characteristics are enhanced, Softmax will amplify significant differences (such as peak attention), making the distribution pattern of the row / column more prominent, which facilitates the capture of structural similarity.

[0060] In the above formula is the structural attention similarity, which is optimized for the characteristics of the Transformer attention map. This measure not only considers the similarity at the element level, but also takes into account the row and column structural information of the attention map. The structural attention similarity considers the similarity of individual attention weights, the similarity at the row level (the attention distribution to a certain position), and the similarity at the column level (the degree to which a certain position is concerned by other positions), and obtains the final similarity by weighted combination of these information.

[0061] Specifically, the function is defined as follows for two attention matrices A and B, both of which have a shape of .

[0062] First, the element-level similarity is calculated, Using improved cosine similarity, a small amount of... Avoid dividing by zero.

[0063] Calculate row-level similarity (compare the distribution of each row). .

[0064] Calculate column-level similarity (compare the distribution of each column). .

[0065] Finally, the structured attention similarity is calculated. .

[0066] In other words .

[0067] in, , is an adjustable weight parameter, which can be set to by default. , Element-level similarity For row-level similarity, For column-level similarity, For the attention matrix A, the first... Line number Column elements, For the attention matrix B, the first... Line number The elements of the column. The characteristics of this metric include: the value range is [0,1], the closer the value is to 1, the higher the similarity; it focuses on both the matching of individual attention weights and the similarity of the overall distribution pattern; it is more sensitive to significant peaks (important associations) in the attention graph; it can be adapted to the needs of different scenarios by adjusting the weight parameters; this metric is suitable for comparing the attention patterns between different layers of the Transformer model or different tasks; and it can capture the structural similarity that traditional element-level metrics may ignore.

[0068] Add an L1 regularization term to the loss function ,in It is a regularization parameter, in this embodiment It can be equal to 1.0. L1 regularization will make... The values ​​tend to be sparse, meaning that most elements tend to be 0. This helps to make it easier for each row to have only one large value (close to 1) after softmax, while the rest of the values ​​are close to 0, thus making it closer to a one-hot vector.

[0069] Based on this, the distillation loss established in this embodiment is ,in, During training, constraint terms and optimization algorithms are used to adjust the values ​​of each element in the linear mapping matrix, thereby enabling the student network to effectively learn and imitate the teacher network's attention map.

[0070] In an example embodiment, the student model and the teacher model each include a plurality of network layers; the knowledge distillation method further includes: for each network layer, calculating a contrastive loss term of the network layer based on a difference between the third set of attention maps and the second set of attention maps of the network layer, calculating a layer distillation loss of the network layer using the contrastive loss term and a regularization term of the network layer, and calculating a distillation loss based on all layer distillation losses. This way of calculating and aggregating losses layer by layer takes into full consideration the differences between the teacher and student models at each level of feature abstraction, and can achieve more refined and in-depth knowledge transfer, effectively alleviating the problem of low overall knowledge transfer efficiency caused by mismatch between model depth and layer head number.

[0071] Specifically, the distillation loss is calculated according to a second relationship ; wherein, is the distillation loss, is the contrastive loss term of the lth layer, is the regularization term of the lth layer, is the first coefficient.

[0072] In this embodiment, the teacher network and the student network are both L layers, and the overall distillation loss expression under the condition that the teacher network and the student network have different head numbers at each layer is defined. The contrastive loss term of the lth layer is defined as , and the regularization term of the lth layer is defined as . Among them, the contrastive loss term is mainly used to measure the difference between the attention maps of the student model and the teacher model at a certain layer, and the regularization term is used to constrain the parameters of the student model to prevent overfitting and improve the generalization ability; the first coefficient is used to balance the influence degree of different layer losses on the overall target.

[0073] The distillation loss function is defined as .

[0074] This distillation loss function calculates and weightedly sums layer by layer, ensuring that knowledge from each layer of the teacher can be effectively transferred to the corresponding layer of the student, and the problem of attention map misalignment and knowledge loss caused by the difference in layer depth and head number at each layer is solved to the greatest extent, significantly improving the integrity of distillation and the performance of the final model.

[0075] In an example embodiment, it further includes: obtaining a prediction result of the student model on the task sample set; comparing the prediction result with the true label of the task sample set, and calculating a task loss according to the comparison result.

[0076] In this embodiment, the task loss is used to ensure that the student model does not deviate from the original downstream task target while absorbing the knowledge of the teacher model.

[0077] In this embodiment, knowledge distillation is performed on a Transformer-based large language model. The task is to predict the next word given a sequence of tokens. Therefore, the task loss function can be the cross-entropy loss function. When the parameters of both the student and teacher networks are optimized during training, they both use the task loss function. If only the student network's parameters are optimized during training, then the student network uses the task loss function. In this embodiment, only the parameters of the student network are optimized during training. This design allows the student model to focus on learning effective attention patterns and other intermediate representations from the teacher model, while maintaining its performance on the final language modeling task through task loss, thereby achieving a balance between knowledge transfer and task performance.

[0078] Suppose there is a containing Given a sequence of n words, for each position p, the model needs to predict the probability distribution of the next word belonging to one of the M words in the vocabulary. Let... It is an M-dimensional one-hot vector representing the category corresponding to the true word at position p. If the model predicts an M-dimensional probability distribution vector, then the cross-entropy loss function can be expressed as: .

[0079] in, It is a vector The q-th element, It is a vector The q-th element. This formula calculates the average difference between the probability distribution predicted by the model and the true label. By minimizing this loss function, the model parameters are adjusted so that the model can better predict the next word, thereby improving the performance of the language model. Introducing task loss and combining it with distillation loss constitutes the overall training objective of the student model. This not only promotes the effective transfer of knowledge within the teacher model (such as the attention mechanism) but also ensures the accuracy and reliability of the student model's output on specific tasks, thus comprehensively improving the overall capabilities of the distilled model.

[0080] In an exemplary embodiment, the loss function is ;in, For distillation losses, For mission losses, These are constant coefficients. This is the loss function.

[0081] In this embodiment, based on the distillation loss between models and the task-related loss described above, the present invention defines... , is a distillation loss function and a task-related loss function as independent variables. In the present application, a linear function is adopted, such as .

[0082] wherein, is a constant coefficient for balancing and contribution, can take a value of 0.5, which functions to adjust the weight ratio between teacher knowledge transfer and student independent task performance. By adjusting better accuracy can be achieved, for example, increasing will prompt the student model to focus more on imitating the internal representation (such as attention map) of the teacher, while decreasing will make the student model focus more on optimizing the output result of the final task. This linear weighting fusion method is computationally efficient and easy to optimize, and can effectively solve the model bias problem that may be caused by simply relying on task loss or distillation loss, ensuring that the student model successfully absorbs the knowledge of the teacher model while maintaining good task performance, thereby comprehensively improving the distillation effect and model generalization ability.

[0083] In an example embodiment, the model parameters of the student model are updated using the loss function, including: calculating the gradient of the loss function with respect to the model parameters of the student model; clipping the gradient to limit the size of the gradient within a threshold range; and updating the model parameters of the student model based on the clipped gradient.

[0084] In this embodiment, the size of the gradient is limited within a threshold range, effectively preventing the problem of gradient explosion during training and improving training stability. By optimizing the algorithm (such as Adam) using the processed gradient to iteratively update the parameters, the total loss function is continuously reduced, thereby optimizing both task performance and knowledge transfer effect.

[0085] In an example embodiment, it further includes: obtaining a data set, and randomly sorting the samples in the data set; and obtaining a preset batch of training samples from the randomly sorted data set.

[0086] In this embodiment, this randomization process can avoid the bias caused by data order and ensure the unbiasedness and generalization ability of model training.

[0087] Specifically, the source model and the target model in the embodiment are large language models based on the Transformer. The data in the training data set is text data. The text data is input into the large language model, thereby generating the intermediate state value (including the attention graph) in the model running and the final output text. In the large language model for question answering, the input is a question and the output is the answer to the question. The embodiment collects a large-scale text data set, which can contain hundreds of millions of sentences, of which the proportion of Chinese sentences is , the proportion of English sentences is , and the proportion of other languages is . Such a multi-language data configuration helps to improve the generalization ability and cross-language understanding ability of the model. In the process of model training, text data is sampled from the large-scale text data set in a random sampling manner. The above language proportion is an example value, and other numerical proportions do not affect the effect of the embodiment.

[0088] In order to avoid the order of samples in the training data set D affecting the performance of the model, all samples in the training data set D are randomly arranged at the beginning of the model training. In each round of training process, the model training algorithm reads a batch of samples, for example, 1024 samples. Such batch processing can not only ensure the calculation efficiency, but also provide sufficient statistical information for gradient update.

[0089] Input the source model M, for example: Llama-3-8B. Read data from the training data set D, assuming that the data has been preprocessed as required, the format meets the model input requirements, the initial training period , the total training period , the random gradient descent algorithm adopts the Adam algorithm, and the related parameters are: the learning rate is , the momentum coefficient , , , the learning rate strategy is the cosine strategy, which can use a larger learning rate to quickly converge in the early stage of training, and use a smaller learning rate for fine tuning in the later stage, and the number of samples in a small batch is 1024.

[0090] The training output is the target model M', wherein the target model M' and the source model M have basically the same structure, and only the number of heads of the multi-head attention mechanism is different. The number of heads of the multi-head attention of each layer of the target model M' is not greater than that of the source model. Such a structural design ensures that the student model maintains the core architecture of the teacher model while realizing the compression of the number of parameters and the improvement of the calculation efficiency.

[0091] The training loss function is constructed The composite loss function effectively balances the knowledge distillation loss and the task-specific loss, ensuring that the student model can learn the internal representation of the teacher model and maintain good downstream task performance. The model training steps are as follows.

[0092] S1: Fix all parameter values of the source model M; construct the target model according to the network structure of the source model M; randomly initialize the target model The elements of the weight matrix in the target model

[0093] S2: When , perform the following operations: a) ; b) randomly shuffle the samples in the data set D; c) select a batch B of training samples from the data set D; d) calculate the loss value according to the above training samples and parameter settings, and then update the model parameters of the target model using the stochastic gradient descent algorithm (Adam algorithm). Before updating the parameters, clip the gradient, set the gradient threshold to 5 (which can be adjusted according to actual conditions), and gradient clipping can effectively prevent training instability and divergence problems; e) repeat steps c) and d) until all samples in the data set D are used. F) End the current training period.

[0094] S3: End training and output the target model . Save the target model to the specified file path for subsequent use. Model saving ensures the persistence of training results and the convenience of subsequent deployment.

[0095] In summary, the present application realizes effective knowledge transfer between the teacher and student networks through a linear combination mechanism. The coefficient matrix constraint and softmax combination simplify the model generation rules, reduce complexity, avoid overfitting, improve generalization ability, and ensure that the student network accurately learns the teacher network attention map knowledge, thereby effectively improving the performance of the student network during knowledge distillation. The present application can be applied to model compression and acceleration in the fields of computer vision, natural language processing, or speech recognition, enabling efficient deployment of high-performance models on resource-constrained devices.

[0096] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a general hardware platform as necessary, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Including but not limited to Application-Specific Integrated Circuit (ASIC), Graphics Processing Unit (GPU) or Field-Programmable Gate Array (FPGA) to support efficient model training and inference.

[0097] Please refer to Figure 3 The embodiments of the present application also provide a knowledge distillation device, comprising: a determination module 11 configured to determine a first set of attention maps generated by a teacher model based on training samples, and a second set of attention maps generated by a student model based on the training samples; the multi-head attention mechanism of the teacher model has a first number of attention heads, and the multi-head attention mechanism of the student model has a second number of attention heads, the first number being greater than or equal to the second number; a processing module 12 configured to process the first set of attention maps to obtain a third set of attention maps matching the second set of attention maps in number; an updating module 13 configured to update model parameters of the student model using a loss function; wherein the loss function is established based on the difference between the third set of attention maps and the second set of attention maps; and a deployment module 14 configured to deploy the student model that meets a deployment condition to a terminal device.

[0098] In an exemplary embodiment, processing the first set of attention maps to obtain a third set of attention maps matching the second set of attention maps in number comprises: determining a linear mapping matrix, the dimension of the linear mapping matrix being , the first number is the second number is; and performing linear transformation on the first set of attention maps using the linear mapping matrix, so as to map the first set of attention maps containing the first number of attention maps into the third set of attention maps containing the second number of attention maps.

[0099] In an exemplary embodiment, performing linear transformation on the first set of attention maps using the linear mapping matrix comprises: performing linear transformation on the first set of attention maps using a first relationship, the first relationship being , , wherein, the i-th attention map in the third set of attention maps is the element in the i-th row and the j-th column of the linear mapping matrix is It is the j-th attention map in the first set of attention maps.

[0100] In one exemplary embodiment, the knowledge distillation apparatus further includes: a construction module for constructing a loss function, the loss function including a distillation loss and a task loss, the distillation loss including a regularization term and a contrastive loss term determined based on the difference between a third attention map set and a second attention map set.

[0101] In an exemplary embodiment, both the student model and the teacher model include multiple network layers; the knowledge distillation apparatus further includes: a first calculation module, configured to calculate a contrastive loss term for each network layer based on the difference between the third attention map set and the second attention map set of the network layer, and calculate the layer distillation loss of the network layer using the contrastive loss term and the regularization term of the network layer; and a second calculation module, configured to calculate the distillation loss based on the distillation loss of all layers.

[0102] In one exemplary embodiment, calculating distillation loss based on the distillation loss of all layers includes: calculating the distillation loss according to a second relationship, the second relationship being... ;in, For distillation losses, For the contrast loss term of the l-th layer, For the regularization term of the l-th layer, It is the first coefficient.

[0103] In an exemplary embodiment, determining a contrast loss term based on the difference between a third attention map set and a second attention map set includes: converting each row of the student network attention map matrix corresponding to the third attention map set into an independent probability distribution to obtain a first matrix, wherein the sum of the elements in each row of the first matrix is ​​1; converting each row of the student network attention map matrix corresponding to the second attention map set into an independent probability distribution to obtain a second matrix, wherein the sum of the elements in each row of the second matrix is ​​1; and using the structured attention similarity between the first matrix and the second matrix as the contrast loss term, wherein the structured attention similarity is obtained by weighted combination of element-level similarity, row-level distribution similarity, and column-level distribution similarity.

[0104] In one exemplary embodiment, the method further includes: obtaining the prediction results of the student model for the task sample set; comparing the prediction results with the true labels of the task sample set; and calculating the task loss based on the comparison results.

[0105] In an exemplary embodiment, the loss function is ;in, For distillation losses, For mission losses, These are constant coefficients. This is the loss function.

[0106] In an example embodiment, the model parameters of the student model are updated using the loss function, including: calculating the gradient of the loss function with respect to the model parameters of the student model; clipping the gradient, limiting the size of the gradient within a threshold range; updating the model parameters of the student model based on the clipped gradient.

[0107] In an example embodiment, further comprising: obtaining a data set, randomly sorting the samples in the data set; obtaining a preset batch of training samples from the randomly sorted data set.

[0108] The features of the embodiments corresponding to the knowledge distillation device can be referred to the related descriptions of the embodiments corresponding to the knowledge distillation method, which will not be repeated here.

[0109] Please refer to Figure 4 The present application also provides an image retrieval method, including: S201: obtaining a query text; S202: inputting the query text into a text encoding module to obtain a text feature embedding vector; the text encoding module is a student model obtained based on any one of the knowledge distillation methods above; S203: calculating the similarity between the text feature embedding vector and each image feature embedding vector in an image feature database; the image feature database includes a plurality of image feature embedding vectors, each image feature embedding vector is obtained by inputting a corresponding image in an image database into an image encoding model, and the image encoding model is a student model obtained using any one of the knowledge distillation methods above; S204: based on the comparison result of the calculated similarity and a preset similarity threshold, retrieving all images with a similarity greater than the preset similarity threshold from the image database as the retrieval result and returning.

[0110] The image retrieval method of the present embodiment retrieves one or more images consistent with the content description of a text query from an image database. For example, when the input query is "a cat", the query system will return images related to "cats", such as an image of an orange cat napping in the sun, an image of a black and white cat playing on the grass, etc. In addition, when the input query is "beautiful scenery", images of majestic mountains and blue sky, images of beautiful sunset scenery, etc. can be returned. Similarly, when the input query is "city night scene", images of flashing city lights and high-rise buildings, images of night scenes on busy streets, etc. can be returned. Such an image retrieval system can help users quickly find images consistent with their description through simple text queries, providing a convenient image search and browsing experience.

[0111] First, a pre-trained text and image encoder is obtained, which is a CLIP (Contrastive Language-Image Pre-training) model trained on a data set of images and texts. A distilled text and image encoder is obtained, which is obtained by using any one of the knowledge distillation method embodiments described above to implement the image encoder and the text encoder in the CLIP model, to obtain a CLIP model after knowledge distillation, including an image encoding module and a distilled text encoding module , as shown in Figure 5 .

[0112] An image database Db to be queried is created, which is an image database to be queried and includes a large number of unlabeled images. The content of these images is unknown, i.e., the object class in the image is unknown. The image database Db is large in size and covers images in multiple fields and scenarios. Each image in the image database Db is an independent instance and has not been classified or labeled in advance. For the convenience of subsequent description, the total number of images in the image database Db is defined as N.

[0113] An image feature database is created according to the image database Db to be queried. Each image in the image database Db to be queried is input into the image encoding module to obtain an output feature embedding vector of each image . In , the output feature embedding vector is taken as the feature of the image . Therefore, the image feature database includes N image features, and the set formed by the N image features is , and the image features correspond one-to-one to the images.

[0114] A query text T is input, which is a natural language text. The language type of the input query text should be consistent with the language type of the text in the training data set of the jointly trained image and text model. In the training data set of the jointly trained image and text model, the image and the corresponding text form an image-text pair, and the language type of the text determines the language type of the input text that can be processed by the text encoding module. In other words, if the text in the training data set of the jointly trained image and text model is Chinese, the input query text should also be Chinese; if the text in the training data set is English, the input query text should also be English; if the text in the training data set covers multiple languages, the input query text can contain multiple languages. Ensuring that the language type of the input query text is consistent with the language type of the text in the training data set ensures the accuracy and reliability of image retrieval.

[0115] The features of the input query text T are obtained, and the query text T is input into the text encoding module. The obtained feature embedding vector is , .exist In the middle, take Features of the input query text.

[0116] Calculate the features of the input query text T and the image feature database. Image feature similarity, image feature database The set consists of N image features. The characteristics of the input query text T are: Therefore, the similarity between the features of the input query text T and each feature in the image feature data is: ,in, u and v represent two column vectors, The transpose of column vector u. Represents the magnitude of the vector.

[0117] The image query system in this embodiment obtains image indexes by finding one or more images from the image database that correspond to the input text. To this end, the present invention sets a similarity threshold. In this embodiment .

[0118] Similarity between features of input query text T and image features for Therefore, based on the similarity threshold... The resulting image index set is , represents returning all A set of indices.

[0119] Returns a set of retrieved images, based on the image index set. It retrieves the corresponding image from the image database Db and returns it.

[0120] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps of any of the knowledge distillation method embodiments described above and / or the steps of the image retrieval method described above.

[0121] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the knowledge distillation method embodiments and / or the steps in the image retrieval method when running.

[0122] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0123] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the knowledge distillation method embodiments and / or the steps in the image retrieval method.

[0124] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the knowledge distillation method embodiments and / or the steps in the image retrieval method.

[0125] The skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0126] The above describes in detail the image retrieval method provided by the present application. The principles and implementation manners of the present application are described by applying specific examples, and the above description of the examples is only used to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. An image retrieval method, characterized in that, include: Get the query text; The query text is input into the text encoding module to obtain the text feature embedding vector; The text encoding module is a student model obtained based on the knowledge distillation method; Calculate the similarity between the text feature embedding vector and each image feature embedding vector in the image feature database; the image feature database includes multiple image feature embedding vectors, and each image feature embedding vector is obtained by inputting the corresponding image in the image database into an image coding model, the image coding model being a student model obtained using the knowledge distillation method; Based on the comparison result between the calculated similarity and the preset similarity threshold, all images with a similarity greater than the preset similarity threshold are retrieved from the image database as retrieval results and returned. The knowledge distillation method includes: A first set of attention maps generated by the teacher model based on the training samples is determined, and a second set of attention maps generated by the student model based on the training samples is determined; the multi-head attention mechanism of the teacher model has a first number of attention heads, and the multi-head attention mechanism of the student model has a second number of attention heads, wherein the first number is greater than or equal to the second number; The first attention map set is processed to obtain a third attention map set that matches the second attention map set in terms of quantity; The model parameters of the student model are updated using a loss function until the student model meets the deployment conditions; wherein, the loss function is established based on the difference between the third attention map set and the second attention map set; The first attention map set is processed to obtain a third attention map set that matches the second attention map set in number, including: Determine the linear mapping matrix, the dimension of which is... , For the first quantity, This is the second quantity; The first attention map set is linearly transformed using the linear mapping matrix, and the first attention map set containing the first number of attention maps is mapped to a third attention map set containing the second number of attention maps.

2. The image retrieval method according to claim 1, characterized in that, Performing a linear transformation on the first attention map set using the linear mapping matrix includes: A linear transformation is performed on the first attention map set using the first relation, which is: , , ,in, For the i-th attention map in the third attention map set, Let be the element in the i-th row and j-th column of the linear mapping matrix. It is the j-th attention map in the first set of attention maps.

3. The image retrieval method according to claim 1, characterized in that, The image retrieval method further includes: Construct the loss function, which includes distillation loss and task loss. The distillation loss includes a regularization term and a contrastive loss term determined based on the difference between the third attention map set and the second attention map set.

4. The image retrieval method according to claim 3, characterized in that, Both the student model and the teacher model include multiple network layers; The image retrieval method further includes: For each network layer, a contrastive loss term for the network layer is calculated based on the difference between the third attention map set and the second attention map set of the network layer, and the layer distillation loss of the network layer is calculated using the contrastive loss term and the regularization term of the network layer. Calculate distillation loss based on distillation loss from all the aforementioned layers; The calculation of distillation loss based on all the aforementioned layer distillation losses includes: Distillation loss is calculated according to the second equation, which is: ; Where L is the total number of network layers, and l is the label of the network layer. For distillation losses, For the contrast loss term of the l-th layer, For the regularization term of the l-th layer, It is the first coefficient.

5. The image retrieval method according to claim 3, characterized in that, The contrast loss term is determined based on the difference between the third attention map set and the second attention map set, including: Each row of the student network attention graph matrix corresponding to the third attention graph set is converted into an independent probability distribution to obtain a first matrix, in which the sum of the elements in each row is 1; Each row of the student network attention graph matrix corresponding to the second attention graph set is converted into an independent probability distribution to obtain a second matrix, in which the sum of the elements in each row is 1; The structured attention similarity between the first matrix and the second matrix is ​​used as the contrast loss term, wherein the structured attention similarity is obtained by weighted combination of element-level similarity, row-level distribution similarity and column-level distribution similarity.

6. The image retrieval method according to claim 3, characterized in that, The image retrieval method further includes: Obtain the prediction results of the student model for the task sample set; The prediction results are compared with the true labels of the task sample set, and the task loss is calculated based on the comparison results. The loss function is: ;in, For the distillation loss, For the loss of the aforementioned task, These are constant coefficients. Let be the loss function.

7. The image retrieval method according to any one of claims 1-6, characterized in that, Updating the model parameters of the student model using the loss function includes: Calculate the gradient of the loss function with respect to the model parameters of the student model; The gradient is clipped to limit its magnitude to a threshold range; The model parameters of the student model are updated based on the clipped gradient.

8. The image retrieval method according to any one of claims 1-6, characterized in that, The image retrieval method further includes: Obtain a dataset and randomly sort the samples in the dataset; The training samples are obtained from a preset batch of randomly sorted data sets.

Images