Model distillation method and apparatus, electronic device, and storage medium

Abstract

The present application discloses a model distillation method and device, electronic equipment and storage medium. The method comprises: obtaining a sample image; obtaining a first image representation and a first text representation output by a teacher model based on the sample image, the teacher model comprising a single-flow model and a double-flow model, and the first image representation and the first text representation being the fusion results of the respective output results of the single-flow model and the double-flow model; obtaining a second image representation and a second text representation output by a student model based on the sample image, the student model being a double-flow model; determining a distillation loss function based on the first image representation and the first text representation, the second image representation and the second text representation; updating the parameters of the student model based on the distillation loss function, and taking the student model when the loss function converges as a target student model. The present application can effectively transmit the advantages of multiple teacher models to the student model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a model distillation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Model distillation is a technique that compresses a complex neural network into a small, fast network. By "distilling" knowledge from a large network into a smaller one, computational resources and time can be reduced while maintaining high accuracy. Current model distillation methods include knowledge distillation, adaptive temperature scaling, distribution matching, and multi-task learning methods. Among these, knowledge distillation is a method based on knowledge transfer between a teacher network and a student network. In this method, the large network is called the "teacher," and the small network is called the "student." The teacher guides the student's learning through its high-precision output. However, the performance of models obtained using current knowledge distillation methods still needs improvement. Summary of the Invention

[0003] This application proposes a model distillation method, apparatus, electronic equipment, and storage medium to improve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide a model distillation method applicable to electronic devices. The method includes: acquiring a sample image; acquiring a first image representation and a first text representation output by a teacher model based on the sample image, wherein the teacher model includes a single-stream model and a two-stream model, and the first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively; acquiring a second image representation and a second text representation output by a student model based on the sample image, wherein the student model is a two-stream model; determining a distillation loss function based on the first image representation, the first text representation, the second image representation, and the second text representation; updating the parameters of the student model based on the distillation loss function, and using the student model at which the loss function converges as the target student model.

[0005] Secondly, embodiments of this application provide a model distillation apparatus that can operate on an electronic device. The apparatus includes: a sample image acquisition module for acquiring sample images; a first output acquisition module for acquiring a first image representation and a first text representation output by a teacher model based on the sample images, wherein the teacher model includes a single-stream model and a two-stream model, and the first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively; a second output acquisition module for acquiring a second image representation and a second text representation output by a student model based on the sample images, wherein the student model is a two-stream model; a loss function construction module for determining a distillation loss function based on the first image representation, the first text representation, the second image representation, and the second text representation; and a model distillation module for updating the parameters of the student model based on the distillation loss function, and using the student model when the loss function converges as the target student model.

[0006] Thirdly, this application provides an electronic device including one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method described in the first aspect above.

[0007] Fourthly, this application provides a computer-readable storage medium storing program code, wherein the method described in the first aspect is executed when the program code is run.

[0008] This application provides a model distillation method, apparatus, electronic device, and storage medium. The method involves: acquiring sample images; acquiring a first image representation and a first text representation output by a teacher model based on the sample images, where the teacher model includes a single-stream model and a two-stream model, and the first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively; acquiring a second image representation and a second text representation output by a student model based on the sample images, where the student model is a two-stream model; determining a distillation loss function based on the first image representation, the first text representation, the second image representation, and the second text representation; updating the parameters of the student model based on the distillation loss function; and using the student model at which the loss function converges as the target student model. This method enables the distillation of a two-stream model using a multimodal teacher model, allowing for the full utilization of the advantages of multiple teacher models and more effectively transferring these advantages to the student model, thereby resulting in a target student model with higher accuracy. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of a model distillation method provided in one embodiment of this application is shown.

[0011] Figure 2 A flowchart of a model distillation method provided in another embodiment of this application is shown.

[0012] Figure 3 This diagram illustrates the computational distribution matrix of the student model for cross-modal alignment distillation in a multi-teacher model provided in an embodiment of this application.

[0013] Figure 4 A schematic diagram illustrating the working principle of the integration module of the teacher model provided in this application embodiment is shown.

[0014] Figure 5 A structural block diagram of a model distillation apparatus provided in an embodiment of this application is shown.

[0015] Figure 6 A structural block diagram of an electronic device provided in an embodiment of this application is shown.

[0016] Figure 7 An embodiment of the present application shows a storage unit for storing or carrying program code that implements the model distillation method according to an embodiment of the present application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0018] Model distillation is a model compression technique that can compress a complex neural network into a lightweight, small-sized network, resulting in a lightweight model. Model miniaturization plays a crucial role in the development of deep learning. Current model distillation methods are typically based on a framework of one teacher model and one student model. While some methods are based on a multi-classroom model and one student model, these do not utilize the strengths of different teacher model network structures, and the framework is difficult to reuse when the teacher model's network structure differs. Therefore, the performance of models obtained using current model distillation methods still needs improvement.

[0019] Through long-term research, the inventors discovered that a distillation loss function can be achieved by: acquiring sample images; obtaining a first image representation and a first text representation output by a teacher model based on the sample images, where the teacher model includes a single-stream model and a two-stream model, and the first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively; acquiring a second image representation and a second text representation output by a student model based on the sample images, where the student model is a two-stream model; determining a distillation loss function based on the first image representation, the first text representation, the second image representation, and the second text representation; updating the parameters of the student model based on the distillation loss function; and using the student model at which the loss function converges as the target student model. This allows for the use of a multimodal teacher model to distill a two-stream model, fully utilizing the advantages of multiple teacher models and more effectively transferring these advantages to the student model, thereby resulting in a target student model with higher accuracy.

[0020] Therefore, in order to improve the above problems, the inventors have proposed a model distillation method, electronic device and computer-readable storage medium that can use a multimodal teacher model to distill a two-stream model, so as to make full use of the advantages of multiple teacher models, so that the advantages of multiple teacher models can be more effectively transferred to the student model, thereby making the target student model obtained by distillation have higher accuracy.

[0021] To facilitate a better understanding of the solutions described in the embodiments of this application, the relevant terms involved in the embodiments of this application will be briefly explained below.

[0022] Knowledge distillation: Knowledge distillation is a commonly used model compression method that trains student models using teacher models.

[0023] KL divergence (also known as relative entropy or information divergence) measures the degree of difference between two probability distributions. A larger KL divergence indicates a greater degree of difference between the two distributions; a smaller KL divergence indicates a smaller degree of difference. If the two distributions are equal, the KL divergence is 0.

[0024] The model distillation method proposed in this application can be applied to scenarios such as image search (searching for one or more images that match a given image in a search engine), image retrieval (finding another image that matches a given image in a file storage device such as a photo album, or finding an image that matches a keyword), and image management (e.g., classifying images). By leveraging the advantages of each teacher model, the advantages of multiple teacher models are more effectively transferred to the student model, thereby enabling the distilled target student model to have a higher inference accuracy, thus improving the accuracy of image search, retrieval, or management.

[0025] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0026] Please see Figure 1 This document illustrates a flowchart of a model distillation method according to an embodiment of this application. This embodiment provides a model distillation method applicable to electronic devices. In one embodiment, the electronic device can be a server. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In this approach, the server can distill student models using multiple teacher models to obtain a target student model, and then send the target student model to a terminal device so that the user can use the target student model through the terminal device. In another embodiment, the electronic device can be a mobile phone, computer, tablet, or wearable electronic device; the specific type of electronic device is not limited. In this approach, the electronic device can directly distill student models using multiple teacher models to obtain a target student model, which the user can then directly use. The method includes:

[0027] Step S110: Obtain sample images.

[0028] In this embodiment, the sample image can be an image to be classified, an image to be searched, or an image to be searched for; the specific content of the sample image is not limited. Optionally, the sample image can be a single frame image.

[0029] Step S120: Obtain the first image representation and the first text representation obtained by the teacher model based on the sample image output. The teacher model includes a single-stream model and a two-stream model. The first image representation and the first text representation are the result of fusing the output results of the single-stream model and the two-stream model respectively.

[0030] The teacher model is a complex model with superior inference performance, while the student model is a simplified and low-complexity model. In this embodiment, the teacher model can include both a single-stream and a two-stream model, or it can include only a single-stream model. In the single-stream model, the text encoder and image encoder share the same encoder, while in the two-stream model, the text encoder and image encoder are two different encoders. This embodiment is illustrated using an example where the teacher model includes both a single-stream and a two-stream model.

[0031] In one implementation, a sample image is input into the teacher model, which performs image encoding and text encoding on it, outputting a first image representation and a first text representation. Both the first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively.

[0032] Specifically, the single-stream model in the teacher model performs image encoding and text encoding on the sample image, outputting a first encoded result (referring to the encoding that combines image recognition and text recognition results). Simultaneously, the two-stream model in the teacher model also performs image encoding and text encoding on the sample image, outputting a second encoded result (referring to the encoding of the image recognition result) and a third encoded result (referring to the encoding of the text recognition result). The teacher model then fuses the first, second, and third encoded results separately: fusing the first and second encoded results to obtain a first image representation; and fusing the first and third encoded results to obtain a first text representation. By fusing the multimodal inference results in the teacher model, the advantages of the multimodal teacher model can be fully utilized, thereby improving the inference ability of the teacher model.

[0033] The process by which the teacher model obtains the first image representation and the first text representation based on the sample images is the process by which the teacher model performs knowledge reasoning. The teacher model can implement this reasoning ability by leveraging artificial intelligence technology.

[0034] Step S130: Obtain the second image representation and the second text representation output by the student model based on the sample image, wherein the student model is a two-stream model.

[0035] In this embodiment, the student model can be a dual-stream model. Dual-stream models offer faster retrieval speeds, making them more suitable for deployment on terminals and allowing for more effective transfer of knowledge from the teacher model to the student model.

[0036] To facilitate knowledge distillation of the student model using a multimodal teacher model, sample images can be input into the student model, and a second image representation and a second text representation can be obtained from the student model's output based on the sample images. This allows for subsequent knowledge distillation of the student model by aligning its output features with those of the teacher model. Aligning the student model's output features with those of the teacher model ensures that the distribution of the student model's output more closely approximates the distribution of the teacher model's output.

[0037] In this application, image representation can be understood as the encoding result of the model performing image recognition on the sample image, and text identifier can be understood as the encoding result of the model performing text recognition on the sample image.

[0038] Step S140: Determine the distillation loss function based on the first image representation and the first text representation, the second image representation and the second text representation.

[0039] To more accurately determine the difference in reasoning ability between the student model and the teacher model, a distillation loss function can be constructed based on the outputs of the teacher model and the student model before using the teacher model to perform knowledge distillation on the student model.

[0040] As one implementation method, the distillation loss function can be determined based on the first image representation and the first text representation, as well as the second image representation and the second text representation. The specific process for determining the distillation loss function can be referred to the relevant descriptions in subsequent embodiments, and will not be repeated here. By utilizing the outputs of the teacher model and the student model to jointly determine the distillation loss function, the ability of the teacher model to reason about knowledge can be more effectively transferred to the student model during the knowledge distillation process, thereby improving the effectiveness of model distillation.

[0041] Step S150: Update the parameters of the student model based on the distillation loss function, and take the student model when the loss function converges as the target student model.

[0042] Once the distillation loss function is constructed, the parameters of the student model (representing parameters related to the student model's reasoning ability, the specific parameter types are not limited here) can be updated based on the distillation loss function, and the student model at which the loss function converges is taken as the target student model. In this embodiment, the reasoning ability of the target student model is improved compared to the reasoning ability of the student model.

[0043] The model distillation method provided in this embodiment acquires sample images; obtains a first image representation and a first text representation output by a teacher model based on the sample images, wherein the teacher model includes a single-stream model and a two-stream model, and the first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively; obtains a second image representation and a second text representation output by a student model based on the sample images, wherein the student model is a two-stream model; determines a distillation loss function based on the first image representation, the first text representation, the second image representation, and the second text representation; updates the parameters of the student model based on the distillation loss function, and uses the student model at which the loss function converges as the target student model. Thus, this method enables the distillation of a two-stream model using a multimodal teacher model, allowing full utilization of the advantages of multiple teacher models and more effectively transferring these advantages to the student model, thereby resulting in a higher accuracy target student model obtained through distillation.

[0044] Please see Figure 2 This document illustrates a flowchart of a model distillation method according to another embodiment of this application. This embodiment provides a model distillation method applicable to electronic devices. Building upon the descriptions in the foregoing embodiments, this embodiment will describe in detail the process of determining a distillation loss function based on a first image representation, a first text representation, a second image representation, and a second text representation. The method includes:

[0045] Step S210: Obtain the first KL divergence based on the second image representation and the second text representation.

[0046] As one implementation method, in order to measure the difference between the output of the student model and the target distribution, the first KL divergence can be obtained based on the second image representation and the second text representation output by the student model.

[0047] In this embodiment, to facilitate cross-modal alignment of the student model in a multi-teacher model, a general form for calculating the distribution matrix and KL divergence can be predefined. Please refer to [link to relevant documentation]. Figure 3 This diagram illustrates the computational distribution matrix of the student model for cross-modal aligned distillation in a multi-teacher model, as provided in an embodiment of this application. Figure 3 As shown, the teacher model's integration module (i.e. Figure 3 The output of Teachers Integration is the first image representation I. T (I stands for the first letter of the English word "Image") and the first text representation T T (T stands for the first letter of the English word "Text"), student model (i.e. Figure 3The output of Student in the image is the second image representation I. S Second text representation T S By integrating the output of the teacher model's integration module with the student model's output, we can obtain... Figure 3 The distribution matrix shown (i.e. Figure 3 (The matrix within the dashed rectangle shown).

[0048] in, These represent the distributions of the image-to-text similarity matrix, respectively. These represent the distribution of the text-to-image similarity matrix, respectively. This represents the distribution matrix obtained from the student model. This represents the distribution matrix obtained after fusing the multimodal teacher models. This represents the alignment operation that aligns the image representation output by the student model to the text representation output by the teacher model. This represents the alignment operation that aligns the text representation output by the student model to the image representation output by the teacher model.

[0049] As one implementation method, based on the second image representation I S Second text representation T S In obtaining the first KL divergence, it can be first based on the second image representation I. S Second text representation T S Obtain the first distribution matrix representing the image-text similarity calculated by the student model. and based on the second image representation I S Second text representation T S Obtain the second distribution matrix representing text-image similarity calculated by the student model. In this embodiment of the application, the image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image. Then, the first KL divergence is obtained based on the first distribution matrix and the second distribution matrix.

[0050] in, τ S Characterizes the temperature coefficient of the student model. Optionally, by... By swapping the input positions of the image representation and the text representation in the formula, we can obtain... The calculation formula, namely Similarly, for forms like The similarity matrix in the format can be calculated using the following formula: Correspondingly, for forms like The similarity matrix in the format can be calculated using the following formula:

[0051] In this embodiment, The similarity parameter can be calculated using the following formula:

[0052]

[0053] Where softmax represents the softmax function, I represents the normalized image representation, T represents the normalized text representation, and τ represents the temperature coefficient. The representation is that I and T are transformed into a dot product. By substituting formula (1) into the above formula for calculating the similarity matrix, the values ​​of each similarity matrix can be calculated.

[0054] Step S220: Obtain the second KL divergence based on the first image representation and the first text representation.

[0055] As one implementation method, based on the first image representation I T And the first text represents T T In obtaining the second KL divergence, we can first base it on the first image representation I. T And the first text represents T T Obtain the third distribution matrix representing image-text similarity calculated by the teacher model. and based on the first image representation I T And the first text represents T T Obtain the fourth distribution matrix representing text-image similarity calculated by the teacher model. Then, the second KL divergence is obtained based on the third and fourth distribution matrices.

[0056] The calculation principle of the third distribution matrix is ​​similar to that of the first distribution matrix, namely... The calculation principle of the fourth distribution matrix is ​​similar to that of the second distribution matrix, namely... I will not go into details here.

[0057] Step S230: Obtain the third KL divergence based on the first text representation and the second image representation.

[0058] As one implementation method, similar to the description above, based on the first text representation T T Second image representation I S In obtaining the third KL divergence, we can first represent the second image as I. S Align to the first text representation T T The alignment operation is performed, and the fifth distribution matrix representing the similarity between the aligned images and text is obtained. And obtain the sixth distribution matrix representing the similarity between text and images after alignment. Among them, the image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image; then, the third KL divergence is obtained based on the fifth distribution matrix and the sixth distribution matrix.

[0059] In this embodiment, the fifth distribution matrix The calculation formula is expressed as follows:

[0060]

[0061] Among them, since the second image output by the student model represents I S Aligned to the first text representation T output by the teacher model T During the process, the two matrices will be fused. In this implementation, the temperature coefficients of the teacher model and the student model will be integrated, that is, the average of the sum of the temperature coefficients of the teacher model and the student model (i.e., (τ)). T +τ S () / 2) is used as a new temperature coefficient in this calculation method to improve the accuracy of the fifth distribution matrix calculation.

[0062] Correspondingly, the sixth distribution matrix The calculation formula can be expressed as:

[0063]

[0064] The calculation principle of the fifth distribution matrix is ​​similar to that of the first distribution matrix, and the calculation principle of the sixth distribution matrix is ​​similar to that of the second distribution matrix. They will not be repeated here.

[0065] Step S240: Obtain the fourth KL divergence based on the second text representation and the first image representation.

[0066] As one implementation method, based on the second text representation T S And the first image represents I T In obtaining the fourth KL divergence, we can first analyze the second text representation T. S Align to the first image representation I T The alignment operation is performed, and the seventh distribution matrix representing the similarity between the aligned images and text is obtained. And obtain the eighth distribution matrix representing the similarity between text and images after alignment. Among them, the image-text similarity represents the similarity between the image and the text, and the text-image similarity represents the similarity between the text and the image; then, the fourth KL divergence is obtained based on the seventh distribution matrix and the eighth distribution matrix.

[0067] In this embodiment, the seventh distribution matrix The calculation formula is expressed as follows:

[0068]

[0069] Similar to the fifth distribution matrix, the seventh distribution matrix will also be the average of the sum of the temperature coefficients of the teacher model and the student model (i.e., (τ)). T +τ S () / 2) is used as a new temperature coefficient under this calculation method to improve the accuracy of the seventh distribution matrix calculation.

[0070] Correspondingly, the eighth distribution matrix The calculation formula can be expressed as:

[0071]

[0072] The calculation principle of the seventh distribution matrix is ​​similar to that of the first distribution matrix, and the calculation principle of the eighth distribution matrix is ​​similar to that of the second distribution matrix. They will not be repeated here.

[0073] Step S250: Obtain the target distribution distillation parameters based on the first KL divergence and the second KL divergence.

[0074] The target distribution distillation parameter can be understood as the distribution of multiple outputs of a multimodal teacher model (i.e., multiple teacher models consisting of single-stream and dual-stream models).

[0075] As one implementation method, the formula for calculating the target distribution distillation parameters can be expressed as:

[0076]

[0077] Among them, f MT This represents the KL divergence, which measures the distribution between the output and the target. Characterizing the first KL divergence, Characterizing the second KL divergence, L MT Characterize the distillation parameters of the target distribution.

[0078] Step S260: Obtain the target feature distillation parameters based on the third KL divergence and the fourth KL divergence.

[0079] The target feature distillation parameter can be understood as the feature obtained by weighting the multiple outputs of the multimodal teacher model (i.e., multiple teacher models consisting of single-stream and dual-stream models). Therefore, when distilling the student model, the student model and the target feature distillation parameter can be distilled together.

[0080] As one implementation method, the formula for calculating the target characteristic distillation parameters can be expressed as:

[0081]

[0082] Among them, f MT This represents the KL divergence, which measures the distribution between the output and the target. Characterizing the third KL divergence, Characterizing the fourth KL divergence, Distillation parameters that characterize the target features.

[0083] Step S270: Use the sum of the target distribution distillation parameters and the target feature distillation parameters as the distillation loss function.

[0084] Given the target distribution distillation parameters and the target feature distillation parameters, the distillation loss function can be determined using the following formula:

[0085]

[0086] in, Characterizes the distillation loss function. and The uniform representation aligns the features output by the student model with the features output by the teacher model, thereby allowing the distribution of the student model's output results to better approximate the distribution of the teacher model's output results.

[0087] As one implementation method, the sum of the target distribution distillation parameters and the target feature distillation parameters can be used as the value of the distillation loss function to obtain the distillation loss function.

[0088] In this embodiment, to more accurately calculate the differences between the outputs of the student model and the teacher model, and to optimize the model distillation process, for Figure 3 The integration module of the teacher model shown can also fuse the outputs of multiple modal teacher models through an aggregation module before the integration module outputs the first image representation and the first text representation, so as to better utilize the features of the multi-teacher model.

[0089] The teacher model includes both single-stream and two-stream models. In the case of a two-stream teacher model, it is assumed that the set of text-image pairs given the input (including inputs to both the single-stream and two-stream models of the teacher model) consists of n text-image pairs. Assume that the image representation and text representation output by the two-stream model (referring to the teacher model) are I and I, respectively. DS ,T DS Therefore, the similarity matrix of the distribution can be expressed as Then, for each similarity matrix, the top k similarity score matrix is ​​calculated. That is, for each image / text, the k most similar texts / images are calculated. The calculation formulas are as follows:

[0090]

[0091]

[0092]

[0093]

[0094] Where, τ DS P represents the temperature coefficient of the two-flow model. i2t This represents the index of each image and its top k most similar texts, while P... t2i This represents each text and its k most similar images.

[0095] Furthermore, the top k scores can be recalculated using a single-stream model. As one implementation, assume the corresponding graph-to-text and text-to-graph score matrices output by the two-stream model (referring to the teacher model) are represented as follows: Their respective calculation methods are as follows:

[0096]

[0097]

[0098] Among them, f SS This represents the process of obtaining similarity scores for text pairs using a single-stream model, (i l ,t m () represents any text pair contained in the top k, where the subscripts l and m represent the image and text subscripts, respectively. It's important to note that in the score matrix... In the middle, it can be used only for These elements are used for calculation, so only these fractions can be used.

[0099] Therefore, based on the above description, the KL divergence between two distribution matrices can be calculated using the following specified formula, which includes:

[0100]

[0101] Among them, the The similarity matrix between the representation graph and the text output by the two-stream model representing the teacher model, the... The similarity matrix between text and image representations output by the dual-stream model representing the teacher model, wherein P i2tThe indexes of the top k most similar texts to each image in the image-to-text similarity matrix, P t2i The similarity matrix between text and image represents the k images most similar to each text, where σ represents the normalization norm. The matrix representing the similarity between images and text, the This is a matrix representing the similarity between text and images.

[0102] Optionally, the The distribution matrix can be characterized by the first distribution matrix, the third distribution matrix, the fifth distribution matrix, or the seventh distribution matrix. It can represent the second distribution matrix, the fourth distribution matrix, the sixth distribution matrix, or the eighth distribution matrix.

[0103] As one implementation method, for the normalization norm σ, when a matrix D is given * When the time is right, its normalization formula is expressed as:

[0104]

[0105] In a specific application scenario, please refer to Figure 4 This diagram illustrates the working principle of the integrated module of the teacher model provided in an embodiment of this application. Figure 4 As shown, when given a text-image pair (i l ,t m ), l, m∈[1,2,...n], assuming the image representation and text representation output by the two-stream model of the teacher model are respectively Assumption This represents the text-image fusion features output by the single-stream model of the teacher model. For example... Figure 4 As shown, the features output by multiple teacher models first pass through functions g with different functionalities. * Mapped to another vector, in this embodiment, g 1 / 2 It can be represented as follows:

[0106]

[0107] Among them, f LP G represents a linear (or nonlinear) mapping. 1 / 2 It can also be responsible for filtering features from different teacher models. Based on the aforementioned description, since the distributions of the two-stream and single-stream models are fused only on the top k text pairs, features from different teacher models can be fused on the top k text pairs. Therefore, the output of the teacher model's Integration module (…) Figure 4 (output in) The calculation method can be expressed as follows:

[0108]

[0109]

[0110] Here, α is a learnable parameter, and norm represents L2 normalization.

[0111] for Figure 4 The output of the integration module of the teacher model shown can be combined with... Figure 4 The outputs of the student models shown on the right are fused. The principle and calculation process of the fusion are similar to those described above, and will not be repeated here.

[0112] The model distillation method provided in this embodiment enables the distillation of a two-stream model using a multimodal teacher model. This allows for the full utilization of the advantages of multiple teacher models, enabling the advantages of multiple teacher models to be more effectively transferred to the student model, thereby resulting in a higher accuracy of the target student model obtained through distillation.

[0113] Meanwhile, by using a multi-teacher model to distill the two-stream model, the advantages of the two-stream model in terms of offline deployment of edge indexes are fully utilized, and the advantages of multiple teacher models are fully taken into account, which can significantly improve the accuracy of the student model obtained by distillation.

[0114] Please see Figure 5 This is a structural block diagram of a model distillation apparatus provided in an embodiment of this application. This embodiment provides a model distillation apparatus 300, which can operate on an electronic device. The apparatus 300 includes a sample image acquisition module 310, a first output acquisition module 320, a second output acquisition module 330, a loss function construction module 340, and a model distillation module 350.

[0115] The sample image acquisition module 310 is used to acquire sample images.

[0116] The first output acquisition module 320 is used to acquire the first image representation and the first text representation obtained by the teacher model based on the sample image. The teacher model includes a single-stream model and a two-stream model. The first image representation and the first text representation are the result of fusing the output results of the single-stream model and the two-stream model respectively.

[0117] The second output acquisition module 330 is used to acquire the second image representation and the second text representation output by the student model based on the sample image, wherein the student model is a two-stream model.

[0118] The loss function construction module 340 is used to determine the distillation loss function based on the first image representation and the first text representation, the second image representation and the second text representation.

[0119] In one implementation, the loss function construction module 340 can be used to obtain a first KL divergence based on a second image representation and a second text representation; obtain a second KL divergence based on the first image representation and the first text representation; obtain a third KL divergence based on the first text representation and the second image representation; obtain a fourth KL divergence based on the second text representation and the first image representation; obtain target distribution distillation parameters based on the first KL divergence and the second KL divergence; obtain target feature distillation parameters based on the third KL divergence and the fourth KL divergence; and use the sum of the target distribution distillation parameters and the target feature distillation parameters as the distillation loss function.

[0120] One implementation of this method is to obtain the first KL divergence based on the second image representation and the second text representation, which may include: obtaining a first distribution matrix of image-text similarity calculated by the student model based on the second image representation and the second text representation, and obtaining a second distribution matrix of text-image similarity calculated by the student model based on the second image representation and the second text representation, wherein the image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image; and obtaining the first KL divergence based on the first distribution matrix and the second distribution matrix.

[0121] One implementation method for obtaining the second KL divergence based on the first image representation and the first text representation may include: obtaining a third distribution matrix of image-text similarity calculated by the teacher model based on the first image representation and the first text representation, and obtaining a fourth distribution matrix of text-image similarity calculated by the teacher model based on the first image representation and the first text representation, wherein the image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image; and obtaining the second KL divergence based on the third distribution matrix and the fourth distribution matrix.

[0122] One implementation method for obtaining the third KL divergence based on the first text representation and the second image representation may include: performing an alignment operation on the second image representation to the first text representation, and obtaining a fifth distribution matrix representing the similarity between the aligned image and text, and obtaining a sixth distribution matrix representing the similarity between the aligned text and image, wherein the image-text similarity represents the similarity between the image and the text-image similarity represents the similarity between the text and the image; and obtaining the third KL divergence based on the fifth distribution matrix and the sixth distribution matrix.

[0123] One implementation method for obtaining the fourth KL divergence based on the second text representation and the first image representation may include: performing an alignment operation on the second text representation to the first image representation, and obtaining a seventh distribution matrix representing the similarity between the aligned text and image, and obtaining an eighth distribution matrix representing the similarity between the aligned text and image, wherein the text-image similarity represents the similarity between the image and the text-image similarity represents the similarity between the text and the image; and obtaining the fourth KL divergence based on the seventh distribution matrix and the eighth distribution matrix.

[0124] It should be noted that this implementation method can calculate the KL divergence between any two distribution matrices according to a specified formula, which includes:

[0125]

[0126] Among them, the The similarity matrix between the representation graph and the text output by the two-stream model representing the teacher model, the... The similarity matrix between text and image representations output by the dual-stream model representing the teacher model, wherein P i2t The indexes of the top k most similar texts to each image in the image-to-text similarity matrix, P t2i The similarity matrix between text and image represents the k images most similar to each text, where σ represents the normalization norm. The matrix representing the similarity between images and text, the This is a matrix representing the similarity between text and images.

[0127] The model distillation module 350 is used to update the parameters of the student model based on the distillation loss function, and to use the student model when the loss function converges as the target student model.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0130] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0131] Please see Figure 6Based on the aforementioned model distillation method and apparatus, this application also provides an electronic device 100 capable of performing the aforementioned model distillation method. The electronic device 100 includes a memory 102 and one or more (only one shown in the figure) processors 104 coupled to each other, with communication lines connecting the memory 102 and the processors 104. The memory 102 stores programs capable of executing the contents of the aforementioned embodiments, and the processors 104 can execute the programs stored in the memory 102.

[0132] The processor 104 may include one or more processing cores. The processor 104 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 102, and by calling data stored in the memory 102. Optionally, the processor 104 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 104 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 104 and may be implemented separately using a communication chip.

[0133] The memory 102 may include random access memory (RAM) or read-only memory (ROM). The memory 102 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 102 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the aforementioned embodiments. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0134] Please refer to Figure 7This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 400 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0135] The computer-readable storage medium 400 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 400 has storage space for program code 410 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 410 may be compressed, for example, in a suitable form.

[0136] In summary, this application provides a model distillation method, apparatus, electronic device, and storage medium. The method is applied to an electronic device and involves: acquiring a sample image; acquiring a first image representation and a first text representation output by a teacher model based on the sample image, where the teacher model includes a single-stream model and a two-stream model, and the first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively; acquiring a second image representation and a second text representation output by a student model based on the sample image, where the student model is a two-stream model; determining a distillation loss function based on the first image representation, the first text representation, the second image representation, and the second text representation; updating the parameters of the student model based on the distillation loss function; and using the student model at which the loss function converges as the target student model. Thus, this method enables the distillation of a two-stream model using a multimodal teacher model, allowing full utilization of the advantages of multiple teacher models and more effectively transferring these advantages to the student model, thereby resulting in a higher accuracy target student model obtained through distillation.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A model distillation method, characterized in that, The method includes: Acquire sample images; The teacher model obtains a first image representation and a first text representation based on the sample image output. The teacher model includes a single-stream model and a two-stream model. The first image representation and the first text representation are both the result of fusing the outputs of the single-stream model and the two-stream model respectively. The fusion process includes: the single-stream model performs image encoding and text encoding on the sample image and outputs a first encoding result, which is a mixture of image recognition result and text recognition result; the two-stream model performs image encoding and text encoding on the sample image and outputs a second encoding result representing the image recognition result and a third encoding result representing the text recognition result; the first encoding result and the second encoding result are fused to obtain the first image representation; and the first encoding result and the third encoding result are fused to obtain the first text representation. Obtain the second image representation and the second text representation output by the student model based on the sample images, wherein the student model is a two-stream model; The distillation loss function is determined based on the first image representation and the first text representation, the second image representation and the second text representation; The parameters of the student model are updated based on the distillation loss function, and the student model when the loss function converges is taken as the target student model.

2. The method according to claim 1, characterized in that, The step of determining the distillation loss function based on the first image representation and the first text representation, the second image representation and the second text representation includes: The first KL divergence is obtained based on the second image representation and the second text representation; The second KL divergence is obtained based on the first image representation and the first text representation; The third KL divergence is obtained based on the first text representation and the second image representation; The fourth KL divergence is obtained based on the second text representation and the first image representation; The target distribution distillation parameters are obtained based on the first KL divergence and the second KL divergence; The target feature distillation parameters are obtained based on the third KL divergence and the fourth KL divergence; The sum of the target distribution distillation parameters and the target feature distillation parameters is used as the distillation loss function.

3. The method according to claim 2, characterized in that, The step of obtaining the first KL divergence based on the second image representation and the second text representation includes: A first distribution matrix of image-text similarity calculated by the student model is obtained based on the second image representation and the second text representation, and a second distribution matrix of text-image similarity calculated by the student model is obtained based on the second image representation and the second text representation, wherein the image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image. The first KL divergence is obtained based on the first distribution matrix and the second distribution matrix.

4. The method according to claim 2, characterized in that, The step of obtaining the second KL divergence based on the first image representation and the first text representation includes: A third distribution matrix of image-text similarity calculated by the teacher model is obtained based on the first image representation and the first text representation, and a fourth distribution matrix of text-image similarity calculated by the teacher model is obtained based on the first image representation and the first text representation, wherein the image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image. The second KL divergence is obtained based on the third distribution matrix and the fourth distribution matrix.

5. The method according to claim 2, characterized in that, The process of obtaining the third KL divergence based on the first text representation and the second image representation includes: The second image representation is aligned to the first text representation, and a fifth distribution matrix representing the image-text similarity after alignment and a sixth distribution matrix representing the text-image similarity after alignment are obtained. The image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image. The third KL divergence is obtained based on the fifth distribution matrix and the sixth distribution matrix.

6. The method according to claim 2, characterized in that, The process of obtaining the fourth KL divergence based on the second text representation and the first image representation includes: Align the second text representation to the first image representation, and obtain the seventh distribution matrix representing the image-text similarity after alignment, and obtain the eighth distribution matrix representing the text-image similarity after alignment. The image-text similarity represents the similarity from image to text, and the text-image similarity represents the similarity from text to image. The fourth KL divergence is obtained based on the seventh distribution matrix and the eighth distribution matrix.

7. The method according to any one of claims 3-6, characterized in that, The method further includes: Calculate the KL divergence between two distribution matrices according to a specified formula, wherein the specified formula includes: Among them, the The similarity matrix from the representation graph to the text output by the student model is represented when the teacher model is a single-stream model. The similarity matrix between text and graph representations output by the student model is represented when the teacher model is a single-stream model. The indices of the top k most similar texts to each image in the image-to-text similarity matrix are represented. The similarity matrix between the text and the image represents the k images most similar to each text. The normalization norm is represented by the The matrix representing the similarity between images and text, the This is a matrix representing the similarity between text and images.

8. A model distillation apparatus, characterized in that, The device includes: The sample image acquisition module is used to acquire sample images; The first output acquisition module is used to acquire a first image representation and a first text representation obtained by the teacher model based on the sample image. The teacher model includes a single-stream model and a two-stream model. The first image representation and the first text representation are the result of fusing the outputs of the single-stream model and the two-stream model respectively. The fusion process includes: the single-stream model performs image encoding and text encoding on the sample image and outputs a first encoding result, which is a mixture of image recognition result and text recognition result; the two-stream model performs image encoding and text encoding on the sample image and outputs a second encoding result representing the image recognition result and a third encoding result representing the text recognition result; the first encoding result and the second encoding result are fused to obtain the first image representation; and the first encoding result and the third encoding result are fused to obtain the first text representation. The second output acquisition module is used to acquire the second image representation and the second text representation output by the student model based on the sample image, wherein the student model is a two-stream model; A loss function construction module is used to determine a distillation loss function based on the first image representation and the first text representation, the second image representation and the second text representation; The model distillation module is used to update the parameters of the student model based on the distillation loss function, and to use the student model when the loss function converges as the target student model.

9. An electronic device, characterized in that, Includes one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method according to any one of claims 1-7.

Images